AWS Autoscaling & Queues

Scott StultsAugust 8, 2013

In GPSN we have this task stereotype of spinning up a group of VMs to process task items in an SQS queue. While reading through the autoscaling docs two things jumped out at me: The minimum size of size of an autoscale group is zero, and autoscaling can be triggered off of the size of a queue. In other words, we could have a fifteen-step process based on fifteen stacks and fifteen queues, and we don’t have to manually start any of them — we can just let the stacks autoscale down to zero instances when their queue is empty.

A great side-effect of this is that each stack will start worker VMs when items start popping up in the queue they’re watching. While not as fine-grained as a full-blown Actor Model framework, this still lets multiple types of tasks through the system in parallel and at their own pace.

More blog articles:

Let's do a project together!

We provide tailored search, discovery and personalization solutions using Solr and Elasticsearch. Learn more about our service offerings