For a recent hobby project, one of the things I’m doing is processing all the items in an S3 bucket. Simple, right? I like to keep things serverless when possible, so here’s what the design looks like:

This works great, but the problem is that these buckets could have millions of items! The listing stage goes pretty quickly as each request returns 1,000 items, but it slows to a crawl when it gets to the processing stage. Why?

The processor Lambda function pulls from the queue, does some basic data manipulation, inserts the result into a DynamoDB table, and deletes the message from the queue. None of that is particularly tough or slow, but the latency between requests adds up quickly when you’re doing it millions of times. There’s no clean way of reliably having a Lambda function process the items immediately as they’re added to the queue, so what else can we do?

There are a few options, but each has significant drawbacks. What would really help is a way to invoke a Lambda function many times at once. If we only had a way of sending a single API call that would result in a hundreds or thousands of concurrent Lambda invocations, we could process our queue in minutes or seconds instead of hours or days.

There must be a better way!

The better way

It’s based on a few premises:

You can subscribe a Lambda function to an SNS topic, but you can’t subscribe the same function multiple times to the same topic.

SNS supports sending notifications to HTTPS endpoints, but they can’t have the exact same URL.

SNS handles notifications for topics with even hundreds of thousands of subscribers all day every day.

API Gateway allows you to expose an HTTPS endpoint that invokes a Lambda function.

Adding a throwaway URL parameter to that endpoint will give you as many unique URLs for invoking that Lambda function as you need.

Do you see where this is going? With a little bit of setup work and configuration, we can set up an SNS topic that, when published to, will cause our Lambda function to be invoked hundreds or thousands of times. And it works!

Massive concurrency achieved!

With this in place, the processing of my SQS queue is much simpler and massively parallel. I simply check approximately how many messages are in my SQS queue and send a single SNS notification which results in my Lambda function being invoked as many times as I want. None of the services utilized (API Gateway, SNS, SQS, DynamoDB, Lambda) have architectural scaling limits, so while I may hit default account limits, there’s no published strict limit I’m approaching.

CloudWatch chart showing over 1,000 concurrent Lambda invocations (17 throttles), as caused by a single SNS publish.

Try it out!

There’s a proof-of-concept CloudFormation template available here:

Caveats

They’re pretty minimal, all things considered: