Receiver Service

This service is relatively simple - it receives the webhook payload from the provider, writes it to an Amazon S3 bucket, then emits a message to an Amazon Simple Queue Service queue with some metadata about the payload, including the path to the S3 object that contains the payload itself.

We chose to use a queue here to gain a few benefits:

It allows us to have higher availability. The Receiver is simple, and rarely changes so needs to be deployed infrequently. All of the logic is in the Handler, which does get redeployed frequently, but doing so doesn't interrupt event processing, since the Receiver continues to add events to the queue.

Having a queue gives us a simple retry mechanism, both in the case of a fault or transient error. We only remove a message from the queue once we have successfully processed it, and if we try to reprocess the event N times, we move the message to a dead letter queue. We can then address the fault (or wait for the transient error to resolve), then move the message from the dead letter queue back to the primary queue for reprocessing.

There is a trade-off for using a queue here, however: SQS provides at-least-once delivery (we will see every message), but does not guarantee at-most-once delivery (we may see the same message multiple times). To handle this, the subsystem is designed to process messages idempotently, so there is no harm in processing the same message multiple times.

We store the payload in S3 instead of including it in the SQS message because maximum allowed SQS message size is 256 KB, and we can't guarantee that all payloads will be less than that size, especially since we don't control the generation of those payloads (there are other benefits to using S3 for the payloads, as we'll see in a bit).