level 200

Queue-Based Event Distribution

Fan-out and distribute events to multiple consumers to allow decoupled distributed processing

Context

Patterns like Queue-Based Load Leveling decouple requests and protect downstream services using an SQS queue to enable asynchronous processing. However, the design of SQS means that messages are removed from the queue when they have been processed. Sometimes a single queue consumer can be responsible for all of the work triggered by a message event. In other cases, we may have several logically unrelated processes that need to be performed. For example, a new Order event may need to add a customer to the CRM, trigger a notification to the warehouse and fulfillment system, perform fraud and risk checks, and send a confirmation email. Merging this logic into a single Lambda function would tightly couple these unrelated processes - escalating both the likelihood and consequence of errors, and increasing code complexity.

We need a mechanism to push events to multiple interested consumers asynchronously, without coupling the senders to the receivers.

Solution

Push events to an SNS Topic connected to multiple SQS subscribers to distribute events.

SNS abstracts the delivery of events to SQS - simply configure the queues as subscriptions to the topic and the platform ensures delivery. Delivery can be filtered by message attribute - subscribers are only sent messages that match the specific filter parameters. Filtering can help reduce scope and complexity in downstream consumers - we can create granular consumers that handle specific types of messages.

Components

  • SNS Topic
  • SQS Queues
SNS Topic
An SNS Topic is configured to push messages to subscriber queues.
SQS Queues
Multiple queues are configured to allow dispatching of events to small, isolated consumers that encapsulate the logic of single process.

Notes

Queue-Based Event Distribution is often coupled with Queue Based Load Leveling. Messages are pushed to multiple queues by SNS and those queues are processed by Lambda consumers.

Durability & Guarantees

SNS provides an “immediate dispatch” message notification service and guarantees delivery to SQS (as of November 2019).

Message data is stored across multiple Availability Zones and storage is confirmed by the platform before returning success when a message is published.

Cost Profile

Service Charge
SNS Inbound Request
SNS No charge for delivery to SQS
SQS Request
CloudWatch Log Data Ingestion

Related Patterns

Links