Idempotent Proxy
Identify and track messages to handle duplicates in event-driven systems

Context
Serverless architectures tend to the asynchronous, distributed and event-driven. Distributed systems have particular constraints that mean in most case our platforms provide “at-least-once-delivery” guarantees. In such cases, the underlying platform can only guarantee that a message has been received, not that the message was only received once.
Our services therefore need to be idempotent and handle duplicated events. In some cases, we may need to layer idempotent behavior onto a downstream integration.
Solution
Use DynamoDB to store incoming the identity of messages as they are processed. Messages will need to have a unique identifier - producers will need to agree on an identification scheme. With such identification in place, DynamoDB can provide the underlying guarantees that ensure duplicate records are not recorded into the table.
Components
- API Gateway and Lambda Function
- DynamoDB Table
- API Gateway and Lambda Function
- Expose an API endpoint. The Lambda function stores messages before processing. A ConditionExpression on the operation ensures that if the message has previously been stored the operation will fail with a ConditionalCheckFailedException. If the storage operation is successful, continue with the event processing.
- DynamoDB Table
- Create a simple table with a Partition Key that will be mapped to the Event identifier. Optionally add a Time To Live field on the table to create an upper bound on table size.
Notes
Failures when actually processing the message may mean that the DynamoDB record needs to be rolled back and deleted.
Cost Profile
Service | Charge |
---|---|
API Gateway | Request |
API Gateway | Data Transfer |
Lambda | Request |
Lambda | (Compute Time x Memory) |
CloudWatch | Log Data Ingestion |
DynamoDB | Read + Write Throughput |
DynamoDB | Data Storage |