Idempotent Proxy

Context

Serverless architectures tend to the asynchronous, distributed and event-driven. Distributed systems have particular constraints that mean in most case our platforms provide “at-least-once-delivery” guarantees. In such cases, the underlying platform can only guarantee that a message has been received, not that the message was only received once.

Our services therefore need to be idempotent and handle duplicated events. In some cases, we may need to layer idempotent behavior onto a downstream integration.

Solution

Use DynamoDB to store incoming the identity of messages as they are processed. Messages will need to have a unique identifier - producers will need to agree on an identification scheme. With such identification in place, DynamoDB can provide the underlying guarantees that ensure duplicate records are not recorded into the table.

Components

• API Gateway and Lambda Function
• DynamoDB Table

API Gateway and Lambda Function

Expose an API endpoint. The Lambda function stores messages before processing. A ConditionExpression on the operation ensures that if the message has previously been stored the operation will fail with a ConditionalCheckFailedException. If the storage operation is successful, continue with the event processing.

DynamoDB Table

Create a simple table with a Partition Key that will be mapped to the Event identifier. Optionally add a Time To Live field on the table to create an upper bound on table size.

Notes

Failures when actually processing the message may mean that the DynamoDB record needs to be rolled back and deleted.

Cost Profile