level 300

Idempotent Proxy

Identify and track messages to handle duplicates in event-driven systems

Context

Serverless architectures tend to the asynchronous, distributed and event-driven. Distributed systems have particular constraints that mean in most case our platforms provide “at-least-once-delivery” guarantees. In such cases, the underlying platform can only guarantee that a message has been received, not that the message was only received once.

Our services therefore need to be idempotent and handle duplicated events. In some cases, we may need to layer idempotent behavior onto a downstream integration.

Solution

Use DynamoDB to store incoming the identity of messages as they are processed. Messages will need to have a unique identifier - producers will need to agree on an identification scheme. With such identification in place, DynamoDB can provide the underlying guarantees that ensure duplicate records are not recorded into the table.

Components

  • API Gateway and Lambda Function
  • DynamoDB Table
API Gateway and Lambda Function
Expose an API endpoint. The Lambda function stores messages before processing. A ConditionExpression on the operation ensures that if the message has previously been stored the operation will fail with a ConditionalCheckFailedException. If the storage operation is successful, continue with the event processing.
DynamoDB Table
Create a simple table with a Partition Key that will be mapped to the Event identifier. Optionally add a Time To Live field on the table to create an upper bound on table size.

Notes

Failures when actually processing the message may mean that the DynamoDB record needs to be rolled back and deleted.

Cost Profile

Service Charge
API Gateway Request
API Gateway Data Transfer
Lambda Request
Lambda (Compute Time x Memory)
CloudWatch Log Data Ingestion
DynamoDB Read + Write Throughput
DynamoDB Data Storage