AWS SQS Message deduplication on FIFO

Message deduplication for FIFO (First-In-First-Out) queues is a process in messaging systems that ensures that duplicate messages are identified and removed from the queue before being processed. In a FIFO queue, the order in which messages are received is maintained, meaning the first message sent into the queue will be the first one processed.

Message deduplication helps prevent situations where identical messages are processed multiple times, which can lead to unintended consequences or duplicate actions. This is especially important in scenarios where processing duplicate messages could cause data inconsistencies or other issues.

When a message is sent to a FIFO queue, the system typically uses a unique identifier, called a deduplication ID, to determine whether the message is a duplicate or not. If a message with the same deduplication ID is already in the queue, the new message is considered a duplicate and is either discarded or not added to the queue.

This mechanism ensures that only unique messages are processed, maintaining the order of messages in the queue and preventing unnecessary duplication of work or actions.

Example:

Let’s consider an e-commerce website that uses a FIFO queue to process orders. The system receives order requests from customers and places them in a FIFO queue for processing. To avoid processing the same order multiple times, message deduplication is implemented.

1. Customer A submits an order for a laptop, and the system generates a unique order ID, let’s say “Order123”. The order request, along with the deduplication ID “D123”, is added to the FIFO queue.
2. Customer B also submits an order for the same laptop shortly afterward. The system generates a different order ID, such as “Order124”, but since the order details (including the laptop model) are identical, the deduplication ID “D123” is used again.
3. Before adding Customer B’s order to the queue, the system checks the deduplication ID “D123”. Since a message with the same deduplication ID is already in the queue (from Customer A’s order), the system recognizes this as a duplicate order and does not add it to the queue again.
4. Later, Customer C submits an order for a different item, a smartphone. This order is assigned a unique order ID “Order125”, and a new deduplication ID “D125” is generated.
5. Customer D places an order for the same smartphone shortly afterward. Again, the order details are the same, so the deduplication ID “D125” is used for this order as well.
6. The system checks the deduplication ID “D125” and sees that there is already a message with this ID in the queue (from Customer C’s order). Like before, it identifies the duplicate and does not add Customer D’s order to the queue.

By implementing message deduplication, the system ensures that even though duplicate orders were attempted (from Customers B and D), only the original orders (from Customers A and C) are processed. This helps maintain order accuracy and prevents unintended duplication of actions, improving the overall reliability of the order processing system.