Step 3: ID-Based Deduplication for Unique Identifiers

When the producer contract defines a stable unique identifier, ID-based deduplication can use that value directly as the cache key.

This pattern skips the need for hashing content and instead uses the event's own ID as the unique key.

The Goal

This configuration uses event_id directly rather than calculating a content hash.

Implementation

1. Start with the Previous Pipeline: Copy the fingerprint-dedup.yaml from Step 2 to a new file named id-dedup.yaml.

   cp fingerprint-dedup.yaml id-dedup.yaml

2. Simplify the Key Creation: Open id-dedup.yaml. You will replace the complex fingerprinting mapping processor with a much simpler one that just uses the event_id.

   Partial snippet

   Replace the first 'mapping' processor in id-dedup.yaml

   # This processor is now much simpler.
   - mapping: |
       root = this
       # The key for deduplication is now just the event's own ID.
       root.dedup_key = this.event_id

   Inside the inherited for_each, replace the lookup branch so it checks dedup_key and preserves the record.

   Partial snippet

   Replace the lookup branch

   - branch:
       request_map: root = this.dedup_key
       processors:
         - cache:
             resource: dedup_cache
             operator: exists
             key: '${!content()}'
       result_map: root.is_duplicate = content().string().bool()

   Inside the same for_each, replace the conditional cache write to store dedup_key. Keep the final duplicate-filtering mapping processor from Step 1 there as well.

   Partial snippet

   Replace the conditional cache write

   - switch:
       - check: '!this.is_duplicate'
         processors:
           - cache:
               resource: dedup_cache
               operator: set
               key: '${!this.dedup_key}'
               value: seen

3. Deploy and Test:

   # --- Send the SAME event twice ---
   curl -X POST http://localhost:8080/ingest \
     -H "Content-Type: application/json" \
     -d '{"event_id": "abc-123", "message": "hello"}'
   
   curl -X POST http://localhost:8080/ingest \
     -H "Content-Type: application/json" \
     -d '{"event_id": "abc-123", "message": "hello again"}'

4. Verify: Check your logs. Even though the message was different in the second request, it was dropped as a duplicate because it had the same event_id. This method is extremely fast and efficient when you can rely on a unique identifier in your events.