Step 2: Fingerprint-Based Deduplication for Semantic Duplicates

Exact-content keys change when any serialized value changes. Fingerprint-based deduplication instead hashes a selected set of fields that defines equivalence for this example.

The example treats two records with different identifiers and timestamps as equivalent when their configured business fields match.

The Goal

The fingerprint uses event_type and user_id. That field choice defines the example's duplicate policy.

Event 1:

{
  "event_id": "abc-123",
  "timestamp": "10:30:01Z",
  "event_type": "login",
  "user_id": "alice"
}

Event 2 (Semantic Duplicate):

{
  "event_id": "def-456",
  "timestamp": "10:30:03Z",
  "event_type": "login",
  "user_id": "alice"
}

Implementation

1. Start with the Previous Pipeline: Copy the deduplicator.yaml from Step 1 to a new file named fingerprint-dedup.yaml.

   cp deduplicator.yaml fingerprint-dedup.yaml

2. Modify the Hashing Logic: Open fingerprint-dedup.yaml. You will modify the first mapping processor to create the hash from a subset of fields instead of the whole message.

   Modify the first 'mapping' processor in fingerprint-dedup.yaml

   # Change the hash creation logic
   - mapping: |
       root = this
   
       # 1. Create an object containing the selected comparison fields.
       let business_fingerprint = {
         "event_type": this.event_type,
         "user_id": this.user_id
       }
   
       # 2. Create the hash from this new object, not the whole message.
       root.dedup_hash = business_fingerprint.json_format().hash("sha256")

   The rest of the pipeline (cache, check, drop) remains exactly the same. It still uses dedup_hash to check for duplicates, but that hash is now much smarter.

3. Deploy and Test:

   # --- Send two SEMANTICALLY identical messages ---
   curl -X POST http://localhost:8080/ingest \
     -H "Content-Type: application/json" \
     -d '{"event_id": "abc-123", "timestamp": "10:30:01Z", "event_type": "login", "user_id": "alice"}'
   
   curl -X POST http://localhost:8080/ingest \
     -H "Content-Type: application/json" \
     -d '{"event_id": "def-456", "timestamp": "10:30:03Z", "event_type": "login", "user_id": "alice"}'

4. Verify: The intended result retains the first record and drops the matching fingerprint. This page does not record an executed result.

Review false-match risk and field evolution before adapting this strategy.