Skip to main content

Interactive Deduplication Explorer

See event deduplication in action! Use the interactive explorer below to step through 5 stages of deduplication techniques. Watch as duplicate events are progressively detected and filtered using different strategies.

How to Use This Explorer​

  1. Navigate using arrow keys (← →) or click the numbered stage buttons
  2. Compare the Input (left) and Output (right) JSON at each stage
  3. Observe how duplicates are rejected (red strikethrough) or accepted (green highlight)
  4. Inspect the YAML code showing exactly what processor was added
  5. Learn from the stage description explaining the technique and compliance benefits

Interactive Deduplication Explorer​

Original Events

Multiple events arrive with potential duplicates from network retries, load balancer failovers, and eventual consistency delays.

Use ← → arrow keys to navigate
📥Input
{
"event_id": "evt_signup_001",
"event_type": "user_signup",
"timestamp": "2025-01-15T10:30:45.123Z",
"user": {
"id": "user_12345",
"email": "[email protected]"
}
}
📤Output
{
"event_id": "evt_signup_001",
"event_type": "user_signup",
"timestamp": "2025-01-15T10:30:45.123Z",
"user": {
"id": "user_12345",
"email": "[email protected]"
}
}
Added/Changed
Removed
Completed Step
Current Step
Not Done Yet
📄New Pipeline Stepinput.json
# Original input - no deduplication yet
input:
  http_server:
    address: "0.0.0.0:8080"
    path: "/webhooks/events"

Try It Yourself​

Ready to build this deduplication pipeline? Follow the step-by-step tutorial:

Deep Dive into Each Strategy​

Want to understand each deduplication technique in depth?

Next: Set up your environment to build this deduplication pipeline yourself