Step 3: Enrich with Time Metadata

The final step in timestamp normalization pre-calculates common time components that an analytics team might otherwise derive in each query. Whether to materialize these fields depends on the downstream query plan and storage model.

The Goal

You will add a time_metadata object to your events that contains useful, pre-calculated fields like the year, month, hour, and day of the week, all extracted from the standardized timestamp_utc field.

Implementation

1. Start with the Previous Pipeline: Copy the convert-timezones.yaml from Step 2 to a new file named enrich-timestamps.yaml.

   cp convert-timezones.yaml enrich-timestamps.yaml

2. Add the Enrichment Logic: Open enrich-timestamps.yaml and add the following mapping processor after the timezone conversion processor.

   The surrounding pipeline context keeps this partial snippet executable. Keep the HTTP input from Step 1 and append this processor to pipeline.processors.

   Partial snippet

   Partial snippet: timestamp metadata mapping

   pipeline:
     processors:
       - mapping: |
           root = this
           let ts = this.timestamp_utc.ts_parse("2006-01-02T15:04:05Z07:00")
           root.time_metadata = {
             "year": $ts.ts_format("2006", "UTC").number(),
             "month": $ts.ts_format("01", "UTC").number(),
             "day": $ts.ts_format("02", "UTC").number(),
             "hour": $ts.ts_format("15", "UTC").number(),
             "day_of_week": {"Sunday": 0, "Monday": 1, "Tuesday": 2, "Wednesday": 3, "Thursday": 4, "Friday": 5, "Saturday": 6}.get($ts.ts_format("Monday", "UTC"))
           }

3. Deploy and Test:

   # --- Send a test event ---
   curl -X POST http://localhost:8080/ingest \
     -H "Content-Type: application/json" \
     -d '{"event_id": "D", "timestamp": "2025-10-20T18:23:45Z"}'

4. Verify: Check your logs. The output will now contain the new time_metadata object with all the useful time components pre-calculated. Your data is now perfectly formatted for efficient analytics.

   Example Output Snippet:

   {
     "timestamp_utc": "2025-10-20T18:23:45Z",
     "time_metadata": {
       "year": 2025,
       "month": 10,
       "day": 20,
       "hour": 18,
       "day_of_week": 1
     }
   }

You have now built a complete timestamp normalization pipeline that can handle multiple formats, standardize timezones, and enrich the data for analytics.