Step 2: Convert Timezones to UTC

Once you have a standard timestamp object, the next step is to convert it to a consistent timezone. The universal standard for data processing is Coordinated Universal Time (UTC).

The Goal

You will modify your pipeline to take the normalized_timestamp from the previous step and convert it to a new timestamp_utc field, ensuring all your events are comparable on a single timeline.

Implementation

1. Start with the Previous Pipeline: Copy the normalize-timestamps.yaml from Step 1 to a new file named convert-timezones.yaml.

   cp normalize-timestamps.yaml convert-timezones.yaml

2. Add the Timezone Conversion Logic: Open convert-timezones.yaml and add the following mapping processor after the existing parsing processor.

   The surrounding pipeline context keeps this partial snippet executable. Keep the HTTP input from Step 1 and append this processor to pipeline.processors.

   Partial snippet

   Partial snippet: timezone conversion mapping

   pipeline:
     processors:
       - mapping: |
           root = this
           root.timestamp_utc = this.normalized_timestamp.ts_format("2006-01-02T15:04:05.999999999Z07:00", "UTC")
           root.original_timezone = if this.original_timestamp.type() == "string" {
             this.original_timestamp.re_find_all("[+-][0-9]{2}:[0-9]{2}$").index(0).or("+00:00")
           } else {
             "+00:00"
           }

3. Deploy and Test:

   # --- Send a timestamp with a non-UTC timezone ---
   curl -X POST http://localhost:8080/ingest \
     -H "Content-Type: application/json" \
     -d '{"event_id": "C", "timestamp": "2025-10-20T14:23:45-04:00"}'

4. Verify: Check your logs. The output will show:

   • original_timestamp: The original string with the -04:00 offset.
   • timestamp_utc: The converted time, which will be 2025-10-20T18:23:45Z.
   • original_timezone: The extracted -04:00 offset.

You have now successfully standardized all incoming timestamps to UTC, which is a critical step for reliable time-based analysis.