Step 2: Parse CSV Data

Another common log format is CSV (Comma-Separated Values). This step teaches you how to parse a raw log line that is a CSV string into a structured JSON object.

The Goal

You will transform a raw message like this, where the message field is a single CSV string:

{
  "message": "2025-10-20T14:23:45Z,temperature,temp-sensor-01,35.5,celsius"
}

Into a structured object with named fields:

{
  "timestamp": "2025-10-20T14:23:45Z",
  "metric_name": "temperature",
  "sensor_id": "temp-sensor-01",
  "value": "35.5",
  "unit": "celsius"
}

The csv Processor

This transformation is done with the csv processor, where you define the names of the columns in the order they appear.

Implementation

1. Create the Parsing Pipeline: Copy the following configuration into a file named csv-parser.yaml.

   Complete pipeline

   csv-parser.yaml

   name: csv-log-parser
   description: A pipeline that parses a string field containing CSV data.
   
   config:
     input:
       generate:
         interval: 1s
         mapping: |
           root.raw_log = "2025-10-20T14:23:45Z,temperature,temp-sensor-01,35.5,celsius"
   
     pipeline:
       processors:
         - mapping: |
             let header = "timestamp,metric_name,sensor_id,value,unit\n"
             root = ($header + this.raw_log).parse_csv().index(0)
   
     output:
       stdout:
         codec: lines

2. Deploy and Observe: Watch the logs. The generate input creates messages with a single raw_log field containing a CSV string. The output will be the structured JSON that was parsed from that string.

Verification

The output will be a stream of structured JSON objects.

Example Output:

{
  "metric_name": "temperature",
  "sensor_id": "temp-sensor-01",
  "timestamp": "2025-10-20T14:23:45Z",
  "unit": "celsius",
  "value": "35.5"
}

You have successfully parsed a CSV log. Notice that all the values are still strings. Subsequent processing steps would be needed to convert value to a number and timestamp to a time object.