Step 3: Parse Syslog Messages

Syslog is a standard format for system logs. Unlike a simple access log, it has a special <priority> field at the beginning that contains two pieces of information combined: the facility (what part of the system generated the message) and the severity (how important it is).

The Goal

You will parse a raw syslog string and then decompose the priority number into its facility and severity components.

Input Message:

{
  "raw_log": "<134>Oct 20 14:23:45 edge-node-01 app[12345]: Database connection established"
}

Desired Output:

{
  "priority": "134",
  "facility": 16,
  "severity": 6,
  "timestamp": "Oct 20 14:23:45",
  "hostname": "edge-node-01",
  "tag": "app",
  "pid": "12345",
  "message": "Database connection established"
}

The "Grok -> Decompose" Pattern

1. Grok: Use a grok processor with a standard syslog pattern to extract the raw fields, including the priority number.
2. Decompose: Use a mapping processor to perform the integer arithmetic to calculate facility = priority / 8 and severity = priority % 8.

Implementation

1. Create the Parsing Pipeline: Copy the following configuration into a file named syslog-parser.yaml.

   Complete pipeline

   syslog-parser.yaml

   name: syslog-parser
   description: A pipeline that parses syslog messages.
   
   config:
     input:
       generate:
         interval: 1s
         mapping: |
           root.raw_log = "<134>Oct 20 14:23:45 edge-node-01 app[12345]: Database connection established"
   
     pipeline:
       processors:
         - mapping: |
             let parts = this.raw_log.re_find_all_submatch("^<([0-9]+)>([A-Z][a-z]{2} +[0-9]+ [0-9:]+) ([^ ]+) ([^ :\\[]+)(?:\\[([0-9]+)\\])?: (.*)$").index(0)
             let pri = $parts.index(1).number()
             root = {
               "priority": $parts.index(1),
               "timestamp": $parts.index(2),
               "hostname": $parts.index(3),
               "tag": $parts.index(4),
               "pid": $parts.index(5),
               "message": $parts.index(6),
               "facility": ($pri / 8).floor(),
               "severity": $pri % 8
             }
   
     output:
       stdout:
         codec: lines

2. Deploy and Observe: Watch the logs. The output will be the structured JSON, including the calculated facility and severity fields.

Verification

The output will be a stream of structured JSON objects. 134 is correctly decomposed into facility 16 (local use 0) and severity 6 (informational).

Example Output:

{
  "facility": 16,
  "hostname": "edge-node-01",
  "message": "Database connection established",
  "pid": "12345",
  "priority": "134",
  "severity": 6,
  "tag": "app",
  "timestamp": "Oct 20 14:23:45"
}

You have now learned how to parse syslog messages and extract the important priority information, which can be used for routing and alerting in a more advanced pipeline.