Step 2: Filter by Severity

Log retention requirements differ by system. This example selects ERROR and WARN records and drops INFO and DEBUG records.

This step teaches you how to use a conditional mapping to filter a stream of messages.

The Goal

You will add a processor to your pipeline that inspects the level of each log message and deletes any message that is not an ERROR or WARN.

The deleted() Function

The key to filtering is the deleted() function. When a mapping processor returns deleted(), the entire message is discarded and does not proceed to the next stage of the pipeline.

Implementation

1. Start with the Previous Pipeline: Copy the robust-parser.yaml from Step 1 to a new file named severity-filter.yaml.

   cp robust-parser.yaml severity-filter.yaml

2. Add the Filtering Logic: Open severity-filter.yaml and add a new mapping processor to the end of the pipeline section.

   Add this to the 'processors' array in severity-filter.yaml

   - mapping: |
       # Normalize the level to uppercase for consistent matching
       let level = this.level.string().uppercase()
   
       # Check if the level is one we want to keep.
       # If not, the message is deleted.
       root = if level == "ERROR" || level == "WARN" {
         this
       } else {
         deleted()
       }

3. Inspect the intended result: The authored output retains ERROR and WARN messages. The configured filter drops the other fixture levels.

Verification

This page does not record an executed result or measured volume effect. Verify the predicate against representative source levels before adapting it.