Skip to main content

Step 2: Filter by Severity

Log retention requirements differ by system. This example selects ERROR and WARN records and drops INFO and DEBUG records.

This step teaches you how to use a conditional mapping to filter a stream of messages.

The Goal​

You will add a processor to your pipeline that inspects the level of each log message and deletes any message that is not an ERROR or WARN.

The deleted() Function​

The key to filtering is the deleted() function. When a mapping processor returns deleted(), the entire message is discarded and does not proceed to the next stage of the pipeline.

Implementation​

  1. Start with the Previous Pipeline: Copy the robust-parser.yaml from Step 1 to a new file named severity-filter.yaml.

    cp robust-parser.yaml severity-filter.yaml

  2. Add the Filtering Logic: Open severity-filter.yaml and add a new mapping processor to the end of the pipeline section.

    Add this to the 'processors' array in severity-filter.yaml
    - mapping: |
        # Normalize the level to uppercase for consistent matching
        let level = this.level.string().uppercase()

        # Check if the level is one we want to keep.
        # If not, the message is deleted.
        root = if level == "ERROR" || level == "WARN" {
          this
        } else {
          deleted()
        }

  3. Inspect the intended result: The authored output retains ERROR and WARN messages. The configured filter drops the other fixture levels.

Verification​

This page does not record an executed result or measured volume effect. Verify the predicate against representative source levels before adapting it.