Skip to main content

Step 2: Filter by Severity

A common requirement for log processing is to filter out "noise". Most of the time, you only care about ERROR or WARN level logs, while INFO and DEBUG logs can be discarded.

This step teaches you how to use a conditional mapping to filter a stream of messages.

The Goal​

You will add a processor to your pipeline that inspects the level of each log message and deletes any message that is not an ERROR or WARN.

The deleted() Function​

The key to filtering is the deleted() function. When a mapping processor returns deleted(), the entire message is discarded and does not proceed to the next stage of the pipeline.

Implementation​

  1. Start with the Previous Pipeline: Copy the robust-parser.yaml from Step 1 to a new file named severity-filter.yaml.

    cp robust-parser.yaml severity-filter.yaml

  2. Add the Filtering Logic: Open severity-filter.yaml and add a new mapping processor to the end of the pipeline section.

    Add this to the 'processors' array in severity-filter.yaml
    - mapping: |
        # Normalize the level to uppercase for consistent matching
        let level = this.level.string().uppercase()

        # Check if the level is one we want to keep.
        # If not, the message is deleted.
        root = if level == "ERROR" || level == "WARN" {
          this
        } else {
          deleted()
        }

  3. Deploy and Observe: Watch the logs. The generate input is creating a mix of info and other logs, but you will only see the ERROR and WARN messages in the final output. All other messages are being silently dropped by the filter.

Verification​

This simple filtering pattern is one of the most powerful tools for managing high-volume log streams. By placing a filter early in your pipeline, you can dramatically reduce the amount of data that needs to be processed, saving costs and improving performance.