Skip to main content

Expanso example

Filter Severity

The problem

Low-value log traffic can overwhelm downstream storage and obscure the events operators need first.

How Expanso solves it

Keep higher-severity log events and route them to local outputs.

Filter logs before shipping them by selecting the records you need at the source. This first run splits three synthetic JSON logs into records, keeps the ERROR event, and prints it locally.

Run your first result​

You need Expanso Edge and the CLI installed and two terminals. No Cloud account or external data service is required. Use only synthetic data for this first check.

In terminal one, start a dedicated local node. Keep it open to see the output:

expanso-edge run --local --no-watch --api-listen 127.0.0.1:19031 --data-dir ./expanso-first-result-data

In terminal two, download and inspect the job, validate its pipeline configuration, then submit it explicitly to the local node:

curl -fsSLo filter-logs.yaml https://examples.expanso.io/files/first-results/filter-logs.yaml
cat filter-logs.yaml
expanso-edge validate filter-logs.yaml
expanso-cli --endpoint http://127.0.0.1:19031 job deploy filter-logs.yaml

Validation checks pipeline syntax and components. Submission alone is not success: confirm the JSON result in terminal one.

The complete job​

The generated input is finite: one batch, then completion.

Download the job file 
name: first-result-filter-logs
type: pipeline
config:
  input:
    generate:
      count: 1
      interval: 1s
      mapping: |
        root = [{"event_id":"log-001","level":"DEBUG","message":"polling"},{"event_id":"log-002","level":"INFO","message":"ready"},{"event_id":"log-003","level":"ERROR","message":"request failed"}]
  pipeline:
    processors:
      - unarchive:
          format: json_array
      - mapping: |
          root = if this.level == "ERROR" { this } else { deleted() }
  output:
    stdout:
      codec: lines

Input​

[
  { "event_id": "log-001", "level": "DEBUG", "message": "polling" },
  { "event_id": "log-002", "level": "INFO", "message": "ready" },
  { "event_id": "log-003", "level": "ERROR", "message": "request failed" }
]

Expected output​

The local node prints the following JSON record alongside its service logs. JSON key order can differ.

{ "event_id": "log-003", "level": "ERROR", "message": "request failed" }

This bounded job was executed with Expanso Edge 2.1.21 on September 10, 2026; the output matched this record. This proof covers local synthetic execution only.

If you do not see the result​

  • Connection refused: start the local node; check both ports match.
  • Port occupied: use another unused port in both commands.
  • No output: inspect terminal one for errors.
  • Rerun: change the completed job’s name before submitting again.

When finished, press Ctrl+C in terminal one to stop the local node. Its state remains in ./expanso-first-result-data; use a fresh directory for an independent later session.

What this proves, and what to check next​

This filter uses an exact, case-sensitive ERROR match. WARN, missing levels, and other values are dropped. Decide your retention policy before adapting it: low-severity events can be necessary for investigation. Do not extrapolate production traffic or billing from this three-record fixture. The existing multi-stage architecture below has a separate verification boundary.

Common questions​

Will this reduce my logging bill?​

It reduces the number of emitted records in this fixture from three to one. Actual cost depends on your traffic, retention policy, billing model, and where filtering runs. Measure representative byte volume and retained diagnostic value before projecting savings.

How do I move from this result to my own data?​

Test the levels your applications actually emit, including missing and unexpected values. Choose which records must survive before replacing the generated input. Configure a real destination only after verifying retained events. The Local Mode Quick Start explains local operation; the Expanso Cloud Quickstart covers managed nodes. Check delivery separately after deployment.

Try another first result​

Explore the extended example​

The extended parsing, normalization, and routing example remains architecture-only. Running the bounded severity filter above does not establish that the full multi-stage architecture has executed.