Filter & Score Logs

Not all logs carry the same diagnostic value. This step applies explicit filtering and scoring rules.

Goal

• Score logs based on severity and content
• Filter out low-value logs (e.g., successful health checks, debug logs)
• Prioritize critical errors

Configuration

1. Severity Scoring

We verify the standard logging levels and can also upgrade the level based on keywords (e.g., "panic" or "exception").

Partial snippet

- mapping: |
    root = this

    # Calculate numeric priority (higher is more critical)
    let priority = match root.level {
      "FATAL" => 10,
      "ERROR" => 8,
      "WARN" => 5,
      "INFO" => 3,
      "DEBUG" => 1,
      _ => 0
    }

    # Upgrade priority if sensitive keywords found
    if root.message.contains("panic") || root.message.contains("crash") {
       priority = 10
       root.level = "FATAL"
    }

    root.priority_score = $priority

2. Intelligent Filtering

We drop logs that fall below a certain priority, or filter out specific noisy patterns.

Partial snippet

- mapping: |
    root = this

    # Drop DEBUG logs unless explicitly enabled
    root = if this.level == "DEBUG" && env("DEBUG_LOGGING_ENABLED") != "true" {
      deleted()
    } else if this.service == "health-check" && this.level == "INFO" {
      deleted()
    } else { this }

Complete Step 4 Configuration

See the complete reference for the maintained input, named rate-limit resource, processors, and failure boundary.

Deployment & Verification

1. Test Dropped Log (DEBUG):

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: $LOG_API_KEY" \
     -d '{"level": "DEBUG", "message": "This should be dropped"}'

   Result: No output (log dropped).

2. Test Kept Log (ERROR):

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: $LOG_API_KEY" \
     -d '{"level": "ERROR", "message": "Keep this"}'

   Result: Log output with score=8.

Next Steps

We've reduced noise. Now we must ensure we don't leak sensitive user data.

👉 Step 5: Redact Sensitive Data