Skip to main content

Advanced Aggregation Patterns

Once you have mastered the three basic window types, you can combine them and add more sophisticated analytics logic.

Pattern 1: Multi-Level Aggregation (Rollups)​

You can chain aggregation pipelines together to create "rollups". For example, you can take the 1-minute tumbling window aggregates and feed them into another pipeline that aggregates them into 1-hour summaries.

Pipeline 1: 1-Minute Tumbling Windows (from Step 1)

  • Input: Raw events
  • Output: 1-minute summaries per sensor

Pipeline 2: 1-Hour Rollups

  • Input: The 1-minute summaries from Pipeline 1
  • Output: 1-hour summaries per sensor

See the complete configuration for the maintained sensor and location rollups. An hourly rollup needs its own buffer and weighted averages; averaging window averages without event counts can distort the result.

Pattern 2: Advanced Trend Analysis​

The basic sliding window provides a moving average. For more advanced trend analysis, you can perform calculations like slope and correlation on the data points within the window.

Partial snippet
Advanced Trend Calculation
- mapping: |
    # This processor assumes it has a group of events from a sliding window
    let temperatures = this.map_each(event -> event.temperature)
    let n = $temperatures.length()
    let first_temp = $temperatures.index(0)
    let last_temp = $temperatures.index(-1)
    let time_span_minutes = 5 # Assuming a 5-minute window

    root = if $n >= 2 {
      {
        "events": this,
        "temperature_change": (($last_temp - $first_temp) * 100).round() / 100,
        "temperature_slope_per_min": ((($last_temp - $first_temp) / $time_span_minutes) * 1000).round() / 1000,
        "temperature_trend": if (($last_temp - $first_temp) / $time_span_minutes) > 0.1 { "increasing" }
          else if (($last_temp - $first_temp) / $time_span_minutes) < -0.1 { "decreasing" }
          else { "stable" }
      }
    } else { {"events": this} }

Pattern 3: Late Events​

Review event-time selection, window boundaries, and late-arrival behavior in the configured system_window buffer. A cache TTL is not a grace period for an accumulating event window. Test late and out-of-order records against the target executor before selecting a policy.