Skip to main content

Setup Environment for Aggregate Time Windows

Before building the time-windowed aggregation pipeline, you'll set up sample IoT sensor data and configure environment variables.

Prerequisites​

This example requires the following services to be running:

Before you begin, please ensure these services are set up and running according to their respective guides. Additionally, ensure you have completed the Local Development Setup guide for general environment configuration.

Step 1: Configure Example-Specific Variables​

After setting up the core services, configure aggregation-specific variables:

# Analytics endpoint for cloud ingestion (optional)
export ANALYTICS_ENDPOINT="https://analytics.example.com/api/v1"
export ANALYTICS_API_KEY="your_analytics_api_key_here"

# Local buffering directory for network outages
export BUFFER_PATH="/var/expanso/buffer"
mkdir -p "$BUFFER_PATH"

# Verify environment
echo "Analytics endpoint: $ANALYTICS_ENDPOINT"
echo "Buffer directory: $BUFFER_PATH"

Step 2: Download Sample IoT Sensor Data​

Download sample sensor data representing 1000 temperature sensors generating 60 events/minute:

# Download sample IoT sensor data (1000 sensors × 60 events/minute)
curl -o sensor-data.jsonl https://github.com/expanso-io/examples/raw/main/data/iot-sensors.jsonl

# Verify download (should show ~60,000 events)
wc -l sensor-data.jsonl

# Inspect sample events
head -5 sensor-data.jsonl

Expected sample format:

{"sensor_id":"temp_001","location":"warehouse_a","temperature":72.3,"humidity":45.2,"timestamp":"2025-01-15T10:23:45.123Z"}
{"sensor_id":"temp_002","location":"warehouse_a","temperature":71.8,"humidity":44.9,"timestamp":"2025-01-15T10:23:45.456Z"}
{"sensor_id":"temp_003","location":"warehouse_b","temperature":73.1,"humidity":46.1,"timestamp":"2025-01-15T10:23:45.789Z"}

Step 3: Verify Cache Configuration​

Before proceeding with cache-based aggregation, verify your system can handle the memory requirements:

# Check available system memory
free -h

# Calculate expected memory usage for 1-minute windows:
# 1000 events/sec × 60 seconds × 1KB per event = ~60MB
echo "Expected memory usage for 1-minute windows: ~60MB"

# For 1-hour windows:
# 1000 events/sec × 3600 seconds × 1KB per event = ~3.6GB  
echo "Expected memory usage for 1-hour windows: ~3.6GB"

Next Steps​

Your environment is now configured for time-windowed aggregation. Choose your learning path: