Advanced Enrichment & Export Patterns

Once you have mastered the basic "enrich, restructure, batch, export" pattern, you can implement more advanced features.

Pattern 1: Hive-Style S3 Partitioning

For large-scale analytics, storing files with a simple date prefix isn't enough. Hive-style partitioning is a naming convention (key=value) that is automatically understood by big data query engines like AWS Athena, Presto, and Spark. This allows them to dramatically prune the data they need to scan, saving time and money.

S3 Output with Hive-Style Partitioning

output:
  aws_s3:
    bucket: ${S3_BUCKET_NAME}
    # This path creates partitions that analytics engines can use for pruning
    path: "logs/year=${!timestamp("2006")}/month=${!timestamp("01")}/day=${!timestamp("02")}/hour=${!timestamp("15")}/${!uuid_v4()}.jsonl.gz"
    # ... other config

This key layout exposes year and month prefixes that a compatible query engine may use for pruning. Verify the behavior in the selected engine.

Pattern 2: Compression

To configure gzip output, place a compress processor inside the batching block. Measure storage and query effects in the target environment.

S3 Output with Gzip Compression

output:
  aws_s3:
    bucket: ${S3_BUCKET_NAME}
    path: 'logs/.../${!uuid_v4()}.jsonl.gz' # Note the .gz extension
    batching:
      count: 100
      period: 60s
      # This processor runs on the batch *before* it gets sent
      processors:
        - compress:
            algorithm: gzip
    # Let S3 know the content is compressed
    content_encoding: gzip

Pattern 3: Multi-Destination Routing

You may want to send different types of logs to different places. For example, ERROR logs might go to a high-priority S3 bucket for immediate alerting, while INFO logs go to a standard bucket for archival.

Multi-Destination S3 Export

output:
  broker:
    pattern: fan_out
    outputs:
      # --- Output 1: ERROR logs to a priority bucket ---
      - processors:
        # This processor retains error records in this branch
          - mapping: `root = this.filter(log -> log.event.level == "ERROR")`
        aws_s3:
          bucket: "my-priority-error-logs"
          path: "errors/${!timestamp_unix_date()}/${!uuid_v4()}.jsonl.gz"
          # ... use a smaller, faster batching policy here

      # --- Output 2: All other logs to a standard bucket ---
      - processors:
          - mapping: `root = this.filter(log -> log.event.level != "ERROR")`
        aws_s3:
          bucket: "my-standard-logs"
          path: "logs/${!timestamp_unix_date()}/${!uuid_v4()}.jsonl.gz"
          # ... use a larger, more cost-effective batching policy here

This pattern uses a broker to fan out the data, with each output having its own processor to filter the batch for the appropriate log level before sending it to S3.