Configure Secure HTTP Input

The first line of defense in any production pipeline is a secure and robust input configuration. In this step, we'll set up a rate-limited HTTP server that authenticates requests using API keys and handles CORS for web-based clients.

Goal

Create a secure entry point for log ingestion that:

• Authenticates clients using API keys
• Rate limits requests to prevent abuse
• Validates request size and timeouts
• Handles CORS for browser-based logging

Configuration

We'll use the http_server input with authentication and performance tuning settings.

1. Basic HTTP Server

First, let's define the basic server structure listening on port 8080.

input:
  http_server:
    address: "0.0.0.0:8080"
    path: /logs/ingest
    timeout: 10s

2. Adding Security

Now we add API key authentication and CORS headers.

    # Authentication
    auth:
      type: header
      header: "X-API-Key"
      required_value: "${LOG_API_KEY}"
    
    # Security headers (CORS)
    cors:
      enabled: true
      allowed_origins: ["*"]
      allowed_methods: ["POST"]

3. Performance & Protection

Finally, we add rate limiting and size constraints to protect the pipeline.

    # Rate Limiting
    rate_limit: "1000/1s"
    
    # Resource Protection
    max_request_size: 1048576  # 1MB
    max_connections: 1000
    keepalive: true

Complete Step 1 Configuration

Combine these into a single configuration file.

production-pipeline-step-1.yaml

# Step 1: Secure HTTP Input
input:
  http_server:
    address: "0.0.0.0:8080"
    path: /logs/ingest
    timeout: 10s
    rate_limit: "1000/1s"
    
    # Security
    auth:
      type: header
      header: "X-API-Key"
      required_value: "${LOG_API_KEY}"
    
    cors:
      enabled: true
      allowed_origins: ["*"]
      allowed_methods: ["POST"]
      
    # Protection
    max_request_size: 1048576
    max_connections: 1000

pipeline:
  processors:
    # Temporary passthrough for testing
    - log:
        level: INFO
        message: "Received log event"

output:
  stdout: {}

Deployment & Verification

1. Set the API Key:

   export LOG_API_KEY="secret-key-123"

2. Test Access (Success):

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: secret-key-123" \
     -d '{"message": "hello"}'

   Expected: 200 OK

3. Test Access (Failure - Wrong Key):

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: wrong-key" \
     -d '{"message": "hello"}'

   Expected: 401 Unauthorized

Next Steps

Now that we have a secure door, we need to ensure the data coming through it is valid.

👉 Step 2: Parse & Validate Logs