Skip to main content

Setup Environment for Production Log Processing

Before building the production log processing pipeline, you'll set up your environment and configure required services.

Prerequisites​

This example requires the following services to be running:

Before you begin, please ensure these services are set up and running according to their respective guides. Additionally, ensure you have completed the Local Development Setup guide for general environment configuration. You will also need Terminal access with curl and jq installed.

Step 1: Configure Environment Variables​

Set up environment variables for external services. Create a file called .env:

Create .env file
cat > .env << 'EOF'
# Elasticsearch configuration (optional)
ELASTICSEARCH_URL="http://localhost:9200"
ELASTICSEARCH_USERNAME=""
ELASTICSEARCH_PASSWORD=""

# AWS S3 configuration (optional)
S3_BUCKET="your-company-logs-$(date +%s)"
AWS_PROFILE="default"
AWS_REGION="us-west-2"

# Node metadata (automatically set by Expanso)
NODE_ID="edge-node-$(hostname)"
NODE_REGION="datacenter-1"

# Pipeline configuration
LOG_PROCESSING_PORT="8080"
LOG_PROCESSING_TIMEOUT="10s"
LOG_PROCESSING_RATE_LIMIT="1000/1s"
EOF

Load the environment variables:

# Load environment variables
source .env

Step 2: Test External Services (Optional)​

If you're using external services, verify they're accessible:

Test Elasticsearch Connection​

# Test Elasticsearch connectivity
if [ ! -z "$ELASTICSEARCH_URL" ]; then
    curl -X GET "$ELASTICSEARCH_URL/_cluster/health?pretty"
fi

Test AWS S3 Access​

# Create S3 bucket if it doesn't exist
if [ ! -z "$S3_BUCKET" ]; then
    aws s3 mb s3://$S3_BUCKET --region $AWS_REGION 2>/dev/null || echo "Bucket already exists"
    
    # Test write permissions
    echo "test" | aws s3 cp - s3://$S3_BUCKET/test.txt
    aws s3 rm s3://$S3_BUCKET/test.txt
    
    echo "✅ S3 access verified"
fi

Step 3: Download Sample Log Data​

Create sample log data for testing:

# Create sample logs directory
mkdir -p sample-logs

# Create diverse log samples
cat > sample-logs/app-logs.json << 'EOF'
{"timestamp":"2025-10-20T10:30:45.123Z","level":"INFO","service":"api-gateway","message":"User login successful","request_id":"req_001","user_id":"user_123","endpoint":"/auth/login","method":"POST","status_code":200,"duration_ms":45,"ip_address":"192.168.1.100"}
{"timestamp":"2025-10-20T10:31:12.456Z","level":"WARN","service":"api-gateway","message":"Slow database query detected","request_id":"req_002","user_id":"user_456","endpoint":"/api/v1/reports","method":"GET","status_code":200,"duration_ms":1850,"ip_address":"192.168.1.105"}
{"timestamp":"2025-10-20T10:32:30.789Z","level":"ERROR","service":"payment-service","message":"Payment processing failed","request_id":"req_003","user_id":"user_789","endpoint":"/api/v1/payments","method":"POST","status_code":500,"duration_ms":3200,"error_code":"PAYMENT_GATEWAY_TIMEOUT","ip_address":"192.168.1.110"}
{"timestamp":"2025-10-20T10:33:00.000Z","level":"DEBUG","service":"api-gateway","message":"Cache hit for user profile","request_id":"req_004","user_id":"user_123","cache_key":"profile:user_123","ip_address":"192.168.1.100"}
EOF

# Create malformed logs for testing error handling
cat > sample-logs/malformed-logs.json << 'EOF'
{"level":"ERROR","message":"Missing timestamp field"}
{"timestamp":"invalid-date","level":"INFO","service":"test","message":"Invalid timestamp format"}
not-json-at-all
{"timestamp":"2025-10-20T10:30:45.123Z","level":"INFO"}
EOF

echo "✅ Sample log data created in sample-logs/"

Next: Configure HTTP Input to build a production-ready log ingestion endpoint.