Step 1: Collect Data Like inputs.conf

If you've configured Splunk before, you know inputs.conf is where data collection begins. This example uses related file path and metadata concepts.

Configuration Examples

Splunk inputs.conf Example

In Splunk, you'd monitor files like this:

# /opt/splunk/etc/apps/myapp/local/inputs.conf
[monitor:///var/log/expanso-demo/app.log]
disabled = false
followTail = 0
sourcetype = json_logs
index = main
host_segment = 1

[monitor:///var/log/expanso-demo/security.log]
disabled = false
followTail = 0
sourcetype = cef
index = security

[monitor:///var/log/expanso-demo/system.log]
disabled = false
followTail = 0
sourcetype = syslog
index = main

Expanso Pipeline Input

The example represents these settings in one pipeline configuration:

# pipeline.yaml
input:
  file_watcher:
    paths:
      - '/var/log/expanso-demo/*.log'

    # File path and multiline options used in this example
    include_file_name: true
    multiline:
      pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
      negate: false
      match: after

    # Parse JSON at collection time
    auto_parse_json: true

    # Metadata tagging
    metadata:
      source_system: 'expanso-edge'
      collection_time: '${timestamp()}'

Configured behavior in this example:

• The glob path selects the fixture log files.
• JSON parsing runs before the records reach the output.
• The mapping adds source and collection metadata.
• The multiline rule groups records that match the configured timestamp pattern.

Create Your First Expanso Pipeline

Let's build a pipeline that monitors our test data just like inputs.conf:

1. Create Pipeline Configuration

cat > ~/splunk-edge-pipeline.yaml << 'EOF'
apiVersion: v1
kind: Pipeline
metadata:
  name: "splunk-edge-collection"
description: "Collect data from the paths shown in the inputs.conf example"

input:
  file_watcher:
    paths:
      - "/var/log/expanso-demo/app.log"
      - "/var/log/expanso-demo/security.log"
      - "/var/log/expanso-demo/system.log"

    # File monitoring settings
    poll_interval: "1s"
    start_from_beginning: true
    include_file_name: true

    # Multiline support for stack traces
    multiline:
      pattern: '^(\d{4}-\d{2}-\d{2}T|\w{3}\s+\d{1,2}\s|\{"timestamp")'
      negate: false
      match: after
      timeout: "5s"

processors:
  # Add source file as field (like Splunk's source field)
  - mapping: |
      root.source_file = file.name
      root.collection_timestamp = timestamp()
      root.host = hostname()

      # Determine sourcetype based on file name (like Splunk)
      root.sourcetype = match file.name {
        this.contains("app.log") => "json_logs"
        this.contains("security.log") => "cef"
        this.contains("system.log") => "syslog"
        _ => "unknown"
      }

      # Determine target index based on content
      root.target_index = match {
        this.sourcetype == "cef" => "security"
        _ => "main"
      }

# For now, just output to console - we'll add Splunk HEC in Step 4
output:
  stdout:
    format: "json"
EOF

2. Deploy and Test the Pipeline

# Deploy the pipeline
expanso pipeline deploy ~/splunk-edge-pipeline.yaml

# Check pipeline status
expanso pipeline status splunk-edge-collection

# Follow pipeline logs
expanso pipeline logs splunk-edge-collection -f

3. Generate Test Data and Observe

In another terminal, generate some test events:

# Add events to test collection
echo '{"timestamp":"'$(date -u +"%Y-%m-%dT%H:%M:%S.%3NZ")'","level":"INFO","message":"New user registration","user":"test.user","source_ip":"192.168.1.150"}' >> $TEST_DATA_DIR/app.log

echo 'CEF:0|Company|WebApp|1.0|600|New Security Event|Medium|src=192.168.1.150 suser=test.user act=register outcome=success' >> $TEST_DATA_DIR/security.log

You should see output like this:

{
  "timestamp": "2024-02-10T10:15:01.123Z",
  "level": "INFO",
  "message": "New user registration",
  "user": "test.user",
  "source_ip": "192.168.1.150",
  "source_file": "/var/log/expanso-demo/app.log",
  "collection_timestamp": "2024-02-10T10:15:01.456Z",
  "host": "edge-node-01",
  "sourcetype": "json_logs",
  "target_index": "main"
}

Configuration Represented by These Snippets

Concern	inputs.conf snippet	Pipeline snippet
File selection	Explicit monitor stanzas	Glob path
Parsing point	Index-time configuration	Collection-time parsing
Metadata	Host, source, and index fields	Metadata mapping
Multiline	Per-source settings	One configured multiline pattern
Routing	Index field per stanza	Conditions in processors and outputs
Execution point	Splunk-managed	Pipeline-managed

Additional Pipeline Options

The following snippets illustrate format selection, log-rotation windows, and error handling.

1. Format Selection

processors:
  - mapping: |
      # Select a fixture log format from record content
      root.log_format = match {
        this.contains("CEF:") => "cef"
        this.contains("{") && this.contains("}") => "json" 
        this.contains("timestamp") => "structured"
        _ => "plain_text"
      }

2. File Rotation Handling

input:
  file_watcher:
    # Configure log-rotation windows
    ignore_older: '24h'
    close_inactive: '1h'
    clean_inactive: '72h'
    scan_frequency: '10s'

3. Error Resilience

input:
  file_watcher:
    # Continue processing if one file has issues
    ignore_errors: true
    retry_interval: '30s'
    max_retries: 3

Monitoring and Troubleshooting

Check Collection Metrics

# View pipeline metrics
expanso pipeline metrics splunk-edge-collection

# Check for any collection errors
expanso pipeline logs splunk-edge-collection --level error

Common Issues and Solutions

1. Permission Denied

   # Ensure expanso user can read log files
   sudo chown -R expanso:expanso /var/log/expanso-demo
   sudo chmod -R 644 /var/log/expanso-demo/*.log

2. File Not Found

   # Verify test data exists
   ls -la /var/log/expanso-demo/

3. Pipeline Won't Start

   # Check configuration syntax
   expanso pipeline validate ~/splunk-edge-pipeline.yaml

What's Next?

You are now collecting from the file paths represented in inputs.conf. Next, add parsing and field extraction corresponding to the props.conf and transforms.conf concepts used in this tutorial.

→ Next Step: Step 2: Parse Data Like props.conf

---

Key Takeaway: This example expresses file collection, parsing, and routing in one pipeline. If you are familiar with inputs.conf, the path and metadata concepts should be recognizable.