Step 4: Route to Storage

Route each table's data to organized cloud storage paths with compression and appropriate storage classes.

The Goal

• Route records to table-specific paths
• Organize by date for easy recovery
• Compress with Parquet for cost efficiency
• Use Nearline storage class for backups

Implementation

step-4-route.yaml

output:
  switch:
    cases:
      # Orders table
      - check: this._table == "orders"
        output:
          gcp_cloud_storage:
            bucket: '${GCS_BACKUP_BUCKET}'
            path: 'backups/orders/${!this._backup_metadata.backup_date}/orders-${!timestamp_unix()}.parquet'
            content_type: application/octet-stream
            storage_class: NEARLINE
            batching:
              count: 10000
              period: 60s
            parquet_encoding:
              compression: SNAPPY

      # Inventory table
      - check: this._table == "inventory"
        output:
          gcp_cloud_storage:
            bucket: '${GCS_BACKUP_BUCKET}'
            path: 'backups/inventory/${!this._backup_metadata.backup_date}/inventory-full.parquet'
            content_type: application/octet-stream
            storage_class: NEARLINE
            batching:
              count: 50000
              period: 120s
            parquet_encoding:
              compression: SNAPPY

      # Order items table
      - check: this._table == "order_items"
        output:
          gcp_cloud_storage:
            bucket: '${GCS_BACKUP_BUCKET}'
            path: 'backups/order_items/${!this._backup_metadata.backup_date}/items-${!timestamp_unix()}.parquet'
            content_type: application/octet-stream
            storage_class: NEARLINE
            batching:
              count: 10000
              period: 60s
            parquet_encoding:
              compression: SNAPPY

      # Fallback for unknown tables
      - output:
          gcp_cloud_storage:
            bucket: '${GCS_BACKUP_BUCKET}'
            path: 'backups/unknown/${!this._table}/${!this._backup_metadata.backup_date}/data-${!timestamp_unix()}.json'
            content_type: application/json
            storage_class: NEARLINE
            batching:
              count: 1000
              period: 30s

Understanding the Code

Component	Purpose
switch.cases	Route based on _table field
${!this._backup_metadata.backup_date}	Dynamic path from record
${!timestamp_unix()}	Unique file suffix
storage_class: NEARLINE	Authored storage-class choice
parquet_encoding.compression: SNAPPY	Authored codec choice

Choose a Storage Class

Provider pricing changes by region, operation, retention period, retrieval pattern, and contract. Use the current provider calculator and your measured access pattern; this repository does not pin a price table.

Evaluate Parquet

Measure object size, encode/decode behavior, restore behavior, and downstream query plans with representative records before choosing an encoding. This example supplies no compression or query benchmark.

Path Organization

gs://backup-bucket/
└── backups/
    ├── orders/
    │   ├── 2024-01-14/
    │   │   ├── orders-1705190400.parquet
    │   │   └── orders-1705194000.parquet
    │   └── 2024-01-15/
    │       └── orders-1705276800.parquet
    ├── inventory/
    │   └── 2024-01-15/
    │       └── inventory-full.parquet
    └── order_items/
        └── 2024-01-15/
            └── items-1705276800.parquet

Production Considerations

AWS S3 Alternative

output:
  aws_s3:
    bucket: '${S3_BACKUP_BUCKET}'
    path: 'backups/orders/${!this._backup_metadata.backup_date}/orders-${!timestamp_unix()}.parquet'
    storage_class: GLACIER_IR # Similar to Nearline
    batching:
      count: 10000

Encryption at Rest

Enable server-side encryption:

gcp_cloud_storage:
  # GCS encrypts by default, but for customer-managed keys:
  kms_key_name: 'projects/.../cryptoKeys/backup-key'

Retention with Object Lifecycle

# Set via gsutil (delete after 1 year)
gsutil lifecycle set lifecycle.json gs://${GCS_BACKUP_BUCKET}

---

Complete Pipeline

You've built all 4 backup steps! See the complete configuration:

📖 Complete Pipeline →