Step 4: Route to Storage
Route each table's data to organized cloud storage paths with compression and appropriate storage classes.
The Goal
- Route records to table-specific paths
- Organize by date for easy recovery
- Compress with Parquet for cost efficiency
- Use Nearline storage class for backups
Implementation
See the complete configuration for the authoritative output routes, batching processors, Parquet schema, and object-key expressions. Each encoded batch gets a unique identifier so later batches cannot overwrite earlier objects.
Understanding the Code
| Component | Purpose |
|---|---|
switch.cases | Route based on _table field |
${!this._backup_metadata.backup_date} | Dynamic path from record |
| Batch identifier | Unique object suffix |
storage_class: NEARLINE | Authored storage-class choice |
parquet_encode | Encode each collected batch |
Choose a Storage Class
Provider pricing changes by region, operation, retention period, retrieval pattern, and contract. Use the current provider calculator and your measured access pattern; this repository does not pin a price table.
Evaluate Parquet
Measure object size, encode/decode behavior, restore behavior, and downstream query plans with representative records before choosing an encoding. This example supplies no compression or query benchmark.
Path Organization
See the storage layout for table and date paths.
Production Considerations
AWS S3 Alternative
Adapt the complete configuration's batching and Parquet encoding boundary to the S3 output, preserving a unique object identifier per batch. Review storage-class and restore behavior independently.
Encryption at Rest
Enable server-side encryption:
gcp_cloud_storage:
bucket: '${GCS_BUCKET:validation-bucket}'
path: 'backups/${!uuid_v4()}.jsonl'
Retention with Object Lifecycle
# Set via gsutil (delete after 1 year)
gsutil lifecycle set lifecycle.json gs://${GCS_BACKUP_BUCKET}
Complete Pipeline
You've built all 4 backup steps! See the complete configuration:
