Explore Nightly Database Backup

How to use this explorer

Move between stages with the arrows or the numbered stage list. Each stage shows the input it receives on the left and the output it produces on the right, with the lines that changed marked. The configuration that makes the change sits below the comparison, and the final stage shows the complete pipeline.

The 4 stages in order

1. Extract multiple tables
   Read the orders, inventory, and order_items extracts in sequence and tag each row with its table and backup type.
2. Add backup metadata
   Attach the backup date, timestamp, source host, database, pipeline version, and node id to every row.
3. Calculate checksum
   Hash the original columns, excluding the backup fields, so a restore can verify each row.
4. Route to storage
   Switch on the table tag to a dated object path per table. Cloud delivery has not been exercised.

Interactive pipeline

NIGHTLY BACKUP REFERENCE

Sample rows run through expanso-edge

Stage1. Extract multiple tables2. Add backup metadata3. Calculate checksum4. Route to storage

Stage 1 of 4

Extract multiple tables

Read the orders, inventory, and order_items extracts in sequence and tag each row with its table and backup type.

6HighlightsAuthored emphasis only—not a computed diff.

Highlights only

Copy & download

Input

 { "order_id": 100231, "customer_id": 4471, "status": "shipped", "total": 189.95, "currency": "USD", "created_at": "2024-01-14T16:42:10Z", "updated_at": "2024-01-15T08:03:22Z" } { "sku": "SKU-20481", "warehouse": "WH-NORTH", "quantity_on_hand": 312, "reorder_point": 80, "updated_at": "2024-01-15T06:00:00Z" } { "item_id": 550912, "order_id": 100231, "sku": "SKU-20481", "quantity": 2, "unit_price": 89.99, "created_at": "2024-01-14T16:42:10Z" }

Output

 {•Authored highlight: "_backup_type": "incremental",•Authored highlight: "_table": "orders", "created_at": "2024-01-14T16:42:10Z", "currency": "USD", "customer_id": 4471, "order_id": 100231, "status": "shipped", "total": 189.95, "updated_at": "2024-01-15T08:03:22Z" } {•Authored highlight: "_backup_type": "full",•Authored highlight: "_table": "inventory", "quantity_on_hand": 312, "reorder_point": 80, "sku": "SKU-20481", "updated_at": "2024-01-15T06:00:00Z", "warehouse": "WH-NORTH" } {•Authored highlight: "_backup_type": "incremental",•Authored highlight: "_table": "order_items", "created_at": "2024-01-14T16:42:10Z", "item_id": 550912, "order_id": 100231, "quantity": 2, "sku": "SKU-20481", "unit_price": 89.99 }

Stage configuration01-extract-multiple-tables.yaml

input:
  # Sequence through multiple tables in one pipeline
  sequence:
    inputs:
      # Orders table
      - sql_select:
          driver: postgres  # Or: mysql, odbc, mssql
          dsn: "postgres://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:5432/${DB_NAME}?sslmode=require"
          table: orders
          columns: ["*"]
          where: "updated_at >= CURRENT_DATE - INTERVAL '1 day'"
        processors:
          - mapping: |
              root = this
              root._table = "orders"
              root._backup_type = "incremental"

      # Inventory table (full backup - smaller table)
      - sql_select:
          driver: postgres
          dsn: "postgres://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:5432/${DB_NAME}?sslmode=require"
          table: inventory
          columns: ["*"]
        processors:
          - mapping: |
              root = this
              root._table = "inventory"
              root._backup_type = "full"

      # Order line items
      - sql_select:
          driver: postgres
          dsn: "postgres://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:5432/${DB_NAME}?sslmode=require"
          table: order_items
          columns: ["*"]
          where: "created_at >= CURRENT_DATE - INTERVAL '1 day'"
        processors:
          - mapping: |
              root = this
              root._table = "order_items"
              root._backup_type = "incremental"