Skip to main content

Explore DB2 to BigQuery Migration

How to use this explorer

Move between stages with the arrows or the numbered stage list. Each stage shows the input it receives on the left and the output it produces on the right, with the lines that changed marked. The configuration that makes the change sits below the comparison, and the final stage shows the complete pipeline.

The 6 stages in order

  1. Add lineage metadata
    Stamp each DB2 row with the source system, table, pipeline, extraction time, and node id for audit.
  2. Normalize currency
    Keep the original amount and currency, then convert to USD with the example rate table.
  3. Mask account numbers
    Replace the account number with a masked suffix and a truncated SHA-256 hash for joins.
  4. Categorize transactions
    Map the merchant category code prefix to a transaction category.
  5. Standardize schema
    Rename the DB2 columns to the BigQuery schema and add the daily partition field.
  6. Validate required fields
    Reject any row missing a transaction id, customer id, or USD amount before the load.

Interactive pipeline

DB2 TO BIGQUERY REFERENCE

Sample row run through expanso-edge

Stage 1 of 6

Add lineage metadata

Stamp each DB2 row with the source system, table, pipeline, extraction time, and node id for audit.

6HighlightsAuthored emphasis only—not a computed diff.
Copy & download

Input

{"TRANSACTION_ID": "TXN-2024-00123456","ACCOUNT_NUMBER": "4532-1234-5678-9012","CUSTOMER_ID": "CUST-789012","TRANSACTION_DATE": "2024-01-15","TRANSACTION_TYPE": "PURCHASE","AMOUNT": 125.5,"CURRENCY": "EUR","MERCHANT_NAME": "ACME Electronics GmbH","MERCHANT_CATEGORY_CODE": "5411","SOURCE_SYSTEM": "CORE_BANKING_EU","CREATED_AT": "2024-01-15T14:32:17Z"}

Output

{"ACCOUNT_NUMBER": "4532-1234-5678-9012","AMOUNT": 125.5,"CREATED_AT": "2024-01-15T14:32:17Z","CURRENCY": "EUR","CUSTOMER_ID": "CUST-789012","MERCHANT_CATEGORY_CODE": "5411","MERCHANT_NAME": "ACME Electronics GmbH","SOURCE_SYSTEM": "CORE_BANKING_EU","TRANSACTION_DATE": "2024-01-15","TRANSACTION_ID": "TXN-2024-00123456","TRANSACTION_TYPE": "PURCHASE",Authored highlight: "_lineage": {Authored highlight: "extracted_at": "2026-10-05T18:35:27.467696-07:00",Authored highlight: "node_id": "unknown",Authored highlight: "pipeline": "db2-to-bigquery-transactions",Authored highlight: "source_system": "DB2_PROD",Authored highlight: "source_table": "TRANSACTIONS"}}
Stage configuration01-add-lineage-metadata.yaml
pipeline:
  processors:
    # Step 1: Add lineage metadata (critical for audit)
    - mapping: |
        root = this
        root._lineage = {
          "source_system": "DB2_PROD",
          "source_table": "TRANSACTIONS",
          "pipeline": "db2-to-bigquery-transactions",
          "extracted_at": now(),
          "node_id": env("NODE_ID").or("unknown")
        }