Skip to main content

Step 3: Calculate Checksum

Add a checksum to each row so you can verify data integrity during recovery.

The Goal​

  • Calculate MD5 hash of the original data fields
  • Exclude metadata fields from the hash
  • Enable integrity verification during restore

Why Checksums Matter​

Corruption Detection: Verify backup files weren't corrupted in storage.

Recovery Validation: Confirm restored data matches original.

Review context: Compare serialized rows during a restore exercise.

Implementation​

See the checksum processor in the canonical reference and its captured input and output in the Explorer.

Understanding the Code​

ExpressionPurpose
this.without(...)Exclude metadata fields
.format_json()Serialize to consistent JSON string
.hash("md5")Calculate MD5 digest bytes
.encode("hex")Encode digest bytes as hexadecimal text

Why Exclude Metadata?​

The checksum should represent the original data, not backup artifacts:

// Compare the checksum for this exact serialized input:
{"order_id": 12345, "customer_id": "CUST-001", "total": 99.99}

// Regardless of when/where it was backed up:
{"_backup_metadata": {"backup_timestamp": "..."}}

Expected Output​

Use the Explorer checkpoints for the sample input and resulting fields.

Verification During Recovery​

After restoring, verify checksums:

import hashlib
import json

def verify_row(row):
    # Extract data fields (exclude metadata)
    data = {k: v for k, v in row.items()
            if not k.startswith('_')}

    # Calculate checksum
    calculated = hashlib.md5(
        json.dumps(data, sort_keys=True).encode()
    ).hexdigest()

    # Compare
    return calculated == row['_checksum']

Production Considerations​

SHA-256 for Stronger Integrity​

For environments that require a cryptographic digest:

Partial snippet
root._checksum = $data_fields.format_json().hash("sha256").encode("hex")

Include Primary Key in Metadata​

For easier debugging:

Partial snippet
root._checksum_context = {
  "primary_key": this.order_id,
  "checksum": $data_fields.format_json().hash("md5").encode("hex"),
  "field_count": $data_fields.keys().length()
}

Batch-Level Checksum​

Also calculate checksum for entire batch:

Partial snippet
# In output batching config
output:
  processors:
    - mapping: |
        # This runs on batched data
        root._batch_checksum = this.format_json().hash("sha256").encode("hex")

Next Step​

📖 Step 4: Route to Storage →