Skip to main content

Step 5: Ship to S3 with Partitioning

The final edge step: write Parquet files to S3 with Hive-style partitioning so DuckLake can auto-discover and index them.

Hive-Style Partitioning​

DuckLake expects partitioned paths like:

s3://my-retail-data/transactions/
  region=NW/date=2026-02-11/batch_1707616800_001.parquet
  region=NW/date=2026-02-11/batch_1707616810_002.parquet
  region=SW/date=2026-02-11/batch_1707616800_001.parquet
  ...

This layout gives downstream engines partition metadata they may use when filtering by region or date. Measure the actual query plan and runtime with representative data.

S3 Output​

Use the output configuration in the canonical reference. It reads the region from message metadata because the encoded payload is Parquet, not JSON.

Local adaptation​

For filesystem inspection, retain the reference batching and encoding processors and use an all-bytes file output. Preserve the region metadata when constructing local partition paths. A separate local copy requires its own retention and replay design; fan-out alone does not configure either.

What Lands in S3​

Illustrative partition layout (not an execution result):

s3://my-retail-data/transactions/
├── region=MW/
│   └── date=2026-02-11/
│       ├── batch_1707616800_001.parquet
│       ├── batch_1707616810_002.parquet
│       └── ...
├── region=NE/
│   └── date=2026-02-11/
│       └── ...
├── region=NW/
│   └── date=2026-02-11/
│       └── ...
├── region=SE/
│   └── date=2026-02-11/
│       └── ...
└── region=SW/
    └── date=2026-02-11/
        └── ...

Object row counts depend on region grouping within each flush. Measure file sizes and configure downstream table registration before querying.

Next Step​

Review the external analytics integration before querying stored objects.

→ Step 6: Connect DuckLake and Query