Step 4: Batch and Encode to Parquet
Individual JSON transactions become compressed, columnar Parquet. Compression ratio and query performance depend on schema, values, row-group sizing, codecs, storage, and query shape; benchmark generated files before making performance claims.
Batching and encoding
The canonical reference owns output batching, region grouping, metadata capture, and the Parquet schema. The Explorer shows the flattened records and per-region objects.
The output flushes at the configured count or period, then groups records by region before encoding. A mixed-region flush produces separate objects, so each file can contain fewer rows than the batch limit. Region metadata survives encoding for use in the object path.
Measure Compression
Record raw JSON bytes, Parquet bytes, row count, schema, codec, row-group settings, and the exact generator seed for each run. The tutorial does not provide a verified compression ratio.
Why Parquet + DuckLake?
- Columnar: Queries can select a field without reading unrelated columns, subject to the engine and file layout
- Typed: No runtime type parsing — DuckDB knows every column type at read time
- Compressed: Zstandard on columnar data compresses extremely well (repeated store IDs, payment methods, etc.)
- DuckLake inlining: Small batches get inlined into the catalog, periodically flushed to Parquet — no small-file overhead
Next Step
Parquet-encoded batches are ready. Next, we write them to S3 with Hive-style partitioning for automatic DuckLake discovery.
