Skip to main content

Step 2: Convert Avro to Parquet

While Avro is excellent for streaming, Parquet is the industry standard for efficient analytics and data warehousing. It uses a columnar storage format that is highly optimized for analytical queries.

This step teaches you how to convert the Avro data from Step 1 into the Parquet format.

The Goal​

You will add processors to your pipeline to take the binary Avro data, convert it back to a structured JSON object, and then re-encode it into the binary Parquet format.

The "De-serialize -> Re-serialize" Pattern​

Converting between two binary formats is a two-step process:

  1. De-serialize from Avro: Use the from_avro processor to convert the binary Avro data back into a temporary, structured JSON object.
  2. Re-serialize to Parquet: Use the to_parquet processor to convert that JSON object into the binary Parquet format.

Implementation​

  1. Start with the Previous Pipeline: Copy the json-to-avro.yaml from Step 1 to a new file named avro-to-parquet.yaml.

    cp json-to-avro.yaml avro-to-parquet.yaml

  2. Add the Conversion Logic: Open avro-to-parquet.yaml. You will add two new processors to the end of the pipeline section.

    Add these processors to your pipeline in avro-to-parquet.yaml
    # --- Processors from Step 1 ---
    # (The existing mapping and to_avro processors remain here)

    # --- START: New additions for Parquet Conversion ---

    # 1. DE-SERIALIZE: Convert the Avro binary data back to a structured object.
    # We must use the same schema we used to encode it.
    - from_avro:
        schema_path: 'file://./sensor.avsc'

    # 2. RE-SERIALIZE: Convert the structured object to the Parquet format.
    # Parquet also uses the Avro schema to understand the data structure.
    - to_parquet:
        schema_path: 'file://./sensor.avsc'
        # For analytics, Snappy is a common and efficient compression choice.
        default_compression: snappy
    # --- END: New additions ---

    Note: The final output of this pipeline is now Parquet data, so you would typically change the output to write to a system that expects Parquet, like Amazon S3 or Google Cloud Storage.

  3. Deploy and Observe: Watch the logs. The final output will be another stream of garbled binary text, but this time it will be in the highly compressed, columnar Parquet format.

Verification​

Unreadable console bytes do not verify a Parquet conversion. Decode the result with the intended schema and consumer before treating it as valid output.

This page records configuration intent, not a current executed result.