Advanced Format Transformation Patterns

Once you have mastered the basic conversions between formats, you can implement more sophisticated transformation patterns.

Pattern 1: Schema Evolution

Your data schemas will change over time. Schema-based formats like Avro and Protobuf have built-in support for schema evolution, allowing you to add or remove fields without breaking downstream consumers, as long as you follow certain rules (e.g., new fields must have default values).

Example: Adding an optional humidity field to an Avro schema.

v1 schema:

{
  "type": "record",
  "name": "Reading",
  "fields": [{ "name": "temperature", "type": "double" }]
}

v2 schema (backward-compatible):

{
  "type": "record",
  "name": "Reading",
  "fields": [
    { "name": "temperature", "type": "double" },
    // New fields must have a default value to be backward-compatible
    { "name": "humidity", "type": ["null", "double"], "default": null }
  ]
}

A consumer with the v2 schema can still read old v1 data (the humidity field will be null). A consumer with the v1 schema can read new v2 data (it will simply ignore the extra humidity field).

A schema registry can store and version schemas. The maintained Node.js encoder uses an embedded schema and does not implement registry lookup or schema evolution. Review producer and consumer compatibility before adapting it.

Pattern 2: Multi-Format Output

Sometimes you need to convert a single input into multiple different output formats simultaneously. This is a common pattern for data distribution.

Use Case: An incoming JSON event needs to be sent to a Kafka topic as Avro (for streaming) and archived to S3 as Parquet (for analytics).

See the maintained Avro integration source and Parquet reference for their separate encoder boundaries. Combining them requires explicit fan-out, schema agreement, and per-destination verification.

This pattern uses a broker output to fan the message out. Each branch of the fan-out has its own processors block to perform the format-specific conversion before sending the data to its final destination.