Step 1: Convert JSON to Avro

A common transformation is to convert from a text-based format like JSON to a schema-based, binary format like Avro. Avro is more compact and efficient for high-volume streaming, especially with systems like Kafka.

The Goal

You will create a pipeline that takes a JSON message, validates it against an Avro schema, and converts it to the binary Avro format.

The "Prepare -> Convert" Pattern

1. Define Schema: Avro requires a schema that describes the data. You'll create a simple .avsc file.
2. Prepare: The incoming JSON data must be structured to perfectly match the schema. This often involves a mapping processor to ensure data types are correct (e.g., converting a timestamp string to an integer).
3. Convert: Select an encoder whose wire format and schema match the consumer. The maintained integration uses a Node.js subprocess.

Implementation

1. Define the Avro Schema: Create a file named sensor.avsc. This schema defines the structure for our sensor data. Notice that the timestamp is a long (a 64-bit integer), not a string.

   sensor.avsc

   {
     "type": "record",
     "name": "SensorReading",
     "fields": [
       { "name": "sensor_id", "type": "string" },
       { "name": "temperature", "type": "double" },
       {
         "name": "timestamp",
         "type": "long",
         "logicalType": "timestamp-millis"
       }
     ]
   }

2. Inspect the maintained integration: See the JSON-to-Avro source. It accepts /sensors/ingest, encodes a fixed sensor schema in a Node.js subprocess, and sends binary records to Kafka. Node.js must be available on each edge node. Its embedded schema differs from the illustrative schema above; do not substitute schemas without changing the encoder and consumer together.

Verification

Decode emitted records with the matching schema and compare field values. Unreadable console bytes do not prove an Avro conversion. The complete HTTP reference is a separate JSON/CSV/XML example.