top of page
BlogPageTop

Apache Avro Schema Example (in Java)

Introduction

  • Avro provides data serialization based on JSON Schema.

  • It is language neutral data serialization system, means a language A can serialize and languages B can de-serialize and use it.

  • Avro supports both dynamic and static types as per requirement.

  • It supports many languages like Java,C, C++, C#, Python and Ruby.


Benefits

  • Producers and consumers are decoupled from their change in application.

  • Schemas help future proof your data and make it more robust.

  • Supports and used in all use cases in streaming specially in Kafka.

  • Avro are compact and fast for streaming.

  • Supports for schema registry in case of Kafka.


Steps to Serialize Object

  • Create JSON schema.

  • Compile the schema in the application.

  • Populate the schema with data.

  • Serialize data using Avro serializer.



Steps to Deserialize Object

  • Use Apache Avro api to read the serialized file.

  • Populate the schema from file.

  • Use the object for application.


 

Sample Example for Avro (in Java)


Step-1: Create a Java project and add the dependencies as below.


Project Structure and Dependencies


Step-2: Create a Schema file as below:


Customer_v0.avsc


{

"namespace": "com.demo.avro",

"type": "record",

"name": "Customer",

"fields": [

{

"name": "id",

"type": "int"

},

{

"name": "name",

"type": "string"

},

{

"name": "faxNumber",

"type": [

"null",

"string"

],

"default": "null"

}

]

}


Step-3: Compile the schema.

java -jar lib\avro-tools-1.8.1.jar compile schema schema\Customer_v0.avsc schema


Step-4: Put the java generated file to the source directory of the project as shown in project structure.


Step-5: Create the Producer.java


package com.demo.producer;


import java.io.File;

import java.io.IOException;

import org.apache.avro.file.DataFileWriter;

import org.apache.avro.io.DatumWriter;

import org.apache.avro.specific.SpecificDatumWriter;


import com.demo.avro.Customer;


public class Producer {


public static void main(String[] args)throws IOException {

serailizeMessage();

}


public static void serailizeMessage()throws IOException{

DatumWriter<Customer> datumWriter = new SpecificDatumWriter<Customer>(Customer.class);

DataFileWriter<Customer> dataFileWriter = new DataFileWriter<Customer>(datumWriter);

File file = new File("customer.avro");

Customer customer=new Customer();

dataFileWriter.create(customer.getSchema(), file);

customer.setId(1001);

customer.setName("Customer -1");

customer.setFaxNumber("284747384343333".subSequence(0, 10));

dataFileWriter.append(customer);

customer=new Customer();

customer.setId(1002);

customer.setName("Customer -2");

customer.setFaxNumber("45454747384343333".subSequence(0, 10));

dataFileWriter.append(customer);

dataFileWriter.close();

}

}


Step-6: Create the Consumer.java


package com.demo.consumer;

import java.io.File;

import java.io.IOException;

import org.apache.avro.file.DataFileReader;

import org.apache.avro.io.DatumReader;

import org.apache.avro.specific.SpecificDatumReader;

import com.demo.avro.Customer;


public class Consumer {


public static void main(String[] args)throws IOException {

deSerailizeMessage();

}

public static void deSerailizeMessage()throws IOException{

File file = new File("customer.avro");

DatumReader<Customer> datumReader = new SpecificDatumReader<Customer>(Customer.class);

DataFileReader<Customer> dataFileReader= new DataFileReader<Customer>(file,datumReader);

Customer customer=null;

while(dataFileReader.hasNext()){

customer=dataFileReader.next(customer);

System.out.println(customer);

}

}

}


Step-7: Run Producer.java

It creates customer.avro file and puts the customer in Avro format.


Step-8: Run Consumer.java

It reads the customer.avro file and get the customer records.


Thank you! If you have any question please mention in comments section below.


[12/09/2019 10:38 PM CST - Reviewed by: PriSin]


3 Comments


daniel caceres
daniel caceres
Oct 14, 2023

Is there a maven version for it or git repo, it is well explained, but new to avro, it would be nice to have a maven version or git repo. thanks for sharing it.

Like

Dataneb Team
Dataneb Team
Sep 11, 2022

There is no git-repo for this.

Like

Thanks for sharing this. Do we have git-repo for this?

Like

Want to share your thoughts about this blog?

Disclaimer: Please note that the information provided on this website is for general informational purposes only and should not be taken as legal advice. Dataneb is a platform for individuals to share their personal experiences with visa and immigration processes, and their views and opinions may not necessarily reflect those of the website owners or administrators. While we strive to keep the information up-to-date and accurate, we make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability, or availability with respect to the website or the information, products, services, or related graphics contained on the website for any purpose. Any reliance you place on such information is therefore strictly at your own risk. We strongly advise that you consult with a qualified immigration attorney or official government agencies for any specific questions or concerns related to your individual situation. We are not responsible for any losses, damages, or legal disputes arising from the use of information provided on this website. By using this website, you acknowledge and agree to the above disclaimer and Google's Terms of Use (https://policies.google.com/terms) and Privacy Policy (https://policies.google.com/privacy).

bottom of page