Sunday, December 23, 2012

Using Apache Avro with Python

Instructions are given on how to use the Python implementation of Avro to load multiple schemas,  stored in separate files, to build an overall composite schema.

The following example uses two Avro schema files, where the second uses the first.  There is also an example of a Python script, which combines and tests the composite schema by outputting to the console.

First schema file:  parameter_types.avsc

{"namespace": "scriptaxe.parameter",
 "type": "enum",
 "name": "types",
 "symbols": [
Second schema file:  parameter.avsc

{"namespace": "scriptaxe",
 "type": "record",
 "name": "parameter",
 "fields": [
     "name": "name",
     "type": "string"
     "name": "description",
     "type": ["null", "string"]
     "name": "type",
     "type": "scriptaxe.parameter.types"
Python file:

import avro.schema
import json
def main():
  """Start of execution"""
  #combine the schemas 
  known_schemas = avro.schema.Names()
  types_schema = LoadAvsc("parameter_types.avsc", known_schemas)
  param_schema = LoadAvsc("parameter.avsc", known_schemas)
  print json.dumps(param_schema.to_json(avro.schema.Names()), indent=2) 
  #test the schema works 
  param_file = open("parameters.avro", "w")
  writer = DataFileWriter(param_file, DatumWriter(), param_schema)
  param_1 = {"name": "test", "description":"An Avro test.", "type":"int"}
  param_2 = {"name": "test", "description":"An Avro test.", "type":"boolean"}
  reader = DataFileReader(open("parameters.avro", "r"), DatumReader())
  for parameter in reader:
      print parameter
def LoadAvsc(file_path, names=None):
  """Load avsc file
    file_path: path to schema file
    names(optional): avro.schema.Names object
  file_text = open(file_path).read() 
  json_data = json.loads(file_text)
  schema = avro.schema.make_avsc_object(json_data, names) 
  return schema 
if __name__ == "__main__":
Output to console:

  "type": "record",
  "namespace": "scriptaxe",
  "name": "parameter",
  "fields": [
      "type": "string",
      "name": "name"
    }, {
      "type": ["null", "string"],
      "name": "description"
    }, {
      "type": {
        "symbols": ["null", "boolean", "int", "long", "float", "double", "bytes", "string"],
        "namespace": "scriptaxe.parameter",
        "type": "enum",
        "name": "types"
      "name": "type"
{u'type': u'int', u'name': u'test', u'description': u'An Avro test.'}
{u'type': u'boolean', u'name': u'test', u'description': u'An Avro test.'}
Avro's documentation is sparse.  This article is intended to help those who are curious to know if the Python implementation of Avro can reuse separately defined schemas.  The answer is yes, and a working example was presented above.

Keep in mind that the data used in this example, param_1 and param_2, have key names which match some of Avro's.  Obviously these key names were chosen to mimic Avro, not because they were specifically required.

It is important to note that the base schemas that must be "inherited", in this case the schema contained in parameter_types.avsc, must be loaded first.

It would be nice to have a bulk schema loader that could take a set schemas in any order and manage the correct loading process for us.

Additional Notes:
Installing Avro for Python is easy:
$sudo pip install avro


  1. Using Apache Avro , Boris Lublinsky on Jan 25, 2011.
  2. Apache Avro™ 1.7.3 Getting Started (Python)
  3. Apache Avro™ 1.7.3 Specification


  1. This is the best Avro example on the web. You have my utmost thanks. Cheers!

  2. This comment has been removed by the author.

  3. Hi Michael,

    Great blog! I had been looking for a solution all over the internet.

    Mind updating the link to your new blog please? seems to be broken.

    Many thanks,

    1. I'm glad you found it useful. I stopped blogging, and this site is all that remains. I deleted the previous comment with the broken link.