Using Apache Avro with Python

Abstract:
Instructions are given on how to use the Python implementation of Avro to load multiple schemas,  stored in separate files, to build an overall composite schema.

Details:
The following example uses two Avro schema files, where the second uses the first.  There is also an example of a Python script, which combines and tests the composite schema by outputting to the console.

First schema file:  parameter_types.avsc

{"namespace": "scriptaxe.parameter",
 "type": "enum",
 "name": "types",
 "symbols": [
    "null",
    "boolean",
    "int",
    "long",
    "float",
    "double",
    "bytes",
    "string"
    ]
}

Second schema file:  parameter.avsc

{"namespace": "scriptaxe",
 "type": "record",
 "name": "parameter",
 "fields": [
   {
     "name": "name",
     "type": "string"
   },{
     "name": "description",
     "type": ["null", "string"]
   },{
     "name": "type",
     "type": "scriptaxe.parameter.types"
   }
 ]
}

Python file:  avro_test.py

import avro.schema
import json 

def main():
  """Start of execution"""

  #combine the schemas 

  known_schemas = avro.schema.Names()

  types_schema = LoadAvsc("parameter_types.avsc", known_schemas)

  param_schema = LoadAvsc("parameter.avsc", known_schemas)

  print json.dumps(param_schema.to_json(avro.schema.Names()), indent=2) 

  #test the schema works 

  param_file = open("parameters.avro", "w")

  writer = DataFileWriter(param_file, DatumWriter(), param_schema)

  param_1 = {"name": "test", "description":"An Avro test.", "type":"int"}

  param_2 = {"name": "test", "description":"An Avro test.", "type":"boolean"}

  writer.append(param_1)

  writer.append(param_2)

  writer.close()

  reader = DataFileReader(open("parameters.avro", "r"), DatumReader())

  for parameter in reader:

      print parameter

  reader.close()  

def LoadAvsc(file_path, names=None):
  """Load avsc file
    file_path: path to schema file
    names(optional): avro.schema.Names object
  """

  file_text = open(file_path).read() 

  json_data = json.loads(file_text)

  schema = avro.schema.make_avsc_object(json_data, names) 

  return schema 

if __name__ == "__main__":
  main()

Output to console:

{
  "type": "record",
  "namespace": "scriptaxe",
  "name": "parameter",
  "fields": [
    {
      "type": "string",
      "name": "name"
    }, {
      "type": ["null", "string"],
      "name": "description"
    }, {
      "type": {
        "symbols": ["null", "boolean", "int", "long", "float", "double", "bytes", "string"],
        "namespace": "scriptaxe.parameter",
        "type": "enum",
        "name": "types"
      },
      "name": "type"
    }
  ]
}

{u'type': u'int', u'name': u'test', u'description': u'An Avro test.'}
{u'type': u'boolean', u'name': u'test', u'description': u'An Avro test.'}

Discussion:
Avro's documentation is sparse.  This article is intended to help those who are curious to know if the Python implementation of Avro can reuse separately defined schemas.  The answer is yes, and a working example was presented above.

Keep in mind that the data used in this example, param_1 and param_2, have key names which match some of Avro's.  Obviously these key names were chosen to mimic Avro, not because they were specifically required.

It is important to note that the base schemas that must be "inherited", in this case the schema contained in parameter_types.avsc, must be loaded first.

It would be nice to have a bulk schema loader that could take a set schemas in any order and manage the correct loading process for us.


Additional Notes:
Installing Avro for Python is easy:

$sudo pip install avro

References:

1. Using Apache Avro , Boris Lublinsky on Jan 25, 2011.
2. Apache Avro™ 1.7.3 Getting Started (Python)
3. Apache Avro™ 1.7.3 Specification