Deprecated

This blog has been deprecated.

Using Apache Avro with Python

Abstract:
Instructions are given on how to use the Python implementation of Avro to load multiple schemas,  stored in separate files, to build an overall composite schema.

Details:
The following example uses two Avro schema files, where the second uses the first.  There is also an example of a Python script, which combines and tests the composite schema by outputting to the console.

First schema file:  parameter_types.avsc

{"namespace": "scriptaxe.parameter",
 "type": "enum",
 "name": "types",
 "symbols": [
    "null",
    "boolean",
    "int",
    "long",
    "float",
    "double",
    "bytes",
    "string"
    ]
}

Second schema file:  parameter.avsc

{"namespace": "scriptaxe",
 "type": "record",
 "name": "parameter",
 "fields": [
   {
     "name": "name",
     "type": "string"
   },{
     "name": "description",
     "type": ["null", "string"]
   },{
     "name": "type",
     "type": "scriptaxe.parameter.types"
   }
 ]
}

Python file:  avro_test.py

import avro.schema
import json 

def main():
  """Start of execution"""

  #combine the schemas 

  known_schemas = avro.schema.Names()

  types_schema = LoadAvsc("parameter_types.avsc", known_schemas)

  param_schema = LoadAvsc("parameter.avsc", known_schemas)

  print json.dumps(param_schema.to_json(avro.schema.Names()), indent=2) 

  #test the schema works 

  param_file = open("parameters.avro", "w")

  writer = DataFileWriter(param_file, DatumWriter(), param_schema)

  param_1 = {"name": "test", "description":"An Avro test.", "type":"int"}

  param_2 = {"name": "test", "description":"An Avro test.", "type":"boolean"}

  writer.append(param_1)

  writer.append(param_2)

  writer.close()

  reader = DataFileReader(open("parameters.avro", "r"), DatumReader())

  for parameter in reader:

      print parameter

  reader.close()  

def LoadAvsc(file_path, names=None):
  """Load avsc file
    file_path: path to schema file
    names(optional): avro.schema.Names object
  """

  file_text = open(file_path).read() 

  json_data = json.loads(file_text)

  schema = avro.schema.make_avsc_object(json_data, names) 

  return schema 

if __name__ == "__main__":
  main()

Output to console:

{
  "type": "record",
  "namespace": "scriptaxe",
  "name": "parameter",
  "fields": [
    {
      "type": "string",
      "name": "name"
    }, {
      "type": ["null", "string"],
      "name": "description"
    }, {
      "type": {
        "symbols": ["null", "boolean", "int", "long", "float", "double", "bytes", "string"],
        "namespace": "scriptaxe.parameter",
        "type": "enum",
        "name": "types"
      },
      "name": "type"
    }
  ]
}

{u'type': u'int', u'name': u'test', u'description': u'An Avro test.'}
{u'type': u'boolean', u'name': u'test', u'description': u'An Avro test.'}

Discussion:
Avro's documentation is sparse.  This article is intended to help those who are curious to know if the Python implementation of Avro can reuse separately defined schemas.  The answer is yes, and a working example was presented above.

Keep in mind that the data used in this example, param_1 and param_2, have key names which match some of Avro's.  Obviously these key names were chosen to mimic Avro, not because they were specifically required.

It is important to note that the base schemas that must be "inherited", in this case the schema contained in parameter_types.avsc, must be loaded first.

It would be nice to have a bulk schema loader that could take a set schemas in any order and manage the correct loading process for us.


Additional Notes:
Installing Avro for Python is easy:

$sudo pip install avro

References:

1. Using Apache Avro , Boris Lublinsky on Jan 25, 2011.
2. Apache Avro™ 1.7.3 Getting Started (Python)
3. Apache Avro™ 1.7.3 Specification

Ubuntu 12.04 Restore Default Scrollbars

Situation:
Ubuntu 12.04 LTS(Precise)

Problem:
Please restore the default scrollbars.

Solution:
Run the following at command line:

sudo su

echo "export LIBOVERLAY_SCROLLBAR=0" > /etc/X11/Xsession.d/80overlayscrollbars
gnome-session-quit #log out and back in

References:

1. http://www.webupd8.org/2011/04/how-to-disable-overlay-scrollbars-in.html

Error "string indices must be integers" When Deserializing Queryset

Situation:
Attempting to deserialize a django 1.5 queryset

from django.core import serializers

queryset = MyModel.objects.all()

data = serializers.serialize('json', queryset)

#... on another server, data loaded using urllib2:
obj = serializers.deserialize('json', data) # error

Problem:
Django returns the error "string indices must be integers"

Solution:
You are not deserializing what you think you are deserializing.  Look at the JSON string and it should be apparent.   I had this problem in two cases:

1. I had accidentally serialized twice (the JSON string contained escaped quotation marks: \")
2. There was an error message instead of the actual object

References:

1. Django Docs:  Serialization

Python: Import a Submodule Programmatically

Situation:
In Python 2.7, there is a need to import a sub-package programmatically by name.

Problem:
I've looked at __import__ and importlib but the solution is not readily apparent.

Solution:

import importlib

module_name = '.'.join(('os','path'))
path = importlib.import_module(module_name)

#or

import sys

module_name = '.'.join(('os','path'))

__import__(module_name)
path = sys.modules[module_name]

References:

1.  importlib – Convenience wrappers for __import__()
2. Source code for importlib

Trouble Starting Ubuntu: System Running in Low Graphics Mode

Situation:
Trying to boot computer into Ubuntu 12.04 LTS(Precise)

Problem:
You reach dialog which says "System running in low graphics mode"

Solution:
<CTRL-ALT-F1> to get to command line
<CTRL-C> to exit current process if needed

sudo apt-get install --reinstall ubuntu-desktop
sudo reboot

References:

1. Ask Ubuntu: Your system is running in low-graphics mode with an ATI Radeon 3200 Graphics card

Ubuntu Software Center is Prompting for Root

Situation:
Trying to install package using Ubuntu Software Center on Ubuntu 12.04 LTS(Precise)

Problem:
The user is prompted for the root password instead of their own

Solution:
Edit  /etc/polkit-1/localauthority.conf.d/51-ubuntu-admin.conf to include the user or their group. Here is an example of the contents of this file:

[Configuration]
AdminIdentities=unix-group:administrators;unix-user:johnsmith

References:

1. pklocalauthority - PolicyKit Local Authority

2. Ubuntu forums

Switch Linux Java Version (Ubuntu 12.04)

Problem:
I installed a new version of Java on Linux.  How do I quickly make it the default?


Solution:
$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (6b24-1.11.1-4ubuntu3)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)


$ update-java-alternatives -l
java-1.6.0-openjdk-amd64 1061 /usr/lib/jvm/java-1.6.0-openjdk-amd64
java-1.7.0-openjdk-amd64 1051 /usr/lib/jvm/java-1.7.0-openjdk-amd64


$ sudo update-java-alternatives -s java-1.7.0-openjdk-amd64

...

$ java -version

java version "1.7.0_03"

OpenJDK Runtime Environment (IcedTea7 2.1.1pre) (7~u3-2.1.1~pre1-1ubuntu3)

OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)

References:

1. Install Sun Java Alternative in Ubuntu

Google Chart API Shows Only One Chart

Problem:
When using Google Charts, a single chart is shown despite adding multiple charts.

Solution:
The <div> element is a non-void(non-empty) element and should not be self closed.

Single graph shows:
  <div id="month_type" class="chg_chart"/>
  <div id="month_cat" class="chg_chart"/>
  <div id="month_region" class="chg_chart"/>

Three graphs show:
  <div id="month_type" class="chg_chart"></div>
  <div id="month_cat" class="chg_chart"></div>
  <div id="month_region" class="chg_chart"></div>


References:

1. stackoverflow: is it possible to have multiple google charts in one page?

Unable to upload files on Redmine installed on a Windows IIS server

The following instructions were used to install Redmine on a Windows server:
http://www.helicontech.com/articles/installing-redmine-on-windows-in-production/

Problem:
There is an error when you try to upload files to a project in Redmine.

Solution:

1. Create a folder in the root of the Redmine installation directory:
   C:\inetpub\wwwroot\redmine\files
2. Add write permission to this directory.  In our case we had a IIS_IUSRS user that was given modify permission for this directory.

References:

• http://www.redmine.org/boards/2/topics/12086

Eclipse won't start on Linux "Could not load SWT library" with Oracle Java 7

Problem:
When starting Eclipse on Linux (64bit Ubuntu 12.04) you get an error message:
An error has occurred.  See the log file
/home/<user_name>/.eclipse/org.eclipse.platform_3.70_155965261/configuration/1335552227148.log.

Opening the log file reveals the following:
!SESSION 2012-04-27 11:43:47.016 -----------------------------------------------
eclipse.buildId=I20110613-1736
java.version=1.7.0_03
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -os linux -ws gtk -arch x86_64


!ENTRY org.eclipse.osgi 4 0 2012-04-27 11:43:48.075
!MESSAGE Application error
!STACK 1
java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons: 
no swt-gtk-3740 in java.library.path
no swt-gtk in java.library.path
Can't load library: /home/maj/.swt/lib/linux/x86_64/libswt-gtk-3740.so
Can't load library: /home/maj/.swt/lib/linux/x86_64/libswt-gtk.so


at org.eclipse.swt.internal.Library.loadLibrary(Library.java:285)
at org.eclipse.swt.internal.Library.loadLibrary(Library.java:194)
at org.eclipse.swt.internal.C.<clinit>(C.java:21)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:63)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:54)
at org.eclipse.swt.widgets.Display.<clinit>(Display.java:132)
at org.eclipse.ui.internal.Workbench.createDisplay(Workbench.java:695)
at org.eclipse.ui.PlatformUI.createDisplay(PlatformUI.java:161)
at org.eclipse.ui.internal.ide.application.IDEApplication.createDisplay(IDEApplication.java:153)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:95)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
at org.eclipse.equinox.launcher.Main.main(Main.java:1386)

Solution:
Execute the following command to create a symlink to the current version of swt:

ln -s /usr/lib/jni/libswt-* ~/.swt/lib/linux/x86_64/

References:

1. StackOverflow: Ubuntu - Eclipse cannot load SWT libraries. Not opening

SmartGit Hangs When Switching Branches

Problem:
On Windows, SmartGit Hangs when switching braches, with the message:

Process Not Responding
SmartGit is waiting for the following process to finish.
C:\Program Files(x86)\Git\bin\git.exe ls-remote origin
If you thnk the process is hanging, click the Exit Process button, otherwise Wait

Solution:
Install an earlier version of msysgit to eliminate this issue.

This error popped up in version 1.7.10 (Git-1.7.10-preview20120409.exe) of msysgit.   This error was confirmed absent from versions 1.7.9 (Git-1.7.9-preview20120201.exe) and 1.7.8 (Git-1.7.8-preview20111206.exe)  of msysgit.

UPDATE (2012-04-24):

This bug was fixed in SmartGit 3.0.4 (2012-04-21):

Switch (Git): hangs for https repositories with (msys)Git 1.7.9.2 or newer

SmartGit SSL Certificate Problem, error:14090086:SSL

Problem:

You get error:14090086:SSL in SmartGit when trying to clone a remote repository using HTTPS (SSL):

Solution (Windows):

1.  Create a new directory called <project_name>

2. Right-Click on the new directory and select "Git Bash Here"

3.  Enter the following command lines:

git init

git config http.sslVerify false

git remote add origin https://</Bonobo.Git.Server/Git.aspx/pylet

4. Close the Git Bash

5. Open SmartGit and open the existing local repository, and click the Pull button

Warning:

This solution is a security risk if ever used with an external site and is intended for intranet servers with self-signed ssl certificates.  Do not use this solution with public servers and if your repository is ever moved to an external site, you will want to switch http.sslVerify back to true.

You might see examples using a global setting (git config --global http.sslverify "false") .  In the solution presented here, we are intentionally disabling SSL verification on a per repository basis so you can still safely use public repositories.

References:

1. How do I set GIT_SSL_NO_VERIFY for specific repos only?

Compiling Bonobo.Git.Server from Source

After downloading Bonobo.Git.Server source code:
https://github.com/jakubgarfield/Bonobo-Git-Server

I experienced a few problems getting started.

I suspect that just installing the following will allow you get to business immediately:

• Microsoft Web Platform Installer 3.0
• ADO.NET 2.0 Provider for SQLite

but here is the path I took...

Problem 1:
I was not able to open the project.  It reported this error in VS2010:
Error : The Project Type is not supported by this Installation

Solution 1:
Edit  Bonobo.Git.Server / Bonobo.Git.Server / Bonobo.Git.Server.csproj and remove
{e53f8fea-eae0-44a6-8774-ffd645390401}; from ProjectTypeGuids


Problem 2:
A lot of web dependencies are missing, and the "Install Web Components" button does not work in VS2010

Solution 2:
Install Microsoft Web Platform


Problem 3:
Error 175: The specified data store provider cannot be found, or is not valid.

Solution 3:
Install ADO.NET 2.0 Provider for SQLite



References:

1. http://social.msdn.microsoft.com/Forums/is/vssetup/thread/e68a080d-b394-4c8b-b593-438b82c284b8
   
   
2. http://social.msdn.microsoft.com/Forums/en/vssetup/thread/90f13749-8eaa-4a1b-b29e-b621b54cbefe
   
   
3. http://stackoverflow.com/questions/4444742/error-175-the-specified-data-store-provider-cannot-be-found

Bonobo Git Server - "The remote end hung up unexpectedly"

Problem:

The following error occurs in SmartGit when trying to push an existing local repository to a new remote repository on Bonobo Git Server:

Push:
The remote end hung up unexpectedly
The remote end hung up unexpectedly
RPC failed; result=22, HTTP code = 404

Solution:

Modify the web.config file in the root of Bonobo.Git.Server (C:\initpub\wwwroot\Bonobo.Git.Server\Web.config) and up the limits on the following lines:

<system.web>
   <httpRuntime maxRequestLength="102400" />


<security>
   <requestFiltering>
     <requestLimits maxAllowedContentLength="102400" />

In both cases "102400" was replaced with "999999999" and the push worked!

References:

1. windows - Issue with GIT hosted on IIS with Bonobo - Stack Overflow

Redirect a Python Import

Problem:
You need to redirect a Python import to a module or package in a relative location.

Solution:
Create a dummy module that replaces its own reference in sys.modules with a reference to a module or package in another location.

# Paste this in the module being imported, in this example a package
# Note that when del sys.modules[__name__] is executed, local variables no longer accessible
# You can store items in sys.argv if exec('import {0}'.format(module_name)) needed,
# but you get the following warning:
# RuntimeWarning: Parent module 'attila' not found while handling absolute import
import os
import sys

RELATIVE_VALUE = -1  # Directories to step up:  -1 = . (packages only), -2 = .. , -3 = ... , etc.
PATH_APPEND = []  # After step up, append this sub path, example: ['foo', 'bar'] -> /foo/bar is appended

pth = os.path.sep.join(__file__.split(os.path.sep)[0:RELATIVE_VALUE] + PATH_APPEND)
sys.path.insert(0, pth)

del sys.modules[__name__]
import foo  #name of this module, and the one you are redirecting to

Reference: 

1. http://stackoverflow.com/questions/4855936/how-can-i-transparently-redirect-a-python-import/9450785#9450785 
2. https://gist.github.com/1983695

FlashBuilder Fails to Launch: JVM terminated. Exit code=-1

Problem:

FlashBuilder fails to launch, with a "JVM terminated.  Exit code=-1" error:

Solution:
Delete the following line from the FlashBuilder.ini file in root of your installation directory:

-XX:MaxPermSize=512m


The default installation directory on 64bit Windows 7 is:

C:\Program Files (x86)\Adobe\Adobe Flash Builder 4\FlashBuilder.ini



References:

1. http://www.blog.techsplice.com/archives/38

VirtualBox - Windows 7 fails to install - Status: 0xc0000225

Problem:

While attempting to install Windows 7 as a guest on a Linux host using VirtualBox, the following error occurs during installation:

Windows failed to start...

Status: 0xc0000225

Solution:

In the VirtualBox settings for the virtual machine, select "Enable IO APIC" under System > Motherboard.

References:

virtualbox.org • View topic - [SOLVED] Windows 7 fails to install; Status: 0xc0000225

Convert Mercurial (hg) Repository to Git

The following instructions are for converting an hg repository to git on Windows 7.

INSTRUCTIONS

1. Install Cygwin with python, hg and git
2. Install fast-export
   
   You can try installing directly from the fast-export repository by issuing this command in the Cygwin console:
   

   git clone git://repo.or.cz/fast-export.git

   If your firewall doesn't like that command and responds with:
   Cloning into fast-export...
   repo.or.cz[0: 195.113.20.142]: errno=Connection timed out
   fatal: unable to connect a socket (Connection timed out)
   
   you can download an archive from here:
   http://repo.or.cz/w/fast-export.git/shortlog/refs/heads/master/
   
   Also, I was getting the following error: 
   IOError: [Errno 32] Broken pipe
   
   and was able to get around it by downloading an older version:
   fast-export-c8a45848968a21c3021489a6ba3ab1ffe5ef2a90.tar.gz
3. Execute these commands in cygwin to create a working directory and initialize a new git repository:
   
   mkdir conversion
   cd conversion
4. In Windows explorer, copy the fast-export folder from inside the archive into:
   
   C:\cygwin\home\<USERNAME>\conversion
5. Also,  copy your existing mercurial repository, we'll call it convert.hg, into the same directory
6. Execute the following commands:
   
   mkdir convert.git
   cd convert.git
   git init
   ~/conversion/fast-export/hg-fast-export.sh -r ~/conversion/convert.hg
   

ADDITIONAL NOTES

The directory structure created for this example was:

~/conversion
    /convert.git  # A new empty git repository
    /convert.hg   # Your existing mercurial repository
    /fast-export  # The fast-export directory we downloaded

REFERENCES

1. http://stackoverflow.com/questions/3267232/converting-a-mercurial-hg-repository-to-git-on-windows-7
2. http://hedonismbot.wordpress.com/2008/10/16/hg-fast-export-convert-mercurial-repositories-to-git-repositories/

Installing Ubuntu Desktop 11.10 on a Dell M4400 with Dock and Dual External Monitors

The following are notes for how I was able to get Ubuntu 11.10 working with the two external monitors attached to my docked Dell M4400 laptop.

1. Before starting the installation, it is best to start by undocking the laptop and connecting directly to the router instead of using wireless. I had an issue where it seemed progress would halt during startup, booting from USB, but the problem didn't occur when using the undocked laptop screen.
2. I went ahead and checked the box for proprietary drivers. This option might install the Nvidia
   driver which ultimately causes an issue(we're going to remove the Nvidia drivers later).  I also checked the box to install updates. If you don't install updates, you will eventually notice an issue where you have to hold down the mouse button to navigate the menus.
3. After installation completes, and you reboot into Ubuntu, execute these commands:
   
      sudo apt-get remove nvidia-*
      sudo apt-get upgrade
   
   Enter "y" when prompted
4. Shutdown
5. Dock your laptop with the lid closed and press the power button.  The external monitors will eventually fire up, but the desktop will be mirrored.
6. To extend the desktop across both monitors, under the same "Dash home" searchbox, enter "displays". Unclick the mirror displays checkbox and make sure moitor resolutions are correct, then click the Apply button.  Click the "Keep this configuration" button in the dialog that appears.
7. All done, you should be able to drag windows across both desktops. At one point there was a glitch when I applied the settings and I needed to restart because the menu bar was missing (click the desktop and push the power button on the dock, a window with restart options will appear).
8. Enjoy Ubuntu 11.10!  It looks slick!  I finally made the switch because Windows 7 kept blue-screening, and I had always wanted to leave the dark side.
9. If you find out the exact instructions to get the Nvidia driver working, please post a comment.  I ultimately gave up after multiple failed attempts at installing the driver both from the Nvidia website and from Ubuntu's package repository. Eventually, I might do some more attempts with a fresh install on a different hard drive, but for now I can get to work!

DateTimer: Python class to easily report stop, start and elapsed time

Here is a simple class to make reporting start, finish and elapsed time easy.  If you are running python scripts at command line, this utility will make it easier to keep track of the progress of your processing.

Here is an example of usage:

from arcpyh import DateTimer
import time

timer = DateTimer()
timer.start()
time.sleep(10)
timer.stop()

Here is an example of the output:

Started: 2012-01-23 14:50:00
Finished: 2012-01-23 14:50:10 (Elapsed: 0:00:10)

Here is the important code.  Save this in a file called “arcpyh.py” and put it in the same directory of the script you are running or elsewhere in your pythonpath.

from datetime import datetime

class DateTimer:
    ''' Handy timer that reports start time, end time and delta time

        report:(default=True) If True, prints automatically

        Example output:

            Started: 2012-01-23 14:40:37
            Finished: 2012-01-23 14:40:47 (Elapsed: 0:00:10)

    '''

    NO_START_MSG = "Timer was never started."
    FINISH_MSG = "Finished: {0} (Elapsed: {1})"
    START_MSG = "Started: {0}"

    def __init__(self, report=True):

        self.report = report
        self.startDateTime = None
        self.endDateTime = None
        self.deltaTime = None

    def start(self):

        self.__init__(self.report)
        self.startDateTime = datetime.now()
        self.endDateTime = None

        if self.report:
            self.printStart()

    def stop(self):
        self.endDateTime = datetime.now()
        if self.startDateTime:
            self.deltaTime = self.endDateTime - self.startDateTime

        if self.report:
            self.printEnd()

    def printStart(self):
        if self.startDateTime:
            print self.START_MSG.format(self.datetimeToString(self.startDateTime))
        else:
            print self.NO_START_MSG

    def printEnd(self):
        if self.deltaTime:
            print self.FINISH_MSG.format(self.datetimeToString(self.endDateTime), self.deltaToString(self.deltaTime))
        else:
            print self.NO_START_MSG

    def datetimeToString(self, dateTimeObject):
        return str(dateTimeObject).split(".")[0]

    def deltaToString(self, delta):
        return str(delta).split(".")[0]