Sunday, December 23, 2012

Deprecated

This blog has been deprecated.

Please visit the new location, where all content has been transferred:
http://blog.majgis.com

Using Apache Avro with Python

Abstract:
Instructions are given on how to use the Python implementation of Avro to load multiple schemas,  stored in separate files, to build an overall composite schema.

Details:
The following example uses two Avro schema files, where the second uses the first.  There is also an example of a Python script, which combines and tests the composite schema by outputting to the console.

First schema file:  parameter_types.avsc

{"namespace": "scriptaxe.parameter",
 "type": "enum",
 "name": "types",
 "symbols": [
    "null",
    "boolean",
    "int",
    "long",
    "float",
    "double",
    "bytes",
    "string"
    ]
}
Second schema file:  parameter.avsc

{"namespace": "scriptaxe",
 "type": "record",
 "name": "parameter",
 "fields": [
   {
     "name": "name",
     "type": "string"
   },{
     "name": "description",
     "type": ["null", "string"]
   },{
     "name": "type",
     "type": "scriptaxe.parameter.types"
   }
 ]
}
Python file:  avro_test.py


import avro.schema
import json
 
def main():
  """Start of execution"""
  #combine the schemas 
  known_schemas = avro.schema.Names()
  types_schema = LoadAvsc("parameter_types.avsc", known_schemas)
  param_schema = LoadAvsc("parameter.avsc", known_schemas)
  print json.dumps(param_schema.to_json(avro.schema.Names()), indent=2) 
  #test the schema works 
  param_file = open("parameters.avro", "w")
  writer = DataFileWriter(param_file, DatumWriter(), param_schema)
  param_1 = {"name": "test", "description":"An Avro test.", "type":"int"}
  param_2 = {"name": "test", "description":"An Avro test.", "type":"boolean"}
  writer.append(param_1)
  writer.append(param_2)
  writer.close()
  reader = DataFileReader(open("parameters.avro", "r"), DatumReader())
  for parameter in reader:
      print parameter
  reader.close()  
def LoadAvsc(file_path, names=None):
  """Load avsc file
    file_path: path to schema file
    names(optional): avro.schema.Names object
  """
  file_text = open(file_path).read() 
  json_data = json.loads(file_text)
  schema = avro.schema.make_avsc_object(json_data, names) 
  return schema 
if __name__ == "__main__":
  main()
Output to console:

{
  "type": "record",
  "namespace": "scriptaxe",
  "name": "parameter",
  "fields": [
    {
      "type": "string",
      "name": "name"
    }, {
      "type": ["null", "string"],
      "name": "description"
    }, {
      "type": {
        "symbols": ["null", "boolean", "int", "long", "float", "double", "bytes", "string"],
        "namespace": "scriptaxe.parameter",
        "type": "enum",
        "name": "types"
      },
      "name": "type"
    }
  ]
}
{u'type': u'int', u'name': u'test', u'description': u'An Avro test.'}
{u'type': u'boolean', u'name': u'test', u'description': u'An Avro test.'}
Discussion:
Avro's documentation is sparse.  This article is intended to help those who are curious to know if the Python implementation of Avro can reuse separately defined schemas.  The answer is yes, and a working example was presented above.

Keep in mind that the data used in this example, param_1 and param_2, have key names which match some of Avro's.  Obviously these key names were chosen to mimic Avro, not because they were specifically required.

It is important to note that the base schemas that must be "inherited", in this case the schema contained in parameter_types.avsc, must be loaded first.

It would be nice to have a bulk schema loader that could take a set schemas in any order and manage the correct loading process for us.


Additional Notes:
Installing Avro for Python is easy:
$sudo pip install avro

References:

  1. Using Apache Avro , Boris Lublinsky on Jan 25, 2011.
  2. Apache Avro™ 1.7.3 Getting Started (Python)
  3. Apache Avro™ 1.7.3 Specification





Sunday, November 4, 2012

Ubuntu 12.04 Restore Default Scrollbars

Situation:
Ubuntu 12.04 LTS(Precise)

Problem:
Please restore the default scrollbars.

Solution:
Run the following at command line:

sudo su
echo "export LIBOVERLAY_SCROLLBAR=0" > /etc/X11/Xsession.d/80overlayscrollbars
gnome-session-quit #log out and back in

References:

Error "string indices must be integers" When Deserializing Queryset


Situation:
Attempting to deserialize a django 1.5 queryset

from django.core import serializers
queryset = MyModel.objects.all()
data = serializers.serialize('json', queryset)
#... on another server, data loaded using urllib2:
obj = serializers.deserialize('json', data) # error

Problem:
Django returns the error "string indices must be integers"

Solution:
You are not deserializing what you think you are deserializing.  Look at the JSON string and it should be apparent.   I had this problem in two cases:

  1. I had accidentally serialized twice (the JSON string contained escaped quotation marks: \")
  2. There was an error message instead of the actual object


References:

  1. Django Docs:  Serialization

Thursday, August 9, 2012

Python: Import a Submodule Programmatically

Situation:
In Python 2.7, there is a need to import a sub-package programmatically by name.

Problem:
I've looked at __import__ and importlib but the solution is not readily apparent.

Solution:
import importlib
module_name = '.'.join(('os','path'))
path = importlib.import_module(module_name)

#or

import sys

module_name = '.'.join(('os','path'))
__import__(module_name)
path = sys.modules[module_name]

References:
  1.  importlib – Convenience wrappers for __import__()
  2. Source code for importlib





Thursday, August 2, 2012

Trouble Starting Ubuntu: System Running in Low Graphics Mode

Situation:
Trying to boot computer into Ubuntu 12.04 LTS(Precise)

Problem:
You reach dialog which says "System running in low graphics mode"

Solution:
<CTRL-ALT-F1> to get to command line
<CTRL-C> to exit current process if needed

sudo apt-get install --reinstall ubuntu-desktop
sudo reboot

References:
  1. Ask Ubuntu: Your system is running in low-graphics mode with an ATI Radeon 3200 Graphics card

Wednesday, August 1, 2012

Ubuntu Software Center is Prompting for Root

Situation:
Trying to install package using Ubuntu Software Center on Ubuntu 12.04 LTS(Precise)

Problem:
The user is prompted for the root password instead of their own

Solution:
Edit  /etc/polkit-1/localauthority.conf.d/51-ubuntu-admin.conf to include the user or their group. Here is an example of the contents of this file:

[Configuration]
AdminIdentities=unix-group:administrators;unix-user:johnsmith

References:
  1. pklocalauthority - PolicyKit Local Authority
    
    
  2. Ubuntu forums

Wednesday, June 27, 2012

Switch Linux Java Version (Ubuntu 12.04)

Problem:
I installed a new version of Java on Linux.  How do I quickly make it the default?


Solution:
$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (6b24-1.11.1-4ubuntu3)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)


$ update-java-alternatives -l
java-1.6.0-openjdk-amd64 1061 /usr/lib/jvm/java-1.6.0-openjdk-amd64
java-1.7.0-openjdk-amd64 1051 /usr/lib/jvm/java-1.7.0-openjdk-amd64


$ sudo update-java-alternatives -s java-1.7.0-openjdk-amd64
...

$ java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.1pre) (7~u3-2.1.1~pre1-1ubuntu3)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)

References:

Sunday, June 17, 2012

Google Chart API Shows Only One Chart

Problem:
When using Google Charts, a single chart is shown despite adding multiple charts.

Solution:
The <div> element is a non-void(non-empty) element and should not be self closed.

Single graph shows:
  <div id="month_type" class="chg_chart"/>
  <div id="month_cat" class="chg_chart"/>
  <div id="month_region" class="chg_chart"/>

Three graphs show:
  <div id="month_type" class="chg_chart"></div>
  <div id="month_cat" class="chg_chart"></div>
  <div id="month_region" class="chg_chart"></div>


References:
  1. stackoverflow: is it possible to have multiple google charts in one page?

Wednesday, May 2, 2012

Unable to upload files on Redmine installed on a Windows IIS server

The following instructions were used to install Redmine on a Windows server:
http://www.helicontech.com/articles/installing-redmine-on-windows-in-production/

Problem:
There is an error when you try to upload files to a project in Redmine.

Solution:

  1. Create a folder in the root of the Redmine installation directory:
    C:\inetpub\wwwroot\redmine\files
  2. Add write permission to this directory.  In our case we had a IIS_IUSRS user that was given modify permission for this directory.


References:

Friday, April 27, 2012

Eclipse won't start on Linux "Could not load SWT library" with Oracle Java 7

Problem:
When starting Eclipse on Linux (64bit Ubuntu 12.04) you get an error message:
An error has occurred.  See the log file
/home/<user_name>/.eclipse/org.eclipse.platform_3.70_155965261/configuration/1335552227148.log.

Opening the log file reveals the following:
!SESSION 2012-04-27 11:43:47.016 -----------------------------------------------
eclipse.buildId=I20110613-1736
java.version=1.7.0_03
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -os linux -ws gtk -arch x86_64


!ENTRY org.eclipse.osgi 4 0 2012-04-27 11:43:48.075
!MESSAGE Application error
!STACK 1
java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons: 
no swt-gtk-3740 in java.library.path
no swt-gtk in java.library.path
Can't load library: /home/maj/.swt/lib/linux/x86_64/libswt-gtk-3740.so
Can't load library: /home/maj/.swt/lib/linux/x86_64/libswt-gtk.so


at org.eclipse.swt.internal.Library.loadLibrary(Library.java:285)
at org.eclipse.swt.internal.Library.loadLibrary(Library.java:194)
at org.eclipse.swt.internal.C.<clinit>(C.java:21)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:63)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:54)
at org.eclipse.swt.widgets.Display.<clinit>(Display.java:132)
at org.eclipse.ui.internal.Workbench.createDisplay(Workbench.java:695)
at org.eclipse.ui.PlatformUI.createDisplay(PlatformUI.java:161)
at org.eclipse.ui.internal.ide.application.IDEApplication.createDisplay(IDEApplication.java:153)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:95)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
at org.eclipse.equinox.launcher.Main.main(Main.java:1386)

Solution:
Execute the following command to create a symlink to the current version of swt:

ln -s /usr/lib/jni/libswt-* ~/.swt/lib/linux/x86_64/



References:

  1. StackOverflow: Ubuntu - Eclipse cannot load SWT libraries. Not opening

Monday, April 16, 2012

SmartGit Hangs When Switching Branches

Problem:
On Windows, SmartGit Hangs when switching braches, with the message:

Process Not Responding
SmartGit is waiting for the following process to finish.
C:\Program Files(x86)\Git\bin\git.exe ls-remote origin
If you thnk the process is hanging, click the Exit Process button, otherwise Wait




Solution:
Install an earlier version of msysgit to eliminate this issue.

This error popped up in version 1.7.10 (Git-1.7.10-preview20120409.exe) of msysgit.   This error was confirmed absent from versions 1.7.9 (Git-1.7.9-preview20120201.exe) and 1.7.8 (Git-1.7.8-preview20111206.exe)  of msysgit.

UPDATE (2012-04-24):

This bug was fixed in SmartGit 3.0.4 (2012-04-21):
  • Switch (Git): hangs for https repositories with (msys)Git 1.7.9.2 or newer





Tuesday, April 10, 2012

SmartGit SSL Certificate Problem, error:14090086:SSL




Problem:
You get error:14090086:SSL in SmartGit when trying to clone a remote repository using HTTPS (SSL):


Command Failed:  SSL certificate problem, verify that the CA cert is OK.  Details:  HTTP request failed: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https


Solution (Windows):

1.  Create a new directory called <project_name>

2. Right-Click on the new directory and select "Git Bash Here"

3.  Enter the following command lines:

git init

git config http.sslVerify false

git remote add origin https://</Bonobo.Git.Server/Git.aspx/pylet

4. Close the Git Bash

5. Open SmartGit and open the existing local repository, and click the Pull button


Warning:
This solution is a security risk if ever used with an external site and is intended for intranet servers with self-signed ssl certificates.  Do not use this solution with public servers and if your repository is ever moved to an external site, you will want to switch http.sslVerify back to true.

You might see examples using a global setting (git config --global http.sslverify "false") .  In the solution presented here, we are intentionally disabling SSL verification on a per repository basis so you can still safely use public repositories.


References:

Thursday, March 8, 2012

Compiling Bonobo.Git.Server from Source

After downloading Bonobo.Git.Server source code:
https://github.com/jakubgarfield/Bonobo-Git-Server

I experienced a few problems getting started.

I suspect that just installing the following will allow you get to business immediately:
but here is the path I took...

Problem 1:
I was not able to open the project.  It reported this error in VS2010:
Error : The Project Type is not supported by this Installation

Solution 1:
Edit  Bonobo.Git.Server / Bonobo.Git.Server / Bonobo.Git.Server.csproj and remove
{e53f8fea-eae0-44a6-8774-ffd645390401}; from ProjectTypeGuids


Problem 2:
A lot of web dependencies are missing, and the "Install Web Components" button does not work in VS2010

Solution 2:
Install Microsoft Web Platform


Problem 3:
Error 175: The specified data store provider cannot be found, or is not valid.

Solution 3:
Install ADO.NET 2.0 Provider for SQLite



References:

  1. http://social.msdn.microsoft.com/Forums/is/vssetup/thread/e68a080d-b394-4c8b-b593-438b82c284b8

  2. http://social.msdn.microsoft.com/Forums/en/vssetup/thread/90f13749-8eaa-4a1b-b29e-b621b54cbefe

  3. http://stackoverflow.com/questions/4444742/error-175-the-specified-data-store-provider-cannot-be-found

Tuesday, March 6, 2012

Bonobo Git Server - "The remote end hung up unexpectedly"

Problem:
The following error occurs in SmartGit when trying to push an existing local repository to a new remote repository on Bonobo Git Server:
Push:
The remote end hung up unexpectedly
The remote end hung up unexpectedly
RPC failed; result=22, HTTP code = 404

Solution:
Modify the web.config file in the root of Bonobo.Git.Server (C:\initpub\wwwroot\Bonobo.Git.Server\Web.config) and up the limits on the following lines:

<system.web>
   <httpRuntime maxRequestLength="102400" />


<security>
   <requestFiltering>
     <requestLimits maxAllowedContentLength="102400" />

In both cases "102400" was replaced with "999999999" and the push worked!

References:

  1. windows - Issue with GIT hosted on IIS with Bonobo - Stack Overflow

Redirect a Python Import

Problem:
You need to redirect a Python import to a module or package in a relative location.

Solution:
Create a dummy module that replaces its own reference in sys.modules with a reference to a module or package in another location.
# Paste this in the module being imported, in this example a package
# Note that when del sys.modules[__name__] is executed, local variables no longer accessible
# You can store items in sys.argv if exec('import {0}'.format(module_name)) needed,
# but you get the following warning:
# RuntimeWarning: Parent module 'attila' not found while handling absolute import
import os
import sys

RELATIVE_VALUE = -1  # Directories to step up:  -1 = . (packages only), -2 = .. , -3 = ... , etc.
PATH_APPEND = []  # After step up, append this sub path, example: ['foo', 'bar'] -> /foo/bar is appended

pth = os.path.sep.join(__file__.split(os.path.sep)[0:RELATIVE_VALUE] + PATH_APPEND)
sys.path.insert(0, pth)

del sys.modules[__name__]
import foo  #name of this module, and the one you are redirecting to


Reference: 

  1. http://stackoverflow.com/questions/4855936/how-can-i-transparently-redirect-a-python-import/9450785#9450785 
  2. https://gist.github.com/1983695

Thursday, March 1, 2012

FlashBuilder Fails to Launch: JVM terminated. Exit code=-1

Problem:
FlashBuilder fails to launch, with a "JVM terminated.  Exit code=-1" error:



Solution:
Delete the following line from the FlashBuilder.ini file in root of your installation directory:

-XX:MaxPermSize=512m


The default installation directory on 64bit Windows 7 is:

C:\Program Files (x86)\Adobe\Adobe Flash Builder 4\FlashBuilder.ini



References:
  1. http://www.blog.techsplice.com/archives/38

VirtualBox - Windows 7 fails to install - Status: 0xc0000225

Problem:
While attempting to install Windows 7 as a guest on a Linux host using VirtualBox, the following error occurs during installation:

Windows failed to start...

Status: 0xc0000225

Solution:
In the VirtualBox settings for the virtual machine, select "Enable IO APIC" under System > Motherboard.

References:
virtualbox.org • View topic - [SOLVED] Windows 7 fails to install; Status: 0xc0000225

Saturday, February 25, 2012

Convert Mercurial (hg) Repository to Git

The following instructions are for converting an hg repository to git on Windows 7.

INSTRUCTIONS
  1. Install Cygwin with python, hg and git
  2. Install fast-export

    You can try installing directly from the fast-export repository by issuing this command in the Cygwin console:
    git clone git://repo.or.cz/fast-export.git
    If your firewall doesn't like that command and responds with:
    Cloning into fast-export...
    repo.or.cz[0: 195.113.20.142]: errno=Connection timed out
    fatal: unable to connect a socket (Connection timed out)


    you can download an archive from here:
    http://repo.or.cz/w/fast-export.git/shortlog/refs/heads/master/

    Also, I was getting the following error: 
    IOError: [Errno 32] Broken pipe


    and was able to get around it by downloading an older version:

    fast-export-c8a45848968a21c3021489a6ba3ab1ffe5ef2a90.tar.gz
  3. Execute these commands in cygwin to create a working directory and initialize a new git repository:

    mkdir conversion
    cd conversion
  4. In Windows explorer, copy the fast-export folder from inside the archive into:

    C:\cygwin\home\<USERNAME>\conversion
  5. Also,  copy your existing mercurial repository, we'll call it convert.hg, into the same directory
  6. Execute the following commands:

    mkdir convert.git
    cd convert.git
    git init
    ~/conversion/fast-export/hg-fast-export.sh -r ~/conversion/convert.hg
ADDITIONAL NOTES

The directory structure created for this example was:

~/conversion
    /convert.git  # A new empty git repository
    /convert.hg   # Your existing mercurial repository
    /fast-export  # The fast-export directory we downloaded
REFERENCES

Saturday, February 11, 2012

Installing Ubuntu Desktop 11.10 on a Dell M4400 with Dock and Dual External Monitors



The following are notes for how I was able to get Ubuntu 11.10 working with the two external monitors attached to my docked Dell M4400 laptop.
  1. Before starting the installation, it is best to start by undocking the laptop and connecting directly to the router instead of using wireless. I had an issue where it seemed progress would halt during startup, booting from USB, but the problem didn't occur when using the undocked laptop screen.
  2. I went ahead and checked the box for proprietary drivers. This option might install the Nvidia
    driver which ultimately causes an issue(we're going to remove the Nvidia drivers later).  I also checked the box to install updates. If you don't install updates, you will eventually notice an issue where you have to hold down the mouse button to navigate the menus.
  3. After installation completes, and you reboot into Ubuntu, execute these commands:

       sudo apt-get remove nvidia-*
       sudo apt-get upgrade


    Enter "y" when prompted
  4. Shutdown
  5. Dock your laptop with the lid closed and press the power button.  The external monitors will eventually fire up, but the desktop will be mirrored.
  6. To extend the desktop across both monitors, under the same "Dash home" searchbox, enter "displays". Unclick the mirror displays checkbox and make sure moitor resolutions are correct, then click the Apply button.  Click the "Keep this configuration" button in the dialog that appears.
  7. All done, you should be able to drag windows across both desktops. At one point there was a glitch when I applied the settings and I needed to restart because the menu bar was missing (click the desktop and push the power button on the dock, a window with restart options will appear).
  8. Enjoy Ubuntu 11.10!  It looks slick!  I finally made the switch because Windows 7 kept blue-screening, and I had always wanted to leave the dark side.
  9. If you find out the exact instructions to get the Nvidia driver working, please post a comment.  I ultimately gave up after multiple failed attempts at installing the driver both from the Nvidia website and from Ubuntu's package repository. Eventually, I might do some more attempts with a fresh install on a different hard drive, but for now I can get to work!



Monday, January 23, 2012

DateTimer: Python class to easily report stop, start and elapsed time


Here is a simple class to make reporting start, finish and elapsed time easy.  If you are running python scripts at command line, this utility will make it easier to keep track of the progress of your processing.
Here is an example of usage:
from arcpyh import DateTimer
import time

timer = DateTimer()
timer.start()
time.sleep(10)
timer.stop()
Here is an example of the output:
Started: 2012-01-23 14:50:00
Finished: 2012-01-23 14:50:10 (Elapsed: 0:00:10)
Here is the important code.  Save this in a file called “arcpyh.py” and put it in the same directory of the script you are running or elsewhere in your pythonpath.
from datetime import datetime

class DateTimer:
    ''' Handy timer that reports start time, end time and delta time

        report:(default=True) If True, prints automatically

        Example output:

            Started: 2012-01-23 14:40:37
            Finished: 2012-01-23 14:40:47 (Elapsed: 0:00:10)

    '''

    NO_START_MSG = "Timer was never started."
    FINISH_MSG = "Finished: {0} (Elapsed: {1})"
    START_MSG = "Started: {0}"

    def __init__(self, report=True):

        self.report = report
        self.startDateTime = None
        self.endDateTime = None
        self.deltaTime = None

    def start(self):

        self.__init__(self.report)
        self.startDateTime = datetime.now()
        self.endDateTime = None

        if self.report:
            self.printStart()

    def stop(self):
        self.endDateTime = datetime.now()
        if self.startDateTime:
            self.deltaTime = self.endDateTime - self.startDateTime

        if self.report:
            self.printEnd()

    def printStart(self):
        if self.startDateTime:
            print self.START_MSG.format(self.datetimeToString(self.startDateTime))
        else:
            print self.NO_START_MSG

    def printEnd(self):
        if self.deltaTime:
            print self.FINISH_MSG.format(self.datetimeToString(self.endDateTime), self.deltaToString(self.deltaTime))
        else:
            print self.NO_START_MSG

    def datetimeToString(self, dateTimeObject):
        return str(dateTimeObject).split(".")[0]

    def deltaToString(self, delta):
        return str(delta).split(".")[0]