import operator
from sqlalchemy.sql import case, func, select

def pivot_report(report, pivot_on=None, pivot_columns=None, pivot_func=func.sum,
                    non_pivot_columns=None, group_by=None):
    """ produce a pivot on a select

    if we have a report: 

        id, type, count
        1, white, 10
        1, black, 20
        2, white, 12
        2, black, 20

    and we want 

        id, black count, white count
        1, 20, 10
        2, 20, 12

    pass in type as the pivot_on, and [count] as pivot columns  

    """

    # find all possible values of the pivot
    pivot_values = map(
        operator.itemgetter(0),
        select([pivot_on], from_obj=[report]).distinct().execute()
    )

    # build the new pivot columns
    new_columns = [
        pivot_func(case([(pivot_on == value, column)])).label("%s %s" % (value, column))
        for value in pivot_values
        for column in pivot_columns
    ]

    return select(
        non_pivot_columns + new_columns,
        from_obj=[report],
        group_by=group_by
    )

# example code
from sqlalchemy import Table, Column, Integer, String, MetaData
from sqlalchemy import create_engine

metadata = MetaData()
example = Table('example', metadata,
    Column('id', Integer),
    Column('type', String),
    Column('count', Integer),
)

engine = create_engine('sqlite:///:memory:')
metadata.bind = engine
example.create()
example.insert().execute(
    dict(id=1, type='white', count=10),
    dict(id=1, type='black', count=20),
    dict(id=2, type='white', count=30),
    dict(id=2, type='black', count=40),
)

report = example.select()

# now build the pivot
report = pivot_report(
    report,
    pivot_on=report.c.type,
    pivot_columns=[report.c.count],
    non_pivot_columns=[report.c.id],
    group_by=[report.c.id])

for r in report.execute():
    print r.items()

Think of the hash of hashes, list of hashes approach that you often find in a perl, python or ruby program. These are so useful, that they have been codified as JSON – which can simply be evaled to return your data in several languages. Ad-hoc data structures like this have a great appeal when hacking something in python, yet you quickly get to a pain point when using them when the data doesn’t look exactly as you would expect and you need to handle exceptions and edge cases.

So why is the hash of hashes approach so tempting? Because it avoids the ceremony around object creation. Things like Python’s “__init__” and Perl’s “bless” are ceremony and are necessarily ceremony because all a type in a dynamic language is just data with some ceremony. Clearly, perl has got a perfect name for the ceremony in “bless”.

The eureka moment comes when you use a typed language that is low ceremony, such as Haskell. Algebraic data types give you the freedom to create complex data structures without ceremony, which you can then process without the hassle involved in unpicking a big blob of JSON, which will typically need lots of switches. Instead, you can pattern match on the type.

So if you went to a dynamic language to avoid the ceremony, you may well be moving in the wrong direction.

First of all, if you haven’t already, install virtualenv and pip. Both are easy installable. Now install virtualenv wrapper.

Now we are going to setup a bioinformatics environment with both biopython and pygr installed so that you can hack on them. Firstly create a new virtualenv, passing the no site packages flag to keep this clean:

james@flapjack:~/Documents/virtualenvs$ mkvirtualenv --no-site-packages bio
New python executable in bio/bin/python
Installing setuptools............done.
(bio)james@flapjack:~/Documents/virtualenvs$ cdvirtualenv
(bio)james@flapjack:~/Documents/virtualenvs/bio$

Now, to install biopython we first use pip to install numpy:

(bio)james@flapjack:~/Documents/virtualenvs/bio$ pip -E . install numpy
Downloading/unpacking numpy
...

Important to remember the ‘-E’ flag which tells pip to use the virtualenv we are in (this should be added to virtualenv_wrapper IMHO). Now we can install biopython from our github fork, using the ‘-e’ flag to keep it editable (i.e we are hacking on it).

(bio)james@flapjack:~/Documents/virtualenvs/bio$ pip -E . install -e git://github.com/jamescasbon/biopython.git#egg=biopython
Obtaining biopython from git+git://github.com/jamescasbon/biopython.git#egg=biopython
Cloning git://github.com/jamescasbon/biopython.git to ./src/biopython
remote: Counting objects: 22719, done.
...

Next up, we want pygr so we need pyrex to build the c files:

(bio)james@flapjack:~/Documents/virtualenvs/bio$ pip -E . install -U pyrex -f http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/Pyrex-0.9.8.5.tar.gz
Downloading/unpacking pyrex
Downloading Pyrex-0.9.8.5.tar.gz (242Kb): 242Kb downloaded
In the tar file /var/folders/Gn/GneSaDeKGaGpZXx+hcopdU+++TI/-Tmp-/tmpTEdhFd/Pyrex-0.9.8.5.tar.gz the member Pyrex-0.9.8.5/Demos/embed/Makefile is invalid: 'filename None not found'
Running setup.py egg_info for package pyrex
Installing collected packages: pyrex
Running setup.py install for pyrex
changing mode of build/scripts-2.5/pyrexc from 644 to 755
changing mode of /Users/james/Documents/virtualenvs/bio/bin/pyrexc to 755
Successfully installed pyrex

Now, to get and editable pygr:

(bio)james@flapjack:~/Documents/virtualenvs/bio$ pip -E . install -e git://github.com/jamescasbon/pygr.git#egg=pygr
Obtaining pygr from git+git://github.com/jamescasbon/pygr.git#egg=pygr
Cloning git://github.com/jamescasbon/pygr.git to ./src/pygr
remote: Counting objects: 6281, done.
...
Successfully installed pygr

Finally, ipython:

(bio)james@flapjack:~/Documents/virtualenvs/bio$ pip -E . install ipython
Downloading/unpacking ipython
Downloading ipython-0.9.1.tar.gz (2.8Mb): 2.8Mb downloaded


We now have a completely isolated environment, where pygr and biopython are editable:

(bio)james@flapjack:~/Documents/virtualenvs/bio$ bin/ipython
Python 2.5.4 (r254:67916, Mar 2 2009, 10:40:04)
Type "copyright", "credits" or "license" for more information.

In [2]: pygr.__file__
Out[2]: '/Users/james/Documents/virtualenvs/bio/src/pygr/pygr/__init__.pyc'


The idea is simple. Pyjamas can create rich ajax applications in the same style as GWT but in python rather than Java. However, it leaves the implementation of the server side up to you. The aim of this entry is to show how easy it is to use pylons to provide a json based service for the client.

First, you need to make sure you’ve got pylons and pyjamas installed, which you can do with easy_install. Next, create the template using paster, and start the server.

/tmp$paster create --template pylons echo
....
/tmp$ cd echo/
/tmp/echo$ paster serve --reload development.ini
Starting subprocess with file monitor
Starting server in PID 6377.
serving on 0.0.0.0:5000

Now, point your browser at http://localhost:5000/ and you should see the default pylons page. To get the client side working. We need to copy the json pyjamas example into the public directory (this contains static files to be served). Then edit build.sh to point to your pyjamas installation build script.

/tmp/echo$ cd echo/public/
/tmp/echo/echo/public$ cp -r /.../pyjamas/examples/jsonrpc/* .
/tmp/echo/echo/public$ echo "python2.4 /.../pyjamas/builder/build.py JSONRPCExample.py" > build.sh
james@jlap:/tmp/echo/echo/public$ ./build.sh
Building 'JSONRPCExample.py' to output directory 'output'
...

If you now point your browser at http://localhost:5000/output/JSONRPCExample.html you should be able to see the example. I recommend that you install the mozilla live http headers plugin. This allows you to see what resources the client is interacting with. If you press “send to python service”, you will see a POST to output/services/EchoService.py. Edit JSONRPCExample.py so that the EchoServicePython class looks like this:

class EchoServicePython(JSONProxy):
    def __init__(self):
        JSONProxy.__init__(self, "/echoservice/json", ["echo", "reverse", "uppercase", "lowercase"])

You now need to rebuild the client with build.sh and restart the server. Unfortunately, the paster server doesn’t reload automatically when changes are made to the public directory so you need to do this manually. We have now set the python rpc service to point at /echoservice/json. In pylons terms, this will be automatically mapped to a method named json on a controller called echoservice. Create a file called echoservice.py in echo/controllers and put this in it.

from echo.lib.base import *

class EchoserviceController(BaseController):
    def index(self):
        return Response('Echo World')

If you now visit http://localhost:5000/echoservice/index you should see “echo world” coming from that controller. Now all we need to add is the json part of the controller. Put this into echoservice.py

from echo.lib.base import *
from pylons.decorators import jsonify
import simplejson

class EchoserviceController(BaseController):
    def index(self):
        return Response('Hello World')

    @jsonify
    def json(self):
        body = request.body.read(int(request.environ['CONTENT_LENGTH']))
        json = simplejson.loads(body)
        data = json['params'][0]
        method = json['method']
        if method == 'reverse':
            data = data[::-1]
        if method == 'uppercase':
            data = data.upper()
        if method == 'lowercase':
            data = data.lower()
        return dict(result=data)

Now try the “send to python service” and you should see the correct results returned. So what have we done here? Firstly, we needed to decode the method body. Import simplejson and use the loads method on the body got from the request object. The json object is a dict, so we can access the method name and parameters and create a dictionary with the answer stored under the result key. To return the data in json format, we just need the jsonify decorator which will translate the dictionary for us. Simple, huh?

When you mess up the service, it is worth knowing that you can inspect the stack using the debug url. This is given both in the console, and the http headers. Note also that storing the source python for the client in the public directory is stupid, but you knew that.

I couldn’t find a decent viewer in python, so I thought I’d knock one up. It seems quick enough and does the job. I even used pylab to allow the use to zoom in on particular nodes to see whats going on in a busy part.

You can’t see the node labels, but they are displayed in the interactive map. I couldn’t think of a decent way of getting them into a node that could be really small.

It also became a nice guinea pig for testing the setuptools automate cheeseshop /pypi upload. Its really good, and worked without a hitch. You should now be able to do easy_install treemap!

I also use nose for testing. Even written a few plugins (eg one that dumps the test results into a nice excel spreadsheet to give to the QA department) and contibuted a few patches. The test attributes are a great feature, and it allows you to mix unittest, simple test methods and doctests without thinking about it. However, I was inspired by this post on automating nose whenever a source file changes to look at doing the same in eclipse. Turns out, its easier than I could have imagined.

Under project properties, define a new ‘program’ builder. Set the location to the nosetests script, set its working directory to ${project loc}, append any command line options you want (eg –with-coverage -a !slow,!gui) and then, under the build options tab, tick the ‘launch in background’ and ‘during auto builds’. Now, whenever you save the changes to a file, the test results will come out in the console.

Unfortunately, it seems eclipse cannot share builders between projects. However, you can setup nose as an external tool. Then, when you need it as a builder you can import it. It’s also quite nice to have an external tool set up that runs nose only on the selected file. To do this set up an external tool as before, but instead set working directory to {$container loc} and put {$resource loc} in the command line arguments.