Article / 20th Feb 2009

Fun with GIS

I've always been somewhat envious when seeing the various sites with nice maps of crime, prices and other things for non-UK regions.

However, over the past week and a bit me and Martin have been changing that. WhoseTurf rolled out of the coding mill some time last week, and it's been improved a lot since, in our quest for useful (and fun) online geographical statistics for the UK.

GeoDjango has been on my list of things to play with properly for a long time, and so it was good fun to get my teeth into it. It works surprisingly well - the admin interface was really helpful when collating all our various area info (and even drawing some districts by hand - not a fun task when your source map is on a different projection).

At the core of WhoseTurf is a load of 'reviews'. Each review has a certain 'fanout' - the radius in which it can affect the ratings - and, accordingly, most of the work in our custom tile renderer is correctly interpolating between the correct review points and their values.

Having circles affecting values rather than the actual discrete shapes of the various UK districts makes for some slight inaccuracy, but for most variables this makes more sense than drawing it discretely (crime doesn't suddenly drop when you change county, for example). Council tax is one of the few exceptions to this, and so accordingly is drawn discretely.

Another clever trick we use is that, because the heatmaps we serve up just consist of tiles of large squares, we simply serve 40x40px images and let the browser scale them up to 256x256. This cuts down on rendering time and bandwidth, makes our cache (Varnish) happier, and also lets me crow on about how clever it is (even though it's reasonably obvious).

It turns out there's some things GeoDjango just can't do - like this massive query that makes correctly-shaped districts out of the various output areas we have (they don't fill the whole space, nor line up perfectly, so otherwise you get holes or hair-thin 'cracks'):

SELECT st_collect(cpoly) FROM (
        SELECT st_simplify(st_makepolygon(st_exteriorring(
        (st_dump(upoly)).geom )), 0.001) AS cpoly FROM (
            SELECT st_union(spoly) AS upoly FROM (
                SELECT st_snaptogrid(mpoly, 0.01) AS spoly
                FROM core_outputarea
                WHERE id LIKE '""" + code + """%%' AND
                st_isvalid(mpoly) = true)
                AS a
            ) AS b
        )
    AS c;

The other small problem was Firefox not having enough JS stack space to load in the border of the Highlands (it has lots of very silly little islands and glens), but I'm willing to let that one slide.

We'll probably be releasing the code behind the thing as open source; I doubt any of it is _that_ transferable, but there is all the code in there to render standard (spherical mercator) tiles using cairo and/or PIL, and if you too want to show lots of heatmaps, and perhaps kill your server, it could be for you.

One last side note: OpenLayers is, unfortunately, still not really useable. Google maps is a) smaller and b) more efficient (drags more smoothly, scroll zoom, although I did have to write our own layers widget). On the flip side, OpenStreetMap is getting more than useable for a site like ours now.