Aeracode

Django-powered Snow

During the planning of our Christmas card at work this year, a mad idea came up. Do we ignore mad ideas? No, we tackle them head-on.

The idea was to build an internet-controlled snow machine - you'd hit the button on the website, and watch a member of the Torchbox team get pelted with snow. When we first came up with the idea, we dismissed it as being "too complex", but after a while, we came around.

Cue three days of frantic development and phoning round to get the parts. I'll be posting a full build article, with all our source code, once we're done.

The brief summary is that we have a Django app which handles rate-limiting of snow, and tracking who has clicked the button, which then communicates with our snow machine using Artnet and DMX. Ustream is used to stream the video back to the internet.

Still, I imagine you want to see it in action, so head over to snow.torchbox.com, and have a go. It's only online 10 - 5 UK time, and only until Tuesday (we can't fill our office up with paper snow forever), but it's still very good fun, even if you don't know any of us!

South 0.6.2

In my neverending quest to save the time of those using RDBMSen, South 0.6.2 is released.

It mostly contains bugfixes, but there's now support for extending introspection to third-party apps without having to edit their source.

I'll probably start working full steam on South 0.7 soon - there's a refactor of the migration engine (mostly done), of startmigration (not started), and a few changes to command names to make them nicer (the old ones will still work), and to the behaviour of default values.

I'm hoping 0.7 will be the last release before 1.0, at which point I will claim I've been working in an octal version numbering sequence all along.

Announcing Heechee

In my opinion, there's just not enough Subversion servers on the internet.

That's an opinion you'll probably never hear me express - I much rather everyone move over to a DVCS (*ahem* mercurial) - but Subversion has, some would say quite rightly, earned its place as the dominant VCS for nearly every part of the IT community (apart from big, lumbering, financial companies).

Recently, I've been slowly switching away from subversion for my personal projects into Mercurial. This, in itself, a painless process, but with my public projects - particularly South - this has the unintended effect of slightly alienating some of my users. Most are alright with adapting - after all, it's quickly becoming the norm to use a DVCS - but there's still a few people left out.

One of the particular issues I have is with svn:externals. A lot of apps - including some we have at work - rely on svn:externals to pull in external dependencies into a libs folder along with the project itself. Externals is one of the few features of subversion that I thought was pretty much perfect, and it was sad to see my move to Mercurial break it.

This week, at DjangoCon, Chris Wanstrath did a nice talk on DVCSen. One of the questions led him onto hg-git - the awesome git backend plugin for Mercurial, that the GitHub guys wrote - and how they first investigated the idea of an svn gateway to expose their repositories transparently to subversion users. From what I gathered, subversion's wire format proved too tricky to deal with, and so they turned elsewhere.

However, the idea of a subversion gateway intrigued me greatly. What better way to transparently serve South so people can still pull it using svn:externals while still developing in Mercurial? With that in mind, I started looking into Subversion's wire protocols and their delta format (svndiff) sometime yesterday afternoon.

After some digging, reading an obscure academic paper and liberal application of Wireshark, I grew confident enough that I could at least implement something. A day later, and I'd like to present the very first version of what I'm calling Heechee (if you get the pun, ten nerd points).

Heechee is a transparent mercurial-as-subversion gateway. It serves a Mercurial repository as a Subversion WebDAV-based repository. It's still in the early stages, but at the moment it will serve its own mercurial repository to subversion in such a way that you can check out the repository, and update to various revisions within it.

You can check it out at BitBucket. It's pretty alpha code, and make sure you have the dependencies mentioned in the README, but it works, which greatly surprises me. I plan to much improve the code to support more 'advanced' features, like being able to do more than checkout and update, as well as exposing tags and branches correctly. There's even the chance I'll stick Git support in, when I've had a play with Dulwich.

Still, hopefully I'll soon have it so it can serve South's code via svn, and thus restore harmony to my land of VCS users.

So, please, take the code and have a play with it. It's not formally licensed and released yet, but it will probably be Apache licensed. Feedback and suggestions for features, as well as how to linearise trees into svn histories, much welcomed!

South 0.6.1

South 0.6.1 is now available for public consumption; it fixes quite a few bugs.

If you want to see what's been fixed, have a gander at the Release Notes - there's a few significant bugs that have been fixed, including problems with SQLite, GeoDjango, and M2M fields.

Sorry for the short but sweet post - I'm in the process of writing my talk for DjangoCon! I'm giving a talk about South, with both a good grounding for those who haven't grokked migrations yet, as well as some tips on more advanced usage. Come watch it on Tuesday morning!

South 0.6

It is with great pleasure (and a measurable sense of relief) that I announce the release of South 0.6, a new release bringing quite a few new features, although most of them aren't immediately obvious.

Perhaps the biggest change is the move from parsing source files to get the field definitions (ugh, I know) to introspecting the fields to get their definitions. As well as being more stable and making my skin crawl less, this also means that dynamically-created models and fields are now fully supported.

South still falls back on the parser module for custom fields if it has to - since it doesn't know how to introspect them - but we encourage you to implement a South-readable interface for custom fields; see our wiki page explaining how to do that.

The other noticeable change is that the migrate command is much zippier, thanks mostly to the hard work and lazy-evaluation-patches of Juan and Simon from Akoha. Note that the SQL for migrations is no longer printed out as they're running any more - it was getting a bit verbose - but you can re-enable it with --verbosity=2. You can also completely shut South up using --verbosity=0, if you desire.

For easier conversion, there's a new one-shot convert_to_south command - take care to read the wiki page on that, however, as the caveat is that you must run a migrate --fake on any other computers running the same codebase. We're going to make that a bit more intuitive for the next release.

The release notes contain the rest of the changes - some big, some small - and remember, we've switched to Mercurial for revision control, so if you're still running off Subversion please change your checkouts.

South 0.6 Release Candidate

After a long few months, during which I've increased my recommendations to run off trunk, South 0.6 is almost here.

This release contains a whole bunch of new features - more of which will be written up on the Release Notes page soon - but in the meantime, I've decided to adopt a more usual release process.

This means two things: First, I'm releasing a release candidate of South 0.6 for general testing and final critical-bug reporting; please see the Download page and follow the instructions to check out the 'stableish' tag, or if you already have a checkout please 'hg pull' and 'hg up' it.

The Git mirror is also, for once, up to date, and there's also a tar.gz available for those who don't fancy either DVCS.

The second change is that I'll actually maintain a stable branch from now on as well, much like Django. This means there will be a 0.6 branch for any fixes that are easy or trivial to backport, and in the unlikely case someone should manage to conceive of a security vulnerability in a migrations library fixes for that will go in too. This should hopefully mean you and your production servers can track a real release number and not miss out on bugfixes.

Note that a few of the planned features haven't made it into this release in time, but I felt I had to get something out there. My much-vaunted fixture migration didn't make it in, but introspection, the major feature, did.

Still, please, test the release! If things look good, I'll officially tag and release 0.6 early next week.

South 0.6 news

It's about time I posted an update on what's going on with South development, so here it is.

Firstly, there hasn't been much work done in the last few weeks, due to exams and there being a Ball to help run. Now that's over, development will crack down. 0.6 is going to focus on the following new features:

Less parsing, more introspection. The default way of getting fields' definitions will be to introspect them; the implementation of this is pretty much complete already, and can be found in the 'noparsing' Mercurial branch. Custom fields (which cannot be introspected withour prior knowledge) have two options; provide a south_field_triple method to tell us their definition (preferred way), or as a fallback the parser will be used. This feature solves many issues, including dynamic models, and also doesn't lose the ability to work with most custom fields right from the start.
Reverse dependencies. Just brought up by one of the Ella developers, this is a good idea that should have been in South a while back. You'll be able to specify 'this other migration depends on us' as well as 'we depend on that other migration' - this helps pluggable apps around a common core.
Fixture migration. Since I know this is going to bite me in the ass at some point, I'll be rolling in the fixture-migration someone mentioned at EuroDjangoCon (uses the test runner to make a temporary database, loads an old fixture set, an old schema, migrates, dumps the new fixture). I haven't fully tested this yet, so it might not make it for reasons of impossibility, but I'm reasonably confident.
Signals. There will be post-migrate and pre-migrate signals added to help developers add in custom code that needs to be run after migrations or before migrations (the latter helps people trying to carry out a deployable app rename, for example)

As well as those new features, there's also around 25 bugs that need fixing, and will be tackled soon, hopefully. I aim to get a beta release of these out in the next month, especially in the light of South's allegedly high penetration rate.

The last piece of news is that I'm moving my main development efforts to Mercurial (and specifically BitBucket), and only pushing mostly-stable trunk phases to Subversion. There's a github mirror of whatever hg-git deems to be main development, but that's for convenience only, and because I can do it in one command.

South's Design

I had a lot of interesting chats with people about South last week at EuroDjangoCon, and several eyebrows have been raised at me both parsing models and then storing their definitions as dicts.

While this is a design that is partially born of backwards compatibility, it was mostly a conscious design choice I had to make, and so I'd like to now describe why I made it.

Firstly, for those who aren't aware, South's magic autodetection and auto-writing-migration code is based off of a file called 'modelsparser.py', which does what it says on the tin; it opens models.py, parses it, and extracts definitions directly.

Now, my first response to this is also 'ew', but there's a reason for it: custom fields. If it wasn't for those, introspection of the existing fields would work just fine, given a long list of special cases (I in fact have a file called modelsinspector.py sitting around that does just that - if you're interested, it's in the 'noparsing' branch on BitBucket. It doesn't do much, but it proves the concept).

However, with custom fields in the game, you have two options: hope they follow Django conventions and try to introspect common options, or put the onus on custom field authors to add a hook that makes it spit out a definition. While I would obviously choose the second one (possibly added to some of the first), it simply doesn't provide the drop-in compatability I want South to provide.

However, as Alex Gaynor and several others have surmised, I could just pickle everything and stick that at the bottom. No introspection needed, no weird namespace errors because the models dictionary is simply a collection of eval-able strings.

However, I don't like pickle (the Python kind; the delicious food kind, on the other hand, is more than welcome in my household). The security issues don't matter here - after all, we're passing raw strings to eval instead - but it has two major features I don't like. Firstly, it's not at all human-readable - and while I don't imagine people editing their migrations by hand, I like to be able to see what is going on, and I like the fact that South still doesn't depend on you using startmigration - a design choice I'm very proud of.

Secondly, I've been burned by pickles in the past. If you remove a module that a pickled thing uses, pickle will crash and burn, and if you've not done them right, you won't know quite where the error happened. (For the purposes of argument, of course, I'm assuming I'd do it right, so this isn't really a point. Still...)

There's yet another option (see, my job isn't easy) - custom-pickling, which Alex suggested last Wednesday; it consists of roughly freezing the __dict__ of the field. This has some advantages - it's more human-readable - but you get the errors similar to my current approach (where datetime.datetime wasn't unfreezing if people didn't have datetime imported).

My main gripe, with both this and pickle, is that the internals of fields aren't guaranteed to be consistent between releases of Django, just the external API, and so South uses a method that lets it use the external API - in my mind, this is less hassle, since that's the thing that's really not going to change very much (and even if it does, you can easily see what South is trying to call, and run that in a shell yourself to see where the problem lies).

So, that's why models are frozen as a massive lump of dictionaries and triples. As with many things, it probably wouldn't be necessary if models had been done slightly differently (so there was a method that always spat out a nice, portable way to reincarnate a field), but with no reason for that previously, I would have been thoroughly impressed by the core developers' foresight if it existed.

As for parsing versus introspection... I'm still divided. Parsing always has, and still feels, wrong, but introspection might just be less reliable. If you're interested, go peek at the branch I mentioned above, and see what you make of it.

And now, I will stop blathering and return to my small hole at the base of the mighty Django Mountain, where I am attempting to fit square pegs into round holes. Good day.

EuroDjangoCon Slides

For those interested, the slides from my migrations talk at EuroDjangoCon are up. I believe the videos will be around soon.

South 0.5

I'm pleased to announce the release of South 0.5; this has been a long and exciting release, with many new features...

ORM Freezing, a new feature inspired by Migratory, which lets you freeze the historical state of your current models into a migration so you can then use them via an ORM as if your models.py file existed at that point in time. There's a good description of how exactly this is used in part 3 of the new tutorial.
Automatic change detection; using the frozen model states, South can now diff your current models.py file against how it looked last time you made a migration, and write a migration to make up the difference. This essentially allows django-evolution style hinting, and means you'll only have to manually write more advanced migrations. It's discussed more in part two of the new tutorial.
A much more robust models.py parser, using the Python parser module. It's a dirty secret that South automatically makes migrations by directly examining your models' source code; however, the new parsing engine hits 99% of all Django use cases, and has provisions for custom hooks so more wacky fields (such as django-denorm's decorated-function-based-fields) can tell South their equivalent definitions.
A move into a south/ subdirectory, to fit with Python packaging guidelines, and make South more friendly with environments such as buildout and pip.

I've been very excited about the new features for a while, as those who have talked to me will tell you, and thanks to those people who used the trunk releases and submitted a great set of bug reports, it's finally ready for general consumption!

Additionally, for those going to EuroDjangoCon, don't miss my talk on migrations next Monday; as well as South and its new features, I'll be covering and comparing django-evolution and a few other migration solutions.

Happy migrating!