Django Diaries / 11th Dec 2013

Working Together

Time for your semi-regular update on how Django's migrations are going! I've implemented two major things since the last blog post: workflow improvements, and ForeignKey change detection.

Workflow

One of the problems that only started emerging with South some years after it was released was the issue of working with distributed VCS systems - like Git and Mercurial - and how their tendency towards branching impacted migration writing.

When South was initially released, the world was mostly a Subversion place, and so the chance of two developers committing conflicting migrations was small. These days, the chance that someone is going to go off in a branch to add new model features while the master branch also gets a model bugfix is quite high.

South didn't really deal with the resultant merge very well; it implicitly ordered migrations by filename, and so whoever had the migration which appeared first alphabetically "won"; the other developer would have to apply migrations out-of-order or back out theirs and then apply both. In short, it was a mess.

My fix to this problem inside django.db.migrations is simple - because migrations now have explicit dependency entries (the numbers at the start of the names are now just there for show), it's trivial to detect when you've merged two branches - the migration dependency graph for an app has two "most recent" migrations.

If South detects this case, it'll now halt and tell you how to proceed - namely, it'll point you to makemigrations --merge, which will print the changes that have happened on both sides and ask if you want to make a merge migration (essentially, you're saying both branches are fine to run independently). If the two sets of changes conflict, you have a bigger problem; you're going to have to fix your codebase and your migrations manually.

Other Niceties

There's a couple of other small workflow changes as well - suggested by loic84 - that have appeared in runserver and migrate.

Basically, if you run runserver while you have unapplied migrations, it'll print a big red warning notifying you of this; this should help remind developers to run migrations before complaining that your commits don't work.

Somewhat similarly, if you just run migrate with changes you haven't even put into a migration yet, it'll warn you and tell you how to make a migration. This was suggested because otherwise people might still just run migrate like they ran syncdb - add a model, run it - and if you've got a migration-enabled app (as all ones made in 1.7 and up are) this isn't going to do anything. Now, at least, it teaches you what to do right there on the command line.

Foreign Keys

The other big change is that ForeignKeys now know when the field they're pointing to has changed and can update themselves appropriately - for example, if you have a model with a primary key of CharField(max_length=20), and update the length to be 40, all the foreign keys pointing to it will now be updated to be of length 40 as well.

This was a very tricky change - foreign keys are probably the place where Django's supported databases vary most wildly - but it does now work. In particular, it will drop the foreign key constraints while it's changing the columns behind them, so if those columns are very large the re-creation phase for those constraints might take a long time. This is also not supported at all on SQLite; since it doesn't really have foreign key integrity, it's less of a problem there.

Next up

The next thing I'm looking at is going to be some way of nicely fixing the swappable models (AUTH_USER_MODEL) situation - I expect we're not going to be able to autodetect you swapping user models, but third-party apps that ship with migrations need to point to the right table when you start using them.

After that, I suspect I'll start work on South 2, which is probably going to consist of some automated extraction of the Django migrations code into a package combined with a monkeypatch suite that makes 1.4 - 1.6 have enough 1.7 features (connection.schema_editor(), Field.deconstruct()) to make it run. Always good to have something ridiculous to look forward to, eh?