It's been a long road, but finally, I am happy to annouce that Channels 2.0 is released to the world, along with its partner packages Daphne 2.0 and channels_redis 2.0 (renamed from asgi_redis). Channels 2 is a major rewrite, entirely changing the way things are run and how the code is structured, hopefully for the better.
Channels 2 isn't entirely polished yet - it's still lacking a few features from Channels 1 (like multiplexing) and it's lacking a in-depth tutorial or updated examples in my channels-examples repo - but it's more than complete and stable enough to write things on top of.
Of course, I have a history of doing major rewrites - look at South to Django Migrations, which was a similar kind of rewrite, preserving most of the basic concepts but ripping out and replacing nearly all of the code. I don't encourage this, and doing it still feels bad - after all, Django is the world of each release being backwards-compatible - but I felt it was necessary, and I want to go into some of the details of why I did this.
Channel Layers
The cornerstone of the design of Channels 1 was running the network termination code - that turned HTTP requests and WebSocket frames on the wire into events - in a separate process from the application code.
This may seem like a slightly odd idea, and it is - it's far more complex than the nice model of WSGI where things import the application and just run it internally. It was necessary, though, because I had sold myself on needing something restricting: support for Python 2.7.
That meant not having access to any of Python's native async features, and so I had to construct Channels out of synchronous components. That meant implementing the application side as a standard event-loop pulling events off a queue - what everyone knows as runworker - and because of that, having to run the Twisted-based server in a separate thread or process so it wasn't being blocked by the synchronous code as it tried to handle lots of open sockets at once.
Of course, one thing you need in any large event-based system is a way of sending events between different processes - what Channels calls the channel layer - and so I re-used that as the main way for the server and worker to talk to each other, as well as using it for broadcast change events (and other tasks it was really meant for, and still handles in Channels 2).
This model worked surprisingly well, considering - the throughput and latency was not nearly as bad as you might expect (and indeed as bad as it was in the first prototypes) - but it meant deploying the project meant two more moving parts (a Redis/RabbitMQ server for the layer and a separate worker process) and, even worse, more dimensions to scale in. Did you run more workers? Add more Redis shards? Run more Daphne instances?
Simplification
I knew around 18 months ago that I wanted to revisit things and simplify them, but I was too scared - and maybe too self-proud - to make such a breaking change without good reason. It took the sage advice of several of my friends in the Django and Python community to tell me what I knew all along; that it was OK to abandon Python 2.7 (something that would have seemed much less feasible when Channels was first conceived in 2014) and go full-async.
With this in mind, I took a step back and redesigned Channels for what I wanted it to be in this new world - not from an internal code perspective, but from a user-facing perspective. I drafted API designs and deployment docs, made basic prototypes I could play around with as an end-developer, and tried to work out what the best approach was.
One of the most firm things I knew was that things needed to run in-process again. If I am to suggest ASGI as a replacement for WSGI - and I am going to start doing that more - it needed to mirror the simplicity and elegance of WSGI, and allow for a similar interface for servers. Thus, I took what Channels 1 called "ASGI" and split it into two - a specification for encoding HTTP and WebSockets as a series of events (which became part of the new ASGI) and one for Channel Layers and cross-event communication, which became the Channel Layer specification.
I then remade ASGI as a true, simple WSGI successor - a callable you can pass stuff to, and that runs as an application inside another process (and moreover, allows you to nest applications inside other ones and write native middleware). Here's a WSGI application's basic signature:
def application(environ, start_response):
...And here is an ASGI application's basic signature:
class Application:
def __init__(self, scope):
...
async def __call__(self, receive, send):
...The environ of WSGI, which tells you about the properties of a connection, becomes the scope of ASGI. That's the easy part. The harder one is allowing ASGI to support more than a simple HTTP response, which means changing its "call the function and it'll call start_response with data" into "run its coroutine and give it send and receive awaitables".
The new ASGI spec has no hint of Django-specific parts, or channel layers, or any of that - it just does network protocol mapping into events, and nothing else. It even ships with a WSGI-to-ASGI converter that lets you host WSGI applications inside an ASGI server, the backwards compatibility that the spec was always intended to have but was hard to get with the Channels 1 version.
Channel Layers are still around inside Channels, but now they're just for server-to-server communication for events - like "there's a new message in a chatroom", or "this user has a notification", and the ASGI application gets to receive and handle them now, rather than replies going straight to the WebSocket. Crucially, they don't have to take the burden of handling all the application's HTTP traffic too, which means things run a lot easier.
I would like to see ASGI adopted in the Python world in general, and I've been having soft talks with some framework and server maintainers over the past year or so, but now things are finalised I intend to focus on this more directly, with the eventual aim of making it into a PEP if people start adopting it.
Turtles All The Way Down
The new ASGI specification is clean enough to allow another thing - something that I have, ever since I saw Simon Willison get excited about the concept at DjangoCon EU 2009, called "Turtles All The Way Down".
What this means, in short, is that every part of Channels 2 is its own valid ASGI application. Each consumer is, the routing classes are, the static files handler is - everything. It's all composable and nestable inside each other, and when you give something your project's ASGI application to serve, all you're really doing is handing it the root routing class.
Not only does this allow easier testing and debugging - as you can use the same tooling on them all - but it lets you easily add and remove parts of Django as you want to use them, a principle I've always been a fan of - batteries are included, but are easily removable.
Even Django's view system is implemented as an ASGI application - you can pass HTTP requests to it and they'll run through middleware and views just like normal (and this is the default). But, if you want to, you can route to it differently, or even run differently-configured versions under different URLs.
It also allows for better reusability and multiplexing, among other things, but overall it's the simplicity that I like. The URL routing system in Channels is just an ASGI application that takes a list of URL routes and other ASGI applications to call for them. Even better, it's all async, which means when you run under Channels 2 you're running a full async stack all the way until you either hit the view system or a synchronous consumer. Talking of which...
Asynchronous Consumers
Consumers are still the basic unit of code in Channels 2, like a view is in Django, but they have a new twist. Consumers in Channels 1 had to be synchronous (because the whole worker side was, as I've described above). Now that consumers - and all ASGI applications, because they are themselves mini-applications - run inside the asynchronous server process, it opens up the chance to have natively-asynchronous consumers.
This means, if you want to hold open a WebSocket connection for 10 seconds, you can just do this, and not worry about blocking a whole thread:
class LongWaitConsumer(AsyncConsumer):
async def websocket_receive(self, event):
await asyncio.sleep(10)
await self.send({"type": "websocket.send", "text": event["text"]})Of course, you shouldn't have to write asynchronous code unless you need to - it's much easier to make errors in - and so there are synchronous consumers as well. What's interesting is that you can go between these two worlds easily, thanks to the new AsyncToSync and SyncToAsync helpers from asgiref. These let you turn any async callable into a synchronous callable, or vice-versa - letting you call the Django ORM (a synchronous interface) from an async consumer, for example.
Allowing you to run natively async code like this has allowed us to remove entire features, like the delay server, that were only there because there was no way to sleep without blocking a whole process. You can even use AsyncToSync to run a bit of asyncio code in the middle of a synchronous consumer, if you want.
There's other major changes - like entirely reworked test helpers, a routing system that supports Django URL pattern objects (both the old url and new pathtypes), better auth integration, and more. You can read more about it all in the new Channels documentation.
An Apology
This does all come at a price though, and that price is one of the highest you can pay as a user of a piece of software - no backwards-compatibility. The concepts in Channels 2 are the same, but you will absolutely have to change your code to make it work if you are porting from Channels 1.
This is not a decision I make lightly, and I tried to find a way to make things backwards-compatible that wouldn't either compromise the design of Channels 2 or drown me in maintenance debt, but in the end, that wasn't possible. Django has always been a bastion of backwards-compatibility, and it's very sad to me that I can't keep that in this case.
If you have asked me why Channels is not in Django core itself at any point, it's reasons like this. Async web frameworks are still a relatively new area in Python, as is the ability to finally set Python 2 support aside. That, along with other changes, means some of the decisions I made four years ago are no longer valid.
Even worse, it's taken me time to overcome my natural stubbornness and give in to make this change; it's hard to admit that you are wrong in public sometimes, and even more so when, as I tend to be, the type of person who works on projects mostly by yourself.
No small part of this process has been played by my friends and fellow maintainers in the Django and Python web ecosystem who never failed to give advice and suggest the right things to do when I was at a loss, and for my fellow members of the Django core team, for having far too much trust in me to get things right with the Django name hanging off of it.
One thing I would like to change as we enter into Channels 2 is how open and welcoming the project is to new maintainers. I have a nasty tendency to work on things solo, and be a controlling perfectionist about design, and this combined with the fact that very few people actually want to work on async networking code means Channels has not had nearly as many contributors as it could have.
Channels 2.0 is a somewhat unfinished release - some of the key nice things I'd like to have are deliberately not done, partially because shipping is better than perfection, and partially so there's stuff left to work on. If you are at all interested in doing something, please get in touch with me directly, and I'll try to help you through the open issues list to find something you can work on, and give some mentorship time and help as I can too if you need it.
What's Next?
With all that said, what does this release mean for Channels?
First off, there's plenty of room to add nicer interfaces for things like long-polling, work on alternative protocols, and generally add more of the basic batteries, as well as write a full introductory tutorial and nice example projects. I would have called this a soft release, but if I have learnt only one thing from over a decade of being an open-source maintainer it's that you're better off just doing a release even if it's not perfectly complete.
Second, there's plenty of work to do around outreach and cross-collaboration. I want to improve things for the Python web ecosystem in general, and while Django is part of that, there's plenty of other parts too. We have a lot to do if we want to keep Python relevant on the back-end over the next decade.
Finally, it means I'll be wrapping up most support for Channels 1, reducing it to just security and data-loss bug fixes. I'd love to keep it more actively maintained because of the lack of backwards-compatibility, but only a few of us work on Channels and we don't really have a lot of spare time, and people were already starting to move over to Channels 2 even before it was out.
Will Channels merge into Django core sometime? Maybe, but that depends on how things evolve both within Django and outside. Will ASGI finally be something that becomes a spiritual successor to WSGI? Not without work, agreement and compromise, but I have all those to offer.
Will I have a nice rest and finally stop worrying about the mistakes I made in Channels 1? Absolutely, but I'll remember that people still found it useful, sites run on it, and I learned a lot along the way as well. Thanks to everyone who's helped out and used it so far, and here's hoping many more sites, projects, and exciting adventures are to come.