Well, last night I gave a talk at one of the ever-brilliant Oxford Geek Nights, and in case you’re baying for the slides I used (all seven of them), you can find them at this wonderful OGN7 LastGraph Slides link. If you live within punting distance of Oxford, you should really try to come to the next one on August 27th. If not; well, they put videos up on the site…
In a related note, I am enjoying the improvement on my slide-making capabilities via Inkscape. Previously, PDFs exported to massive file sizes and tying the individual pages together wasn’t easy. Now, the new Cairo export in Inkscape (the same PDF export library LastGraph uses, yay) makes reasonably-sized PDFs, and pdftk munges them together easily and in record time. One day I’ll give in and use an actual presentation program.
Yes, this is my new official term for the server going up and down. It seems that some part of the system causes 100% CPU usage on the box it’s running on, and in fact is so nasty to the system that it even makes the serial console lag. This, for those who are not server geeks, is Bad.
I’m on the case, and have brought the full force of monitoring tools to bear to see if I can track down the cause. In the meantime, please accept my apologies for any downtime.
After much work, lastgraph3’s beta is now out. Please be warned that it may not work, may explode, and may abduct your cat, dog or goldfish in the process of failing.
Apart from that, please feel free to give it a try at lastgraph3.aeracode.org. Report any and all problems back to me, either here, or at my email or something similar.
Oh, and it’s also very slow at fetching until I re-negotiate fast fetching with last.fm, since I’ve moved IP.
Update: I’ve decided to make this one of those public, neverending betas, so LastGraph is now basically open for anyone and everyone to use. The last feature I added was LastGraph Premium, my new way of soliciting donations (this time, you can donate, and in return get a few extra features). I’m still not sure about it, but I’ve had some emails before from people who would have liked it, so we’ll see how it turns out.
It really isn’t, honest. I’m working on the final part - the graph scheduler, which replicates the previous queued graph functionality - and I’ll hopefully be pushing it all live soon.
I was tempted to just push the live artist browsing part live, which all works currently, but it’s only interesting for, oh, the first few hours…
LastGraph has been given a much needed refresh, including a tweak to the render nodes to stop them running out of file handles/memory/disk space, and the main site so it in fact remembers when it deletes XML caches to free up space rather than wandering around going “I’m sure I put that file somewhere…”.
I may also actually implement expiry soon, as my S3 bill is finally above $5. Yay…
One of the overwhelming horrors of designing for the web (or so it would appear from a lot of the mockups I’ve seen) is that designers (and people who are just bored of Arial and Times New roman) want to make their titles on web pages using non-standard fonts. “But that’s shockingly non-accessible and uses more bandwidth”, I hear you cry; well, there’s a reason the alt tag is around, and why broadband is much more common.
Well, perhaps this isn’t the sole reason, but nonetheless it’s a more than feasible idea these days to have headings, titles and short lines of text using fonts a user doesn’t have installed on their system. And, until the spec for embedding fonts is finialised in around 2065, there are two main options:
So, we need header images. One horribly labour-intensive way of doing this is making them manually in Generic Graphics Editor 8.6. However, since we’re sensible people, we’ll generate them on the fly. And, since we’re sensible people, we’ll be using Django*, so we need to write some nice Python code.
I’ll be using Cairo to generate graphics, in part because it’s a nice library, and is pretty common these days. You’ll need the Python Cairo bindings; on debian-like systems, this is the package python-cairo; in other places, your mileage may vary.
The key to making Cairo work with Django is wrapping a Cairo canvas in a django view. For this reason, I have this function lying around:
def render_image(drawer, width, height):
import os, tempfile, cairo
# We render to a temporary file, since Cairo can’t stream nicely
filename = tempfile.mkstemp()[1]
# We render to a generic Image, being careful not to use colour hinting
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, int(width), int(height))
font_options = surface.get_font_options()
font_options.set_antialias(cairo.ANTIALIAS_GRAY)
context = cairo.Context(surface)
# Call our drawing function on that context, now.
drawer(context)
# Write the PNG data to our tempfile
surface.write_to_png(filename)
surface.finish()
# Now stream that file’s content back to the client
fo = open(filename)
data = fo.read()
fo.close()
os.unlink(filename)
return HttpResponse(data, mimetype=”image/png”)
The idea is, you pass it a function which will draw the image onto a context, and the image’s width and height, and it takes care of all the boring tedium of wrapping cairo and django together.
Now, that’s not very useful by itself, is it? Time to draw some text!
Firstly, as an aside, we need a way of seeing how big a certain text string will be for a given font and size, so we can render an image just big enough for it. This function achieves that:
def text_bounds(text, size, font="Sans", weight=cairo.FONT_WEIGHT_NORMAL, style=cairo.FONT_SLANT_NORMAL):
import cairo
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 1, 1)
context = cairo.Context(surface)
context.select_font_face(font, style, weight)
context.set_font_size(size)
width, height = context.text_extents(text)[2:4]
return width, height
Yes, yes, it’s somewhat cryptic, but it does the job. Now, we can write a text-rendering view!
def render_title(request, text, size=60):
# Get some variables pinned down
size = int(size) * 3
font = “Meta”
width, height = text_bounds(text, size, font)
def draw(cr):
import cairo
# Paint the background white. Replace with 1,1,1,0 for transparent PNGs.
cr.set_source_rgba(1,1,1,1)
cr.paint()
# Some black text
cr.set_source_rgba(0,0,0,1)
cr.select_font_face(font, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)
cr.set_font_size(size)
# We need to adjust by the text’s offsets to center it.
x_bearing, y_bearing, width, height = cr.text_extents(text)[:4]
cr.move_to(-x_bearing,-y_bearing)
# We stroke and fill to make sure thinner parts are visible.
cr.text_path(text)
cr.set_line_width(0.5)
cr.stroke_preserve()
cr.fill()
return render_image(draw, width, height)
Here, we construct the draw function with a simple text drawing command, and run the wrapper.
There’s some interesting text positioning going on up there; for more on this, and cairo in general, read through the excellent Cairo Tutorial for Python Programmers.
The last thing is to add an appropriate URL into your URLconf, such as
(r'^title/([^\/]+)/(\d+)/$’, “render_title”)
And then, when you browse to /title/HelloWorld/20/, you’ll hopefully get a nice PNG of your new title! Then, you can just use img tags instead of titles, in this sort of style:
<img src="/title/{{ item.title }}/20" alt="{{ item.title }}" />
This process is quite quick, but not without a small cost of processing power; if you’re using it a lot, think about some sort of caching. Apart from that, be happy with your newfound title freedom…
* Or possily Pylons. As long as you don’t go and cavort with those Gems On Guiderails people, or heaven forbid the [PH/AS]P guys…
Yes, people of the internet, LastGraph has returned. After over two weeks of beta testing and bugfixing, it’s finally in a useable state, and so I’m pushing it out to lastgraph.aeracode.org as I type this. If it doesn’t work for you yet, wait for the DNS change to propagate.
This version will, inevitably, contain bugs, so I would appreciate it if any bugs could be emailed to lastgraph at aeracode.org.
There have been some improvements, such as much better error handling, reduced PDF sizes, faster rendering (and more render nodes), and detailed progress. There are still some missing features, though, notably the ability to set the plays threshold and to remove a graph from the queue (this will only be possible if you’ve provided an email address).
Plans for the future include custom colourschemes, more notification methods (jabber, and perhaps some kind of twitter reply thing). If you have more ideas, send them in to that address above (or poke me via one of the many methods found on the contact page).
LastGraph will be down for a few days while I sort out somewhere better for it to run. It’s been using this servers’ resources massively, and there’s about 35GB of data now (over a gig of which is raw XML). The other sites on the server are suffering from the load, and we’re down to Not Very Much diskspace.
I’ll be relaunching with the new rewritten version, and automatic expiry of graph storage, to hopefully keep things more sane. If I can afford it, I may move to using something like Amazon S3 for storage, to avoid these issues.
In the meantime, please hang tight! You will be rewarded with your graphs soon enough (I plan to even spend my time on the coach down to London fixing this). If you’re feeling generous, you may want to consider donating below; hopefully I can get enough funds to invest in more storage and processing power!
If you’d like to help with the costs of servers and storage, why not donate now?
The download queue is back in the hundreds for the first time since the initial launch, and to make things worse the poor Last.fm API server is being a bit unreliable again today.
Still, it’s nice to see it being used! I may have to speed up my plans for lastgraph2, and look into using Amazon Web Services for some more rendering oomph (if the pennies add up, that is). Stay tuned!
After my first attempt at providing some way for people to style graphs in Graphication, which ended up being a rather ugly system with an odd set of nested dictionaries, a thought struck me; we already have a language for specifying presentation, and which has inheritance and other nifty time-saving shortcuts: CSS.
I could only find one python CSS library, cssutils, and while that seemed to have very decent CSS2 support for parsing into a document tree, I couldn’t see any immediate way of using it for retrieving the applicable values for, say, a grid object with class “minor” inside a wavegraph object.
First, I wrote a very lightweight CSS parser and rule matcher. Code examples always show off these things best; first you do something like
css_string = """wavegraph {color: #369; font-size: 12; }
grid.minor { color: #eee; } """
import css
stylesheet = css.CssStylesheet.from_css(css_string)
If you’re feeling like using stylesheets a lot, you can make them external (e.g. a file “default_css.css”) and use the import hook:
import css
css.install_hook()
import graphication.default_css as stylesheet
Then, querying properties is pretty simple:
>>> stylesheet.props("wavegraph")
{"color": "#369", "font-size": "12"}
>>> stylesheet.props("wavegraph").get("color")
"#369"
>>> stylesheet.props("wavegraph").get_int("font-size")
12
>>> stylesheet.props("wavegraph grid.minor line")
{"color": "#eee", "font-size": "12"}
This means all the styling crud previously used can be replaced with these simple css-selector-ish queries, and different graph styles can simply ship as different css files.
So, hopefully, graph styling will be a lot more accessible once I roll this fully into the graph system, as well as a lot nicer to deal with for most uses. In the meantime, if you want to look at this CSS parser code, have a look at the current subversion copy.
