However, as with anything these days, a single app isn’t sufficient no matter how good the app is. The expectation to end users is that everything should play together nicely and you should be able to get data from A into B with a minimum of fuss. And as a developer, you clearly want to enable this- it means your app has more happy users, and if your app is commercial that translates to more cash. If you make a really, really good API then you can end up with a veritable ecosystem around your app; other companies pouring money into development, all supporting your business. So APIs are good for business.

But more importantly APIs are good for consumers. If you’re listening on your DAB radio or (in our case soon, with any luck) an RDS equipped FM radio, you want some metadata. Who are the presenters you’re listening to? What’s this song? Listening online? Then you want the song title. This is all pretty basic stuff we’ve come to expect thanks to media players on computers. If you’re a big company like the BBC you just adjust your tools and systems to support the APIs you need, or specify them in your requirements to outside vendors and get them to add support. But what about the little guys?

P Squared have made a good step in the right direction as of their last major release of Myriad; they added a TCP/IP interface that lets you query Myriad for some basic variables. Through a fairly bodged-together little set of scripts we can get data out of Myriad like what song is playing now, and what’s coming up. But it’s still awkward.

So I whipped up a little Sinatra webapp that does the appropriate little dance to act as a gateway to that awkward TCP/IP gateway from the land of HTTP. Everything speaks HTTP, or can be prodded to speak HTTP quite easily. It’s far and away the best tool for interacting between applications, since it’s well understood and simple to understand. Now I can have a script poke Myriad and our website’s API, combine the two, and suddenly all our metadata can be updated: “There Will Be Cake with James Harrison: Around The World (Radio Edit) – Daft Punk”. And while there’s absolutely no need to tell people the name of that particular song, we’re still adding a lot of value by providing more interactivity to our listeners through updated content. And hell, it’s just more professional. Click through for the scripts.

About 11AM, just as I was trundling along the M25 on my way to Egham, TheyWorkForYou made a change to their API method, getMP, which PoliticsPosters uses to find your MP from your postcode (or at least did- more on that in a tick). Basically since our MPs aren’t technically our MPs, the method started to return nothing, a case I hadn’t considered. Fortunately they provided the always_return option, so when I got back from a successful day in Egham at around 7PM, I quickly fixed that.

Next bug: In any election, constituency boundaries are likely to change. In this one, we got plenty. My lookup had previously been this:

Trouble is, this doesn’t work if your boundary has changed (and thus your last MP has changed). To do that, we need to use the getConstituency call with the new ‘future’ flag.

And now we’re good! Easily fixed, overall, and it removed the two biggest problems. The other problems are a little trickier. One remains- handling of special characters in MP names. There’s a few MPs with circumflexes and the like in their name, which is fine if you have a language/framework/tools that support UTF8 encoding (which Ruby/Sinatra/Prawn does). But the font I’d chosen, Chunk, didn’t have any of the characters it needed to render. This remains an issue, and my current planned work-around is to degrade to another font for pages and posters that handle those names. I’ll sort that out tomorrow, though- this post is, after all, being put together at 3AM, and I have to sleep sometime.

The only other thing people wanted was more posters. I have been happy to oblige. I combined the PublicWhip policy feed provided through TheyWorkForYou, wrote a tiny scraper to get the titles off the PublicWhip site (since there’s no API for that I’m aware of), stuffed it together in PostgreSQL through DataMapper, and now I can do something like this:

Which is very, very neat, and means that developing stuff is very easy. I dug around in the source code for TheyWorkForYou and pulled out their code for rendering PublicWhip data to get the same policy IDs and descriptions, wrote some methods to scale color and size, and stuffed it all together to produce the new and updated constituency page as well as new posters- one for all policies, and one for each individual policy. The individual policy posters are rendered on demand, whereas the common constituency ones are rendered on the first view of a constituency page.

The Linode VPS has been handling the load without even noticing. Despite the site being inoperative most of the day, it got 10,000 hits from nearly 4,000 unique users, who downloaded nearly 2,000 posters. Not bad for a first day.

Tomorrow I hope to sort out UTF8 character handling and maybe even get around to publishing the source code to the site if I have a spare moment.

In total it was a fairly mammoth operation by our standards; we transferred in excess of 100 gigabytes of data between the servers over the course of 12 hours, shifted over 20 websites and 3 major webapps, and got everything up and running again in under a day once we’d moved it all to the new box. The downtime has been annoying and I’ve certainly learned some lessons for next time, but here’s the flipside…

We’re now running on a much, much roomier machine. We’ve not got the environment perfectly set up and we’ll no doubt spend the next week tuning everything, adjusting things till they’re just right and fixing bugs, as well as adjusting and rewriting chunks of applications to make use of the extended caching capabilities of our new environment. We’re already using this to great effect in the EVE Metrics APIs but we can make better use of caching throughout our apps.

Once we’ve gotten settled in, we should be performing much better and more reliably than previously. We’ve already seen huge performance gains on our database (we can process more than twice as many uploads per second, for example) and we hope to have things even faster soon.

Of course, to achieve this I have been running on more or less an empty tank as far as sleep is concerned and working things in around my life at university, which has been interesting. Still, we’re at the point now where it’s more or less stable and everything basically works, so now I’m going to grab a few hours of sleep before lectures tomorrow, before a long long lie-in on Saturday. Enjoy!