More on PoliticsPosters

Okay, so day one of that site is over, with some lessons learned and some serious improvements made!

About 11AM, just as I was trundling along the M25 on my way to Egham, TheyWorkForYou made a change to their API method, getMP, which PoliticsPosters uses to find your MP from your postcode (or at least did- more on that in a tick). Basically since our MPs aren’t technically our MPs, the method started to return nothing, a case I hadn’t considered. Fortunately they provided the always_return option, so when I got back from a successful day in Egham at around 7PM, I quickly fixed that.

Next bug: In any election, constituency boundaries are likely to change. In this one, we got plenty. My lookup had previously been this:

  1. mp = PoliticsPosters::API.twfy.mp(:postcode=>params[:postcode], :always_return=>true)

Trouble is, this doesn’t work if your boundary has changed (and thus your last MP has changed). To do that, we need to use the getConstituency call with the new ‘future’ flag.

  1. constituency = PoliticsPosters::API.twfy.constituency(:postcode=>params[:postcode], :future=>1)
  2. mp = PoliticsPosters::API.twfy.mp(:constituency=>constituency.name, :always_return=>true)

And now we’re good! Easily fixed, overall, and it removed the two biggest problems. The other problems are a little trickier. One remains- handling of special characters in MP names. There’s a few MPs with circumflexes and the like in their name, which is fine if you have a language/framework/tools that support UTF8 encoding (which Ruby/Sinatra/Prawn does). But the font I’d chosen, Chunk, didn’t have any of the characters it needed to render. This remains an issue, and my current planned work-around is to degrade to another font for pages and posters that handle those names. I’ll sort that out tomorrow, though- this post is, after all, being put together at 3AM, and I have to sleep sometime.

The only other thing people wanted was more posters. I have been happy to oblige. I combined the PublicWhip policy feed provided through TheyWorkForYou, wrote a tiny scraper to get the titles off the PublicWhip site (since there’s no API for that I’m aware of), stuffed it together in PostgreSQL through DataMapper, and now I can do something like this:

  1. PoliticsPosters::MP.first(:full_name=>'David Cameron').policies #=> [<#PoliticsPosters::MPPolicy policy_id=4 distance=0.05 etc>]

Which is very, very neat, and means that developing stuff is very easy. I dug around in the source code for TheyWorkForYou and pulled out their code for rendering PublicWhip data to get the same policy IDs and descriptions, wrote some methods to scale color and size, and stuffed it all together to produce the new and updated constituency page as well as new posters- one for all policies, and one for each individual policy. The individual policy posters are rendered on demand, whereas the common constituency ones are rendered on the first view of a constituency page.

The Linode VPS has been handling the load without even noticing. Despite the site being inoperative most of the day, it got 10,000 hits from nearly 4,000 unique users, who downloaded nearly 2,000 posters. Not bad for a first day.

Tomorrow I hope to sort out UTF8 character handling and maybe even get around to publishing the source code to the site if I have a spare moment.

Welcome to Pandora

We’ve successfully moved all sites, email, DNS, and everything else on our old server, Highpoint, to our brand new machine, Pandora.  This has entailed a lot more downtime than we’d anticipated; this has mostly been due to lack of preparation on my part, a glorious DNS cock-up and the added complexity of having Highpoint’s backhaul fail three times as we tried to move across all the data.

In total it was a fairly mammoth operation by our standards; we transferred in excess of 100 gigabytes of data between the servers over the course of 12 hours, shifted over 20 websites and 3 major webapps, and got everything up and running again in under a day once we’d moved it all to the new box. The downtime has been annoying and I’ve certainly learned some lessons for next time, but here’s the flipside…

We’re now running on a much, much roomier machine. We’ve not got the environment perfectly set up and we’ll no doubt spend the next week tuning everything, adjusting things till they’re just right and fixing bugs, as well as adjusting and rewriting chunks of applications to make use of the extended caching capabilities of our new environment. We’re already using this to great effect in the EVE Metrics APIs but we can make better use of caching throughout our apps.

Once we’ve gotten settled in, we should be performing much better and more reliably than previously. We’ve already seen huge performance gains on our database (we can process more than twice as many uploads per second, for example) and we hope to have things even faster soon.

Of course, to achieve this I have been running on more or less an empty tank as far as sleep is concerned and working things in around my life at university, which has been interesting. Still, we’re at the point now where it’s more or less stable and everything basically works, so now I’m going to grab a few hours of sleep before lectures tomorrow, before a long long lie-in on Saturday. Enjoy!