Okay, so, Backchat was hugely interesting as a project. Eventually, I produced a set of graphs using the classifier that showed sentiment over time. These graphs aren’t too accurate but are fairly good at showing how things were going. However, after this I pretty much dropped the project. This was mainly due to exams cropping up and stealing my time away, but also because of how difficult it was to approach a sensible level of accuracy.

In my ‘final’ design I ended up using a bigram classifier. I added parsing of the tweets to pull out mentions of words, URLs and users, and then used this to generate my training sets, which improved things a lot. This gave me several thousand tweets for each training set, which worked okay. However, even with this classifier, which was doing a lot better than most others, my results weren’t very reliable on a tweet-by-tweet basis. Still, it wasn’t too shoddy, and the graphs on the right are fairly reliable I think in terms of general sentiment.

The AMQP-linked network of processors worked extremely well, and resulted in good throughput- I used two parsers, two classifiers and one classifier loader in the end; I was unable to achieve realtime performance due to network constraints. Sadly my ISP at home had decided that I’d used too much bandwidth and clamped me down to 128 kilobits a second. That said, thanks to the streaming API I did not (as far as I know, except for a few hundred to ratelimiting) lose any tweets, I just received them out of order and then reconstructed the correct order using the timestamps for each tweet. The machine I was using for this also pretty much went flat out on disk I/O and CPU usage, but was able to keep up- it’s a fairly old box, only a Pentium 4 with a couple of gigs of RAM.

In any case this was an interesting project and I’ll be open sourcing the data and source in the coming weeks if anyone wants to have a poke at it. While the debates are now gone and done, I’m sure people can come up with some great uses for sentiment analysis outside of UK politics.

We have got some things in the works for Dominion, and we hope you’ll find them useful; we’ve not had much time to work on EVE Metrics, and we’re being distracted by another project at the moment, but we’ll have more time to work on EVE Metrics in a few weeks time around Christmas. I’m still evaluating what we’ll spend our time on, though, and we’d like to get more feedback via the feedback button on the site- you can vote for other people’s suggestions, so please do so!

Other than that, not much to report. We’ll have Dominion items loaded into the site by release day so you can start using the site straight away with the new items. We’ve been so far very successful with some performance improvements on the site; this has mostly been tuning our database server and working on improving the performance of queries through better indexes, clustering indexes, and so on. Hopefully you’ll notice this in the form of improved page responsiveness and less ‘slow loading’ pages. Enjoy!

This means the site will be down for several hours on Thursday as we arduously migrate all the old data (though not market data- we’re starting fresh!), set up the new database backend and test everything. After it comes back up there will be a few changes- notably that the APIs will have moved and changed format. They’re now much more useful, including providing historic data and some other useful bits of info. And they’re documented properly now! There is no web push framework and we’re looking at if we’re going to offer it again or go with an alternative.

The new site features a lot of the features you’re used to, but removes a few. For the time being, regional limiting is not available- however, it’s not as needed in EM2. We’ve redesigned the market UI considerably and it performs a lot better. If you like your graphs, you’ll love EM2- there’s plenty of them, sparklines and full graphs.

Our plan with EM2 is to iteratively add more features incorporating feedback we get from users as we go- hence why this release focuses on making things performant and fast, rather than packing in features aplenty. There will be paid (ISK, of course) versions of EM2 for large scale traders and corporations which will provide more advanced tools, but none of this is in EM2 at release, and we do not intend to make any of the basics cost ISK at any stage, nor will any APIs require payment to use. And one of the best bits about this? We’ll be using the ISK to run competitions for our uploader community, and to reward uploaders with ISK. We’re not doing this at launch because we need to analyse how everything works out in the real world, but uploaders will no longer be credited for getting lots of uploads- they’ll be rewarded for information density of uploads, be it stuff we’ve not seen in a while or stuff that changes often. Uploading the same 10 orders once a day isn’t going to get you a lot of credits- but it won’t get you nothing, either.

What we’re trying to really push and develop is our uploader community. We already have a huge collection of uploaders who regularly upload information to the site, and we’d like to reward them properly in a way that encourages having fresh data in the database for as many items as we can manage. This means anyone who uses the site will be getting better information and more of it, which is of course good. Good data means we can use the data to make more useful tools and provide better statistics.

On the other side of awesome, the song written by Jed Whedon and Maurissa Tancharoen for Episode 13 of Joss Whedon’s Dollhouse (“Epitaph One”) is out on Amazon- and it’s bloody fantastic. Even if you’re not a Dollhouse fan (and if you’re not, why are you here? Go watch it!), you’ll probably love Remains. For 80p, it’s hardly worth not downloading it.

And another awesome thing: Makurid, the MMMetrics resident caching/performance tweaking guru, has been applying some extra caching to Charactr- everything’s now a whole lot snappier after a good helping of memcached, so you should be able to enjoy seeing your character list anywhere, faster.

However, my DIY-fu is fairly strong, and my usual urge to try and do stuff in a somewhat hackish fashion kicked in…

I’m lucky enough to have a nice little stream outside, with a fairly decent flow rate and cold water. I had some copper microbore pipe left over from a plumbing project, and figured that I could rig up a heat exchanger in the room and in the stream, link them and pump water round the system, thus removing heat from the room.

I originally planned to buy some PVC tubing (flexible) but couldn’t find a local supplier, so I turned to garden hose. Very strong, quite flexible, and not too expensive. A 50 metre roll cost £40, and I had 20ish metres left over from another project, more than enough in total. All that remained was to get a heat exchanger and pump. The heat exchanger ended up being a radiator off a BMW from a breaker’s yard (£25), and the pump was a £60 central heating pump (including adaptors to convert the 1/12″ plumbing connectors to 15mm pipe).

The Diet Coke bottle is connected to the radiator overflow

The hoses fitted onto the 15mm pipe after being heated with hot water just fine, with jubilee clips to hold them on securely (likewise on the microbore). To get into the radiator I used a set of 22mm->15mm adaptors with self-algamating tape to pad the diameter out a bit, with jubilee clips clamping down the radiator hoses onto the fat end of the adaptors. Basically what I had kicking about.

The result? With 3 people and 4 (hot) computers in a room the system kept it not only comfortable but cold! For about £125 I not only have a cold room but for the fraction of the power draw- with a fan and the CH pump, it draws around 50 watts instead of the 800+ watts of most air conditioners.

The app is Charactr, a web based skill training monitor and planner with both email and SMS notifications. Go check it out!

The team (MMMetrics) is the same team who will be attacking the problem of EVE Metrics 2. A quick rundown of the features of Charactr:

• Support for as many API keys as you want
• Flexible notification framework
  • Send notifications via text message or email
  • Get notifications on specific characters, accounts, or everything
  • Receive daily summaries, messages before or on completion
• Powerful skill planner
  • ‘New Character’ training speed bonus support
  • Learning skill suggestions
  • Reorder, add and delete skills
  • Training for item/ship
• Secure and fast access anywhere in the world
• SMS messaging to over 750 networks in over 200 countries

Enjoy!