Okay, so this blog, back in the days of old, used to talk about EVE a lot. Now, I don’t play EVE – I haven’t for well over a year. I stopped playing because I lost all faith in the company that runs it, CCP Games hf. At one point I was choosing an education and career path that would set me up nicely to apply for a job at CCP, which should give you an idea of how big a change it was from huge advocate of EVE to what’s known in the community as a bittervet – a bitter veteran. That a term exists for this type of player says a lot. Continue reading The real problems behind CCP’s botched forum launch
We use a program called Myriad for all our broadcast automation, asset management and general playout purposes. It’s made by a company called P Squared, who are a very able and friendly bunch of people and who make ultimately a fairly decent product.
However, as with anything these days, a single app isn’t sufficient no matter how good the app is. The expectation to end users is that everything should play together nicely and you should be able to get data from A into B with a minimum of fuss. And as a developer, you clearly want to enable this- it means your app has more happy users, and if your app is commercial that translates to more cash. If you make a really, really good API then you can end up with a veritable ecosystem around your app; other companies pouring money into development, all supporting your business. So APIs are good for business.
But more importantly APIs are good for consumers. If you’re listening on your DAB radio or (in our case soon, with any luck) an RDS equipped FM radio, you want some metadata. Who are the presenters you’re listening to? What’s this song? Listening online? Then you want the song title. This is all pretty basic stuff we’ve come to expect thanks to media players on computers. If you’re a big company like the BBC you just adjust your tools and systems to support the APIs you need, or specify them in your requirements to outside vendors and get them to add support. But what about the little guys?
P Squared have made a good step in the right direction as of their last major release of Myriad; they added a TCP/IP interface that lets you query Myriad for some basic variables. Through a fairly bodged-together little set of scripts we can get data out of Myriad like what song is playing now, and what’s coming up. But it’s still awkward.
So I whipped up a little Sinatra webapp that does the appropriate little dance to act as a gateway to that awkward TCP/IP gateway from the land of HTTP. Everything speaks HTTP, or can be prodded to speak HTTP quite easily. It’s far and away the best tool for interacting between applications, since it’s well understood and simple to understand. Now I can have a script poke Myriad and our website’s API, combine the two, and suddenly all our metadata can be updated: “There Will Be Cake with James Harrison: Around The World (Radio Edit) – Daft Punk”. And while there’s absolutely no need to tell people the name of that particular song, we’re still adding a lot of value by providing more interactivity to our listeners through updated content. And hell, it’s just more professional. Click through for the scripts. Continue reading The importance of APIs in broadcast
Well, it’s been a while. I’ve been busy with things in this nasty place called reality which has been kicking my ass as a result with all sorts of fun ailments. Recently though I was forced to pay a bit more attention to EVE and specifically EVE Metrics.
A couple of months back we worked with E-ON to put an advert in the magazine as well as running a login screen advert, and this has now run. So for a few days anyone logging into EVE will have been greeted with this fine advert created by Zapatero at MMM Publishing.
Of course, this suddenly meant more people going to and using the site. Fortunately one of the primary concerns when myself and Makurid put EM2 and later EM3 together was scalability. Whereas EM1 would’ve fallen over and died, EM3 has soldiered on like a champ. The only intervention we’ve had to perform was to fix the API processor, which was hanging regularly and causing problems as a result. That’s fixed, and we’re now stable and responsive. So, of course, we now have some numbers! These are the statistics for the most recent 30 days as of this post.
- ~85,000 visits
- ~500,000 page views
- ~5 pages per visit
- ~29,000 unique visitors
- ~1,500 additional account registrations as a result of login screen advert (Above ~20 account registrations per day baseline)
In terms of the site’s dataset, we’re getting fairly huge.
- 6,000 EVE API keys, of which 4,000 are full keys
- 75,000 EVE API methods enabled
- 32,000 EVE API calls per hour (about 10 every second)
- 75,000 characters (includes characters noted in market and other data)
- 15,000,000 wallet journal entries
- 6,300,000 wallet transactions
- 200,000 EVE Mails
- 1,600,000 active market orders
- 6,400,000 EVE API loaded trades
- 52,000,000 Inferred trades
- 11,000,000 processed uploads
- 146,000,000,000 total skillpoints of loaded characters
- 4,100,000,000,000 total ISK of loaded characters
Which is… scary. But! We’ve grown hugely (a threefold increase in requests per second) and we’re still performing well. The site is not throwing errors, users are generally happy with their experience with the site, and we’re stable – no crashes, no fires breaking out, no climbing resource usage. Our caching and design philosophies have worked well and the extra effort invested earlier in development has really paid off. We’ve just handled a huge amount of growth with no effort at all. I’d estimate we’re good to around 12,000-15,000 users on our current hardware.
Of course, we’re now wondering what to do. I stopped playing EVE long ago aside from the odd spot of tinkering, but I’ve even let accounts lapse and stopped updating my skill queue as of a month or two ago. Makurid’s just started playing again recently. We’re not really heavily invested in EM in terms of motivation, other than making a cool webapp.
Capsuleer, as I’m sure many of you know, is an EVE iPhone app that recently shut down because it couldn’t be monetized and the time and money invested in it by the developers was simply unreasonable. A couple of weeks back I was looking at the realities of what EM costs to run, and what it costs in terms of time to maintain. And the logistics of, if needed, shutting the site down. This is still somewhat in my mind but the site will be sticking around for a little while yet. I’m very much hoping CCP will come to their senses in terms of how they choose to support third party developers (a free account subscription appears to be the greatest gift given by CCP, but that’s a far cry from the ~£130 per month it costs to run EVE Metrics – and that’s just our costs _now_, not including new hardware costs or upgrade costs. If we grow much more we’re going to be looking at ~£200/mo to run EM, possibly more). Advertising isn’t an option, and donations have failed every time we’ve tried to support ourselves with them. Timecode sale affiliations barely made a mark on my accounting sheets. We do enjoy writing sites and producing something big that people find useful, but there are certain realities to be faced – both myself and Makurid are students, with no full-time jobs. I’ve recently picked up some part time work which means I can now afford to buy bacon on a weekly basis again.
In the meantime we will continue to support the EVE community by maintaining EVE Metrics. We’re working on the codebase right now to upgrade it to the latest and greatest Rails release and Makurid’s beavering away at the API processor to split it up into a puller and a processor, enabling us to achieve much higher throughput on API calls to CCP’s (slow) servers. Now that we’ve breached 30,000 requests per hour, that 10 request per second figure is actually beyond our capacity right now because of our single-threaded processor/puller mechanism. This is actually a CCP limit- we just need to work around it by hitting them with more requests simultaneously, thus spreading our load over multiple servers on their end. I’ll also be making performance and usability improvements wherever I can and incorporating as many bug reports as possible into what we release. This will likely be a slowish process, but in the long run, it should be worth it. The question really is, will CCP make it worthwhile for us to be investing our time and energy into longer term planning?