A couple of months back we worked with E-ON to put an advert in the magazine as well as running a login screen advert, and this has now run. So for a few days anyone logging into EVE will have been greeted with this fine advert created by Zapatero at MMM Publishing.

Of course, this suddenly meant more people going to and using the site. Fortunately one of the primary concerns when myself and Makurid put EM2 and later EM3 together was scalability. Whereas EM1 would’ve fallen over and died, EM3 has soldiered on like a champ. The only intervention we’ve had to perform was to fix the API processor, which was hanging regularly and causing problems as a result. That’s fixed, and we’re now stable and responsive. So, of course, we now have some numbers! These are the statistics for the most recent 30 days as of this post.

• ~85,000 visits
• ~500,000 page views
• ~5 pages per visit
• ~29,000 unique visitors
• ~1,500 additional account registrations as a result of login screen advert (Above ~20 account registrations per day baseline)

In terms of the site’s dataset, we’re getting fairly huge.

• 6,000 EVE API keys, of which 4,000 are full keys
• 75,000 EVE API methods enabled
• 32,000 EVE API calls per hour (about 10 every second)
• 75,000 characters (includes characters noted in market and other data)
• 15,000,000 wallet journal entries
• 6,300,000 wallet transactions
• 200,000 EVE Mails
• 1,600,000 active market orders
• 6,400,000 EVE API loaded trades
• 52,000,000 Inferred trades
• 11,000,000 processed uploads
• 146,000,000,000 total skillpoints of loaded characters
• 4,100,000,000,000 total ISK of loaded characters

Which is… scary. But! We’ve grown hugely (a threefold increase in requests per second) and we’re still performing well. The site is not throwing errors, users are generally happy with their experience with the site, and we’re stable – no crashes, no fires breaking out, no climbing resource usage. Our caching and design philosophies have worked well and the extra effort invested earlier in development has really paid off. We’ve just handled a huge amount of growth with no effort at all. I’d estimate we’re good to around 12,000-15,000 users on our current hardware.

Of course, we’re now wondering what to do. I stopped playing EVE long ago aside from the odd spot of tinkering, but I’ve even let accounts lapse and stopped updating my skill queue as of a month or two ago. Makurid’s just started playing again recently. We’re not really heavily invested in EM in terms of motivation, other than making a cool webapp.

Capsuleer, as I’m sure many of you know, is an EVE iPhone app that recently shut down because it couldn’t be monetized and the time and money invested in it by the developers was simply unreasonable. A couple of weeks back I was looking at the realities of what EM costs to run, and what it costs in terms of time to maintain. And the logistics of, if needed, shutting the site down. This is still somewhat in my mind but the site will be sticking around for a little while yet. I’m very much hoping CCP will come to their senses in terms of how they choose to support third party developers (a free account subscription appears to be the greatest gift given by CCP, but that’s a far cry from the ~£130 per month it costs to run EVE Metrics – and that’s just our costs _now_, not including new hardware costs or upgrade costs. If we grow much more we’re going to be looking at ~£200/mo to run EM, possibly more). Advertising isn’t an option, and donations have failed every time we’ve tried to support ourselves with them. Timecode sale affiliations barely made a mark on my accounting sheets. We do enjoy writing sites and producing something big that people find useful, but there are certain realities to be faced – both myself and Makurid are students, with no full-time jobs. I’ve recently picked up some part time work which means I can now afford to buy bacon on a weekly basis again.

In the meantime we will continue to support the EVE community by maintaining EVE Metrics. We’re working on the codebase right now to upgrade it to the latest and greatest Rails release and Makurid’s beavering away at the API processor to split it up into a puller and a processor, enabling us to achieve much higher throughput on API calls to CCP’s (slow) servers. Now that we’ve breached 30,000 requests per hour, that 10 request per second figure is actually beyond our capacity right now because of our single-threaded processor/puller mechanism. This is actually a CCP limit- we just need to work around it by hitting them with more requests simultaneously, thus spreading our load over multiple servers on their end. I’ll also be making performance and usability improvements wherever I can and incorporating as many bug reports as possible into what we release. This will likely be a slowish process, but in the long run, it should be worth it. The question really is, will CCP make it worthwhile for us to be investing our time and energy into longer term planning?

So here’s a post about EVE Metrics 3.

We’re getting close to having everything polished and ready for release. The main issue at hand thus far has been the homepage; I say this like it’s a small thing but it’s required me to learn some new and interesting things about CSS, Makurid’s done some excellent work to produce some feed scrapers and elements for the lower portion of the page… there’s a lot to it.

I thought that it’d be good to list a few of the changes we’ve made for version 3.

• Complete redesign of the site thanks to Rettic
• Market detail pages have been entirely renovated
• Various pages which haven’t been improved in some time have now been tidied up
• API key management has been improved
• API key permissions management has been improved
• Backend processors for API functions and upload processing have been improved and made more reliable
• My Metrics has been entirely renovated, now with sparklines for wallets and a new layout
• Orders and transactions have been moved from My Metrics into their own detail pages, with a summary on My Metrics
• Journal information has been added and given it’s own detail page
• Player Owned Structure integration has been added, though still in it’s infancy
• Sensitive portions of the site now make use of SSL transport encryption (HTTPS) automatically
• Wormhole pages have been updated
• Improvements to the corporation pages through refactoring to share view code between character and corporation pages
• Graph improvements
• Complete test coverage of every line of code (Nah, just kidding, we’re still pretty thin on those test things for large chunks of UI code)
• 0.2% more cowbell
• 5% other features I’ve not listed above, plus 100% more polish overall

Excited? We are! There’s a lot of work in the lines above and I think you’ll like the results. We’re not sticking to any firm release schedule because we’re terrible at sticking to them; we’re students, not full-time developers (incidentally, if anyone’s got any jobs available for temporary/contract work, SE UK preferred (or work-at-home), 6 weeks max, let me know!). That said, we hope to have a release before the end of July.

We’ve also been rewriting our uploader! That’s right- Linux, Mac and Windows support all in one neat package. The GUI’s not anything special but it works, and we’ll be polishing it and getting it release-ready before long. Huge thanks should be directed to TTimo, who has been the driving force behind this with some welcome Python experience, and Makurid for assisting him in developing the new client.

Once we’ve gotten that polished, packaged and rolled out, we’ll be running a 5 billion ISK contest to promote it- the three winners (each receiving a portion of the 5 billion pool) will be selected from the most active uploaders for the week or two after the competition is announced. We’ll finalize all the details and have it posted up when we’re ready to go ahead with that, of course. If you’d like to make that 5 billion figure larger, you can contribute ISK to the character MMMetrics Agent ingame! So far, thanks go out to Rilcon, Chribba and Entity for contributing to the current pool. The new uploader will be released after the new site – we’d like to change one thing at a time so we can iron out all the kinks. Once we’ve gotten it out and tested it thoroughly we’ll roll out the upgrade- your existing uploader will prompt you to update when you start it.

One last thing – I’ve personally submitted a proposal to the CSM regarding development efforts from CCP surrounding the API and EVE Gate. If you’ve not done so already, I strongly urge you to read my proposal and support it if you feel, like I do, that CCP have made some serious mistakes lately in this regard. The thread can be found here. It has been picked up and supported by at least 3 CSM members so far, but the community support will help considerably to drive CCP to consider it seriously.

This post is a tale of how we’ve been running into issues with memory usage recently, how we’ve been solving and diagnosing it, and the design decisions that have lead to it.

In the past week or so we’ve been puzzling over some unusually high memory usage of our application servers (we use the Thin server for EVE Metrics and Passenger for everything else). Specifically, our thins were getting fat; chewing up memory, not responding to new requests and churning on locks.

On a hunch and some history, we pulled out RMagick, the Ruby interface to ImageMagick, which had been known to cause issues with memory usage by way of some leaks. We switched the code that we had to use ImageMagick for to use the mini_magick library, and where we could (notably the sparkline graphs on the statistics panel) we switched from server-side to client-side rendering. Those sparklines are now using the excellent jQuery Sparklines plugin. They look better, too!

That did solve our memory issues; we still had problems with sudden spikes in memory, and we pinned down the cause using the Oink diagnostics plugin. Someone (and if you’re reading this, let me know who you are, I’d like to talk, not to bite your head off, but to ask you why the CSVs didn’t leap out at you quite as they should have…) decided to make lots of history requests to the API, for all 59 regions at a time and for 25 items at a time. Admittedly only for one day, but there’s a kicker here.

Those 25 items across 59 regions is not a lot of historical data. But, those of you with EVE Metrics API experience should be shouting, “James, the history API also includes the item API data!” Which it does! This entails loading the market orders. Now, there is a problem here. Let’s say you request 25 items for 59 regions. That is a _lot_ of market orders, potentially. Remember we have 1.5 million orders over the whole market and the really active market is spread across a few thousand items. But there’s a few items with vast numbers of orders. That number of orders can rise quite easily into the tens or hundreds of thousands.

We have to load all those orders into memory to work with them, then allocate the raw data into Gnu Scientific Library vectors, then process that data (fast, thanks to C), and then render out the XML response. Because we have to have all these orders in RAM at once, Ruby’s VM allocates the process more and more memory, which is removed from the OS’s spare resources which are used for caching the filesystem in memory. This cache is the primary mechanism for PostgreSQL, our database engine, to keep data in RAM; subtract from this and you’re hurting DB performance and increasing disk IO. Everything slows down. Eventually, Ruby allocates enough that the OS itself is out of cache and has to start swapping to disk; we’ve not yet run into this since I’ve been monitoring this all closely with Munin, but it would eventually happen. Ruby (along with most other VM-based languages) has a pretty crap GC, and even once the request is over, this memory is not returned to the OS from the VM. This means that things can escalate pretty dramatically if left unchecked.

So, what can we do to fix this? Well, in time I want to remove the item API data from the history API. That’ll improve things greatly in our own application. I’ve also implemented a limit on the number of items and regions that can be checked at once to stop people generating huge memory spikes.

There’s a nasty side-effect here: While these thins are churning away, nomming up memory like there’s no tomorrow, they’re locked up and handling a long running request. If we get enough requests this means the website becomes unavailable, something I personally find intolerable. So I’ve increased the number of app servers we run and split the requests up as they come in between two pools, one of which has a few servers dedicated to handling web site requests, and likewise one of which has a few dedicated to handling API requests, with a few servers shared between the pools. This will help stability in the long run and ensure that at any moment, requests can be fulfilled in a timely manner. Varnish is also getting some tweaks to serve up old content when the backends are busy where a cached object exists.