## EVE Metrics and popularity

Well, it’s been a while. I’ve been busy with things in this nasty place called reality which has been kicking my ass as a result with all sorts of fun ailments. Recently though I was forced to pay a bit more attention to EVE and specifically EVE Metrics.

A couple of months back we worked with E-ON to put an advert in the magazine as well as running a login screen advert, and this has now run. So for a few days anyone logging into EVE will have been greeted with this fine advert created by Zapatero at MMM Publishing.

Of course, this suddenly meant more people going to and using the site. Fortunately one of the primary concerns when myself and Makurid put EM2 and later EM3 together was scalability. Whereas EM1 would’ve fallen over and died, EM3 has soldiered on like a champ. The only intervention we’ve had to perform was to fix the API processor, which was hanging regularly and causing problems as a result. That’s fixed, and we’re now stable and responsive. So, of course, we now have some numbers! These are the statistics for the most recent 30 days as of this post.

• ~85,000 visits
• ~500,000 page views
• ~5 pages per visit
• ~29,000 unique visitors
• ~1,500 additional account registrations as a result of login screen advert (Above ~20 account registrations per day baseline)

In terms of the site’s dataset, we’re getting fairly huge.

• 6,000 EVE API keys, of which 4,000 are full keys
• 75,000 EVE API methods enabled
• 32,000 EVE API calls per hour (about 10 every second)
• 75,000 characters (includes characters noted in market and other data)
• 15,000,000 wallet journal entries
• 6,300,000 wallet transactions
• 200,000 EVE Mails
• 1,600,000 active market orders
• 146,000,000,000 total skillpoints of loaded characters
• 4,100,000,000,000 total ISK of loaded characters

Which is… scary. But! We’ve grown hugely (a threefold increase in requests per second) and we’re still performing well. The site is not throwing errors, users are generally happy with their experience with the site, and we’re stable – no crashes, no fires breaking out, no climbing resource usage. Our caching and design philosophies have worked well and the extra effort invested earlier in development has really paid off. We’ve just handled a huge amount of growth with no effort at all. I’d estimate we’re good to around 12,000-15,000 users on our current hardware.

Of course, we’re now wondering what to do. I stopped playing EVE long ago aside from the odd spot of tinkering, but I’ve even let accounts lapse and stopped updating my skill queue as of a month or two ago. Makurid’s just started playing again recently. We’re not really heavily invested in EM in terms of motivation, other than making a cool webapp.

Capsuleer, as I’m sure many of you know, is an EVE iPhone app that recently shut down because it couldn’t be monetized and the time and money invested in it by the developers was simply unreasonable. A couple of weeks back I was looking at the realities of what EM costs to run, and what it costs in terms of time to maintain. And the logistics of, if needed, shutting the site down. This is still somewhat in my mind but the site will be sticking around for a little while yet. I’m very much hoping CCP will come to their senses in terms of how they choose to support third party developers (a free account subscription appears to be the greatest gift given by CCP, but that’s a far cry from the ~£130 per month it costs to run EVE Metrics – and that’s just our costs _now_, not including new hardware costs or upgrade costs. If we grow much more we’re going to be looking at ~£200/mo to run EM, possibly more). Advertising isn’t an option, and donations have failed every time we’ve tried to support ourselves with them. Timecode sale affiliations barely made a mark on my accounting sheets. We do enjoy writing sites and producing something big that people find useful, but there are certain realities to be faced – both myself and Makurid are students, with no full-time jobs. I’ve recently picked up some part time work which means I can now afford to buy bacon on a weekly basis again.

In the meantime we will continue to support the EVE community by maintaining EVE Metrics. We’re working on the codebase right now to upgrade it to the latest and greatest Rails release and Makurid’s beavering away at the API processor to split it up into a puller and a processor, enabling us to achieve much higher throughput on API calls to CCP’s (slow) servers. Now that we’ve breached 30,000 requests per hour, that 10 request per second figure is actually beyond our capacity right now because of our single-threaded processor/puller mechanism. This is actually a CCP limit- we just need to work around it by hitting them with more requests simultaneously, thus spreading our load over multiple servers on their end. I’ll also be making performance and usability improvements wherever I can and incorporating as many bug reports as possible into what we release. This will likely be a slowish process, but in the long run, it should be worth it. The question really is, will CCP make it worthwhile for us to be investing our time and energy into longer term planning?

## Nebulous

A few days ago I said I’d blog about Nebula, a project myself, Makurid and others have been working on on and off for a few weeks now. So here we go:

After seeing Mynxee get some wheels in motion with CSM5, I felt a bit annoyed at my inability to contribute a lot to the workings of the CSM. At my age, I am not eligible to run for candidacy, as the drinking age is 21 in Iceland and CCP don’t want delegates they can’t take down to the pub (At least, I guess – the age of majority is 18, so it’s not a legal thing). Anyway, I decided to use my normal approach to fixing problems: There’s an app to be written for that…

So with the help of Selene from EOH Poker, we rallied a whole 3 CSM and ex-CSM people into a collaborative text editor, and wrote a spec. The tool in question was to manage proposals as they went through the CSM process – with the wiki and AH forums being identified as inadequate by both the CSM and CCP, but with no activity from CCP on providing a solution, I saw this as an opportunity to get the community involved and really push to get some CSM processes streamlined. We called it Nebula, and formed a new team of developers, with an open source codebase and instructions on how to contribute. We actively sought out new developers and tried to encourage other third party developers to get involved, and were met with mostly positive results. Team Excellence was born.

The CSM involvement we had was good. We ended up with a spec that was, as the title of this blog suggests, a little nebulous, but had most of the major concepts well sketched out and defined. We got to work.

It’s now been a fairly long time since we last had CSM contact, though those who have been trying to get involved in this (Mynxee and  Trebor to name a couple) have done so well, and continue to provide input; the main issue we have is that out of the 8 (since Ankh was removed) active CSM members and 5 reserves (or 9 and 4, depending how you look at it- either way, a total of 13 players), we’ve gotten in touch with and received input from three at most.  This is worrying for a number of reasons.

If the chairwoman of the CSM is having this much difficulty in managing to have some people meet and discuss matters for even a one-off meeting or event, then I start to wonder how well the communication works between meetings on other issues. For example; I posted a long while ago about the API in the assembly hall forum. The matter has been taken up by two CSM members, including Mynxee, and may be raised at a future meeting. I got an evemail from Dierdra Vaal a couple of weeks later, asking me if I thought anything was wrong with the API and if I’d support him raising an issue about it. This is the sort of thing that irks and worries me – if there’s no real communication between CSM members except during their meetings with CCP, then how the hell do they hope to present any sort of unified face to CCP or address issues that are concerns to the playerbase, and not individuals within the CSM? With no communication outside of those meetings, issues aren’t being discussed till it’s far too late to be discussing them, people are missing entire issues, and this all has a knock-on effect on how much the CSM actually achieves.

Frankly, I couldn’t imagine being a member of the CSM without there being a mandatory-usage CSM delegate IRC channel or some similar chat mechanism – heck, even a mailing list or forum – where the CSM can talk amongst itself, and get the ego trips out of the way before CCP gets involved. And I suspect that the fact that a lot of people on the CSM see no problem with the current state of affairs is a good indication of just where the priorities of those members lie – because it’s certainly not with the playerbase as a whole. The CSM is bigger than EVE’s petty squabbling of alliances and corporations; we’re talking about an elected council of people who can help steer the course of a company which is putting food on the table for hundreds, and innovating hugely. The CSM as an entity and as an idea does not deserve the majority of the people it has been saddled with so far.

Nebula as it stands is frozen, awaiting information from the CSM and the motivation of the developers to work on it. I know that myself and Makurid, who thus far have written the vast majority of the code, are having more and more difficulty finding the motivation to work on any EVE Online projects, let alone projects that involve such a depressing facet of EVE, and indeed force us to try and interact with it. Certainly for now, I will be stepping aside as a developer of Nebula, and reducing the amount of time I spend on my other EVE projects. Aside from anything else, as fun as it used to be, EVE apps don’t put food on the table, and wherever my career may go, developing the apps I’ve built further may not be the smartest move – there’s other projects, other opportunities. It’s just a shame that at the moment, EVE doesn’t seem to be working out for myself and other third party devs.

## EVE Scalability Explained

OK, I’ve seen a bunch of posts on the EVE blogosphere about this recently and it’s always been a tricky topic to understand. This post aims to demystify EVE’s architecture and explain in simple terms what EVE’s current issues with scaling for fleet fights are, and approaches for fixing them. So first a disclaimer: I do not work for CCP, I don’t get behind the scenes information. This is a post compiled from several years working on EVE third party development and talking to people who do work at CCP, people who have worked at CCP, and the community at large. To the best of my knowledge this is mostly correct, but I make no promises. If you’re looking for an exact technical description, look elsewhere.

So, let’s start with the basics. This is the (somewhat simplified) hardware layout for Tranquility (click to enlarge).

To sum up in words: There are proxy servers that receive your data and route you to the appropriate sol server, which is running on a sol node or reinforced sol node. These servers communicate with a single, shared database server, which is also used for web services like the API and the MyEVE website (and, soon, Spacebook).

There’s an important distinction to be made here and one that is vital to understanding EVE’s architecture- nodes and servers are not the same thing. Nodes refer to the actual physical hardware (at time of writing, IBM Blade servers) that may run one or more sol servers. Each sol server is, as the name implies (Sol is the name for our sun) responsible for one solar system in EVE. It is a software server process, handling everything that goes on in a system- combat, mining, market, and so on.

EVE’s scalability issues stem from this design, but let’s look at what those issues are. Can EVE handle 56,000 players? Yep, easily. Tranquility will be able to handle many more than that without issue, and because of this design the capacity can be easily expanded by increasing the number of sol nodes for sol servers to run on, spreading the load efficiently and easily. Will you be able to fit 3000 people onto a gate? Nope. Why? Well, because EVE was designed so that the capacity of the whole cluster expanded well, not individual systems. This was a design decision made back in the early days of EVE and it has served EVE well, with the exception of fleet combat and Jita. So how to handle the edge cases?

Well, where does lag come from? Proxy servers have an easy job and they are not a bottleneck in the vast majority of circumstances. The main issues they cause are disconnects; when a proxy server fails, a good chunk of EVE’s inhabitants disappear till they reconnect. The lag is in combat and in high concurrency systems- like Jita, where loads of people trade, talk in local, and fly around suicide ganking each other. This lag stems from intensive processes that have to be done; mathematical steps like calculating transversal velocities between objects, things that have complexity values (algorithmically speaking) of $O(n^2)$ or worse. If you didn’t understand that- well, it just means the more ships you have, the more difficult things get, exponentially.

Obviously, there are optimisations that can be done, better algorithms, and CCP uses them, but the fact remains; this is a lot of work for a computer. Loads. Absolutely shedloads. And that’s all this challenge gets- one computer, at most. In bad cases, it won’t even get that-most sol nodes run multiple servers, the reason why lag sometimes seems to cross between systems- it really can, and does. Reinforced nodes just have more firepower and a guarantee of exclusivity, but they’re still only one computer. And as Google has taught industry, lots of small computers are cheaper, easier to fix, and faster than a single box computer.

True scalability will come to EVE when a sol server can be distributed seamlessly (without rebooting or dropping clients) and near-instantly across multiple sol nodes. That will mean that fleet fights can take all the resources they need, will mean that CCP gets to maintain cheaper hardware, making scaling the hardware cheaper and easier. And you maintain the scalability of the cluster, assuming you keep some hardware spare for sol nodes to grow onto in the event of a fight.

What needs to be done to achieve this? Why haven’t we got this yet? Well, it’s a heck of a lot of work. It’s a huge technical challenge, leaving internet spaceships out of it. Then there’s the hardware prerequisites; you need insanely fast low-latency networking (Infiniband, Fibre Channel, etc), and the extra nodes. It’s a huge investment for CCP, but one they’ll have to make eventually unless they find another way of solving the problem; but any other solution is likely to break immersion and cohesion in the game (grid sharding, etc), and so unlikely.

I hope that helps explain some of the thinking behind EVE’s architecture and why you lost that titan last night. And why it’s likely you’ll lose a few more before it’s fixed.

Minor second disclaimer: It’s 2:30 AM and I’m tired as hell, so this may contain errors. Feel free to point any out in the comments.