Recently, CCP decided to ditch their old forums (ASP, around 10 years old with little to no updates in that time) and started work on a new forum system. They’d just launched EVE Gate, which is a fairly minimal but working social network for EVE, and allows you to view some in-game features out of game. Many people decried EVE Gate, saying CCP should have just focused on building a better API first- then making their own CCP-owned apps on top of the API. Still, EVE Gate was done and launched to a minor fanfare. Some people use it a lot (in-game email is a handy thing to have at work), but many people ignore it outright.

What was shouted about most on the forums was the investment of time and effort from CCP to produce a very minimal site, features-wise. EVE has a stellar (no pun intended) community, with some very bright minds in it, who have built some fantastic apps on top of a fairly limiting API, and done so in their own time as a hobby project. Sites like EVE Metrics, EVE Commander, and such were all cited as examples of this- people were whipping up massively more complex and powerful sites than EVE Gate in their spare time in a matter of months, so what took CCP so long?

Well, the same people who developed EVE Gate (to my knowledge) were applied again to a web based problem; new forums. Now, a logical thing to do would have been to build on EVE Gate, and to add forums directly to that. This means you’d have only one core framework of software to debug and worry about, your authentication (a huge consideration for games like EVE) is all in one place, and in terms of validation and security testing there’s less to go wrong and fewer places to look when the shit hits the fan.

But CCP’s Web Cell did not. They sank a colossal amount of time into the project; the number being thrown around is 72,000 man-hours, which sounds about right to me given when they started and how many people are working on it. 72,000 man-hours is huge. Months and months and months and months. Forums are not complicated beasts; they’re essentially a simple relational DB application. You have to apply some thought about formatting (implementing bbcode or something like Markdown/Textile), accounts, profiles, all that sort of thing, but everything boils down to very simple structures, and simple code. There’s nothing complex, really.

CCP has one extra thing to think about atop of most forums: Integration of EVE accounts and characters. You have to be able to log in, and select a character to post as. But with EVE Gate, they had all that already done and dusted. And, crucially, fairly well tested- exposed to the world for a good long while. But again, this is all quite simple stuff.

But the web cell still managed to botch the job. And not just a half-arsed botch. Oh no.

They didn’t decide to integrate with EVE Gate. They completely ignored it, in fact. They took an existing ASP.NET forum software package, Yet Another Forum, and then skinned it. They just made a CCP skin for it. But that’s not all they did- they also gutted parts of it to tie in the authentication system, filling massive chunks with serious security flaws.

At approximately 21:00 UTC on Friday, April 8 we were made aware of some security issues with the new EVE forums which needed to be addressed. These issues were as follows:

• We discovered that it was possible to access some forums which certain users should not have been able to access
• Users could make and edit posts as another user’s character
• It was possible to inject some HTML code into signatures

At this stage the competent web developers (and software developers in general) are staring blank-faced in incomprehensible confusion. How could a 600-man industry-leading corporation like CCP let this sort of thing slip?

Authentication was done by cookies. That means there was a client-side cookie which stored your character ID. Change that, and you could appear to be logged in as anyone. And act like it, too- this wasn’t just a display bug. Change yourself to the CEO of CCP’s character, and you could see all the private internal forums. And the admin panel. And you could ban people! Handy.

Editing posts was unauthenticated. You could be logged in as you, edit your post but then change the post ID in the edit URL to the post you wanted to edit. And hey, no questions asked – go right ahead!

And finally, at least one HTML injection flaw. Why? Because HTML is used for formatting.

Now, these are all mind-blowingly simple ‘My First Website’ cock-ups. Any competent developer who had knowledge of programming web applications would never have made any of these mistakes. So why are there no competent developers in the web cell?

Here’s a clue. CCP Games hf is based in Iceland. Chances are, you’re relocating to take a job, and that relocation is putting you in a country far, far away from wherever you used to live. That’s a massive downer for potential employees. Next, the other part of the puzzle: I currently do freelancing. With a few clients on the go at once, doing roughly 20 hours of work a week during holidays as a student, per year I earn well in excess of what full time employees of CCP’s web cell get paid. I’m not even out of university yet. There’s one other thing to consider, especially now: CCP is not the sort of company that looks good on your CV any more. There’s some incredibly smart people working at CCP on some incredible stuff. But their reputation is tarnished, almost beyond repair, by this sort of fuck-up. Having CCP on your CV is something you want to think twice about. And working somewhere that triggers that sort of thinking is not looking great, is it?

And last but not least, have a look at the CCP Jobs website. Check out the requirements for a web developer:

Required Experience/Background/Skills:

• B.Sc. in Computer Science or related field, or equivalent training and professional experience
• 2+ years of experience developing & implementing .NET based web solutions
• JQuery, JSON, JavaScript, ASP.NET, MVC, C#, Visual Studio 2010, HTML/DHTML, MS SQL Server 2008, T-SQL
• Strong communication skills – verbal and written
• Strong technical and analytical ability
• Ability to complete projects on a timely basis with an attention to detail with minimal supervision

Now, there’s a bunch of problems here. Firstly, their HR people clearly do not know what makes a good web developer. Listing DHTML is cause for concern – it’s a buzzword from the 90s, nothing more. Personally speaking I have great concern for people who have only been using Microsoft toolsets, particularly for websites, and I’ll go into this a bit more in a second. The other problem with the above list is that, BSc aside, if they were willing to overlook my lack of experience with things that didn’t exist back then, I’d have been qualified at age 15 or so. And then the last bit: “attention to detail with minimal supervision”. That’s very worrying.

It’s worrying because it implies that these underqualified, fresh-out-of-school/university undergraduates who have potentially never written a public-facing website are not going to have their work checked. That’s a clear, utterly obvious problem. That’s saying “You’re confident you’re good enough to not need a supervisor keeping track of your output” to someone who probably has a healthy ego on them but in reality could be utterly useless.

The all-Microsoft toolchain that CCP uses and requires people to know is all well and good, but security has never been in the forefront of Microsoft’s brain. Specifically, tools like Visual Studio and ASP.NET for web developers focus on making life easier. Reducing the amount of work you have to do and increasing the amount of bolting together existing bits and pieces. This hides the underlying reality somewhat, though, and can lead to people just not being aware of things like client-side cookie tampering or cross-site scripting. Malformed HTML just isn’t something most MS web devs really think about- not till they’ve had a few rounds of learning the hard way, at least.

This is just my personal experience, based on people I’ve worked with and projects I’ve had to work with in the past. If you hire people who did their first major websites using a text editor, you’re likely going to get people with more knowledge about why things work, and can better understand how people can attack their nice shiny working things. And that makes you a better programmer. It’s one of the reasons I prefer working with Django/Rails/Sinatra- none of these things hide code from you. They may let you generate code with helpers, but even that is merely convenience- you have to know what the helpers to do to be able to use them, at a code level. Dragging text boxes onto forms does not give you the same experience of interacting with the code directly, and abstracts a lot of important stuff away from you. The result? Worse code.

CCP was very proud at fanfest of the fact they’ve now got over 600 people in the organization. But what sort of people? And why so many? Sure, a goodly number of those people will be DUST and Incarna developers and artists and so on. And you’ve got admin people to go with that all. But CCP’s first website was done by far fewer people than are currently on the web cell. And say what you like about 10-year-old forums, they work, a point made loudly by many forum users. CCP could perhaps do better by hiring fewer people, but people with better real-world experience.

So, the real screw-up here was not a purely technical one; it’s much more about the people who are behind the tech. And CCP is going to find themselves in a bad place on that front in very short order, because nobody except fresh-faced undergraduates is stupid enough to want to work for CCP right now. And I hate myself a bit for saying that, but it’s true. CCP is becoming a liability to itself through its own actions. And like all bittervets, I only whine and complain and point out this sort of thing because I fundamentally love EVE. It remains the only MMORPG that transcends being an actual game and takes on a life of its own. But with CCP at the helm, the world is in perilous danger of being torn to shreds by the very entity that created it. Which would be a terrible shame.

However, as with anything these days, a single app isn’t sufficient no matter how good the app is. The expectation to end users is that everything should play together nicely and you should be able to get data from A into B with a minimum of fuss. And as a developer, you clearly want to enable this- it means your app has more happy users, and if your app is commercial that translates to more cash. If you make a really, really good API then you can end up with a veritable ecosystem around your app; other companies pouring money into development, all supporting your business. So APIs are good for business.

But more importantly APIs are good for consumers. If you’re listening on your DAB radio or (in our case soon, with any luck) an RDS equipped FM radio, you want some metadata. Who are the presenters you’re listening to? What’s this song? Listening online? Then you want the song title. This is all pretty basic stuff we’ve come to expect thanks to media players on computers. If you’re a big company like the BBC you just adjust your tools and systems to support the APIs you need, or specify them in your requirements to outside vendors and get them to add support. But what about the little guys?

P Squared have made a good step in the right direction as of their last major release of Myriad; they added a TCP/IP interface that lets you query Myriad for some basic variables. Through a fairly bodged-together little set of scripts we can get data out of Myriad like what song is playing now, and what’s coming up. But it’s still awkward.

So I whipped up a little Sinatra webapp that does the appropriate little dance to act as a gateway to that awkward TCP/IP gateway from the land of HTTP. Everything speaks HTTP, or can be prodded to speak HTTP quite easily. It’s far and away the best tool for interacting between applications, since it’s well understood and simple to understand. Now I can have a script poke Myriad and our website’s API, combine the two, and suddenly all our metadata can be updated: “There Will Be Cake with James Harrison: Around The World (Radio Edit) – Daft Punk”. And while there’s absolutely no need to tell people the name of that particular song, we’re still adding a lot of value by providing more interactivity to our listeners through updated content. And hell, it’s just more professional. Click through for the scripts.

A couple of months back we worked with E-ON to put an advert in the magazine as well as running a login screen advert, and this has now run. So for a few days anyone logging into EVE will have been greeted with this fine advert created by Zapatero at MMM Publishing.

Of course, this suddenly meant more people going to and using the site. Fortunately one of the primary concerns when myself and Makurid put EM2 and later EM3 together was scalability. Whereas EM1 would’ve fallen over and died, EM3 has soldiered on like a champ. The only intervention we’ve had to perform was to fix the API processor, which was hanging regularly and causing problems as a result. That’s fixed, and we’re now stable and responsive. So, of course, we now have some numbers! These are the statistics for the most recent 30 days as of this post.

• ~85,000 visits
• ~500,000 page views
• ~5 pages per visit
• ~29,000 unique visitors
• ~1,500 additional account registrations as a result of login screen advert (Above ~20 account registrations per day baseline)

In terms of the site’s dataset, we’re getting fairly huge.

• 6,000 EVE API keys, of which 4,000 are full keys
• 75,000 EVE API methods enabled
• 32,000 EVE API calls per hour (about 10 every second)
• 75,000 characters (includes characters noted in market and other data)
• 15,000,000 wallet journal entries
• 6,300,000 wallet transactions
• 200,000 EVE Mails
• 1,600,000 active market orders
• 6,400,000 EVE API loaded trades
• 52,000,000 Inferred trades
• 11,000,000 processed uploads
• 146,000,000,000 total skillpoints of loaded characters
• 4,100,000,000,000 total ISK of loaded characters

Which is… scary. But! We’ve grown hugely (a threefold increase in requests per second) and we’re still performing well. The site is not throwing errors, users are generally happy with their experience with the site, and we’re stable – no crashes, no fires breaking out, no climbing resource usage. Our caching and design philosophies have worked well and the extra effort invested earlier in development has really paid off. We’ve just handled a huge amount of growth with no effort at all. I’d estimate we’re good to around 12,000-15,000 users on our current hardware.

Of course, we’re now wondering what to do. I stopped playing EVE long ago aside from the odd spot of tinkering, but I’ve even let accounts lapse and stopped updating my skill queue as of a month or two ago. Makurid’s just started playing again recently. We’re not really heavily invested in EM in terms of motivation, other than making a cool webapp.

Capsuleer, as I’m sure many of you know, is an EVE iPhone app that recently shut down because it couldn’t be monetized and the time and money invested in it by the developers was simply unreasonable. A couple of weeks back I was looking at the realities of what EM costs to run, and what it costs in terms of time to maintain. And the logistics of, if needed, shutting the site down. This is still somewhat in my mind but the site will be sticking around for a little while yet. I’m very much hoping CCP will come to their senses in terms of how they choose to support third party developers (a free account subscription appears to be the greatest gift given by CCP, but that’s a far cry from the ~£130 per month it costs to run EVE Metrics – and that’s just our costs _now_, not including new hardware costs or upgrade costs. If we grow much more we’re going to be looking at ~£200/mo to run EM, possibly more). Advertising isn’t an option, and donations have failed every time we’ve tried to support ourselves with them. Timecode sale affiliations barely made a mark on my accounting sheets. We do enjoy writing sites and producing something big that people find useful, but there are certain realities to be faced – both myself and Makurid are students, with no full-time jobs. I’ve recently picked up some part time work which means I can now afford to buy bacon on a weekly basis again.

In the meantime we will continue to support the EVE community by maintaining EVE Metrics. We’re working on the codebase right now to upgrade it to the latest and greatest Rails release and Makurid’s beavering away at the API processor to split it up into a puller and a processor, enabling us to achieve much higher throughput on API calls to CCP’s (slow) servers. Now that we’ve breached 30,000 requests per hour, that 10 request per second figure is actually beyond our capacity right now because of our single-threaded processor/puller mechanism. This is actually a CCP limit- we just need to work around it by hitting them with more requests simultaneously, thus spreading our load over multiple servers on their end. I’ll also be making performance and usability improvements wherever I can and incorporating as many bug reports as possible into what we release. This will likely be a slowish process, but in the long run, it should be worth it. The question really is, will CCP make it worthwhile for us to be investing our time and energy into longer term planning?