The real problems behind CCP’s botched forum launch

Okay, so this blog, back in the days of old, used to talk about EVE a lot. Now, I don’t play EVE – I haven’t for well over a year. I stopped playing because I lost all faith in the company that runs it, CCP Games hf. At one point I was choosing an education and career path that would set me up nicely to apply for a job at CCP, which should give you an idea of how big a change it was from huge advocate of EVE to what’s known in the community as a bittervet – a bitter veteran. That a term exists for this type of player says a lot.

Recently, CCP decided to ditch their old forums (ASP, around 10 years old with little to no updates in that time) and started work on a new forum system. They’d just launched EVE Gate, which is a fairly minimal but working social network for EVE, and allows you to view some in-game features out of game. Many people decried EVE Gate, saying CCP should have just focused on building a better API first- then making their own CCP-owned apps on top of the API. Still, EVE Gate was done and launched to a minor fanfare. Some people use it a lot (in-game email is a handy thing to have at work), but many people ignore it outright.

What was shouted about most on the forums was the investment of time and effort from CCP to produce a very minimal site, features-wise. EVE has a stellar (no pun intended) community, with some very bright minds in it, who have built some fantastic apps on top of a fairly limiting API, and done so in their own time as a hobby project. Sites like EVE Metrics, EVE Commander, and such were all cited as examples of this- people were whipping up massively more complex and powerful sites than EVE Gate in their spare time in a matter of months, so what took CCP so long?

Well, the same people who developed EVE Gate (to my knowledge) were applied again to a web based problem; new forums. Now, a logical thing to do would have been to build on EVE Gate, and to add forums directly to that. This means you’d have only one core framework of software to debug and worry about, your authentication (a huge consideration for games like EVE) is all in one place, and in terms of validation and security testing there’s less to go wrong and fewer places to look when the shit hits the fan.

But CCP’s Web Cell did not. They sank a colossal amount of time into the project; the number being thrown around is 72,000 man-hours, which sounds about right to me given when they started and how many people are working on it. 72,000 man-hours is huge. Months and months and months and months. Forums are not complicated beasts; they’re essentially a simple relational DB application. You have to apply some thought about formatting (implementing bbcode or something like Markdown/Textile), accounts, profiles, all that sort of thing, but everything boils down to very simple structures, and simple code. There’s nothing complex, really.

CCP has one extra thing to think about atop of most forums: Integration of EVE accounts and characters. You have to be able to log in, and select a character to post as. But with EVE Gate, they had all that already done and dusted. And, crucially, fairly well tested- exposed to the world for a good long while. But again, this is all quite simple stuff.

But the web cell still managed to botch the job. And not just a half-arsed botch. Oh no.

They didn’t decide to integrate with EVE Gate. They completely ignored it, in fact. They took an existing ASP.NET forum software package, Yet Another Forum, and then skinned it. They just made a CCP skin for it. But that’s not all they did- they also gutted parts of it to tie in the authentication system, filling massive chunks with serious security flaws.

At approximately 21:00 UTC on Friday, April 8 we were made aware of some security issues with the new EVE forums which needed to be addressed. These issues were as follows:

  • We discovered that it was possible to access some forums which certain users should not have been able to access
  • Users could make and edit posts as another user’s character
  • It was possible to inject some HTML code into signatures

At this stage the competent web developers (and software developers in general) are staring blank-faced in incomprehensible confusion. How could a 600-man industry-leading corporation like CCP let this sort of thing slip?

Authentication was done by cookies. That means there was a client-side cookie which stored your character ID. Change that, and you could appear to be logged in as anyone. And act like it, too- this wasn’t just a display bug. Change yourself to the CEO of CCP’s character, and you could see all the private internal forums. And the admin panel. And you could ban people! Handy.

Editing posts was unauthenticated. You could be logged in as you, edit your post but then change the post ID in the edit URL to the post you wanted to edit. And hey, no questions asked – go right ahead!

And finally, at least one HTML injection flaw. Why? Because HTML is used for formatting.

Now, these are all mind-blowingly simple ‘My First Website’ cock-ups. Any competent developer who had knowledge of programming web applications would never have made any of these mistakes. So why are there no competent developers in the web cell?

Here’s a clue. CCP Games hf is based in Iceland. Chances are, you’re relocating to take a job, and that relocation is putting you in a country far, far away from wherever you used to live. That’s a massive downer for potential employees. Next, the other part of the puzzle: I currently do freelancing. With a few clients on the go at once, doing roughly 20 hours of work a week during holidays as a student, per year I earn well in excess of what full time employees of CCP’s web cell get paid. I’m not even out of university yet. There’s one other thing to consider, especially now: CCP is not the sort of company that looks good on your CV any more. There’s some incredibly smart people working at CCP on some incredible stuff. But their reputation is tarnished, almost beyond repair, by this sort of fuck-up. Having CCP on your CV is something you want to think twice about. And working somewhere that triggers that sort of thinking is not looking great, is it?

And last but not least, have a look at the CCP Jobs website. Check out the requirements for a web developer:

Required Experience/Background/Skills:

  • B.Sc. in Computer Science or related field, or equivalent training and professional experience
  • 2+ years of experience developing & implementing .NET based web solutions
  • JQuery, JSON, JavaScript, ASP.NET, MVC, C#, Visual Studio 2010, HTML/DHTML, MS SQL Server 2008, T-SQL
  • Strong communication skills – verbal and written
  • Strong technical and analytical ability
  • Ability to complete projects on a timely basis with an attention to detail with minimal supervision

Now, there’s a bunch of problems here. Firstly, their HR people clearly do not know what makes a good web developer. Listing DHTML is cause for concern – it’s a buzzword from the 90s, nothing more. Personally speaking I have great concern for people who have only been using Microsoft toolsets, particularly for websites, and I’ll go into this a bit more in a second. The other problem with the above list is that, BSc aside, if they were willing to overlook my lack of experience with things that didn’t exist back then, I’d have been qualified at age 15 or so. And then the last bit: “attention to detail with minimal supervision”. That’s very worrying.

It’s worrying because it implies that these underqualified, fresh-out-of-school/university undergraduates who have potentially never written a public-facing website are not going to have their work checked. That’s a clear, utterly obvious problem. That’s saying “You’re confident you’re good enough to not need a supervisor keeping track of your output” to someone who probably has a healthy ego on them but in reality could be utterly useless.

The all-Microsoft toolchain that CCP uses and requires people to know is all well and good, but security has never been in the forefront of Microsoft’s brain. Specifically, tools like Visual Studio and ASP.NET for web developers focus on making life easier. Reducing the amount of work you have to do and increasing the amount of bolting together existing bits and pieces. This hides the underlying reality somewhat, though, and can lead to people just not being aware of things like client-side cookie tampering or cross-site scripting. Malformed HTML just isn’t something most MS web devs really think about- not till they’ve had a few rounds of learning the hard way, at least.

This is just my personal experience, based on people I’ve worked with and projects I’ve had to work with in the past. If you hire people who did their first major websites using a text editor, you’re likely going to get people with more knowledge about why things work, and can better understand how people can attack their nice shiny working things. And that makes you a better programmer. It’s one of the reasons I prefer working with Django/Rails/Sinatra- none of these things hide code from you. They may let you generate code with helpers, but even that is merely convenience- you have to know what the helpers to do to be able to use them, at a code level. Dragging text boxes onto forms does not give you the same experience of interacting with the code directly, and abstracts a lot of important stuff away from you. The result? Worse code.

CCP was very proud at fanfest of the fact they’ve now got over 600 people in the organization. But what sort of people? And why so many? Sure, a goodly number of those people will be DUST and Incarna developers and artists and so on. And you’ve got admin people to go with that all. But CCP’s first website was done by far fewer people than are currently on the web cell. And say what you like about 10-year-old forums, they work, a point made loudly by many forum users. CCP could perhaps do better by hiring fewer people, but people with better real-world experience.

So, the real screw-up here was not a purely technical one; it’s much more about the people who are behind the tech. And CCP is going to find themselves in a bad place on that front in very short order, because nobody except fresh-faced undergraduates is stupid enough to want to work for CCP right now. And I hate myself a bit for saying that, but it’s true. CCP is becoming a liability to itself through its own actions. And like all bittervets, I only whine and complain and point out this sort of thing because I fundamentally love EVE. It remains the only MMORPG that transcends being an actual game and takes on a life of its own. But with CCP at the helm, the world is in perilous danger of being torn to shreds by the very entity that created it. Which would be a terrible shame.

6 thoughts on “The real problems behind CCP’s botched forum launch”

  1. Tough words, with a strong undercurrent of emotion. Understandable however. To many people, and it seems to you as well, CCP and EVE Online represent more than just a studio with a game. They represent something different, something more. CCP does not sell you a game. They sell you something more. Something which allows you to engage in so deeply, between pixels, that the we connect with both the concept as well as the people behind the pixels so deeply that it becomes real. We sometimes joke that CCP sells us a second job. But think of it, the more you love EVE, the more you buy the dream, the more real it becomes and the better CCP is able to sell you what really comes down to a life.

    When a dream meets reality, particularly easily avoided realities, that hits hard. Especially when this becomes commonplace, by CCP saying one thing, doing another, and delivering something entirely different.

    What bites people hard here, is exactly because of that factor of a dream bought into. Dreams are what people strive towards, what people share, and these are all things that people take care of. CCP unfortunately, and this is something most players figure out after a few months to one expansion cycle, only takes care of the presentation. The delivery, it has thusfar increasingly only taken care of what some people at CCP want for themselves on the basis of their own wants, wishes and hopes. And even then, it was delivered badly.

    You could say why play. Very simple. Because again of that dream, and of that life. Think of it, how hard is it to quit life. CCP does not deal in gameplay (no pun intended here), they deal in the currency of trust. When EVE began, people trusted CCP. Sure, trust in EVE (pun semi intended) is a story on its own, but think about it for a moment. Exactly because of the metagame and absence of trust mechanisms in game design we connect with others even harder and deeper. Which contributes immensely to both the factor of dream and life.

    Strong emotions are very easily understood when considering their origins. The very reason why CCP stood out and why EVE as a virtual world gained its momentum, is the same reason why people share the dream.

    I do think you are making some mistakes in your analysis, but those are mistakes from an outsiders perspective and thus only natural. Still, at the heart of it you are touching on the very same symptoms, conflicts and desillusions described by CCP current and former employees on and other corporation review sites. But also increasingly so among their peers in industry, sofar still during the walk to the coffee machine, but it is a worrying trend in its own right.

    CCP aims to be big, for the sake of being big, and seen as big. Perfectly understandable, and even a common practice in business as long as this is something rooted in organisation as well as culture. And this is simply not the case at CCP, has not been for some years now. The stories from players speaking with staff at the recent fanfest highlighting awareness yet total apathy among staff over circumstances and challenges faced are one indicator. But the best indicator is CCP’s own delivering on promises.

    And I’m not talking about the syndrome of players taking a word muttered by devs as the holy bible, to run off with it and interpret and spin tales of promises with. No. I’m referring to something very simple. CCP as a company, always aims to the the right thing for the company. In practice, there is however more than one CCP. They are not united. Where great parts of the company work hard and strive to be innovative and leading, each time the company sets out on a course there is the same part of the company which shoots the rest in the foot, and refuses to consider that even though it matters not that much for a company without competition, it does matter a lot for the cost over time of replacing core subscriber groups with new market reach. Just as it matters for the perception of the company by outsiders.

    To a company like EA, that would matter little. To a company CCP once set out to be, and still claims to be, there is a choice to make there.

    In many ways CCP is not just at the point of making choices for them as a venture. But also at a point of how to deal with people, theirs, and those they attract. And those they serve, in more than just one way. I do not envy them.

    There are several distinct sides of CCP, each with their own push and pull, and as we saw in the publications in media over the past few months there is a part of CCP which sees the challenges on many levels. But there is also a part of CCP which simply does not give a shit about anyone else at CCP but their own wishes and wants. We can think of examples like Planetary Interaction, the technology projects of Walking in Stations prior to Incarna, or how commercial prospects like customer created content are discarded simply on the basis of opinion of one man in Art driving his hammer through the table. But we best shouldn’t. Let us hope, that the side which really and honestly wants to grow up and be different and on top without floating on a cloud, manages to get through to the that side of CCP which makes the decisions. Because yes, that has grown to be a part of CCP which talks the talk, but doesn’t do the walk, and thus has become its own seperate part of the company.

    Can they do it? I think they can. Some really do want to. And there are so many good and smart people there who want to do more than play Facebook and chase the latest grand idea of the inner sanctum before commercial reality once again kicks things back to the workfloor. But they will have to shed the dead weight of apathy and inner sanctum and learn a few things of precedents they are repeating of the business model they follow. Before they crash into those walls.

    1. I entirely agree with you – and to be clear, I think CCP as a whole has got a huge amount of great talent in there producing some awesome stuff. But there is this problem of the company shooting itself in the foot, and that’s only going to keep happening if they don’t make some tough choices and some hard decisions, and fast. I really, really want them to, because a bit of me still thinks that working at CCP would be fantastic, because EVE has so much potential. It’s one of the most exciting projects of the decade, in my eyes. Like I said: I whine because I care.

      I do think you are making some mistakes in your analysis, but those are mistakes from an outsiders perspective and thus only natural.

      Absolutely – this is written by someone who has only seen things from the outside, and who has never been to a fanfest or any other event. My closest experiences with CCP have been working with EVETV back when that existed, but I was only a lowly audio engineering volunteer and CCP did not directly handle EVETV, just approved scripts.

      If you’d like to elaborate on specific mistakes you think I’ve made I’d be happy to explain my reasoning further.

  2. For starters, you are presuming connections which may or may not be there. CCP’s methods are – to say the least – not commonplace. Nor are their operational practices. I do think there’s a good chance you could be spot on, but I do not think that players can ever fully know. Then again, it has always been surprising how players figure out more than what most employees know about their company. Even without drunk staff stripping after a conference.

    Also I do not think one could conclude that there is such a seperation between EVE Gate and the Forums on a project level. There might be on a technical level, but it strikes me that you might very well see this as something which because it is technically disconnected it will also be disconnected on that level. This I doubt to be the case, for that CCP – in spite of how they always are proud publicly of shuffling people around – is not tuned into. I do realise that does not help the situation at all. On the contrary. It makes it worse. But it does have different consequences. For example, I could make a reasonably safe statement that nobody in spite of it all will get fired. And that the causes of what resulted in circumstances leading to these issues, will not float upstairs within the company circles.

    I also don’t think you can make a connection between CCP’s HR and other parts of it. For that there has been too much presentation by CCP on that topic. Not to customers mind you. At CCP HR is a passthrough function. Nothing else. To presume there would be a deeper interaction for addressing company requirements in HR is a very likely mistake.

  3. While I agree with most points, esp. the HR part, I have to disagree with your statement about using microsoft technology and your security concerns.
    There are some pretty decent security mechanisms built into .NET/ASP.NET, CAS is a powerful way of restricting access within your application.
    Plus, a company with the size of CCP *hopefully* has ALM-Tools built into their CI/Release-infrastructure. They should have a solid workflow of designing, developing, testing and releasing, supported by their infrastructure.

    On the other hand: although all these patterns and tools are proven, it’s the human who usually fails in applying them properly.

    1. Marc: Sure, and I’m well aware that in order to have screwed up this badly, the devs at CCP actually must have bypassed manually some of the automatic filters that exist to protect against this sort of thing.

      My point is more that people who _only_ have experience with the MS toolkit are, in my experience, far more likely to do such a thing without realizing the implications (or design a login system that trusts the client – all these little lower-level distinctions that the MS stuff hides from you a little bit to make your life easier can be mighty dangerous to forget) than someone who has been brought up on other languages, or a variety. I’d rather hire someone who’s worked with PHP, Python _and_ ASP.NET than someone who has worked with ASP.NET for three times as long, even if I know that they’re going to be writing nothing but ASP.NET.

      You would really hope that CCP would have some form of formal ALM process and that security would be one thing on the list of pre-release checks for anything they publish, especially web applications (where the potential for abuse and attacks is high). Unfortunately, there’s no evidence of this, and without CCP telling us we can’t know one way or the other.

  4. Weird thing is, is that EveGate was created using mvc. Which like Rails Django does not hide any code and has plenty of security built in. Had they build upone EveGate using MVC I doubt they would have made these silly mistakes.

    It also forces you to understand a little better what exactly you’re doing. It’s almost like the Evegate people left (which wasn’t all that great to begin with) and were replaced by people who dont have a clue.

    And I do agree your comment that good people won’t be moving to Iceland wfor a not so freat salary. They might have a problem there.

Comments are closed.