Architecture for the future

After that EVE-centric post on scalability (thanks to HighScalability.com for linking in, hope it was an interesting read), I figured it was time to return to EVE Metrics and other sites- accVIEW and ISKsense.

In the next week we will be migrating to a new server. It’s in the same datacenter with the same host, is a slightly faster machine but has four times as much RAM (8GB) and an additional 10kRPM hard drive. As part of the migration to the new server we’ll be making some changes to the software architecture running the show.

The main difference is that we’re moving away from Passenger, also known as mod_rails. It has some advantages in low-memory conditions, but we’ve had more trouble than it’s worth, so we’ll be moving back to running application servers manually as daemons. For this we’ll be using the excellent Thin application server. For the sites running PHP on the server (this blog, for example), we’ll be using PHP FPM as we are currently; we’ve had no issues with that. Both of those will be sitting as reverse proxies behind nginx. Nginx has done very well as a web server and it’s very fast, as well as being easy to configure.

There is only one other major change; we’ll be sitting nginx itself behind Varnish, a high performance HTTP cache. This will let us more efficiently leverage HTTP caching in our applications and speed up requests dramatically. Right now we don’t use HTTP caching that much; we’d like to change this, particularly in EVE Metrics’ API so we can let Varnish handle a good portion of the thousands of API calls we get asking for the price of trit or what have you. All in all it’ll mean reduced load on the application cluster, which means we can keep that smaller and lighter, which in turn means more room for the database in memory.

That translates to better performance on the more complex components in the site, ie market pages, your account page, corporate pages, and that better performance means we can build more- we’re waiting for the new capacity before we add asset support, one of the things we’re really looking forward to adding, since it will let us add a whole new level of functionality by giving lots more information to processes like our inferred trade detector and our planned fulfilled orders listings. Plus we’ll be adding asset valuation tools, of course.

The architecture I’ve described above will basically be ‘it’ for now; we have more complication at the application and DB layer (We still use MySQL for a few legacy applications, so we have a tiny MySQL server running). The complication at the app layer mainly consists of things like background processing tools, and for EVE Metrics tasks that are actually executed on a VPS and the results uploaded back to the server (we now do all the major CSV dumps on Makurid’s VPS).

As the guy who ends up fixing all this when it goes wrong, simplicity is always my main priority, but the added complexity of Thin and Varnish should be well worth it in the long run.

EVE Mail and training make an appearance

At last, we get an EVE mail API! It’s a bit rubbish as APIs go – no message bodies yet- but it’s a great step in the right direction. And of course we’ve got it implemented and polished already over at EVE Metrics.

All you need to do is head over to EVE Metrics, log in (or sign up if you’ve not got an account yet), add your API key(s) if you haven’t already, and then enable the EVE mail API method. And voila- EVE mails, in your browser, updated as often as CCP lets us.

The icing on the cake is that we’ve also provided a feed for RSS readers for your EVE mails. Google Reader/iGoogle or any other ATOM-compatible reader (which is basically all of them) can now monitor your ingame EVE mails at the click of a button.

We’ve also gotten around to doing skill training- you can see what you’re training (queue support of course is included) on all your accounts.

The next logical step from here is notification support- get an email or SMS whenever your characters can train a new skill, whenever you get a new EVE mail, whenever one of your market orders is outbid or fullfilled. You name it, I’d love to see it notifiable. We’re still in the early days with that, but that’s where we’d like for that to end up.

We’ll be improving on these and implementing other APIs in the coming days- we want to get notifications loading for all you corporate types, and we’re looking forward to bringing more skill monitoring/information into the UI. I’ve got a lot of ideas bubbling around- we’re getting to the point where we’ve got loads of little snippets of data that can all tie in with each other, creating something really fantastic for you guys and girls, the users. And that’s awesome.

Of course, we need your help to make all this run smoothly and perform well, which it has problems doing at the moment. We’re still asking for donations here, you can buy GTCs in support of us here, and we’ve just opened up advertising on the site through Project Wonderful. Any form of help is hugely appreciated.

Blog Banter 12 – Mobile EVE

Welcome to the twelfth installment of the EVE Blog Banter, the monthly EVE Online blogging extravaganza created by CrazyKinux. The EVE Blog Banter involves an enthusiastic group of gaming bloggers, a common topic within the realm of EVE Online, and a week to post articles pertaining to the said topic. The resulting articles can either be short or quite extensive, either funny or dead serious, but are always a great fun to read! Any questions about the EVE Blog Banter should be directed here. Check out other EVE Blog Banter articles at the bottom of this post!

This month’s banter comes to us from CrazyKinux himself, who asks the following: First there was the MMO on the PC, and now with the recent announcement of DUST 514, EVE will soon be moving onto consoles. But what about mobile? Allow your imagination to run wild for a second and describe how you would see EVE being ported to mobile devices, whether the iPhone/iPod touch, Blackberrys or Android-based devices. Dream the impossible for us!

No.

What do I mean, ‘no’? What I mean is, I would actively dislike it if CCP decided their next project was iEVE. Why? Well, it’s quite simple, really. It’s immersion. It’s partly why CCP won’t let people write UI addons in the style of WoW. EVE is very distinctive, and extremely immersive. It’s part of why it’s such a brilliant game. You just simply can’t replicate that immersion on a mobile platform, no matter the technology involved. I guess what I’m saying is that EVE shouldn’t be played on a mobile device.

If we did get anything for mobile platforms, then obviously EVEmail and other information would be what you’d want. Some people would argue skill training should be accessible from outside the game. I’d have to disagree with that one, again on the grounds of immersion. You’re doing something very much tied to the game there, and it should remain in the game.

Of course, the optimal way to do all this is just to open up some APIs and let people get to work on it. Look at the current offering of mobile apps- there’s loads out there and some (Capsuleer) are of an exceptionally high standard. Why should CCP try and replicate all this effort in-house when they could just open the appropriate APIs to developers? If CCP wanted to really fuel development of certain tools, they could take on the popular third-party devs for given platforms as contractors and task them with developing the features full-time, thus leveraging existing apps and talent while getting exactly what they want. It’d be good for the 3rd-party developer ecosystem and better for CCP, who get what they want at vastly reduced effort, cost and so on.