So here’s a post about EVE Metrics 3.

We’re getting close to having everything polished and ready for release. The main issue at hand thus far has been the homepage; I say this like it’s a small thing but it’s required me to learn some new and interesting things about CSS, Makurid’s done some excellent work to produce some feed scrapers and elements for the lower portion of the page… there’s a lot to it.

I thought that it’d be good to list a few of the changes we’ve made for version 3.

• Complete redesign of the site thanks to Rettic
• Market detail pages have been entirely renovated
• Various pages which haven’t been improved in some time have now been tidied up
• API key management has been improved
• API key permissions management has been improved
• Backend processors for API functions and upload processing have been improved and made more reliable
• My Metrics has been entirely renovated, now with sparklines for wallets and a new layout
• Orders and transactions have been moved from My Metrics into their own detail pages, with a summary on My Metrics
• Journal information has been added and given it’s own detail page
• Player Owned Structure integration has been added, though still in it’s infancy
• Sensitive portions of the site now make use of SSL transport encryption (HTTPS) automatically
• Wormhole pages have been updated
• Improvements to the corporation pages through refactoring to share view code between character and corporation pages
• Graph improvements
• Complete test coverage of every line of code (Nah, just kidding, we’re still pretty thin on those test things for large chunks of UI code)
• 0.2% more cowbell
• 5% other features I’ve not listed above, plus 100% more polish overall

Excited? We are! There’s a lot of work in the lines above and I think you’ll like the results. We’re not sticking to any firm release schedule because we’re terrible at sticking to them; we’re students, not full-time developers (incidentally, if anyone’s got any jobs available for temporary/contract work, SE UK preferred (or work-at-home), 6 weeks max, let me know!). That said, we hope to have a release before the end of July.

We’ve also been rewriting our uploader! That’s right- Linux, Mac and Windows support all in one neat package. The GUI’s not anything special but it works, and we’ll be polishing it and getting it release-ready before long. Huge thanks should be directed to TTimo, who has been the driving force behind this with some welcome Python experience, and Makurid for assisting him in developing the new client.

Once we’ve gotten that polished, packaged and rolled out, we’ll be running a 5 billion ISK contest to promote it- the three winners (each receiving a portion of the 5 billion pool) will be selected from the most active uploaders for the week or two after the competition is announced. We’ll finalize all the details and have it posted up when we’re ready to go ahead with that, of course. If you’d like to make that 5 billion figure larger, you can contribute ISK to the character MMMetrics Agent ingame! So far, thanks go out to Rilcon, Chribba and Entity for contributing to the current pool. The new uploader will be released after the new site – we’d like to change one thing at a time so we can iron out all the kinks. Once we’ve gotten it out and tested it thoroughly we’ll roll out the upgrade- your existing uploader will prompt you to update when you start it.

One last thing – I’ve personally submitted a proposal to the CSM regarding development efforts from CCP surrounding the API and EVE Gate. If you’ve not done so already, I strongly urge you to read my proposal and support it if you feel, like I do, that CCP have made some serious mistakes lately in this regard. The thread can be found here. It has been picked up and supported by at least 3 CSM members so far, but the community support will help considerably to drive CCP to consider it seriously.

This post is a tale of how we’ve been running into issues with memory usage recently, how we’ve been solving and diagnosing it, and the design decisions that have lead to it.

In the past week or so we’ve been puzzling over some unusually high memory usage of our application servers (we use the Thin server for EVE Metrics and Passenger for everything else). Specifically, our thins were getting fat; chewing up memory, not responding to new requests and churning on locks.

On a hunch and some history, we pulled out RMagick, the Ruby interface to ImageMagick, which had been known to cause issues with memory usage by way of some leaks. We switched the code that we had to use ImageMagick for to use the mini_magick library, and where we could (notably the sparkline graphs on the statistics panel) we switched from server-side to client-side rendering. Those sparklines are now using the excellent jQuery Sparklines plugin. They look better, too!

That did solve our memory issues; we still had problems with sudden spikes in memory, and we pinned down the cause using the Oink diagnostics plugin. Someone (and if you’re reading this, let me know who you are, I’d like to talk, not to bite your head off, but to ask you why the CSVs didn’t leap out at you quite as they should have…) decided to make lots of history requests to the API, for all 59 regions at a time and for 25 items at a time. Admittedly only for one day, but there’s a kicker here.

Those 25 items across 59 regions is not a lot of historical data. But, those of you with EVE Metrics API experience should be shouting, “James, the history API also includes the item API data!” Which it does! This entails loading the market orders. Now, there is a problem here. Let’s say you request 25 items for 59 regions. That is a _lot_ of market orders, potentially. Remember we have 1.5 million orders over the whole market and the really active market is spread across a few thousand items. But there’s a few items with vast numbers of orders. That number of orders can rise quite easily into the tens or hundreds of thousands.

We have to load all those orders into memory to work with them, then allocate the raw data into Gnu Scientific Library vectors, then process that data (fast, thanks to C), and then render out the XML response. Because we have to have all these orders in RAM at once, Ruby’s VM allocates the process more and more memory, which is removed from the OS’s spare resources which are used for caching the filesystem in memory. This cache is the primary mechanism for PostgreSQL, our database engine, to keep data in RAM; subtract from this and you’re hurting DB performance and increasing disk IO. Everything slows down. Eventually, Ruby allocates enough that the OS itself is out of cache and has to start swapping to disk; we’ve not yet run into this since I’ve been monitoring this all closely with Munin, but it would eventually happen. Ruby (along with most other VM-based languages) has a pretty crap GC, and even once the request is over, this memory is not returned to the OS from the VM. This means that things can escalate pretty dramatically if left unchecked.

So, what can we do to fix this? Well, in time I want to remove the item API data from the history API. That’ll improve things greatly in our own application. I’ve also implemented a limit on the number of items and regions that can be checked at once to stop people generating huge memory spikes.

There’s a nasty side-effect here: While these thins are churning away, nomming up memory like there’s no tomorrow, they’re locked up and handling a long running request. If we get enough requests this means the website becomes unavailable, something I personally find intolerable. So I’ve increased the number of app servers we run and split the requests up as they come in between two pools, one of which has a few servers dedicated to handling web site requests, and likewise one of which has a few dedicated to handling API requests, with a few servers shared between the pools. This will help stability in the long run and ensure that at any moment, requests can be fulfilled in a timely manner. Varnish is also getting some tweaks to serve up old content when the backends are busy where a cached object exists.

Implementing was actually really easy. I grabbed the RiCal gem off Github, threw in the config.gem line in environment.rb, and added this to the characters controller index action:

So as you can see, it’s a snap. Rails ships with a preconfigured MIME type for the :ics format, so it’s handled properly by nginx automatically. Still having some issues making Google Calendar accept the feed’s name as Charactr, mind you, but the rest works flawlessly and is available for all Charactr users right now.

On a side note, we’ve set up the @mmmetrics Twitter account as a shared account between all the MMMetrics team; if you’re having problems with any of the sites and use Twitter that’s a pretty good port of call. We’ll also be using that for announcements and so on, so it might make sense to follow if you’re interested in what we do.

It’s been a busy day on EVE Metrics 2, so I leave you with a few teaser screenshots. We’re almost done with the basic market view pages, and it’s now a matter of implementing the smaller features- market favourites, map features, and the APIs. EM2 won’t be an instant fullfilment of every feature promised over the past year or so; we’re taking the development slowly. EM2 at release won’t be as feature-packed as EM1, but it’ll work a whole lot better! Once we’ve gotten it released (hopefully in just under a week) we’ll be adding features and maintaining the site continuously to get all the features you want implemented without affecting performance.

Charactr has been doing quite well. We originally released it deployed on Apache with Passenger (also known as mod_rails). While the performance was tolerable it wasn’t the snappiest thing in the world, and on a memory-limited VPS even with Ruby Enterprise Edition we were often running into memory limits. Right now however we’re on nginx and thin, the first production environment I’ve used this combination for.

Performance-wise, I’m impressed. Not only is the static asset hosting snappy as anything, but thin handles remarkably well and does it without chewing up much RAM. I’m pretty sure Passenger/REE would win on a box with a few gigs of RAM, but on something this small (540 megs, including the db server on the same box) the extra overhead from Passenger’s spawner was too much.

Configuration is where nginx really wins out, however…

There’s nothing wrong per-se with Apache’s configuration, but nginx makes life a whole lot easier for hosting Rails apps. First off, many thanks to Igor Sysoev for some recommendations in this config and help debugging an issue with 502 catching (So I can display a pretty page to the users while the cluster restarts; ideally we’d avoid this entirely but I’m not sure how I’d go about doing it on a budget).

See? That’s all of Charactr’s configs. Of course, there’s still the server config but that’s pretty simple, and the default will do just fine for that.

So, overall I’m impressed with nginx. Thin I’ve used before and will be doing so with EVE Metrics 2; the combination of excellent stability with an Evented architecture hasn’t failed me yet.

In EVE Metrics 2.0 I’ve been looking to replace the previous delayed_job system with a more robust and fault-tolerant system. A while ago I attended the Ruby Manor conference, and saw an excellent talk on Nanite by George Palmer. It’s a very neat way of doing background processing- it depends on a mapper, agents and actors. Agents have many actors and announce their presence to the mapper regularly along with (by default) system load info, and the mapper then sends jobs to the least loaded agent which supports a given actor. It also works on AMPQ, and by default the RabbitMQ server, a very high performance queue server written in Erlang. It’s fast enough that you can stream video over it quite well, apparently, and it scales well with large queues, so should fit the bill nicely for EVE Metrics.

Getting started, however, can be kinda tricky. So here’s the simple ‘getting started’ that is sadly absent from the Nanite documentation.

First, we need RabbitMQ. So, install Erlang and RabbitMQ- if you’re on Debian/Ubuntu then Erlang is in the default repositories (apt-get install erlang), and while RabbitMQ is also there I recommend grabbing the most recent version in .deb format from their site. Windows users can find a handy bundle with Erlang and RabbitMQ and a nice readme, so go ahead and sort that out. I’ll wait here.

Now that’s all done, it’s a simple matter of installing Nanite and it’s dependencies. Best way to do this is to clone nanite (git clone git://github.com/ezmobius/nanite.git), cd in there and run ‘rake gem’. Then install the gem it creates with ‘gem install pkg/*.gem’. Dependencies (AMPQ and EventMachine) will get autoinstalled. Note that simply running ‘gem install ezmobius-nanite’ fails to install correctly and does not create documentation.

Not much further, I promise!

Now that we’ve got it all installed we need to tie it into Rails. While googling for this, I tripped over the highly useful and recently created nanite-rails plugin. While it’s not perfect it’s a whole lot better than nothing and takes care of some of the nastier bits of Nanite’s workings. Install it as a plugin with script/plugin, and then generate nanite (‘ruby script/generate nanite’). Voila! Now there’s only one last thing to do, and that’s to add the following line to config/environment.rb:

config.gem "nanite"

And we’re done! There’s a rake task to automagically configure RabbitMQ, and so on, but you should be set up with a nanite initializer to kick off the mapper as well as a file structure and agent generator (nanite in RAILS_ROOT). And from there, it’s back to learning. I’m still getting to grips with the actor/agent side of things and working out how to set that all up. I’ll update this post if I find some more useful info, but this lot was fairly scarcely spread over the ‘net so I figure it’ll be some use to people at any rate. If anyone has any guides on how to work with Nanite that I’m hugely overlooking then let me know, but it’s a shame as far as I can see that Nanite has such sparse documentation on actual real-world usage. Ho hum. Still looks like it could work really well if I can get it working.

What I hadn’t really considered was the market. Read on to read a short bit of background and find out where the site’s heading.

EVE Metrics originally started out as a statistics aggregator for everything. I figured I could pull in killboard data, market data, map data, everything the API offered, and roll it into a huge database and make pretty statistics. After sitting down and thinking I thought that tackling one bit at a time would be a good plan- so I started with the market. It seemed logical; the largest bit of the project, in theory, and interconnected with all other elements of EVE at some level.

The market is huge. And it’s really interesting, if you’re into large-scale datasets, statistics and computational economics. After about 6 months of dabbling in the various APIs and getting Salvis working with me on the uploader, I started to harbour increasing envy towards Dr. Eyjólfur Guðmundsson – CCP’s resident EVE economist. Having a job working on the market has gotta be awesome, honestly.

And then with the project in a ‘hey, stuff sorta works!’ state, Hexxx from EBANK went ahead and plugged it. We recently hit the 1,000 user mark, which I’m hugely proud of. But there’s a catch, of course…

Right now the database holds:

• 1,936,686 unique market orders
• 11,082,158 historic price indexes
• 10,330,669 historic movement records
• 606,091 EVE Metrics API call records
• 117,914 bits of uploaded data still waiting to be imported
• 2,069,769 historic system jump histories
• 1,968,425 historic system kill histories
• 165,798 non-market price reports

It’s a lot of data. But the way in which it’s organised and structured, and most importantly the processes involved in getting the data in and analysing it, are not designed for this kind of volume. I figured I’d have a few thousand orders in this, but the cache uploader changed that. We got some people in Jita and other trade hubs interested and they’re just churning out uploads- over half a million distinct uploads so far- which results in a lot of data.

Right now, EVE Metrics grabs the upload when it hits our API, stuffs it in a job queue, and forgets about it. Some behind-the-scenes workers perform an upload filtering algoritm which marks any outliers in the uploaded data, mark old orders in the database as expired, and then add any new orders and update existing ones. Then the raw data is thrown back in the queue to be sent out to users of the Webhook service, and then discarded. When you go to a page, then, we get to do the following:

• Retrieve all the market orders for that item that aren’t expired or outliers for the regions the user has set as favourites or default market regions
• Split the dataset up into Jita/non-Jita and buy/sell arrays
• Crunch numbers to generate averages for those arrays
• Render all the market orders on the webpage, including loading in all the appropriate system/station/region data for each order

The first three steps take a little while. On the larger orders the mere step of retrieval from the database can take multiple seconds. And that’s no good. We want millisecond response times. EVE Metrics should be snappy, responsive. Upload processing can take a whole 20 seconds for a Jita upload of tritanium- the backlog of nearly 120,000 jobs should speak worlds about how ungood that is.

And all this leads to unhappy servers, which lead to broken websites and unhappy users, and that’s no good.

So, I’m hitting the reset button.

The data will remain, your user accounts will be ported over, and I will endeavour to keep legacy APIs supported and old links redirecting to their new homes. But the codebase and website will be entirely new. The frontend will be getting a new skin which is minimalist and flexible and has proven to be very popular with users of accVIEW in the past month.

I’ll be writing everything for speed, performance, and general scalability. I want this to perform well under all circumstances and have a delay from viewing an item in EVE to having the new data on the site of under a minute. And I’m going to be writing tests for everything to ensure behaviour of the site is predictable and doesn’t randomly break.

On the downside, EVE Metrics will be hobbling along unmaintained for a month or two while I write EM2. It’s managing fine now, mind you, but in half a year’s time it may not be working quite as well…

I’ll be posting up a survey for EVE Metrics users to let them decide what they’d like to see in 2.0. API integration is an absolute certainty, but things like a JS-sidebar-style-market-browser and competition filters are also on the table. I’d rather hear your opinions now than after I’ve gotten halfway through doing something that would make your idea really tricky to implement

Problem: It didn’t power up. No lights. Nothing. Nada. OK, I figured that was a power supply issue. Stripped it down to it’s component parts (left the optical engine and ballast), spotted the bad connection- this projector had been damaged by being dropped, and the whole frame had warped a bit. This connection, between the IEC connector on the side for power and the horizontal power supply board, had fallen out entirely. Reconnected and resoldered, and subsequently reassembled- and I get a ‘circuit error’ when trying to power up.

So, back to the disassembly. Took out the ballast unit on a hunch and spotted a microswitch which a service manual claims detects when the lamp housing is open. Unfortunately, this doesn’t work too well when the frame’s a bit bent, so I soldered over that on the mainboard. Reassembled, again.

Getting a little bored with the constant dis/reassembly at this point- and it still doesn’t start. Bulb struck, which was a great morale booster, but it then shut down and burned it’s fans at maximum for 5 minutes before turning off. Not too useful.

Poked around a bit and considered these symptoms were similar to what my other projector had- lo and behold, the fan on the smoothing unit is jammed! One of the support struts for the fan was snapped, and a quick spot of superglue fixed that in place and let it spin freely again. Reassembled, and bam! Working projector #2. I’ll post pics sometime.

All in all that’s two projectors for £0 which I’ve gotten working again with long remaining bulb lifes in each. Total resources required: Three screwdrivers, a Torx driver, three forceps (Seriously- best tool I’ve ever had for electronics work), some narrownose pliers, antistatic gloves and static control kit (to avoid frying the mainboard), soldering iron (Got a new Weller temperature controlled iron, which is doing great), approximately 4mm of solder, a dab of Loctite superglue, an IEC power lead and some spare time. Oh, and lots of caffeine.

If I get a chance tomorrow I’ll see about throwing a post up about some of the under-the-hood changes I’ve been making and have planned for EVE Metrics and other sites. I’m looking more seriously at a single-sign-on solution between all the MMMetrics sites (ISKsense, EM, AccVIEW)- but the question is, if I do a SSO solution why not make that a global SSO solution with OpenID support for pluging in other sites? More to the point, allowing centralised API key management between applications that need it (EVE Metrics would use this to store/retrieve keys) would be great- give application your MMMetrics ID, MMMetrics prompts you to give access to your account data to that site, optionally letting you provide one or more API keys to the site. Kinda like a badass Gatecamper 2 that doesn’t suck so badly. Thoughts?

Most Wanted Features

1. API import of losses and kills
2. In-detail fitting information/fitting database
3. Accurate price data
4. Scoring/points
5. Standings import

Most Wanted Fixes

1. Needs to be quick
2. Needs to be easily themed and styled
3. Needs good documentation
4. Needs a better admin panel than EDK
5. Needs easier installation

Medals and ranks were a close runner up, as were better engagement summary tools.

So, let’s go into a little more detail and see how we’re covering each of these for Nexus.

API import of losses and kills

Already done! Works a treat. Does corporate feeds as well as personal.

In-detail fitting information/fitting database

This is built into the killboard. Instead of a kill having associated modules and a ship, we store that as a fitting and reference it from the kill. So we have a fitting database built on the side from the word ‘go’. Of course, kills aren’t the only way to load in fittings.

Accurate price data

We have a simple-to-extend framework implemented in Nexus for price sources. Here’s our EVE Central source.

So, there you go. A simple updater. You get a web interface which lets you add sources which use these updaters- setting up the URL, priority, and what that source should be used for. We have this set up in a way which enables you to display different prices to corporation members, alliance members, and so on. If your corporation has in-house shops, those shops can be easily added and used for corporate members.

Scoring/points

Ooer. We’ve not gone near this, yet. It’s always a difficult subject, and plenty of people have plenty of different ideas about how you should do this. Whatever we end up doing, rest assured there will be plenty of flexibility about how the system is configured. Possibly a modular system, though that might be overcomplicating things.

Standings Import

Again, we’ve not gone near this yet. We’re currently using the restful_authentication plugin with some extra filters (fc_required, admin_required, and so on). This isn’t great. One of the things we need to do is to move over to a more flexible framework (we’re looking at Lockdown) for authentication, and set up a proper usergroup/permission system. For example, you might want everyone in certain alliances/corps to have access to reimbursement, but not everyone you’re blue to or your alliance is blue to. It’s a can of worms, programatically, and we’re leaving it till we’ve got the basics sorted.

Needs to be quick

Ok, no problems here. Not only is the system just faster in design, we’re also providing extensive support for action and fragment caching in pages, counter caches in the database, and so on. We’re experiencing great performance for a large number of users on an already overworked server, even without any caching enabled. Caching will support a variety of backends, notably memcached.

There’s no risk of servers being bogged down by large killmails taking ages to parse and be added as can sometimes occur- when a mail is submitted, it gets put on a queue, and a worker will pick it up in the background. It’ll appear on the site when the worker’s done building the kill, checking it’s sane, and retrieving prices/portraits/corp logos associated with the kill and attackers.

Needs to be easily themed/styled

Ok, again, not much work done here. We’re using Haml for our views by and large, but nothing is stopping you writing Erb templates or anything like that. Views are split into partials where it makes sense (the killboard is so far around 20 templates), and there’s a global layout which is easily adjusted. Stylesheets are authored with Sass, which gets automatically compiled into a minified .css file. Javascript extras can be done with the included script.aculo.us library, which comes with a lot of effects and handy tools.

Themes are certainly doable and we might add support for this later if there’s demand for it.

On the same topic of styling/theming, the whole system is written with internationalisation (I18n) support for every aspect of the interface. So if you’ve got a non-English corp or alliance, you’ll be able to have Nexus in French, Dutch, German… whatever people voluntarily translate Nexus into. There’s around the order of 450 strings to translate, though we’ve reused quite a few of those so they might need splitting up in places for other languages.

Needs good documentation

We document all the code inline, and build RDoc from that, so there’s a goodly starting point there. The official site (when it’s up) will include user documentation for all the features, as well as administrator guides and tutorials for installation.

Needs a better admin panel than EDK

Right now I’d say EDK is winning on this in some ways. We’re currently lacking a central admin panel. But it’s a few templates away, and I might get around to it this evening.

Needs easier installation

I can’t promise anything! It all depends on the host, really. If your host will let you install gems for your Ruby setup, and will let you run an always-running rake task in the background, and will let you set up a cronjob (Basically: Any VPS host, most shared hosts, any dedicated host) then it’s a piece of cake. But the always-running Rake task might be a problem for some people, notably those running on shared hosts.

Installation really hasn’t been looked at in any detail whatsoever. We have a ‘developer friendly’ writeup on the Nexus development wiki, but it’s not for the shared hosters amongst us, and goes through building the Nexus database from the EVE database sources, a step which we want to avoid by providing prepackaged starting templates.

So, there you have it- what you want and what we’re doing to accomodate you. Of course, we’re now going to go away and work on what we’re not doing so well on. I’ll be pushing up the MMMetrics website in the next week or so, complete with some Nexus screenshots and so on.

Git (hosted at github.com) stores the project source code. For a given problem, I fork, make changes, merge back in, and push. That results in the Git repository on Github having the most up-to-date version on the ‘master’ branch.

At this point, CruiseControl.rb (A great continuous integration tool) picks up the update, and tries to run the full RSpec test suite on the project. Emails are sent out to all developers when the tests pass or fail, with associated error information if it’s the latter, as well as a HTML report on our CCrb instance.

Assuming we’re ready to deploy, I fire up Capistrano (an excellent pure-Ruby tool for deployment) on my development box, and run ‘cap deploy’. Capistrano connects to the server, makes a Git clone of the most recent master copy, copies in keys/database configs (So they’re kept out of Git), migrates the database, and then makes a symlink to the new project so Apache (running Passenger) is updated as to where it should be looking. And voila, we’re deployed!

I’d be interested to hear what workflows other people use for Rails development. This works fine, and Git means it scales fairly easily to accept more developers, as well as providing incredibly flexible distributed control (So I can work on my laptop away from the internet, committing, branching and so on as needed). But I’d be at a loss to make this work on a more complex deployment, say with EC2 or other virtualisation software. A continuous integration tool that played better with RSpec would be awesome, too.

VAF Logo ftw!

I figured it was high time something was done about the lack of such a tool- a tool to generate the icons for corporations in EVE Online, which are constructed from three pictures with their related shaders, given the data from the API corporation sheet. So here it is. With many thanks to Entity for doing the hard work getting the image data out of EVE and working out the finer details of EVE’s image formatting.

Really can’t be bothered to host your own? Take a look at my freely available corporation logo generator server, which uses this library to provide corporation logos for those of a less technical nature!

This is a pure-Ruby library with a bundle of prerendered corporation image components, which are combined at runtime to display the image. It is recommended that you cache corporation logos generated with this plugin.

Transparent Logos are also supported

Browse source/Download (Github)

While it’s designed as a Rails plugin it should work equally well as a standalone pure-Ruby plugin. The only dependency is on RMagick, a Ruby interface to ImageMagick. The library is hosted on Github, so you’ll need git to grab it, or you can hit the download link on Github to grab a tarball or zip.

Usage is as simple as the following:

If you use this and want to say thanks, please do so with ISK- both to Entity and Ix Forres. As far as I know there should be no legal issues with this, other than to give CCP credit as with all ingame assets used out of game. And, of course, many thanks to CCP for adding the data needed into the API. If you really must, the Python version by Entity is over here.