I’ve spent most of my time working on EVE Metrics. There’s some very cool, very powerful changes coming up soon; early Feb saw the introduction of much more accurate prices and indexes, with a newly improved algorithm for calculating the average prices of items. But even better is some of the new stuff coming in the next few weeks- notably the implementation of a fully-fledged OLAP warehouse for EVE Metrics, which will open up some very awesome possibilities in the long run.

Also under the scalpel this month has been the API system. EVE Metrics will support full and limited keys when it comes to the API. However, there will be a quirk; if you want to make use of other people’s API data, for example to see more detailed market analysis with transactions hooked in and so on, then you’ll have to share your data. This means if you want to benefit from other people’s data, you must reciprocate and share your data for the benefit of others. Your data will, in all cases, be used for global averages, but entirely anonymously; for example, the number of transactions per day on a given item may include data from your API, but nobody would know it. This system will hopefully encourage users to share data more often than hide it. I’m planning to make this an opt-out system, with the choice to opt-out given on the API key page as part of the form. It’ll be really hard to miss, and those who are paranoid or wish to hide their activity completely can check the box to opt out.

I released accVIEW a few days ago, and it’s had quite rapid takeup from corporations. It’s a service that lets you perform background checks on prospective new members to your corporation- the basic tool lets you view skills, characters on an account, and various bits of information like their corp details, CEO, and so on. The premium version (For the low cost of 150mISK) lets you see the applicant’s wallet journal (with tools to show suspcious transactions and filter the results), as well as their recent kills/losses.

EVE’s M10 expansion, Apocrypha, should be awesome. Tech 3 is going to be great- lots of people complain about the skill loss and the fact that it’ll make FCing a nightmare. Well, no. The skill loss makes sense, and provides an interesting new dynamic to EVE. FCing- well, those who complain about T3 making FCing impossible are evidently not up to scratch as FCs. It’ll give FCs a real change and challenge for the first time in years. The wormhole stuff will be an interesting thing to watch pan out- there’s lots ot potential there, and it could end up being a lot of fun…

Nexus is progressing slowly but surely; the large amount of data we’re gleaning via an API-scraping installation for Sc0rched Earth is helping no end with development, and we’re busy tidying things up behind the scenes and refining some of the interface to make more sense. Once I’ve gotten ActiveWarehouse’s ETL library working properly, my next step will be to break out Photoshop and my text editor- EVE Metrics, ISKsense, maybe accVIEW, and the new MMMetrics site (Which will be launched soon) are all going under the knife and getting a serious facelift. And then it’s on to even more awesome stuff for EVE Metrics!

Most Wanted Features

1. API import of losses and kills
2. In-detail fitting information/fitting database
3. Accurate price data
4. Scoring/points
5. Standings import

Most Wanted Fixes

1. Needs to be quick
2. Needs to be easily themed and styled
3. Needs good documentation
4. Needs a better admin panel than EDK
5. Needs easier installation

Medals and ranks were a close runner up, as were better engagement summary tools.

So, let’s go into a little more detail and see how we’re covering each of these for Nexus.

API import of losses and kills

Already done! Works a treat. Does corporate feeds as well as personal.

In-detail fitting information/fitting database

This is built into the killboard. Instead of a kill having associated modules and a ship, we store that as a fitting and reference it from the kill. So we have a fitting database built on the side from the word ‘go’. Of course, kills aren’t the only way to load in fittings.

Accurate price data

We have a simple-to-extend framework implemented in Nexus for price sources. Here’s our EVE Central source.

So, there you go. A simple updater. You get a web interface which lets you add sources which use these updaters- setting up the URL, priority, and what that source should be used for. We have this set up in a way which enables you to display different prices to corporation members, alliance members, and so on. If your corporation has in-house shops, those shops can be easily added and used for corporate members.

Scoring/points

Ooer. We’ve not gone near this, yet. It’s always a difficult subject, and plenty of people have plenty of different ideas about how you should do this. Whatever we end up doing, rest assured there will be plenty of flexibility about how the system is configured. Possibly a modular system, though that might be overcomplicating things.

Standings Import

Again, we’ve not gone near this yet. We’re currently using the restful_authentication plugin with some extra filters (fc_required, admin_required, and so on). This isn’t great. One of the things we need to do is to move over to a more flexible framework (we’re looking at Lockdown) for authentication, and set up a proper usergroup/permission system. For example, you might want everyone in certain alliances/corps to have access to reimbursement, but not everyone you’re blue to or your alliance is blue to. It’s a can of worms, programatically, and we’re leaving it till we’ve got the basics sorted.

Needs to be quick

Ok, no problems here. Not only is the system just faster in design, we’re also providing extensive support for action and fragment caching in pages, counter caches in the database, and so on. We’re experiencing great performance for a large number of users on an already overworked server, even without any caching enabled. Caching will support a variety of backends, notably memcached.

There’s no risk of servers being bogged down by large killmails taking ages to parse and be added as can sometimes occur- when a mail is submitted, it gets put on a queue, and a worker will pick it up in the background. It’ll appear on the site when the worker’s done building the kill, checking it’s sane, and retrieving prices/portraits/corp logos associated with the kill and attackers.

Needs to be easily themed/styled

Ok, again, not much work done here. We’re using Haml for our views by and large, but nothing is stopping you writing Erb templates or anything like that. Views are split into partials where it makes sense (the killboard is so far around 20 templates), and there’s a global layout which is easily adjusted. Stylesheets are authored with Sass, which gets automatically compiled into a minified .css file. Javascript extras can be done with the included script.aculo.us library, which comes with a lot of effects and handy tools.

Themes are certainly doable and we might add support for this later if there’s demand for it.

On the same topic of styling/theming, the whole system is written with internationalisation (I18n) support for every aspect of the interface. So if you’ve got a non-English corp or alliance, you’ll be able to have Nexus in French, Dutch, German… whatever people voluntarily translate Nexus into. There’s around the order of 450 strings to translate, though we’ve reused quite a few of those so they might need splitting up in places for other languages.

Needs good documentation

We document all the code inline, and build RDoc from that, so there’s a goodly starting point there. The official site (when it’s up) will include user documentation for all the features, as well as administrator guides and tutorials for installation.

Needs a better admin panel than EDK

Right now I’d say EDK is winning on this in some ways. We’re currently lacking a central admin panel. But it’s a few templates away, and I might get around to it this evening.

Needs easier installation

I can’t promise anything! It all depends on the host, really. If your host will let you install gems for your Ruby setup, and will let you run an always-running rake task in the background, and will let you set up a cronjob (Basically: Any VPS host, most shared hosts, any dedicated host) then it’s a piece of cake. But the always-running Rake task might be a problem for some people, notably those running on shared hosts.

Installation really hasn’t been looked at in any detail whatsoever. We have a ‘developer friendly’ writeup on the Nexus development wiki, but it’s not for the shared hosters amongst us, and goes through building the Nexus database from the EVE database sources, a step which we want to avoid by providing prepackaged starting templates.

So, there you have it- what you want and what we’re doing to accomodate you. Of course, we’re now going to go away and work on what we’re not doing so well on. I’ll be pushing up the MMMetrics website in the next week or so, complete with some Nexus screenshots and so on.

Git (hosted at github.com) stores the project source code. For a given problem, I fork, make changes, merge back in, and push. That results in the Git repository on Github having the most up-to-date version on the ‘master’ branch.

At this point, CruiseControl.rb (A great continuous integration tool) picks up the update, and tries to run the full RSpec test suite on the project. Emails are sent out to all developers when the tests pass or fail, with associated error information if it’s the latter, as well as a HTML report on our CCrb instance.

Assuming we’re ready to deploy, I fire up Capistrano (an excellent pure-Ruby tool for deployment) on my development box, and run ‘cap deploy’. Capistrano connects to the server, makes a Git clone of the most recent master copy, copies in keys/database configs (So they’re kept out of Git), migrates the database, and then makes a symlink to the new project so Apache (running Passenger) is updated as to where it should be looking. And voila, we’re deployed!

I’d be interested to hear what workflows other people use for Rails development. This works fine, and Git means it scales fairly easily to accept more developers, as well as providing incredibly flexible distributed control (So I can work on my laptop away from the internet, committing, branching and so on as needed). But I’d be at a loss to make this work on a more complex deployment, say with EC2 or other virtualisation software. A continuous integration tool that played better with RSpec would be awesome, too.

All this info will go towards deciding our priorities going forward with Nexus. Internally we’re coming very close to a complete working killboard (price loading is the only thing on the menu that still needs work, and that’s only for a few of our importers), and once we’ve done a few more milestones worth of work towards other bits of Nexus we’ll be returning to the killboard and forking out the open source, freely-available version when we’re done.

So, if you use killboards in any capacity, click through to answer this survey and let us know where we should be focusing our efforts, or tell us about a killer feature you think all KBs should have had from day one.

cforms contact form by delicious:days

As always you can use the buttons below to submit this to Twitter or other social sites.

Rails 2.2 comes with I18n built in, so I’ve been using that. It’s a slow process, turning every string into it’s own categorised, indexed line. It’s all in YAML though, so I can easily work on it freehand without needing to worry about complex systems for storage. The one downside is I get to reload my development server to see changes to the pages, but that’s not too big an issue. It’s easy to use and works like a charm, so it fits the bill for my purposes.

Also, Rails 2.2 is thread safe! I’m interested to see what this does performance-wise on Passenger. Load on my server isn’t a problem but memory is my main concern- Passenger with Ruby Enterprise Edition works great by sharing common elements in memory to reduce usage, but thread safe Rails means I should be able to get away with running fewer servers in the first place. I’ll spend some time with JMeter and New Relic RPM at some point to see what the real-world gains are.

My next challenge is going to be creating a decent looking fittings/kill display interface. I’ve learned a lot working on the TacMap, and have some ideas for how I’m going to display fittings. I don’t think I’ll use the ingame fittings screen picture with overlays; it’s overly complex and imo there’s better ways to show fittings. Considering one of the main goals with Nexus is high performance and given the slashdot-effect style killboard spammings that occur whenever a big kill gets scored, I’d like to try and keep the SQL loading down to a minimum.

I’m also trying to work out what the best way of handling caching will be. My whole configuration system thus far is getting looked at sternly- I’m using Configatron, but think I need something a little beefier than that which supports some kind of webinterface for editing configuration values. Still, that’s a problem for another day.

On the EVE front, Shrike lost his titan. Again. In exactly the same way (cloaked off a gate). Yay.