I made it public and popularity grew. I added some features, added the premium option for those who wanted a bit more, and it’s been ticking along, occasionally throwing horrible errors and falling over, the background worker regularly falling over and dying, and running on a Quantum Rise datadump. And there was a major security glitch- we didn’t store API keys, making it impossible to validate people regularly, meaning people who left corporations could still view their old corp’s requests. And they couldn’t update their account to their new corporation.

No more.

accVIEW has gotten a fresh new facelift, skill distribution graphs, a fundamental API key change, some improved code throughout and a new database dump update. I’ve also added a ‘forgot password’ feature for those who don’t remember their logins too well, and fixed a few outstanding bugs.

If you’re an accVIEW user, next time you log in you will be prompted for your API key again. This is to be expected; the reason we’re doing this is so we have a copy we can re-validate regularly (once a day) to ensure that you are still in the corporation you were in last time we looked. If you change corporations, your main character will be dissociated and you’ll have to reenter your API keys next time you log in and choose a new main character.

Enjoy!

There’s the top-down view for you. We’ve never really not had enough data. We get good market coverage in most regions and we’re fairly up to date in the grand scheme of things. But compare the actual market of EVE to EVE Metrics and we’re still a long way off having a truly accurate picture. EVE moves quickly- in some markets, from minute to minute orders will be shuffling around and changing price and being bought out.

With Dominion we got a new browser. This means you can now use the full EVE Metrics website ingame, but also (through some Javascript client hook additions) lets us provide a fantastic new tool to help us get an even better picture of the market in EVE.

If you fire up the IGB and head over to the upload suggestions page, you’ll be given a list of 10 items, and a few options for automatic checking. Choosing this option will prompt EVE Metrics for a list of items to check, and will automatically go and look at those items. It’s slow, but it works. In the space of a few hours with one user, we can get data for an entire region across all the items on the market. This is utterly fantastic and we’re really looking forward to the larger volume of data this is bringing to the site.

So, if you’ve got a spare moment, or you need to go AFK for an hour, or you want to help out while you’re mining, or you’re just tired of clicking the next item in the list, install the uploader and visit the page ingame to get started. Every upload counts and helps us build the biggest, best picture of EVE’s market we can manage to produce. Uploads to EVE Metrics are also syndicated to other websites and tools, of course. Your uploads and contribution of time help hundreds of users who use the site, and tens of thousands more who rely on our pricing, history and order APIs for their applications.

Oh, and if you’re a developer, we now have a server status API with all the information you could possibly want on TQ, Sisi and the API servers. It can be found here (docs here). Enjoy!

Anyway. We got a lot of fluff, this year. Aside from further elaboration on stuff already announced, there were actually no major announcements made at fanfest. We did have some interesting info about New Eden, CCP’s EVE-Online-Online website. And there was some evidence (gasp!) that CCP were listening to third party developer suggestions at the API roundtable.

There was almost enough minor stuff announced to make it worthwhile. We did get a release date for Dominion – 1st December 2009. But no New Eden with the launch. And knowing CCP we’ll probably not get API changes till a bit after that. What’s really awesome though is that we will be getting new APIs. I’m just hoping they’re useful APIs…

Anyway, while I was sitting at home being mostly bored, I decided I’d had enough pressing F5 on the Twitter search page, and put together a website (ff.mmmetrics.co.uk – it’s down now) to grab EVE fanfest feeds from Twitter and Flickr. This became popular enough within a few hours that we had to rip it off the server and give it it’s own Amazon EC2 virtual server, as it was in danger of crashing ISKsense and EVE Metrics. Doh. A wild success, in any case, for a simple but handy website. What the website did make us realise is how little headroom we have on our current server. We kinda knew that already but it did make the point quite well.

EVE Metrics 2.1 has launched mostly well but we’re still having issues with the API processing code. Makurid has been working hard to pin down the cause of the problems and destroy it while I’ve been fixing up servers and moving sites around, and we’re getting a bit closer to having a complete fix. We’re not there yet, but we will be soon with any luck.

Half the point of EVE is the userbase is a very mature one and the sort of crowd who sit in their room playing with internet spaceships. Does EVE really want to get the Halo players of the world contributing to the game? I honestly don’t think so. It’s a very snobbish view, I know, but the average console gamer probably doesn’t want to spend their time liasing with alliances and planning the takeover of space from other alliances with internet spaceships they can’t see. What’s in it for the Dust 514 players, anyway? A mission system that could easily be done with some clever AI, from what I hear. I remain highly sceptical and look forward to seeing what CCP has planned in further detail- if they make it work it’ll be fantastic. But it’s a big if.

On a much lighter note, I’ve got some snippets from EM2′s new API integration. We’re being really thorough with this so far; we have seperate workers to download and process the API, meaning we can get around the API-being-slow bottleneck by having 3-5 workers just downloading and a few doing the processing (which is quick). What we also wanted to do was give you, the user, tons of control over what information we load into EVE Metrics. Read on for detailed information.

Right now we don’t have the ability for you to set information as private. It’s either in the system or out. That applies only to market orders, of course; everything else is private anyway. We may well be using people’s transactions and asset data in an anonymous manner for statistics, but we’ll have a complete privacy policy drawn up before this goes live so that’s more concrete. What we haven’t added, though, is a way to have your orders show up in the system just for you and not for anyone else. It’s a technically huge hurdle, and so it’s unlikely to happen.

Edit: Apologies for the odd yellow lines in the screenshots below. My computer is currently totally fubar with graphics card issues and software problems and I’m writing this blog in safe mode with graphics artefacts all over the damn monitor. They’re not there on the site, anyway.

What we do have is the aforementioned shedload of controls over everything we access. Here’s the API list view:

The EM2 API key list page, complete with permissions link (Click image to expand)

This is exactly the same as on other websites I’ve produced, with one extra addition- the permissions button. Click on that and you’ll be taken to a page that looks something along the lines of this (BETA ALERT: The UI here is unfinished and the API methods are not representative of what EM2 will actually use. The character list method will reside in it’s own API-key-wide section instead of on each character, too, and you won’t be able to block character sheet calls):

API Permissions example page with character sheets blocked on two characters (Click image to expand)

Now, this is where things get interesting. EVE Metrics will only pull data for the methods enabled here. By default, everything that is not required for basic operation (Which will be Character List, Character Sheet and Corp Member Security for corp directors- more on that in a bit) will be turned off. As you browse EVE Metrics you might stumble across features that need data from the API; for example you might try and use an asset valuator. At that point you’d be prompted to enable that bit of the API. You can also go in and fiddle on a per-character basis at any time to keep things locked down to your heart’s content.

Of course, this means nothing. We could be pulling your asset APIs for your corps and selling them on eBay. Now, I’d like to think we’re quite well trusted in the EVE community, but of course the more paranoid amongst you will want to keep an eye on your API request logs. Our server’s IP is 77.74.196.169 and we’ll be maintaining a list of all IP addresses API requests could have come from if we ever expand to more than one server. We may also add our own API access list to log everything we do with your API, but that’s subject to performance constraints.

We do think this system will let even the most paranoid of players and CEOs of large corporations use EVE Metrics safely and securely, without compromising any data they don’t want shared with us. Of course, I’d be interested to hear feedback on the system, but that’s what comments and Twitter are for.

I promised some statistics up in that title, so here’s some juicy stats for you now the teasing is over.

• The Forge Market Coverage: 65% (Active Orders)
• Active Orders: 480,734
• Expired Orders for July and August: 1,608,408
• Total Orders: 2,089,151
• Historic Datapoints: 6,949,494
• Computed Trades for August: 892,990

And for the technical readers, some interesting statistics on our caching performance:

• Cache hits: 72% (438,140 of 609,386 gets)
• Cache datastore size: 41 megabytes

We’re quite proud of that cache hit rate. The vast majority of the time you look at a page on EVE Metrics, the database is barely involved; you’re looking at a pregenerated snippet of HTML. But there’s never stale data on the site; every time new data comes in the relevant caches are wiped and next time you look, the new data is there right away. It works really well, and is part of the reason why EM2 is so damn snappy compared to EM1.

Most Wanted Features

1. API import of losses and kills
2. In-detail fitting information/fitting database
3. Accurate price data
4. Scoring/points
5. Standings import

Most Wanted Fixes

1. Needs to be quick
2. Needs to be easily themed and styled
3. Needs good documentation
4. Needs a better admin panel than EDK
5. Needs easier installation

Medals and ranks were a close runner up, as were better engagement summary tools.

So, let’s go into a little more detail and see how we’re covering each of these for Nexus.

API import of losses and kills

Already done! Works a treat. Does corporate feeds as well as personal.

In-detail fitting information/fitting database

This is built into the killboard. Instead of a kill having associated modules and a ship, we store that as a fitting and reference it from the kill. So we have a fitting database built on the side from the word ‘go’. Of course, kills aren’t the only way to load in fittings.

Accurate price data

We have a simple-to-extend framework implemented in Nexus for price sources. Here’s our EVE Central source.

So, there you go. A simple updater. You get a web interface which lets you add sources which use these updaters- setting up the URL, priority, and what that source should be used for. We have this set up in a way which enables you to display different prices to corporation members, alliance members, and so on. If your corporation has in-house shops, those shops can be easily added and used for corporate members.

Scoring/points

Ooer. We’ve not gone near this, yet. It’s always a difficult subject, and plenty of people have plenty of different ideas about how you should do this. Whatever we end up doing, rest assured there will be plenty of flexibility about how the system is configured. Possibly a modular system, though that might be overcomplicating things.

Standings Import

Again, we’ve not gone near this yet. We’re currently using the restful_authentication plugin with some extra filters (fc_required, admin_required, and so on). This isn’t great. One of the things we need to do is to move over to a more flexible framework (we’re looking at Lockdown) for authentication, and set up a proper usergroup/permission system. For example, you might want everyone in certain alliances/corps to have access to reimbursement, but not everyone you’re blue to or your alliance is blue to. It’s a can of worms, programatically, and we’re leaving it till we’ve got the basics sorted.

Needs to be quick

Ok, no problems here. Not only is the system just faster in design, we’re also providing extensive support for action and fragment caching in pages, counter caches in the database, and so on. We’re experiencing great performance for a large number of users on an already overworked server, even without any caching enabled. Caching will support a variety of backends, notably memcached.

There’s no risk of servers being bogged down by large killmails taking ages to parse and be added as can sometimes occur- when a mail is submitted, it gets put on a queue, and a worker will pick it up in the background. It’ll appear on the site when the worker’s done building the kill, checking it’s sane, and retrieving prices/portraits/corp logos associated with the kill and attackers.

Needs to be easily themed/styled

Ok, again, not much work done here. We’re using Haml for our views by and large, but nothing is stopping you writing Erb templates or anything like that. Views are split into partials where it makes sense (the killboard is so far around 20 templates), and there’s a global layout which is easily adjusted. Stylesheets are authored with Sass, which gets automatically compiled into a minified .css file. Javascript extras can be done with the included script.aculo.us library, which comes with a lot of effects and handy tools.

Themes are certainly doable and we might add support for this later if there’s demand for it.

On the same topic of styling/theming, the whole system is written with internationalisation (I18n) support for every aspect of the interface. So if you’ve got a non-English corp or alliance, you’ll be able to have Nexus in French, Dutch, German… whatever people voluntarily translate Nexus into. There’s around the order of 450 strings to translate, though we’ve reused quite a few of those so they might need splitting up in places for other languages.

Needs good documentation

We document all the code inline, and build RDoc from that, so there’s a goodly starting point there. The official site (when it’s up) will include user documentation for all the features, as well as administrator guides and tutorials for installation.

Needs a better admin panel than EDK

Right now I’d say EDK is winning on this in some ways. We’re currently lacking a central admin panel. But it’s a few templates away, and I might get around to it this evening.

Needs easier installation

I can’t promise anything! It all depends on the host, really. If your host will let you install gems for your Ruby setup, and will let you run an always-running rake task in the background, and will let you set up a cronjob (Basically: Any VPS host, most shared hosts, any dedicated host) then it’s a piece of cake. But the always-running Rake task might be a problem for some people, notably those running on shared hosts.

Installation really hasn’t been looked at in any detail whatsoever. We have a ‘developer friendly’ writeup on the Nexus development wiki, but it’s not for the shared hosters amongst us, and goes through building the Nexus database from the EVE database sources, a step which we want to avoid by providing prepackaged starting templates.

So, there you have it- what you want and what we’re doing to accomodate you. Of course, we’re now going to go away and work on what we’re not doing so well on. I’ll be pushing up the MMMetrics website in the next week or so, complete with some Nexus screenshots and so on.

Over the next few weeks I’ll be reskinning EVE Metrics, ISKsense, and various other things to have a much much nicer theme and one that is consistent across websites, giving each site their own visual take on the theme. Those sites will also be getting some much-needed optimisation, graphics, and new layouts.

Nexus is the other thing this brief post will touch on. Myself and flexd, the other developer working on the project and new member of VAF, have decided that open-sourcing the whole thing would be mad. It’s hideously complex to set up in places (though much easier than it used to be), and in any case a lot of the functionality Nexus provides is heavily aimed at alliances rather than corporations. We’ll still be offering Nexus to some alliances on an ask-and-we-might-let-you basis, but we will be forking Nexus, stripping it back to the killboard and fittings components, and releasing that as open source on Github.

That means if you’re an average corp you won’t get things like the tactical/intel overviews and IGB components, and all the alliance-level member managemet, capfleet management, and so on will be out. Because 99% of people don’t need it, to be honest. By keeping things more limited we’re keeping it much more hackable, more secure, easier to maintain and easier to extend, which means we’ll get it out of the door much quicker than we would have if we’d kept it all bundled together. So we’re looking at somewhere early-to-mid 2009 as our first release.

I think that just about covers it. Oh, other than to say happy holidays and a happy new year in advance, so I don’t forget as I obviously will do.

Long story short, one of the cronjob tasks that goes through and closes expired orders for EVE Metrics did so, then crashed and burned quite rapidly, chewing up all the RAM on the server and expertly smashing some queries into MySQL which promptly made MySQL cough up blood and fall over. Several uncacheable SQL-heavy API calls to EVE Metrics later and Apache became inundated with slow-completing requests and started spawning more threads, so on and so forth, and there goes the server. What _should_ have happened is that the process resource monitor script should have caught the rake task eating a gig of RAM and killed that, but for some reason I’m still trying to work out it didn’t. In either case, the server is now back up and stable again.

I run this server entirely out of my own pocket for services I provide to the community, and with the exception of one person who bought a GTC through the link on this site, I don’t get any financial aid on that front. So for now, a single point of failure is the way it’ll have to be unless I can convince my host to provide me with a second server and load balancer for free

If you’re an ISKsense-using site, then I do apologise when ISKsense javascript includes never load, and I am trying to work out a solution in javascript which will avoid that by rendering some predefined ads if it can’t pull the JS down in 2 seconds or so. As for CLGS and so on, there’s not much I can do other than to keep this server ticking along as happily as I can with all the applications it’s running. Needless to say I’ve been pretty busy over the past few days, and keeping close eyes on my server has taken second priority against keeping close eyes on the Entirely Harmless gangs floating around Fade in the past week

If you want to help me manage costs and improve my stability, then buying GTCs via the link on this blog is a good way to go about that; alternatively, feel free to donate via the link on the sidebar, where I’ve set up a Pledgie campaign for the donations. Every little helps (Unless it’s a really little chunk in which case PayPal will steal all of it and give me pennies, which aren’t very useful), so please do donate. Anything left over at the end of each month will be donated to Cancer Research UK rather than being spent on mroe bree.

http://clg.eve-metrics.com/<corporationID>.png
Example: http://clg.eve-metrics.com/853748831.png

Nice and simple, no?

There’s just one caveat- this currently only works for corporations in alliances. The API has some very odd ideas about letting out-of-alliance corps have their corporation sheets grabbed, meaning I can’t retrieve shape data from those corporations. I have made a bug report to CCP, and I’ll keep you updated when/if this gets fixed. In the meantime, I’ve got a workaround- visit the CLG website and you can submit your limited key and user ID to add your corporation’s logos to the database.

I’m making no charge for this service, though of course donations in ISK are hugely appreciated if you use this. If you’re a heavy user and want to start using this, please drop me a line first.

One other thing you might want to know if you’re caching- this will actually redirect you with a 302 error code. This means that your average request doesn’t even need to hit the extremely light Rack application which generates the images if the file’s there already, thus making my server and your response times both happier things. It shouldn’t make any difference, but I’m mentioning it just in case.

If you cook up any mods or tools to use this (I’d love to see an EDK mod to add this) do let me know and I’ll drop a link in this post. Edit: There’s an EDK mod in the works! Woot!