After seeing Mynxee get some wheels in motion with CSM5, I felt a bit annoyed at my inability to contribute a lot to the workings of the CSM. At my age, I am not eligible to run for candidacy, as the drinking age is 21 in Iceland and CCP don’t want delegates they can’t take down to the pub (At least, I guess – the age of majority is 18, so it’s not a legal thing). Anyway, I decided to use my normal approach to fixing problems: There’s an app to be written for that…

So with the help of Selene from EOH Poker, we rallied a whole 3 CSM and ex-CSM people into a collaborative text editor, and wrote a spec. The tool in question was to manage proposals as they went through the CSM process – with the wiki and AH forums being identified as inadequate by both the CSM and CCP, but with no activity from CCP on providing a solution, I saw this as an opportunity to get the community involved and really push to get some CSM processes streamlined. We called it Nebula, and formed a new team of developers, with an open source codebase and instructions on how to contribute. We actively sought out new developers and tried to encourage other third party developers to get involved, and were met with mostly positive results. Team Excellence was born.

The CSM involvement we had was good. We ended up with a spec that was, as the title of this blog suggests, a little nebulous, but had most of the major concepts well sketched out and defined. We got to work.

It’s now been a fairly long time since we last had CSM contact, though those who have been trying to get involved in this (Mynxee and  Trebor to name a couple) have done so well, and continue to provide input; the main issue we have is that out of the 8 (since Ankh was removed) active CSM members and 5 reserves (or 9 and 4, depending how you look at it- either way, a total of 13 players), we’ve gotten in touch with and received input from three at most.  This is worrying for a number of reasons.

If the chairwoman of the CSM is having this much difficulty in managing to have some people meet and discuss matters for even a one-off meeting or event, then I start to wonder how well the communication works between meetings on other issues. For example; I posted a long while ago about the API in the assembly hall forum. The matter has been taken up by two CSM members, including Mynxee, and may be raised at a future meeting. I got an evemail from Dierdra Vaal a couple of weeks later, asking me if I thought anything was wrong with the API and if I’d support him raising an issue about it. This is the sort of thing that irks and worries me – if there’s no real communication between CSM members except during their meetings with CCP, then how the hell do they hope to present any sort of unified face to CCP or address issues that are concerns to the playerbase, and not individuals within the CSM? With no communication outside of those meetings, issues aren’t being discussed till it’s far too late to be discussing them, people are missing entire issues, and this all has a knock-on effect on how much the CSM actually achieves.

Frankly, I couldn’t imagine being a member of the CSM without there being a mandatory-usage CSM delegate IRC channel or some similar chat mechanism – heck, even a mailing list or forum – where the CSM can talk amongst itself, and get the ego trips out of the way before CCP gets involved. And I suspect that the fact that a lot of people on the CSM see no problem with the current state of affairs is a good indication of just where the priorities of those members lie – because it’s certainly not with the playerbase as a whole. The CSM is bigger than EVE’s petty squabbling of alliances and corporations; we’re talking about an elected council of people who can help steer the course of a company which is putting food on the table for hundreds, and innovating hugely. The CSM as an entity and as an idea does not deserve the majority of the people it has been saddled with so far.

Nebula as it stands is frozen, awaiting information from the CSM and the motivation of the developers to work on it. I know that myself and Makurid, who thus far have written the vast majority of the code, are having more and more difficulty finding the motivation to work on any EVE Online projects, let alone projects that involve such a depressing facet of EVE, and indeed force us to try and interact with it. Certainly for now, I will be stepping aside as a developer of Nebula, and reducing the amount of time I spend on my other EVE projects. Aside from anything else, as fun as it used to be, EVE apps don’t put food on the table, and wherever my career may go, developing the apps I’ve built further may not be the smartest move – there’s other projects, other opportunities. It’s just a shame that at the moment, EVE doesn’t seem to be working out for myself and other third party devs.

So here’s a post about EVE Metrics 3.

We’re getting close to having everything polished and ready for release. The main issue at hand thus far has been the homepage; I say this like it’s a small thing but it’s required me to learn some new and interesting things about CSS, Makurid’s done some excellent work to produce some feed scrapers and elements for the lower portion of the page… there’s a lot to it.

I thought that it’d be good to list a few of the changes we’ve made for version 3.

• Complete redesign of the site thanks to Rettic
• Market detail pages have been entirely renovated
• Various pages which haven’t been improved in some time have now been tidied up
• API key management has been improved
• API key permissions management has been improved
• Backend processors for API functions and upload processing have been improved and made more reliable
• My Metrics has been entirely renovated, now with sparklines for wallets and a new layout
• Orders and transactions have been moved from My Metrics into their own detail pages, with a summary on My Metrics
• Journal information has been added and given it’s own detail page
• Player Owned Structure integration has been added, though still in it’s infancy
• Sensitive portions of the site now make use of SSL transport encryption (HTTPS) automatically
• Wormhole pages have been updated
• Improvements to the corporation pages through refactoring to share view code between character and corporation pages
• Graph improvements
• Complete test coverage of every line of code (Nah, just kidding, we’re still pretty thin on those test things for large chunks of UI code)
• 0.2% more cowbell
• 5% other features I’ve not listed above, plus 100% more polish overall

Excited? We are! There’s a lot of work in the lines above and I think you’ll like the results. We’re not sticking to any firm release schedule because we’re terrible at sticking to them; we’re students, not full-time developers (incidentally, if anyone’s got any jobs available for temporary/contract work, SE UK preferred (or work-at-home), 6 weeks max, let me know!). That said, we hope to have a release before the end of July.

We’ve also been rewriting our uploader! That’s right- Linux, Mac and Windows support all in one neat package. The GUI’s not anything special but it works, and we’ll be polishing it and getting it release-ready before long. Huge thanks should be directed to TTimo, who has been the driving force behind this with some welcome Python experience, and Makurid for assisting him in developing the new client.

Once we’ve gotten that polished, packaged and rolled out, we’ll be running a 5 billion ISK contest to promote it- the three winners (each receiving a portion of the 5 billion pool) will be selected from the most active uploaders for the week or two after the competition is announced. We’ll finalize all the details and have it posted up when we’re ready to go ahead with that, of course. If you’d like to make that 5 billion figure larger, you can contribute ISK to the character MMMetrics Agent ingame! So far, thanks go out to Rilcon, Chribba and Entity for contributing to the current pool. The new uploader will be released after the new site – we’d like to change one thing at a time so we can iron out all the kinks. Once we’ve gotten it out and tested it thoroughly we’ll roll out the upgrade- your existing uploader will prompt you to update when you start it.

One last thing – I’ve personally submitted a proposal to the CSM regarding development efforts from CCP surrounding the API and EVE Gate. If you’ve not done so already, I strongly urge you to read my proposal and support it if you feel, like I do, that CCP have made some serious mistakes lately in this regard. The thread can be found here. It has been picked up and supported by at least 3 CSM members so far, but the community support will help considerably to drive CCP to consider it seriously.

In total it was a fairly mammoth operation by our standards; we transferred in excess of 100 gigabytes of data between the servers over the course of 12 hours, shifted over 20 websites and 3 major webapps, and got everything up and running again in under a day once we’d moved it all to the new box. The downtime has been annoying and I’ve certainly learned some lessons for next time, but here’s the flipside…

We’re now running on a much, much roomier machine. We’ve not got the environment perfectly set up and we’ll no doubt spend the next week tuning everything, adjusting things till they’re just right and fixing bugs, as well as adjusting and rewriting chunks of applications to make use of the extended caching capabilities of our new environment. We’re already using this to great effect in the EVE Metrics APIs but we can make better use of caching throughout our apps.

Once we’ve gotten settled in, we should be performing much better and more reliably than previously. We’ve already seen huge performance gains on our database (we can process more than twice as many uploads per second, for example) and we hope to have things even faster soon.

Of course, to achieve this I have been running on more or less an empty tank as far as sleep is concerned and working things in around my life at university, which has been interesting. Still, we’re at the point now where it’s more or less stable and everything basically works, so now I’m going to grab a few hours of sleep before lectures tomorrow, before a long long lie-in on Saturday. Enjoy!

All you need to do is head over to EVE Metrics, log in (or sign up if you’ve not got an account yet), add your API key(s) if you haven’t already, and then enable the EVE mail API method. And voila- EVE mails, in your browser, updated as often as CCP lets us.

The icing on the cake is that we’ve also provided a feed for RSS readers for your EVE mails. Google Reader/iGoogle or any other ATOM-compatible reader (which is basically all of them) can now monitor your ingame EVE mails at the click of a button.

We’ve also gotten around to doing skill training- you can see what you’re training (queue support of course is included) on all your accounts.

The next logical step from here is notification support- get an email or SMS whenever your characters can train a new skill, whenever you get a new EVE mail, whenever one of your market orders is outbid or fullfilled. You name it, I’d love to see it notifiable. We’re still in the early days with that, but that’s where we’d like for that to end up.

We’ll be improving on these and implementing other APIs in the coming days- we want to get notifications loading for all you corporate types, and we’re looking forward to bringing more skill monitoring/information into the UI. I’ve got a lot of ideas bubbling around- we’re getting to the point where we’ve got loads of little snippets of data that can all tie in with each other, creating something really fantastic for you guys and girls, the users. And that’s awesome.

Of course, we need your help to make all this run smoothly and perform well, which it has problems doing at the moment. We’re still asking for donations here, you can buy GTCs in support of us here, and we’ve just opened up advertising on the site through Project Wonderful. Any form of help is hugely appreciated.

There’s the top-down view for you. We’ve never really not had enough data. We get good market coverage in most regions and we’re fairly up to date in the grand scheme of things. But compare the actual market of EVE to EVE Metrics and we’re still a long way off having a truly accurate picture. EVE moves quickly- in some markets, from minute to minute orders will be shuffling around and changing price and being bought out.

With Dominion we got a new browser. This means you can now use the full EVE Metrics website ingame, but also (through some Javascript client hook additions) lets us provide a fantastic new tool to help us get an even better picture of the market in EVE.

If you fire up the IGB and head over to the upload suggestions page, you’ll be given a list of 10 items, and a few options for automatic checking. Choosing this option will prompt EVE Metrics for a list of items to check, and will automatically go and look at those items. It’s slow, but it works. In the space of a few hours with one user, we can get data for an entire region across all the items on the market. This is utterly fantastic and we’re really looking forward to the larger volume of data this is bringing to the site.

So, if you’ve got a spare moment, or you need to go AFK for an hour, or you want to help out while you’re mining, or you’re just tired of clicking the next item in the list, install the uploader and visit the page ingame to get started. Every upload counts and helps us build the biggest, best picture of EVE’s market we can manage to produce. Uploads to EVE Metrics are also syndicated to other websites and tools, of course. Your uploads and contribution of time help hundreds of users who use the site, and tens of thousands more who rely on our pricing, history and order APIs for their applications.

Oh, and if you’re a developer, we now have a server status API with all the information you could possibly want on TQ, Sisi and the API servers. It can be found here (docs here). Enjoy!

We have got some things in the works for Dominion, and we hope you’ll find them useful; we’ve not had much time to work on EVE Metrics, and we’re being distracted by another project at the moment, but we’ll have more time to work on EVE Metrics in a few weeks time around Christmas. I’m still evaluating what we’ll spend our time on, though, and we’d like to get more feedback via the feedback button on the site- you can vote for other people’s suggestions, so please do so!

Other than that, not much to report. We’ll have Dominion items loaded into the site by release day so you can start using the site straight away with the new items. We’ve been so far very successful with some performance improvements on the site; this has mostly been tuning our database server and working on improving the performance of queries through better indexes, clustering indexes, and so on. Hopefully you’ll notice this in the form of improved page responsiveness and less ‘slow loading’ pages. Enjoy!

I’ve been busy with reality, by and large. Lots of stuff has been happening in the real world, mostly relating to my current studies, which has removed my spare time and distracted me from things like blog posts. This is going to be a bit of a summary post before I go back to my usual programme of ranting, rambling, mad ideas and neat snippets of code.

Recently I’ve been going to some gigs; first off was Mitch Benn, who is fantastic live and whom I can strongly recommend you go and see if you like comedy, rock/metal and cleverness. And on the cleverness note, I also went to see Jonathan Coulton with Paul & Storm. You probably have heard JoCo’s work before- he wrote Still Alive (don’t watch if you haven’t played Portal yet!), the ending song of Portal. He writes other songs, too, about evil geniuses in love, office romances, geeks, programmers and giant squids. Oh, and robots. Don’t forget robots. The live gig was a great experience- here, have some pictures to go with that…

12►

OK, so that’s about all on the live gigs side of things. Back to EVE and code. I’ve rejoined my old corporation, Vanguard Frontiers, after a brief leave of absence to have a bit of a highsec break. Mostly doing trading in the interim, which has been useful for getting another perspective for EVE Metrics. And hey, neat segway there, yeah?

EVE Metrics has been going well; we’re still mostly rewriting things for performance at the moment. But we have been adding features and improving the site in other ways- we’ve improved the My Metrics functionality no end, and added wallet ledgers, both corporate and personal. We’ve got a lot planned for EVE Metrics but we really need your help to let us do what we want to do- we’re now asking for donations for our new server, and we’d also appreciate it if people used the GTC link on the bottom of all EVE Metrics pages if they need to buy GTCs; you’ll be supporting us with your purchase. Get your friends to help out too with donations and GTC purchases, and we’ll be closer to getting our own hardware sorted out. Myself and Makurid are basically a pair of very poor students, so we really can’t do this without your help. If you want to see EVE Metrics progress faster and perform better, this is the way to make that vision a reality.

Other than that, I’ve been quite slow on the code side of life- mostly, I’ve been fixing bugs, tweaking server software and working on maintainence scripts to keep everything running smoothly. Hopefully in the next few months we’ll have some new toys out for you to play with. Until then, keep giving us feedback on EM and we’ll keep fixing bugs and improving things- we’ve done a lot in the past few weeks, including new history dumps, data freshness display, realtime data feed, and more, including a whole host of performance improvements. Keep the comments coming!

Anyway. We got a lot of fluff, this year. Aside from further elaboration on stuff already announced, there were actually no major announcements made at fanfest. We did have some interesting info about New Eden, CCP’s EVE-Online-Online website. And there was some evidence (gasp!) that CCP were listening to third party developer suggestions at the API roundtable.

There was almost enough minor stuff announced to make it worthwhile. We did get a release date for Dominion – 1st December 2009. But no New Eden with the launch. And knowing CCP we’ll probably not get API changes till a bit after that. What’s really awesome though is that we will be getting new APIs. I’m just hoping they’re useful APIs…

Anyway, while I was sitting at home being mostly bored, I decided I’d had enough pressing F5 on the Twitter search page, and put together a website (ff.mmmetrics.co.uk – it’s down now) to grab EVE fanfest feeds from Twitter and Flickr. This became popular enough within a few hours that we had to rip it off the server and give it it’s own Amazon EC2 virtual server, as it was in danger of crashing ISKsense and EVE Metrics. Doh. A wild success, in any case, for a simple but handy website. What the website did make us realise is how little headroom we have on our current server. We kinda knew that already but it did make the point quite well.

EVE Metrics 2.1 has launched mostly well but we’re still having issues with the API processing code. Makurid has been working hard to pin down the cause of the problems and destroy it while I’ve been fixing up servers and moving sites around, and we’re getting a bit closer to having a complete fix. We’re not there yet, but we will be soon with any luck.

Login and Authentication

• Internal login system changed from restful_authentication to authlogic
• OAuth support added for login, registration and client authentication
• Twitter login support added
• Chromabar password field colour checking support added

New Sections and General Changes

• API keys section added
• My Metrics section is now significantly more useful and contains information relating to your characters with the API enabled
• Corporate section added
• Statistical methods are now mostly implemented using the Gnu Scientific Library for performance, accuracy and additional statistical features
• Statistical methods are now weighted by quantity where appropriate for improved accuracy
• Server-side Ruby environment upgraded to 1.9.1 from 1.8.7 for performance gains
• ‘Contribute’ pages added
• Content which was missing from privacy policy, terms of service and credits pages have been added

Visual/UI changes

• Layout inconsistencies between operating systems, notably Mac OS X and Windows, affecting text fields have been resolved
• A new landing page has been added for 2.1
• Individual order rows now use row background colour cycling in the market
• Font selection on OSX and Linux now uses a Sans-serif font for greater visual consistency
• Visual inconsistencies due to long values distorting fields have been removed
• Times now do not show the timezone for each value, as all values on the site use UTC
• The display of notices, errors and warnings has been visually improved

Uploader and 3rd party APIs

• Upload statistics added to My Metrics
• Upload log added to My Metrics
• Credit system based on formula considering usefulness of uploaded data implemented for uploader rewards
• Top Uploader rankings added to uploader section
• Quantity-weighted values for kurtosis, skew, variance and standard deviation added to price APIs
• Average price is now weighted by quantity

• API statistics now use only the middle 90% of values (top/bottom 5% are ignored).
  This can be overriden with min_quantity=0 to use all data, but using all data may affect the accuracy of statistical values,
  as the calculations used to generate average, kurtosis, skew, variance and standard deviation are not statistically robust.

• Mac and Linux uploaders are not available with the release of EVE Metrics 2.1, but will be making an appearance soon.

EVE Online API support

• API support has been added for market and wallet APIs, plus additional support APIs
• EVE Metrics now supports corporation-level security policies that are based on ingame roles and used to protect corporate data
• API request logging added
• Per-request/feature per-character API security switching for absolute security granularity is included
• Server monitoring support added, including sisi and API servers, plus API functionality
• API server monitoring integrates into API functionality to gracefully degrade service during periods of API downtime

Market

• Changes have been made to the market page generation code to improve performance and to use the Gnu Scientific Library for statistics
• Quantity-weighted average prices are shown for each region, along with deviation from global average
• Descriptions of items are now only partially shown by default, and can be fully shown via a new link added to the description
• Display of values and dates has been adjusted
• Larger figures (>10000) in less critical fields are now shown abbreviated with kISK/mISK/bISK/tISK standard formatting

Trades

• EVE Metrics now has support for inferred and API-generated trades
  • Inferred trades are generated from observed changes in orders between uploads
  • API trades are 100% accurate but only available when traders of an item have the feature enabled
  • Trades are an uploader feature to encourage uploaders to join EVE Metrics
• Trading statistics for last 24 hours added

On a personal note, I’m starting a course in Computer Science at Royal Holloway, University of London in a week’s time. As a result I’ve been running around sorting out accomodation and so on rather than writing blogs. I’ve been spending the rest of my time working (the sort that pays), and the tiny little nooks and crannies of spare time are crammed with work on EVE Metrics 2.

In the past week I also received some donated server hardware; this will be taking residence in my new home and hopefully in the future the cluster will be spending most of it’s CPU cycles doing reports and more complicated econometrics using the EVE Metrics dataset- stuff that takes way too long to do online, but is achievable offline.

Anyway, back to business. Read on for the lowdown on EM2.1, new features, new uploader info and more!

EVE Metrics 2.1, the API release, is a bit off schedule. We were hoping to have it done last weekend, but events have conspired against us. We’re 70% there on the bug tracker but it’s more like 95% in terms of time; the majority of the remaining tasks involve some of the finer points of the upload reward system (which has involved so far hours of Makurid poring over data and projections in MATLAB) and that whole front-end thing. On release, 2.1 will let you view market orders and wallet transactions/journal entries online, and the market orders will be autoloaded into the database.

Trades and an improved uploader

Those of you who are heavy EM users will have known for some time the ‘trading’ tab has been inactive. 2.1 opens that, allowing you to view all our inferred trade data. We’ll be adding in the API-aquired trade data sometime soon after release but it’s not in yet. By inferred data, we mean trades we can infer have occurred by looking at how orders change between uploaded datasets; it’s far from perfect but it’s a good indicator and can be useful.

There’s one other notable thing we have planned for 2.1/2.2- 2.1 will introduce OAuth, and 2.2 will fully implement it including components of the OpenSocial API to provide widgets for both iGoogle and AvatarsUnited. This is part of a partnership between MMMetrics and EnemyUnknown, and we’re very excited to see how we can make the market more of a social tool through these widgets and using social networking technologies. What we will have through API data in EM2 is a lot of useful information about how people trade, allowing us to develop better systems, algorithms and predictions about the market in the future.

What OAuth also means, though, is that we finally have a way to securely link API applications with EVE Metrics. This opens up the possibility of personal API feeds but also gives us a better way to do the uploader. So what we’ll be publishing in the near future is a new uploader, with total support for Mac and Linux as well as Windows. We’re still working out the technicalities and helping to develop some of the technologies involved (libevecache etc) to make the uploader as lightweight and fast as possible while being even more efficient and secure for uploading fantastic data. When we release the new uploader we’ll also be opening the upload protocol specification to let you lot make your own clients if you want to.

Realtime with RabbitMQ, plus feature creep!

One of the most-requested features for EVE Metrics has always been the real-time upload feed. We think email is the wrong way to go about this. It’s not designed for it, it’s overcomplex to parse, it’s a lot of overhead, etc. So we’re going for a more modern solution- a messaging queue based on AMQP and RabbitMQ. We’ll be providing example clients in Ruby and hopefully a few other languages, so if the technicalities escape you there’s no problem. But the point is, it’s fast, resilient, easy to use, and lightweight. Technical details and code samples will come later, of course.

And last but certainly not least, we will be putting new features in. That means trading tools, favourite-based price lists, UI changes to the item display page, and more. As ever, if you want features, hit the ‘feedback’ link on the site and tell us. Not everything will be coming out with 2.1; it’s mostly a framework release, with some very large changes to authentication and lots of new stuff in the market code, plus of course API support. New features will be coming on a more regular basis after that; once we’ve got the basics done, features are often pretty simple in comparison.

Pictures speak a thousand words

Which, incidentally, is roughly the length of this blog post. Speaking of posts- an article I wrote back in 2008 on delayed_job has been published in Rails Magazine issue 4! If you’re interested in background processing, you might want to give it a read.

Anyway, I promised pictures, so here you have the servers, plus some of my latest purchase, a fantastic DT109 headset.

DT109s and The Box

◄ Back

Next ►

Picture 1 of 5

New headset and my latest electronics project