OK, so it’s been a while since I posted a blog update. I’ve been busy- Band of Brothers declared war against the Northern Coalition so I’ve gotten swept up in the fighting and commanding. EVE Metrics has progressed well with constant work, and my holiday in Scotland was a success on all counts. And now it’s August. Blimey.
Anyway, back to Rails I go. I’ve been pondering the usual problems. REST is great for doing stuff on publicly accessible models- news posts and the like, for example. And scaffolds/make_resourceful can do that really, really well. But what about when you want to cater for private resources?
Continue Reading…
Posted by James Harrison on Aug 03, 2008. 1 comments.
For EVE Metrics we ran into some scalability and other issues right away with the market. Uploading market history data was causing some major problems, so we had to break out the big guns in terms of distributed workers right away.
Basically, whenever our upload client throws a chunk of data (Say, 100 different orders for a type and the price history for the last month), we’re running at a minimum 128 SQL insert queries- this all adds up, even with bulk inserts. Then you have to factor in delta calculations for histories, which adds another two SQL queries for each history value- again, more queries, more time. As the complexity increased, we ended up with around 3 seconds to run all the queries for larger logfiles, which is no good in reality- the uploader can sometimes be trying to push 3 to 5 files at a time up to the cluster, and this hangs half the app servers immediately for 3 seconds or so.
So, we now avoid MySQL entirely, and use some enterprise-grade messaging to handle our distributed processing of logs, entirely asynchronously from the upload controllers and Mongrel. Details after the break.
Continue Reading…
Posted by James Harrison on Jul 04, 2008. 0 comments.
This is a quick post (I’m going to be busy today and tomorrow) to throw out a request for testers for the EVE Metrics desktop application.
We’re looking for industrialists and people out in 0.0/lowsec PvPing to use the beta client to collect price data for the development of the EVE Metrics market system.
If you’re interested, please send me an email- the details are on the about page – and I’ll send you a link to get the beta copy of the software from.
Please note that the current program is Windows-only, although we hope to provide Linux and Mac OS applications in the near future. This updater uses both logfile and cache reading methods, so simply browsing the market is all you need to do (Although pressing ‘export log’ in the market a few times wouldn’t hurt), with the program running in the background. This is to test the scalability of the EVE Metrics website’s upload API and to confirm that we can handle the amounts of data we’re looking at. More on that in another post I’m about to write…
Continue Reading…
Posted by James Harrison on Jul 04, 2008. 0 comments.
