This is kinda a documentation post to describe the move and how you, too, can migrate your databases with a relative amount of sanity remaining intact, unlike me.

Export from MySQL

This is the easy bit.

mysqldump -B yourdb --compatible=postgresql --skip-opt --quick --extended-insert -t > dump.sql

Easy. Except that –compatible=postgresql doesn’t help much.

Working around MySQL’s crap

OK, so now we have some issues to deal with. Let’s get the simple ones out of the way first. Fire up vi and run these:

:%s/,'/,E'/g
:%s/\\0/0/g

That does proper string escaping and removes null bytes, which PostgreSQL can’t handle as it uses C-style string termination internally. You may face consistency issues if you’re going from, say, LATIN1 to UTF8 in your migration; if that’s the case, add this to the top of your dumpfile:

SET CLIENT_ENCODING TO 'LATIN1';

Do you use booleans in MySQL? Heavily? You’re going to want to stab a MySQL Core Developer in a moment, so make sure the knives are somewhere you can’t find them when you’re angry. I’ll wait.

OK. MySQL doesn’t have a boolean. It’s actually TINYINT(1), with 1 being a true and 0 being a false. Which makes a sort of sense. But MySQL dump doesn’t export ‘true’ or ‘false’, it exports 1s and 0s. PostgreSQL expects true or false values, and won’t cast integers to booleans. Welcome to hell.

The easiest(!) solution myself, Makurid and the combined efforts of #mysql and #postgresql on Freenode could come up with was to set the column types in PostgreSQL to be character varying, load the dataset, rename the columns as c_columnname, make a new column which is the boolean column columnname and run the following SQL:

UPDATE table SET columnname=(CASE WHEN c_columnname::int=1 THEN true ELSE false END)

Repeat for each column, in each table. Once done, you’re good. Pat yourself on the back and go get the knives out again.

Importing back into PostgreSQL

Now you’re on the home straight.

However, one note. If you’re like me and use an insanely long password for the database user (more than 100 characters), you’ve got a problem! psql accepts passwords as a hidden entry field, but it silently cuts off at around 48 characters. There’s a workaround, though, and this is the command you’d use to import back in:

PGPASSWORD=longpasshere psql -U username database < dump.sql

The PGPASSWORD envvar passes the password in directly, no matter how long. Problem solved! You should now have a database full of data. You may also now have some issues with sequences not being correctly set. For each table:

ALTER SEQUENCE table_id_seq RESTART WITH (SELECT MAX(id) FROM table);

This will set the current value of the sequence to be the current maximum ID in that table, meaning the next assigned id will be that, plus one. Problem solved, again! You now have a database ready to face the world. Feel free to get the knives out again and find that MySQL core developer.

I’ve spent most of my time working on EVE Metrics. There’s some very cool, very powerful changes coming up soon; early Feb saw the introduction of much more accurate prices and indexes, with a newly improved algorithm for calculating the average prices of items. But even better is some of the new stuff coming in the next few weeks- notably the implementation of a fully-fledged OLAP warehouse for EVE Metrics, which will open up some very awesome possibilities in the long run.

Also under the scalpel this month has been the API system. EVE Metrics will support full and limited keys when it comes to the API. However, there will be a quirk; if you want to make use of other people’s API data, for example to see more detailed market analysis with transactions hooked in and so on, then you’ll have to share your data. This means if you want to benefit from other people’s data, you must reciprocate and share your data for the benefit of others. Your data will, in all cases, be used for global averages, but entirely anonymously; for example, the number of transactions per day on a given item may include data from your API, but nobody would know it. This system will hopefully encourage users to share data more often than hide it. I’m planning to make this an opt-out system, with the choice to opt-out given on the API key page as part of the form. It’ll be really hard to miss, and those who are paranoid or wish to hide their activity completely can check the box to opt out.

I released accVIEW a few days ago, and it’s had quite rapid takeup from corporations. It’s a service that lets you perform background checks on prospective new members to your corporation- the basic tool lets you view skills, characters on an account, and various bits of information like their corp details, CEO, and so on. The premium version (For the low cost of 150mISK) lets you see the applicant’s wallet journal (with tools to show suspcious transactions and filter the results), as well as their recent kills/losses.

EVE’s M10 expansion, Apocrypha, should be awesome. Tech 3 is going to be great- lots of people complain about the skill loss and the fact that it’ll make FCing a nightmare. Well, no. The skill loss makes sense, and provides an interesting new dynamic to EVE. FCing- well, those who complain about T3 making FCing impossible are evidently not up to scratch as FCs. It’ll give FCs a real change and challenge for the first time in years. The wormhole stuff will be an interesting thing to watch pan out- there’s lots ot potential there, and it could end up being a lot of fun…

Nexus is progressing slowly but surely; the large amount of data we’re gleaning via an API-scraping installation for Sc0rched Earth is helping no end with development, and we’re busy tidying things up behind the scenes and refining some of the interface to make more sense. Once I’ve gotten ActiveWarehouse’s ETL library working properly, my next step will be to break out Photoshop and my text editor- EVE Metrics, ISKsense, maybe accVIEW, and the new MMMetrics site (Which will be launched soon) are all going under the knife and getting a serious facelift. And then it’s on to even more awesome stuff for EVE Metrics!

Over the next few weeks I’ll be reskinning EVE Metrics, ISKsense, and various other things to have a much much nicer theme and one that is consistent across websites, giving each site their own visual take on the theme. Those sites will also be getting some much-needed optimisation, graphics, and new layouts.

Nexus is the other thing this brief post will touch on. Myself and flexd, the other developer working on the project and new member of VAF, have decided that open-sourcing the whole thing would be mad. It’s hideously complex to set up in places (though much easier than it used to be), and in any case a lot of the functionality Nexus provides is heavily aimed at alliances rather than corporations. We’ll still be offering Nexus to some alliances on an ask-and-we-might-let-you basis, but we will be forking Nexus, stripping it back to the killboard and fittings components, and releasing that as open source on Github.

That means if you’re an average corp you won’t get things like the tactical/intel overviews and IGB components, and all the alliance-level member managemet, capfleet management, and so on will be out. Because 99% of people don’t need it, to be honest. By keeping things more limited we’re keeping it much more hackable, more secure, easier to maintain and easier to extend, which means we’ll get it out of the door much quicker than we would have if we’d kept it all bundled together. So we’re looking at somewhere early-to-mid 2009 as our first release.

I think that just about covers it. Oh, other than to say happy holidays and a happy new year in advance, so I don’t forget as I obviously will do.