The lack of fault management and good process management is a problem. Quite often workers would not be used for an hour or two, and MySQL would drop the connection. You’d expect AM to bottle out or reconnect at that point, but instead it just silently carries on, grabbing messages into a black hole. Not ideal. Getting the processes working well was difficult too- I ended up with a few shellscripts to manage them, running on a cronjob to restart them every hour to avoid the MySQL timeout problem. Not exactly ideal.

So, I’m moving the processing on EVE Metrics and Nexus (Which previously used AM/ActiveMQ for alert sending, amongst other things like API crawling) to a pure-Ruby solution from the guys who did Shopify, delayed_job. As well as being pure-Ruby (and thus not needing any external servers like ActiveMQ, simplifying install/setup) it’s got real provisions for locking, persistence even after working, and fault management. And it’s all ActiveRecord based, making an admin interface a cinch to add on. This should help loads, as debugging the AM setup was a bit hit and miss, and there are still bugs in there- see the lack of updates on the Orca/PLEX pages!

So, hopefully by the end of the weekend I’ll have Nexus ported, EVE Metrics ported, and some more stable queues….

Off to RubyManor in London on Saturday, if Boris doesn’t close Bakerloo as well as District/Circle. Looking forward to George’s talk on Nanite, which I was looking at as an alternative to AM/ActiveMQ before I stumbled upon delayed_job- Nanite looks very cool, though for different jobs, and it’s worth looking at if you haven’t already.