Monitoring radio, and the joys of realtime feedback

Monitoring and logging of broadcast systems is often overlooked in smaller setups like university radio stations. Which is a shame. If we’d had monitoring in place last year at Insanity, we’d have known exactly how much dead air went out. Now, we’ve got a much better setup. It’s cheap(ish), a little bit homebrew, but there’s nothing fundamentally wrong about that. It works, and I know we’ve had 53m 34s of dead air in the last year – and it’s all accounted for. We also know when we were running on backup audio – showing that we’d have been on dead air for a goodly 1 day, 17 hours, if it weren’t for our silence detector and backup playout unit.

So, what’s the best way to approach all this? With a new set of equipment to monitor, and a lot of old equipment still unmonitored (our desk’s GPIO and the AM transmission gear, for instance), hardware GPIO lines are order of the day. Mostly, this can be done with relays, pull-up resistors, optoisolators and a steady hand with a soldering iron (bolted onto an Arduino – see my last post). This is all well and good – but now you need to get that into a sensible format, record it, log it, and take action depending on the state (emails, indicators in the studio, etc). Which is where Nagios comes in.

Nagios is scarily flexible. It’s also extremely lightweight for small setups, and very easy to configure, with excellent documentation. And since it’s widely used in larger IT outfits, there’s an infinitude of addons, plugins, and so on to expand it. It’s at heart a monitoring program, with support for active checks (where Nagios polls the host or service), and passive checks (where the host/service reports in to Nagios itself). Once you’ve gotten things inside Nagios you can use a flexible set of rules for notifications, and use some of the many addons (we use check_mk’s livestatus) to get data out into other programs. We use CoffeeSaint on dashboards to provide single heads-up displays (with camera displays combined), and we’ll soon have Nagios integrated into the Insanity website’s admin backend so that staff can see a general overview of the system alongside all the other vital information like listening figures and cameras from wherever they are in the world.

At the moment because we’re poor, we don’t have ethernet shields for our Arduinos, so we have an Arduino Mega and an Arduino doing two things- driving a matrix LED display, and monitoring our silence detector. The latter is done by the Mega, which will later be expanded with ethernet and various boards to break out the DB25/DB15/DB9 connectors from the mixing desk into Arduino-compatible levels. For now though it’s just hooked up on USB. The Mega also drives a set of Bliptronics LED RGB modules- these are currently spread along the top of the mixing desk in the studio where presenters can see it.

The Mega is set up to simply watch the two inputs of the silence detector and their alarm states, and to display one of four messages describing that state to serial over USB, where it’s picked up by a small Ruby script, which then sends a NSCA (Nagios Status Check Acceptor) packet to the NSCA daemon on the monitoring box. This packet contains the information about the silence detector in a format that maps to the host and services described in Nagios. This means we get very rapid updates on the silence detector compared to active monitoring. Even more rapid is the LED strip, which is controlled wholly in C. If the station’s output gets switched to the backup source because the silence detector thinks the studio is too quiet, then the strip goes red (normally green) – and if a presenter is in there, he/she now knows that there’s a problem, they’re no longer going out on air, and they can either work out the problem by themselves (previously the staff wouldn’t know that this had even happened without going out and listening to the radio) or call for support.

It’s one thing to have standards, and another thing entirely to monitor your performance and hold yourself to them; and when there’s a problem with your metrics, then you can fix it. We’ve dramatically improved the actual quality of broadcast radio that Insanity puts out in the last year; partly it’s a human change, we’ve had some great presenters join us this year and a very determined board, but there’s an element of technical change and a lot of instant feedback for presenters to help them; and this has really helped people spot problems where previously they wouldn’t even have known they were there.

The £800 microswitch… and some other things.

OK, so fixing the other projector (an Epson EMP-50) was a little bit trickier than the first one.

Problem: It didn’t power up. No lights. Nothing. Nada. OK, I figured that was a power supply issue. Stripped it down to it’s component parts (left the optical engine and ballast), spotted the bad connection- this projector had been damaged by being dropped, and the whole frame had warped a bit. This connection, between the IEC connector on the side for power and the horizontal power supply board, had fallen out entirely. Reconnected and resoldered, and subsequently reassembled- and I get a ‘circuit error’ when trying to power up.

So, back to the disassembly. Took out the ballast unit on a hunch and spotted a microswitch which a service manual claims detects when the lamp housing is open. Unfortunately, this doesn’t work too well when the frame’s a bit bent, so I soldered over that on the mainboard. Reassembled, again.

Getting a little bored with the constant dis/reassembly at this point- and it still doesn’t start. Bulb struck, which was a great morale booster, but it then shut down and burned it’s fans at maximum for 5 minutes before turning off. Not too useful.

Poked around a bit and considered these symptoms were similar to what my other projector had- lo and behold, the fan on the smoothing unit is jammed! One of the support struts for the fan was snapped, and a quick spot of superglue fixed that in place and let it spin freely again. Reassembled, and bam! Working projector #2. I’ll post pics sometime.

All in all that’s two projectors for £0 which I’ve gotten working again with long remaining bulb lifes in each. Total resources required: Three screwdrivers, a Torx driver, three forceps (Seriously- best tool I’ve ever had for electronics work), some narrownose pliers, antistatic gloves and static control kit (to avoid frying the mainboard), soldering iron (Got a new Weller temperature controlled iron, which is doing great), approximately 4mm of solder, a dab of Loctite superglue, an IEC power lead and some spare time. Oh, and lots of caffeine.

If I get a chance tomorrow I’ll see about throwing a post up about some of the under-the-hood changes I’ve been making and have planned for EVE Metrics and other sites. I’m looking more seriously at a single-sign-on solution between all the MMMetrics sites (ISKsense, EM, AccVIEW)- but the question is, if I do a SSO solution why not make that a global SSO solution with OpenID support for pluging in other sites? More to the point, allowing centralised API key management between applications that need it (EVE Metrics would use this to store/retrieve keys) would be great- give application your MMMetrics ID, MMMetrics prompts you to give access to your account data to that site, optionally letting you provide one or more API keys to the site. Kinda like a badass Gatecamper 2 that doesn’t suck so badly. Thoughts?

The £800+ screw

I’m currently very happy. And yes, I will write a real blog post soon. It’ll have chaos, misery, confusion and intruige, I promise.

I just got two broken projectors for free. This is so I can finally attempt my multitouch display project and have a go with some tangible interfaces. Of course, they were broken, hence the lovely £0 pricetag.

Exhibit A is a Sanyo PLC-SU20E, a fairly old but decent projector. It’s a solid workhorse, capable of 1300x760ish 16:9 resolution. I get it home, plug it in and watch- the projector turns on, strikes the bulb, then shuts down. A little Googling reveals that fan faults can trigger the PSU to shut down the projector, and sure enough a quick inspection of the back fans as I power it up shows one fan not spinning up.

At this point I got somewhat scared by leaning on everything too hard, then threw caution to the wind and gave it a good shove; voila, fan tray pops out, and fan is shown to be siezed up. A quick prod, and a screw falls out of the fan blades, after which the fan rotates freely. Slapped it all back together and it powers up, runs fine and looks great. £800+ of projector for half an hour’s work and £0.

So of course next step is the projector mount and frame for the multitouch table, along with the associated gubbins to make it all play nicely. We’ll have to see how well that all works out but it shouldn’t be hard. I’ll throw some pictures up sometime soon.