IRIS was written to replace MACIS, a system I bodged up out of necessity. At Insanity, we had a computer failure weeks before we went on air at the start of the first term, and lost all the data- including the entire playout system. Lessons have been learned (I made sure we replaced that machine with a box that had RAID, for starters) since, but we had the unenviable challenge of repopulating a student radio playout system from scratch with little to no staff. Enter MACIS!

MACIS was dumb. It talked to our playout system (PSquared’s Myriad 3.5) via the not-very-documented TCP/IP interface, had a web interface and drop box, and some background processing magic. It was implemented as a Ruby on Rails web application, since we already used Rails and Ruby for various tasks around the station (the website is all Rails, and Ruby was chosen for most scripting tasks because of its user friendliness to people not too familiar with programming. You passed it files, it converted them (the main purpose- Myriad doesn’t support AAC, MP4 or many other formats we were using for ingest), did a basic stab at normalization, and then imported the files. Myriad is slow at importing files- 1-2 minutes per file on average, so we let MACIS distribute workload across all four of our Myriad machines, speeding things up massively.

This was good and got us running, but then term started. We’re a student station and have specialist music shows, who upload their own content to the playout system every week, and new content for new music and chart shows came in regularly. MACIS was used for this, as it did the conversion automatically, saving our presenters loads of time. It also did batch imports quickly and efficiently, which sped things up. However, after a while, we stopped using it, and just provided a simple Ruby dropbox on a shared file server for conversion. MACIS was useful, but too buggy and inflexible. In addition, while it did a better job of getting metadata into Myriad than Myriad managed on its own, it had issues with some formats and material.

What was needed was a system that would let presenters upload content in any format, would sort out the metadata, handle conversion, and fire it off to the playout system for usage.

An example upload showing the log and graphs (huge image)

So, while presenters got back to using Myriad directly, I went back to the drawing board and scrapped MACIS. At this stage we were considering transitioning to the Rivendell open source playout system to replace Myriad, so I decided whatever I was going to make had to support both Myriad and Rivendell, and any other system you could imagine.

I also wanted to solve the loudness problem. Even doing normalization to every track imported, we had huge loudness level differences between some tracks, making life quite tricky for presenters and impacting our on-air sound. Especially given our lack of transmitter processing (we only have a limiter and preemphasis box on the AM system- no AGC or multiband comps), I wanted to do all I could to get everything as perceptually loud as everything else. Enter EBU Recommendation 128 for loudness measurement- with the help of some great libraries (libebur128 in particular), I implemented a simple version of the recommended processing system for loudness normalization, including LRA correction using a compressor. Thus, everything you run through IRIS comes out sounding about the same as everything else in terms of perceptual levels – as much as is possible without impacting the sound. Wish You Were Here is still going to have a quiet bit at the beginning- but IRIS will gently compress the track to make the difference less severe, and will then use R128′s standards to normalize to -23 LUFS.

Next up, user authentication. This was almost an afterthought, but added after talking to people about security. You register an account, and that account is either able to upload content only, upload and review (more on that in a sec), or administrate the system (ie modify users etc). This is done by user groups, which are pretty flexible, and easily adapted for your own usage via a simple permissions file. Uploads are linked to users- users can only see their uploads (unless they have permission to see more than that) and admins can see who owns a specific upload. You can also have emails sent on error conditions being met- so presenters know if a file they uploaded failed to make it to the playout system before they turn up to do their show and wonder where it is.

Metadata was one of the big issues I wanted to solve. Let’s say I have a track from a CD- I’ve ripped it and the ripper has embedded title/artist, maybe album metadata from Gracenote. For a playout system for radio this isn’t perfect- really we want to know the record label, copyright info, and so on. Enter MusicBrainz- a huge collaborative open database of music metadata. With some clever tools, IRIS matches up the track’s metadata to a MusicBrainz identifier and fills in the blanks. For most tracks, it can get everything- including ISRCs, year of release, and so on. This is great for music librarians and makes copyright reconciliation for PRS/PPL much simpler, since you’ve now got all this in a database.

Of course, if we know what the track is, we can do another useful step- especially so for student stations. Using the MusixMatch API service (commercial but free for nonprofits at present), we can get the lyrics to a track. This means we can do a quick once-over for words we don’t want on air (swearing). Of course, this assumes the track isn’t a radio edit. We do a quick check and skip the lyric pass for tracks that look like a radio edit, but if not, the track will be flagged for review.

Additionally, and this is intended specifically for situations where you have no control over the ingest quality that presenters are using to rip CDs or vinyl, tracks are flagged for review if they fail to meet quality restrictions. This can be specified as a function of sample rate and bit rate. This stops people trying to upload 96kbps MP3 at 32k sample rate. We don’t want that in the system. Not now, not ever. All of these parameters can be changed easily and simply by changing a single configuration file and restarting the app.

Overview of the system architecture

Once an upload is flagged for review an administrator or music librarian can review the track and either permanently reject it or approve it (in case of false positive lyric matches or odd uploads where quality restrictions aren’t able to be met).

Everything in IRIS is done through a web interface- uploads, management and monitoring. Every track has its own log of events, allowing administrators to debug and diagnose problems with ease, and giving clear and simple feedback to users. There are convenience functions such as automatic display of R128 loudness graphs pre/post normalization and compression, and display of all metadata available for tracks, plus lyrics if they were found.

The backend to IRIS is all Ruby and Rails, using a simple database server (PostgreSQL recommended) to store everything. Background processing is distributable over multiple computers with shared storage, allowing for CPU-intensive tasks to be spread across multiple machines. Given the R128 metering process includes a fourfold upsampling, this is particularly useful. You can run workers without running the whole web application, allowing you to install copies of the app onto lots of low-cost general purpose machines and have your own distributed ingest processing cluster on even a tight bugdet.

Of course, now you’ve got a track with lots of metadata and some normalized audio in WAV format (archived to FLAC pre-normalization, just in case you need the original audio). Now you need to get it into your playout system. Rivendell is supported, Myriad 3.6 now has dropbox support so you can just tell IRIS to export files to that dropbox in a Myriad-supported format, and you can also just do an export in any format you choose to an arbitrary folder. Export formats supported include FLAC, MP3, WAV, BWF (Broadcast WAVE Format) and AAC. Most of these flavours come with embedded metadata.

IRIS isn’t a perfect system, and it’s not an instant drop-in system; I don’t have the time to maintain it as such. What it is, though, is a flexible and powerful system that any average Linux user can install and have running in hours, and which can be used by any station looking to improve their import process.  The entire project is open source, and can be obtained here- there’s also a bugtracker and wiki with some documentation (unfinished) on it. If you’d like to contribute, feel free- as I’ve stepped out from student radio to work on student television, I’ve not got a huge amount of time to work on it at the moment.

Well, here’s one thing- there’s so many fantastic audio editing, production and routing tools for Linux. And even better, there’s loads that can be run on a command line, without needing a desktop computer- so we can do audio processing on our server hardware, of which we’ve got plenty. But they’re a bit hard to find so I figured I’d just do a quick summary of the top few tools we’re using that you might’ve heard of, and a few you probably haven’t. So, in no particular order…

sox

sox is an awesome little tool which bills itself as the swiss-army knife of audio tools. It’s not far off, either. While it’s not so great on the format support side of things, this little thing can do pretty much any common operation you want it to do (and a few not-so-common ones). We’re using it to automatically split up hour long MP3 files into half-hour chunks, as well as using it for trimming silence off audio and performing companding on audio before things get imported to our playout system. We use ffmpeg to get things into WAV for it, though.

ffmpeg/ffprobe

ffmpeg is one of those legendary tools. This thing will read anything and lets you seamlessly turn it into anything else. I’ve thrown the nastiest, strangest and most obscure files at this thing and had it spit out completely usable WAVs time after time. Learn to love this puppy. Oh, and it’s fast, too. ffprobe is it’s companion tool, able to quickly give you the rundown on any file and return metadata.

exiftool

If ffmpeg is the king of reading audio data, exiftool is the queen of metadata. While it’s aimed mostly at image files, it’s also incredibly good at pulling useful metadata out of various formats stored in audio files- MP3 ID3 tags, AAC metadata, you name it. We use this to get metadata out of pretty much anything and normalize it into MP3 ID3 tags for our playout system, which can’t recognize much unless you spoon-feed it in the right format.

mp3info

A blissfully simple little tool for querying and writing MP3 ID3 tags. Very handy, fast, and simple. What’s not to like?

JACK

JACK, the Jack Audio Connection Kit is yet another one of the more well-known tools. Capable of routing audio signals inside a virtual patchbay it’s very powerful, and we use it extensively all over the place. NetJACK is an interesting development that is really starting to take shape now, and is one to watch in the coming years. JACK is supported by tons of stuff, including a lot of the software aimed at the broadcast and professional audio crowd.

jack_meter

jack_meter is a really nice little script that can do command-line monitoring of levels on a given set of JACK ports. It’ll draw you a nice little meter, or spit out numerical information. We’ve actually now got this hooked up to a Nagios script to check to ensure our self-operated studio is in the right ballpark levels-wise and isn’t driving the net TX system overly hard- works a treat!

rotter

Not so handy for _everyone_, but certainly has its place. Rotter is a recording of transmission tool, intended for radio stations and other people who want to make a continuous recording of an audio source (using JACK again) and store it in files split up based on a fairly flexible partitioning system. We’re required to do regulatory logging and such, and this is used for both that and for presenters who want to make podcasts of their shows – and this is a darn sight cheaper (and better!) than the £1500+ commercial products to do this.

audacity

Okay, not a command-line tool, but it’s by far one of the best audio editors out there right now, in my opinion matched only by Pro Tools for just doing quick and simple composition and editing of assets. And since it’s crossplatform and free, the presenters can take it home with them and work there instead of having to use our production room to do simple edits!

I’m sure I’ll remember a few after I post this but that’s certainly a good start. Do you have any tools you couldn’t live without for audio processing?

So, what’s the best way to approach all this? With a new set of equipment to monitor, and a lot of old equipment still unmonitored (our desk’s GPIO and the AM transmission gear, for instance), hardware GPIO lines are order of the day. Mostly, this can be done with relays, pull-up resistors, optoisolators and a steady hand with a soldering iron (bolted onto an Arduino – see my last post). This is all well and good – but now you need to get that into a sensible format, record it, log it, and take action depending on the state (emails, indicators in the studio, etc). Which is where Nagios comes in.

Nagios is scarily flexible. It’s also extremely lightweight for small setups, and very easy to configure, with excellent documentation. And since it’s widely used in larger IT outfits, there’s an infinitude of addons, plugins, and so on to expand it. It’s at heart a monitoring program, with support for active checks (where Nagios polls the host or service), and passive checks (where the host/service reports in to Nagios itself). Once you’ve gotten things inside Nagios you can use a flexible set of rules for notifications, and use some of the many addons (we use check_mk’s livestatus) to get data out into other programs. We use CoffeeSaint on dashboards to provide single heads-up displays (with camera displays combined), and we’ll soon have Nagios integrated into the Insanity website’s admin backend so that staff can see a general overview of the system alongside all the other vital information like listening figures and cameras from wherever they are in the world.

At the moment because we’re poor, we don’t have ethernet shields for our Arduinos, so we have an Arduino Mega and an Arduino doing two things- driving a matrix LED display, and monitoring our silence detector. The latter is done by the Mega, which will later be expanded with ethernet and various boards to break out the DB25/DB15/DB9 connectors from the mixing desk into Arduino-compatible levels. For now though it’s just hooked up on USB. The Mega also drives a set of Bliptronics LED RGB modules- these are currently spread along the top of the mixing desk in the studio where presenters can see it.

The Mega is set up to simply watch the two inputs of the silence detector and their alarm states, and to display one of four messages describing that state to serial over USB, where it’s picked up by a small Ruby script, which then sends a NSCA (Nagios Status Check Acceptor) packet to the NSCA daemon on the monitoring box. This packet contains the information about the silence detector in a format that maps to the host and services described in Nagios. This means we get very rapid updates on the silence detector compared to active monitoring. Even more rapid is the LED strip, which is controlled wholly in C. If the station’s output gets switched to the backup source because the silence detector thinks the studio is too quiet, then the strip goes red (normally green) – and if a presenter is in there, he/she now knows that there’s a problem, they’re no longer going out on air, and they can either work out the problem by themselves (previously the staff wouldn’t know that this had even happened without going out and listening to the radio) or call for support.

It’s one thing to have standards, and another thing entirely to monitor your performance and hold yourself to them; and when there’s a problem with your metrics, then you can fix it. We’ve dramatically improved the actual quality of broadcast radio that Insanity puts out in the last year; partly it’s a human change, we’ve had some great presenters join us this year and a very determined board, but there’s an element of technical change and a lot of instant feedback for presenters to help them; and this has really helped people spot problems where previously they wouldn’t even have known they were there.