IRIS was written to replace MACIS, a system I bodged up out of necessity. At Insanity, we had a computer failure weeks before we went on air at the start of the first term, and lost all the data- including the entire playout system. Lessons have been learned (I made sure we replaced that machine with a box that had RAID, for starters) since, but we had the unenviable challenge of repopulating a student radio playout system from scratch with little to no staff. Enter MACIS!

MACIS was dumb. It talked to our playout system (PSquared’s Myriad 3.5) via the not-very-documented TCP/IP interface, had a web interface and drop box, and some background processing magic. It was implemented as a Ruby on Rails web application, since we already used Rails and Ruby for various tasks around the station (the website is all Rails, and Ruby was chosen for most scripting tasks because of its user friendliness to people not too familiar with programming. You passed it files, it converted them (the main purpose- Myriad doesn’t support AAC, MP4 or many other formats we were using for ingest), did a basic stab at normalization, and then imported the files. Myriad is slow at importing files- 1-2 minutes per file on average, so we let MACIS distribute workload across all four of our Myriad machines, speeding things up massively.

This was good and got us running, but then term started. We’re a student station and have specialist music shows, who upload their own content to the playout system every week, and new content for new music and chart shows came in regularly. MACIS was used for this, as it did the conversion automatically, saving our presenters loads of time. It also did batch imports quickly and efficiently, which sped things up. However, after a while, we stopped using it, and just provided a simple Ruby dropbox on a shared file server for conversion. MACIS was useful, but too buggy and inflexible. In addition, while it did a better job of getting metadata into Myriad than Myriad managed on its own, it had issues with some formats and material.

What was needed was a system that would let presenters upload content in any format, would sort out the metadata, handle conversion, and fire it off to the playout system for usage.

An example upload showing the log and graphs (huge image)

So, while presenters got back to using Myriad directly, I went back to the drawing board and scrapped MACIS. At this stage we were considering transitioning to the Rivendell open source playout system to replace Myriad, so I decided whatever I was going to make had to support both Myriad and Rivendell, and any other system you could imagine.

I also wanted to solve the loudness problem. Even doing normalization to every track imported, we had huge loudness level differences between some tracks, making life quite tricky for presenters and impacting our on-air sound. Especially given our lack of transmitter processing (we only have a limiter and preemphasis box on the AM system- no AGC or multiband comps), I wanted to do all I could to get everything as perceptually loud as everything else. Enter EBU Recommendation 128 for loudness measurement- with the help of some great libraries (libebur128 in particular), I implemented a simple version of the recommended processing system for loudness normalization, including LRA correction using a compressor. Thus, everything you run through IRIS comes out sounding about the same as everything else in terms of perceptual levels – as much as is possible without impacting the sound. Wish You Were Here is still going to have a quiet bit at the beginning- but IRIS will gently compress the track to make the difference less severe, and will then use R128′s standards to normalize to -23 LUFS.

Next up, user authentication. This was almost an afterthought, but added after talking to people about security. You register an account, and that account is either able to upload content only, upload and review (more on that in a sec), or administrate the system (ie modify users etc). This is done by user groups, which are pretty flexible, and easily adapted for your own usage via a simple permissions file. Uploads are linked to users- users can only see their uploads (unless they have permission to see more than that) and admins can see who owns a specific upload. You can also have emails sent on error conditions being met- so presenters know if a file they uploaded failed to make it to the playout system before they turn up to do their show and wonder where it is.

Metadata was one of the big issues I wanted to solve. Let’s say I have a track from a CD- I’ve ripped it and the ripper has embedded title/artist, maybe album metadata from Gracenote. For a playout system for radio this isn’t perfect- really we want to know the record label, copyright info, and so on. Enter MusicBrainz- a huge collaborative open database of music metadata. With some clever tools, IRIS matches up the track’s metadata to a MusicBrainz identifier and fills in the blanks. For most tracks, it can get everything- including ISRCs, year of release, and so on. This is great for music librarians and makes copyright reconciliation for PRS/PPL much simpler, since you’ve now got all this in a database.

Of course, if we know what the track is, we can do another useful step- especially so for student stations. Using the MusixMatch API service (commercial but free for nonprofits at present), we can get the lyrics to a track. This means we can do a quick once-over for words we don’t want on air (swearing). Of course, this assumes the track isn’t a radio edit. We do a quick check and skip the lyric pass for tracks that look like a radio edit, but if not, the track will be flagged for review.

Additionally, and this is intended specifically for situations where you have no control over the ingest quality that presenters are using to rip CDs or vinyl, tracks are flagged for review if they fail to meet quality restrictions. This can be specified as a function of sample rate and bit rate. This stops people trying to upload 96kbps MP3 at 32k sample rate. We don’t want that in the system. Not now, not ever. All of these parameters can be changed easily and simply by changing a single configuration file and restarting the app.

Overview of the system architecture

Once an upload is flagged for review an administrator or music librarian can review the track and either permanently reject it or approve it (in case of false positive lyric matches or odd uploads where quality restrictions aren’t able to be met).

Everything in IRIS is done through a web interface- uploads, management and monitoring. Every track has its own log of events, allowing administrators to debug and diagnose problems with ease, and giving clear and simple feedback to users. There are convenience functions such as automatic display of R128 loudness graphs pre/post normalization and compression, and display of all metadata available for tracks, plus lyrics if they were found.

The backend to IRIS is all Ruby and Rails, using a simple database server (PostgreSQL recommended) to store everything. Background processing is distributable over multiple computers with shared storage, allowing for CPU-intensive tasks to be spread across multiple machines. Given the R128 metering process includes a fourfold upsampling, this is particularly useful. You can run workers without running the whole web application, allowing you to install copies of the app onto lots of low-cost general purpose machines and have your own distributed ingest processing cluster on even a tight bugdet.

Of course, now you’ve got a track with lots of metadata and some normalized audio in WAV format (archived to FLAC pre-normalization, just in case you need the original audio). Now you need to get it into your playout system. Rivendell is supported, Myriad 3.6 now has dropbox support so you can just tell IRIS to export files to that dropbox in a Myriad-supported format, and you can also just do an export in any format you choose to an arbitrary folder. Export formats supported include FLAC, MP3, WAV, BWF (Broadcast WAVE Format) and AAC. Most of these flavours come with embedded metadata.

IRIS isn’t a perfect system, and it’s not an instant drop-in system; I don’t have the time to maintain it as such. What it is, though, is a flexible and powerful system that any average Linux user can install and have running in hours, and which can be used by any station looking to improve their import process.  The entire project is open source, and can be obtained here- there’s also a bugtracker and wiki with some documentation (unfinished) on it. If you’d like to contribute, feel free- as I’ve stepped out from student radio to work on student television, I’ve not got a huge amount of time to work on it at the moment.

Trouble is, there is literally no budget (yet, at least), but we wanted to cover events during the Fresher’s Week here at Royal Holloway. So, I spent a week putting together our own YouTube+UStream. This turned out to be surprisingly complex purely in terms of lack of good contiguous information and documentation. Once the parts are all together, it’s actually remarkably simple.

First, let’s examine the goals of the station. From there, we can work out what we need to have in order to achieve this.So, we wanted:

• Live coverage of some events (optionally while recording)
• Live roaming camera capability
• Publishing of recorded content
• Support for low bitrate devices/connections and as many target devices as possible
• Support for up to quite high qualities for desktops and laptops with solid connections

In terms of equipment to pull this off, we had:

• A single laptop, obtained off eBay for £130 (a HP nw8200 mobile workstation, which has a 2Ghz Celeron D processor, and which I upgraded to 2GB of DDR2 RAM)
• A camera, borrowed from our Media Arts department (a Sony PD-150 DV camera)
• A 4-6 pin firewire cable and a 6-to-4 adaptor (allowing us to get the DV feed from the camera into the laptop. A 4-4 cable would have been better, but we didn’t have one)
• A Shure SM58 microphone, an AKG C535EB, both lent to us by people, plus some XLR cable
• A server rented in a Maidenhead datacenter with oodles of bandwidth per month (terabytes) on a 100Mbps pipe
• Wifi coverage in the areas we were shooting in, shared with everyone else in the area (lots of people) with internet connectivity

So, not a lot, especially by professional broadcast standards. But we’ve got enough! Specifically we’ve got a way to capture video, a way to communicate that video to a laptop, and we have the makings of a distribution system.
Onwards, then. First stop was to get the laptop ready- for this we used Ubuntu 11.04 (in the Ubuntu Classic session, of course- never will understand how people use Unity), and installed the latest versions of ffmpeg, libtheora, libvpx and libx264 as detailed in this smashing post here. But in addition to that post we install the librtmp-dev package and enable it when building ffmpeg with –enable-rtmp. This will let us encode RTMP streams directly with ffmpeg. That post installs everything as packages, which is great- just make sure you uninstall the system versions first.
Now our laptop is ready. Let’s crack on – we’ve now got a way to encode the video but we need to be able to fetch it from the camera- dvgrab comes into play here, so install that (sudo apt-get install dvgrab). This lets us capture DV video as an AVI.

Let’s just step back and consider our workflow for a moment. We take DV video out of the camera in realtime, feed that into ffmpeg. Then we use ffmpeg to encode the video and audio (because raw AVI is quite large), and encapsulate it in a RTMP stream. The RTMP stream then gets distributed to RTMP clients, such as Flash player (Flowplayer, jwplayer, etc). This is our live workflow. For recorded content we’re going to do roughly the same, but instead of encoding RTMP, we have ffmpeg turn our recorded AVI file into an encoded FLV file, which our RTMP distribution server will handle streaming over RTMP for us. Piece of cake.
Right. That RTMP server is the next thing- we’ll use crtmpserver, as it’s both open source and very lightweight. Simply visit the site, follow the install instructions, and for live streaming, configure the server such that you can send it a RTMP feed (InboundRtmp). You also want the flvplayback application set up. This is where it gets fun! At this point you should be able to go get Flowplayer (or whichever RTMP-supporting Flash player/other client you want to use) and configure that.

So here’s the tricky bits- making ffmpeg do the right thing. First, let’s consider a script which will encode our recorded videos. We pass this script one argument, the filename to encode, and off it goes, making bitrate versions for 400k, 800k, 1200k and 1600k to be switched between by Flowplayer or similar, written to FLV containers so crtmpd can read them:

#!/bin/bash
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 400k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 450k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 96k -ar 44100 -s 360x288 -f flv "`echo "$1" | sed -e 's/\..../\-400k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 800k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 850k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 128k -ar 44100 -f flv "`echo "$1" | sed -e 's/\..../\-800k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 1200k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 1300k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 192k -ar 44100 -f flv "`echo "$1" | sed -e 's/\..../\-1200k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 1600k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 1800k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 192k -ar 44100 -f flv "`echo "$1" | sed -e 's/\..../\-1600k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";

Okay, so that’s recorded video. What about live, though? Well, for live we’re going to do a simpler encode, because we’re trying to do this in realtime on a slow laptop (the above encodes take a long time. An hour of video came out at about 6 hours of encoding). We’re also using a lower audio bitrate, using the same sample rate as the camera (to save us having to do sample rate conversion), and we’re using a basic preset from ffmpeg. We’re also using the AAC codec as it’s performance-equivalent to MP3 but gives a much better quality for the bitrate, at the cost of some interoperability.

#!/bin/bash
dvgrab -f dv1 -noavc - | ffmpeg -i - -re -deinterlace -s 360x288 -acodec libfaac -ar 48000 -ac 1 -ab 64k -vcodec libx264 -vpre normal -f flv rtmp://yourstreamserver/live/streamName

Note also the dvgrab command writes to -, the standard output, and does not control the camera (-noavc). This means we can pipe the output directly to ffmpeg (which reads stdin with -i -) without messing around with any silly intermediary files. Fast and efficient! Don’t forget that -, though, or you’ll spend hours working out why it doesn’t work like I did. Doh. We also drop the resolution by half which allows us to run things a whole lot faster without a huge quality drop. We don’t specify a bitrate to target, and elect to let libx264 figure out what it needs (combined with -vpre normal to say “We don’t want anything too fancy”). This got around 400-600kbps in practice, which is quite reasonable for most users. It would be fairly trivial to reencode on the server a low bitrate version if desired using ffmpeg, just taking input from RTMP and outputting to another RTMP target, though delays must be considered when doing this. Also worth noting is the ‘-ac 1′ command- this tells ffmpeg to downmix the audio to one channel. On our camera we have two channels, one from a camera-mounted electret microphone (not very good) and one from the handheld mic (an SM58 most of the time); we just sum these and mix on the camera. When recording we get both channels individually on the tape and can mix properly afterwards.

This live encode command is actually fast enough to do realtime streaming from what amounts to a very reasonable laptop. Any modern bit of kit should be able to do similar with ease. For the Fresher’s Fayre, we had the laptop slung in a shoulder satchel on the same arm as the camera, with a firewire lead going into the bag from the camera. Taking some care to leave the air vents able to ventilate, we had a smashing little portable rig at this point which would let us record direct to tape on the camera while also playing out live. Admittedly only for an hour till the battery on the laptop died, but for £130 of laptop and borrowed gear for the rest, it’s not half bad.
The downsides so far I’ve noticed:

• No simple way to fall back on a static file when a stream is unavailable
• iPhone/iPad requires RTSP for streaming; crtmpserver will produce RTSP, but I’ve not got an iPad/iPhone to test with so can’t figure this out. If someone wants this documenting, buy me an iPad.
• No way to mux logos, graphics or anything else onto the stream that I can see so far- easiest way would probably involve streaming to one point, inserting graphics with another ffmpeg copy, then streaming that for distribution
• Multiple bitrates and adaptive bitrate hopping aren’t easily done with this sort of setup

There’s a lot of good here, though. Notably, we’ve just pulled off a complete end-to-end video broadcast chain supporting realtime and recorded content using (almost- Flowplayer’s the exception, and there are usable alternatives) open source software for a total cost of £0. Wowza Media Server, Flash Media Server and friends are around $999. They may be more complete solutions, but then they’ve got the money to spend on it. Red5 is worth noting (lots of big players use it, including Facebook), but is too badly documented to be usable, sadly. I’m sure it’s very capable, but since nobody can explain to anybody how to make use of this wonderful program, it’s utterly useless. For editing, we’re also using a free bit of software which, come the end of the year, will also be open source; an ex-commercial system called Lightworks. It’s utterly fantastic, free, and developed by some great people. It also has a fantastic community around it, and is well worth a look. I’m almost certainly sticking with it over using Avid Media Composer for the SU TV project at present (AVCHD support and better export formats are the two biggies, but they’re being solved as we speak).

The fruits of my efforts here can be viewed on the rhubarbTV website (name liable to change). This site also contains demonstration code for Flowplayer in various roles, though not using adaptive bandwidth control just yet.
WebRTC and other frameworks are worth keeping an eye on; codecs like VP8, WebM, and Theora are rapidly advancing, and while h264 currently represents one of the most rapidly adopted codecs, it is patent-encumbered and thus worth avoiding if possible. HTML5 video is going to be the way to go for recorded content, but RTMP/RTSP remain the only usable systems for realtime (until WebRTC gains traction).

SilentJack is an awesome little utility from the king of ‘oh, that’s a handy little program for broadcast’, Nicholas Humfrey. This guy’s getting a beer if I ever meet him. But it’s not a simple drop-in tool for monitoring, sadly – we need to do a bit of work to make it so.

We use Nagios at Insanity – it wasn’t made for broadcasting shops, but it’s perhaps the best thing out there for monitoring large complex systems. And let’s face it, even simple stations are. For Insanity we have 17 hosts (one of which is a virtual host representing our hardware silence detector) and 95 services configured. All of those are prodded every few minutes to check they’re alive and working. The sort of thing we monitor are load averages of Linux boxes, disk space, process counts, pings, time synchronization, HTTP responses (on our online streaming servers), POP3/IMAP/SMTP responses. For the Windows boxes we also check if Myriad is running (though this is a bit useless, as if Myriad’s crashed it will still be running, just frozen). We also monitor things like SNMP responses from our switch and use cluster checks to monitor dumb switches without SNMP support. GPIO monitoring at the moment is done with some kinda awkward Arduino scripting. More on that in another post.

Anyway, back to the task at hand. SilentJack lets us run a command after a given amount of silence. It will wait till the command finishes, and then go back to listening for silence.

This is awkward. What we really want is a command to run on silence, and a command to run after we get good audio for a certain period. Well, we can almost do that with what we’ve got. Perhaps the ideal solution is some hacking on the source for silentjack- I’ll have a look at that in a bit…

Here’s what we have for now, though. We’ve got Redis running at Insanity, which is a very lightweight key-value server. Go grab it and install it. Now we’ll need some scripts!

So, this is a tiny little script that does only one thing – it spawns our main script in a screen window, and takes very little time to do it. This escapes the fact that we can’t fork a long-running process from silentjack without stopping it listening to the audio. Hence, silenthack.sh! Note you need GNU Screen to run the above, packaged as ‘screen’ on most distros. Now we need something a little more heavyweight. Break out python, send_nsca and redis! We’re going to construct a small script that runs for 12 seconds; we’ll be making silentjack angry on more than 10 seconds of audio silence. This script will use a mutex lock to see if another copy of the script has been started since it started; this means we can see if there’s a continuing outage of audio. With that knowledge we can both maintain a simple flag in Redis (to be queried by Nagios for active checks) and send a passive check result to Nagios. We will always send the failure result to let Nagios know that we’ve still got a problem each time the script fires, but we only send the “All okay” result if we don’t see silentjack spawn another process after we’ve waited for greater than the silentjack silence duration. This keeps Nagios nice and clean and ensures you don’t get unneeded flapping and loads of emails.

The script to do this is above. Note you’ll have to change a few parameters here and there, and you need the send_nsca command-line utility installed, not to mention the nsca daemon running on another machine and all that. You’ll also need the python Redis package. But if you’re using Nagios already you’ll probably already have that lot, and if you’re not, it’s pretty simple – the Nagios documentation is great, and your distro probably already has packages.

So, this now gives us a rather nice simple display of our levels being okay. But don’t forget to set up a process check to ensure that silentjack is actually running. Once you’ve done that, you’re sorted. You can also whip up a quick Nagios plugin to check the flag actively.

There’s what I use – it’s quick and dirty, but it works. We simply check to see if the silentjack_silent flag is true or false. Then all that remains is to define the command in Nagios…

define command{
        command_name    check_redis
        command_line    $USER1$/check_redis $ARG1$ $ARG2$
        }

And to define the service, making sure we enable passive checks…

define service{
        use                     generic-service
        host_name               paxman
        service_description     SilenceDetector
        check_command           check_redis!silentjack_silent!False
        passive_checks_enabled  1
        }

And now to start silentjack, here using Rivendell’s main output.

screen -dmS sj silentjack -l -30 -c rivendell_0:playout_0L -p 10 -v \
         /home/insanity/rdscripts/silenthack.sh

Voila! We’ve now got both active and passive monitoring enabled for our silence detector in software. It’s a bit rough around the edges- I’d love any suggestions on how to improve it.