So, lately I’ve been moving out from the broadcast area and getting back into webapp development, but some of the things I’ve been working on touch quite heavily on deduplication, of images and music. This is quite an interesting topic, so let’s have a look at what we can do now.

Doing exact deduplication – stopping someone uploading the same file twice to a website – is pretty easy. You just hash the uploaded file (or de-encapsulate the data and hash that if you want to be a little more resilient) with something like SHA256 or SHA512. It’s fast, effective and easy. Lookups are as fast as your RDBMS is. This works with images, audio, videos, you name it.

What’s much harder is doing perceptual deduplication, or content deduplication. If I upload two files which are the same except one’s a PNG and one’s a JPEG, I want to be able to say “Hang on, you’ve already uploaded that!” when you upload the second file. Similarly, what if we resize an image? We want something resistant to that sort of attack.

There are already perceptual hashing libraries like pHash out there which work by generating hashes which look very similar to your average md5sum, but are actually generated perceptually – the Hamming distance between two hashes is a measure of the similarity of the images represented by those two hashes. This is a great thing, but it’s pretty useless on large datasets without some specialist software to manage databases of hashes and querying based on Hamming distance. The pHash guys will of course sell you this solution, but there’s the problem – there’s quite a bit of money to be made with this sort of product, and useful open source implementations seem to be quite rare if not non-existent.

pHash is also a bit of an odd thing in that it works on multiple media types – audio, video and images. More specifically for audio are techniques for audio fingerprinting, like AcoustID which aim to generate a specific fingerprint for each recording of a song. Distance between songs isn’t something we’re very interested in, because audio releases are typically few in number, and rarely are we looking for a song which sort of sounds like X – if it sounds different, it’s a different recording of the song, or has been mastered.

Images are very different because people often make small changes, or change the formatting or file type of an image, and these circulate and get thrown around all over the place. We want to be able to accept things that are legitimately changed, but flag up things that are almost perfectly identical to something we already have in a database.

So, what can we do? Well, we can use simpler techniques to reduce the number of images to compare down to a small number – say, 50 – and then compare the Hamming distance on full pHashes using that technique. But do we actually need this? Certainly for some databases, merely using those simpler techniques may well be sufficient.

I’ve been building an image board, lately, which I’m using Dragonfly for, an awesome on-the-fly image serving system. We’re storing everything in MongoDB, with GridFS for file storage. Here’s what we’re doing.

First, we need some analysis of the image contents. I’m using a roughly perceptually weighted average intensity metric. Here’s a tiny little Python script using Python-OpenCV’s SWIG bindings to perform that analysis rapidly.

Now we need to get Dragonfly in on this, and register a function we’ll use to handle GIFs. This goes in our Dragonfly initializer.

And finally we need to store all this and actually do the find-duplicates step. Note the aspect ratio stuff – with the ImageMagick analyser, Dragonfly will handle storing this for you, which makes life easier.

In our find_duplicates function we just look for images that have both a similar intensity and a similar aspect ratio. If both of these things are very close we’ve got ourselves a potential duplicate. We’re not doing exact matching on intensity/aspect ratio, more fuzzy matching, because compression changes and resizes can often affect both slightly.

Of course, the proof is in the pudding- this site only has a hundred images in it now, and we may need to adjust the distance variable, but so far it’s working very well and isn’t bringing up any false positives while correctly identifying duplicates. The way we’re doing this from a workflow perspective is important, too – when we do find duplicates, we ask the uploader of the image to check and make sure they’re not uploading anything we already have by presenting the duplicate images to them. They can then mark the duplicate (which deletes the image they just uploaded and leaves a redirect to the image their upload duplicates) or confirm it’s not a duplicate. The statistics from those actions will be interesting to see over time as a metric of success for this approach!

So, that’s how I’ve done image dedupe in a Rails app – what approach are you using?

Trouble is, there is literally no budget (yet, at least), but we wanted to cover events during the Fresher’s Week here at Royal Holloway. So, I spent a week putting together our own YouTube+UStream. This turned out to be surprisingly complex purely in terms of lack of good contiguous information and documentation. Once the parts are all together, it’s actually remarkably simple.

First, let’s examine the goals of the station. From there, we can work out what we need to have in order to achieve this.So, we wanted:

• Live coverage of some events (optionally while recording)
• Live roaming camera capability
• Publishing of recorded content
• Support for low bitrate devices/connections and as many target devices as possible
• Support for up to quite high qualities for desktops and laptops with solid connections

In terms of equipment to pull this off, we had:

• A single laptop, obtained off eBay for £130 (a HP nw8200 mobile workstation, which has a 2Ghz Celeron D processor, and which I upgraded to 2GB of DDR2 RAM)
• A camera, borrowed from our Media Arts department (a Sony PD-150 DV camera)
• A 4-6 pin firewire cable and a 6-to-4 adaptor (allowing us to get the DV feed from the camera into the laptop. A 4-4 cable would have been better, but we didn’t have one)
• A Shure SM58 microphone, an AKG C535EB, both lent to us by people, plus some XLR cable
• A server rented in a Maidenhead datacenter with oodles of bandwidth per month (terabytes) on a 100Mbps pipe
• Wifi coverage in the areas we were shooting in, shared with everyone else in the area (lots of people) with internet connectivity

So, not a lot, especially by professional broadcast standards. But we’ve got enough! Specifically we’ve got a way to capture video, a way to communicate that video to a laptop, and we have the makings of a distribution system.
Onwards, then. First stop was to get the laptop ready- for this we used Ubuntu 11.04 (in the Ubuntu Classic session, of course- never will understand how people use Unity), and installed the latest versions of ffmpeg, libtheora, libvpx and libx264 as detailed in this smashing post here. But in addition to that post we install the librtmp-dev package and enable it when building ffmpeg with –enable-rtmp. This will let us encode RTMP streams directly with ffmpeg. That post installs everything as packages, which is great- just make sure you uninstall the system versions first.
Now our laptop is ready. Let’s crack on – we’ve now got a way to encode the video but we need to be able to fetch it from the camera- dvgrab comes into play here, so install that (sudo apt-get install dvgrab). This lets us capture DV video as an AVI.

Let’s just step back and consider our workflow for a moment. We take DV video out of the camera in realtime, feed that into ffmpeg. Then we use ffmpeg to encode the video and audio (because raw AVI is quite large), and encapsulate it in a RTMP stream. The RTMP stream then gets distributed to RTMP clients, such as Flash player (Flowplayer, jwplayer, etc). This is our live workflow. For recorded content we’re going to do roughly the same, but instead of encoding RTMP, we have ffmpeg turn our recorded AVI file into an encoded FLV file, which our RTMP distribution server will handle streaming over RTMP for us. Piece of cake.
Right. That RTMP server is the next thing- we’ll use crtmpserver, as it’s both open source and very lightweight. Simply visit the site, follow the install instructions, and for live streaming, configure the server such that you can send it a RTMP feed (InboundRtmp). You also want the flvplayback application set up. This is where it gets fun! At this point you should be able to go get Flowplayer (or whichever RTMP-supporting Flash player/other client you want to use) and configure that.

So here’s the tricky bits- making ffmpeg do the right thing. First, let’s consider a script which will encode our recorded videos. We pass this script one argument, the filename to encode, and off it goes, making bitrate versions for 400k, 800k, 1200k and 1600k to be switched between by Flowplayer or similar, written to FLV containers so crtmpd can read them:

#!/bin/bash
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 400k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 450k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 96k -ar 44100 -s 360x288 -f flv "`echo "$1" | sed -e 's/\..../\-400k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 800k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 850k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 128k -ar 44100 -f flv "`echo "$1" | sed -e 's/\..../\-800k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 1200k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 1300k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 192k -ar 44100 -f flv "`echo "$1" | sed -e 's/\..../\-1200k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";
ffmpeg -i "$1" -deinterlace -vcodec libx264 -b 1600k -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +wpred -maxrate 1800k -i_qfactor 0.71 -keyint_min 25 -b_strategy 1 -g 150 -r 25 -acodec libmp3lame -ab 192k -ar 44100 -f flv "`echo "$1" | sed -e 's/\..../\-1600k\.flv/g' | sed -e 's/[^a-zA-Z0-9\.\-]//g'`";

Okay, so that’s recorded video. What about live, though? Well, for live we’re going to do a simpler encode, because we’re trying to do this in realtime on a slow laptop (the above encodes take a long time. An hour of video came out at about 6 hours of encoding). We’re also using a lower audio bitrate, using the same sample rate as the camera (to save us having to do sample rate conversion), and we’re using a basic preset from ffmpeg. We’re also using the AAC codec as it’s performance-equivalent to MP3 but gives a much better quality for the bitrate, at the cost of some interoperability.

#!/bin/bash
dvgrab -f dv1 -noavc - | ffmpeg -i - -re -deinterlace -s 360x288 -acodec libfaac -ar 48000 -ac 1 -ab 64k -vcodec libx264 -vpre normal -f flv rtmp://yourstreamserver/live/streamName

Note also the dvgrab command writes to -, the standard output, and does not control the camera (-noavc). This means we can pipe the output directly to ffmpeg (which reads stdin with -i -) without messing around with any silly intermediary files. Fast and efficient! Don’t forget that -, though, or you’ll spend hours working out why it doesn’t work like I did. Doh. We also drop the resolution by half which allows us to run things a whole lot faster without a huge quality drop. We don’t specify a bitrate to target, and elect to let libx264 figure out what it needs (combined with -vpre normal to say “We don’t want anything too fancy”). This got around 400-600kbps in practice, which is quite reasonable for most users. It would be fairly trivial to reencode on the server a low bitrate version if desired using ffmpeg, just taking input from RTMP and outputting to another RTMP target, though delays must be considered when doing this. Also worth noting is the ‘-ac 1′ command- this tells ffmpeg to downmix the audio to one channel. On our camera we have two channels, one from a camera-mounted electret microphone (not very good) and one from the handheld mic (an SM58 most of the time); we just sum these and mix on the camera. When recording we get both channels individually on the tape and can mix properly afterwards.

This live encode command is actually fast enough to do realtime streaming from what amounts to a very reasonable laptop. Any modern bit of kit should be able to do similar with ease. For the Fresher’s Fayre, we had the laptop slung in a shoulder satchel on the same arm as the camera, with a firewire lead going into the bag from the camera. Taking some care to leave the air vents able to ventilate, we had a smashing little portable rig at this point which would let us record direct to tape on the camera while also playing out live. Admittedly only for an hour till the battery on the laptop died, but for £130 of laptop and borrowed gear for the rest, it’s not half bad.
The downsides so far I’ve noticed:

• No simple way to fall back on a static file when a stream is unavailable
• iPhone/iPad requires RTSP for streaming; crtmpserver will produce RTSP, but I’ve not got an iPad/iPhone to test with so can’t figure this out. If someone wants this documenting, buy me an iPad.
• No way to mux logos, graphics or anything else onto the stream that I can see so far- easiest way would probably involve streaming to one point, inserting graphics with another ffmpeg copy, then streaming that for distribution
• Multiple bitrates and adaptive bitrate hopping aren’t easily done with this sort of setup

There’s a lot of good here, though. Notably, we’ve just pulled off a complete end-to-end video broadcast chain supporting realtime and recorded content using (almost- Flowplayer’s the exception, and there are usable alternatives) open source software for a total cost of £0. Wowza Media Server, Flash Media Server and friends are around $999. They may be more complete solutions, but then they’ve got the money to spend on it. Red5 is worth noting (lots of big players use it, including Facebook), but is too badly documented to be usable, sadly. I’m sure it’s very capable, but since nobody can explain to anybody how to make use of this wonderful program, it’s utterly useless. For editing, we’re also using a free bit of software which, come the end of the year, will also be open source; an ex-commercial system called Lightworks. It’s utterly fantastic, free, and developed by some great people. It also has a fantastic community around it, and is well worth a look. I’m almost certainly sticking with it over using Avid Media Composer for the SU TV project at present (AVCHD support and better export formats are the two biggies, but they’re being solved as we speak).

The fruits of my efforts here can be viewed on the rhubarbTV website (name liable to change). This site also contains demonstration code for Flowplayer in various roles, though not using adaptive bandwidth control just yet.
WebRTC and other frameworks are worth keeping an eye on; codecs like VP8, WebM, and Theora are rapidly advancing, and while h264 currently represents one of the most rapidly adopted codecs, it is patent-encumbered and thus worth avoiding if possible. HTML5 video is going to be the way to go for recorded content, but RTMP/RTSP remain the only usable systems for realtime (until WebRTC gains traction).

SilentJack is an awesome little utility from the king of ‘oh, that’s a handy little program for broadcast’, Nicholas Humfrey. This guy’s getting a beer if I ever meet him. But it’s not a simple drop-in tool for monitoring, sadly – we need to do a bit of work to make it so.

We use Nagios at Insanity – it wasn’t made for broadcasting shops, but it’s perhaps the best thing out there for monitoring large complex systems. And let’s face it, even simple stations are. For Insanity we have 17 hosts (one of which is a virtual host representing our hardware silence detector) and 95 services configured. All of those are prodded every few minutes to check they’re alive and working. The sort of thing we monitor are load averages of Linux boxes, disk space, process counts, pings, time synchronization, HTTP responses (on our online streaming servers), POP3/IMAP/SMTP responses. For the Windows boxes we also check if Myriad is running (though this is a bit useless, as if Myriad’s crashed it will still be running, just frozen). We also monitor things like SNMP responses from our switch and use cluster checks to monitor dumb switches without SNMP support. GPIO monitoring at the moment is done with some kinda awkward Arduino scripting. More on that in another post.

Anyway, back to the task at hand. SilentJack lets us run a command after a given amount of silence. It will wait till the command finishes, and then go back to listening for silence.

This is awkward. What we really want is a command to run on silence, and a command to run after we get good audio for a certain period. Well, we can almost do that with what we’ve got. Perhaps the ideal solution is some hacking on the source for silentjack- I’ll have a look at that in a bit…

Here’s what we have for now, though. We’ve got Redis running at Insanity, which is a very lightweight key-value server. Go grab it and install it. Now we’ll need some scripts!

So, this is a tiny little script that does only one thing – it spawns our main script in a screen window, and takes very little time to do it. This escapes the fact that we can’t fork a long-running process from silentjack without stopping it listening to the audio. Hence, silenthack.sh! Note you need GNU Screen to run the above, packaged as ‘screen’ on most distros. Now we need something a little more heavyweight. Break out python, send_nsca and redis! We’re going to construct a small script that runs for 12 seconds; we’ll be making silentjack angry on more than 10 seconds of audio silence. This script will use a mutex lock to see if another copy of the script has been started since it started; this means we can see if there’s a continuing outage of audio. With that knowledge we can both maintain a simple flag in Redis (to be queried by Nagios for active checks) and send a passive check result to Nagios. We will always send the failure result to let Nagios know that we’ve still got a problem each time the script fires, but we only send the “All okay” result if we don’t see silentjack spawn another process after we’ve waited for greater than the silentjack silence duration. This keeps Nagios nice and clean and ensures you don’t get unneeded flapping and loads of emails.

The script to do this is above. Note you’ll have to change a few parameters here and there, and you need the send_nsca command-line utility installed, not to mention the nsca daemon running on another machine and all that. You’ll also need the python Redis package. But if you’re using Nagios already you’ll probably already have that lot, and if you’re not, it’s pretty simple – the Nagios documentation is great, and your distro probably already has packages.

So, this now gives us a rather nice simple display of our levels being okay. But don’t forget to set up a process check to ensure that silentjack is actually running. Once you’ve done that, you’re sorted. You can also whip up a quick Nagios plugin to check the flag actively.

There’s what I use – it’s quick and dirty, but it works. We simply check to see if the silentjack_silent flag is true or false. Then all that remains is to define the command in Nagios…

define command{
        command_name    check_redis
        command_line    $USER1$/check_redis $ARG1$ $ARG2$
        }

And to define the service, making sure we enable passive checks…

define service{
        use                     generic-service
        host_name               paxman
        service_description     SilenceDetector
        check_command           check_redis!silentjack_silent!False
        passive_checks_enabled  1
        }

And now to start silentjack, here using Rivendell’s main output.

screen -dmS sj silentjack -l -30 -c rivendell_0:playout_0L -p 10 -v \
         /home/insanity/rdscripts/silenthack.sh

Voila! We’ve now got both active and passive monitoring enabled for our silence detector in software. It’s a bit rough around the edges- I’d love any suggestions on how to improve it.