So, lately I’ve been moving out from the broadcast area and getting back into webapp development, but some of the things I’ve been working on touch quite heavily on deduplication, of images and music. This is quite an interesting topic, so let’s have a look at what we can do now.

Doing exact deduplication – stopping someone uploading the same file twice to a website – is pretty easy. You just hash the uploaded file (or de-encapsulate the data and hash that if you want to be a little more resilient) with something like SHA256 or SHA512. It’s fast, effective and easy. Lookups are as fast as your RDBMS is. This works with images, audio, videos, you name it.

What’s much harder is doing perceptual deduplication, or content deduplication. If I upload two files which are the same except one’s a PNG and one’s a JPEG, I want to be able to say “Hang on, you’ve already uploaded that!” when you upload the second file. Similarly, what if we resize an image? We want something resistant to that sort of attack.

There are already perceptual hashing libraries like pHash out there which work by generating hashes which look very similar to your average md5sum, but are actually generated perceptually – the Hamming distance between two hashes is a measure of the similarity of the images represented by those two hashes. This is a great thing, but it’s pretty useless on large datasets without some specialist software to manage databases of hashes and querying based on Hamming distance. The pHash guys will of course sell you this solution, but there’s the problem – there’s quite a bit of money to be made with this sort of product, and useful open source implementations seem to be quite rare if not non-existent.

pHash is also a bit of an odd thing in that it works on multiple media types – audio, video and images. More specifically for audio are techniques for audio fingerprinting, like AcoustID which aim to generate a specific fingerprint for each recording of a song. Distance between songs isn’t something we’re very interested in, because audio releases are typically few in number, and rarely are we looking for a song which sort of sounds like X – if it sounds different, it’s a different recording of the song, or has been mastered.

Images are very different because people often make small changes, or change the formatting or file type of an image, and these circulate and get thrown around all over the place. We want to be able to accept things that are legitimately changed, but flag up things that are almost perfectly identical to something we already have in a database.

So, what can we do? Well, we can use simpler techniques to reduce the number of images to compare down to a small number – say, 50 – and then compare the Hamming distance on full pHashes using that technique. But do we actually need this? Certainly for some databases, merely using those simpler techniques may well be sufficient.

I’ve been building an image board, lately, which I’m using Dragonfly for, an awesome on-the-fly image serving system. We’re storing everything in MongoDB, with GridFS for file storage. Here’s what we’re doing.

First, we need some analysis of the image contents. I’m using a roughly perceptually weighted average intensity metric. Here’s a tiny little Python script using Python-OpenCV’s SWIG bindings to perform that analysis rapidly.

Now we need to get Dragonfly in on this, and register a function we’ll use to handle GIFs. This goes in our Dragonfly initializer.

And finally we need to store all this and actually do the find-duplicates step. Note the aspect ratio stuff – with the ImageMagick analyser, Dragonfly will handle storing this for you, which makes life easier.

In our find_duplicates function we just look for images that have both a similar intensity and a similar aspect ratio. If both of these things are very close we’ve got ourselves a potential duplicate. We’re not doing exact matching on intensity/aspect ratio, more fuzzy matching, because compression changes and resizes can often affect both slightly.

Of course, the proof is in the pudding- this site only has a hundred images in it now, and we may need to adjust the distance variable, but so far it’s working very well and isn’t bringing up any false positives while correctly identifying duplicates. The way we’re doing this from a workflow perspective is important, too – when we do find duplicates, we ask the uploader of the image to check and make sure they’re not uploading anything we already have by presenting the duplicate images to them. They can then mark the duplicate (which deletes the image they just uploaded and leaves a redirect to the image their upload duplicates) or confirm it’s not a duplicate. The statistics from those actions will be interesting to see over time as a metric of success for this approach!

So, that’s how I’ve done image dedupe in a Rails app – what approach are you using?

IRIS was written to replace MACIS, a system I bodged up out of necessity. At Insanity, we had a computer failure weeks before we went on air at the start of the first term, and lost all the data- including the entire playout system. Lessons have been learned (I made sure we replaced that machine with a box that had RAID, for starters) since, but we had the unenviable challenge of repopulating a student radio playout system from scratch with little to no staff. Enter MACIS!

MACIS was dumb. It talked to our playout system (PSquared’s Myriad 3.5) via the not-very-documented TCP/IP interface, had a web interface and drop box, and some background processing magic. It was implemented as a Ruby on Rails web application, since we already used Rails and Ruby for various tasks around the station (the website is all Rails, and Ruby was chosen for most scripting tasks because of its user friendliness to people not too familiar with programming. You passed it files, it converted them (the main purpose- Myriad doesn’t support AAC, MP4 or many other formats we were using for ingest), did a basic stab at normalization, and then imported the files. Myriad is slow at importing files- 1-2 minutes per file on average, so we let MACIS distribute workload across all four of our Myriad machines, speeding things up massively.

This was good and got us running, but then term started. We’re a student station and have specialist music shows, who upload their own content to the playout system every week, and new content for new music and chart shows came in regularly. MACIS was used for this, as it did the conversion automatically, saving our presenters loads of time. It also did batch imports quickly and efficiently, which sped things up. However, after a while, we stopped using it, and just provided a simple Ruby dropbox on a shared file server for conversion. MACIS was useful, but too buggy and inflexible. In addition, while it did a better job of getting metadata into Myriad than Myriad managed on its own, it had issues with some formats and material.

What was needed was a system that would let presenters upload content in any format, would sort out the metadata, handle conversion, and fire it off to the playout system for usage.

An example upload showing the log and graphs (huge image)

So, while presenters got back to using Myriad directly, I went back to the drawing board and scrapped MACIS. At this stage we were considering transitioning to the Rivendell open source playout system to replace Myriad, so I decided whatever I was going to make had to support both Myriad and Rivendell, and any other system you could imagine.

I also wanted to solve the loudness problem. Even doing normalization to every track imported, we had huge loudness level differences between some tracks, making life quite tricky for presenters and impacting our on-air sound. Especially given our lack of transmitter processing (we only have a limiter and preemphasis box on the AM system- no AGC or multiband comps), I wanted to do all I could to get everything as perceptually loud as everything else. Enter EBU Recommendation 128 for loudness measurement- with the help of some great libraries (libebur128 in particular), I implemented a simple version of the recommended processing system for loudness normalization, including LRA correction using a compressor. Thus, everything you run through IRIS comes out sounding about the same as everything else in terms of perceptual levels – as much as is possible without impacting the sound. Wish You Were Here is still going to have a quiet bit at the beginning- but IRIS will gently compress the track to make the difference less severe, and will then use R128′s standards to normalize to -23 LUFS.

Next up, user authentication. This was almost an afterthought, but added after talking to people about security. You register an account, and that account is either able to upload content only, upload and review (more on that in a sec), or administrate the system (ie modify users etc). This is done by user groups, which are pretty flexible, and easily adapted for your own usage via a simple permissions file. Uploads are linked to users- users can only see their uploads (unless they have permission to see more than that) and admins can see who owns a specific upload. You can also have emails sent on error conditions being met- so presenters know if a file they uploaded failed to make it to the playout system before they turn up to do their show and wonder where it is.

Metadata was one of the big issues I wanted to solve. Let’s say I have a track from a CD- I’ve ripped it and the ripper has embedded title/artist, maybe album metadata from Gracenote. For a playout system for radio this isn’t perfect- really we want to know the record label, copyright info, and so on. Enter MusicBrainz- a huge collaborative open database of music metadata. With some clever tools, IRIS matches up the track’s metadata to a MusicBrainz identifier and fills in the blanks. For most tracks, it can get everything- including ISRCs, year of release, and so on. This is great for music librarians and makes copyright reconciliation for PRS/PPL much simpler, since you’ve now got all this in a database.

Of course, if we know what the track is, we can do another useful step- especially so for student stations. Using the MusixMatch API service (commercial but free for nonprofits at present), we can get the lyrics to a track. This means we can do a quick once-over for words we don’t want on air (swearing). Of course, this assumes the track isn’t a radio edit. We do a quick check and skip the lyric pass for tracks that look like a radio edit, but if not, the track will be flagged for review.

Additionally, and this is intended specifically for situations where you have no control over the ingest quality that presenters are using to rip CDs or vinyl, tracks are flagged for review if they fail to meet quality restrictions. This can be specified as a function of sample rate and bit rate. This stops people trying to upload 96kbps MP3 at 32k sample rate. We don’t want that in the system. Not now, not ever. All of these parameters can be changed easily and simply by changing a single configuration file and restarting the app.

Overview of the system architecture

Once an upload is flagged for review an administrator or music librarian can review the track and either permanently reject it or approve it (in case of false positive lyric matches or odd uploads where quality restrictions aren’t able to be met).

Everything in IRIS is done through a web interface- uploads, management and monitoring. Every track has its own log of events, allowing administrators to debug and diagnose problems with ease, and giving clear and simple feedback to users. There are convenience functions such as automatic display of R128 loudness graphs pre/post normalization and compression, and display of all metadata available for tracks, plus lyrics if they were found.

The backend to IRIS is all Ruby and Rails, using a simple database server (PostgreSQL recommended) to store everything. Background processing is distributable over multiple computers with shared storage, allowing for CPU-intensive tasks to be spread across multiple machines. Given the R128 metering process includes a fourfold upsampling, this is particularly useful. You can run workers without running the whole web application, allowing you to install copies of the app onto lots of low-cost general purpose machines and have your own distributed ingest processing cluster on even a tight bugdet.

Of course, now you’ve got a track with lots of metadata and some normalized audio in WAV format (archived to FLAC pre-normalization, just in case you need the original audio). Now you need to get it into your playout system. Rivendell is supported, Myriad 3.6 now has dropbox support so you can just tell IRIS to export files to that dropbox in a Myriad-supported format, and you can also just do an export in any format you choose to an arbitrary folder. Export formats supported include FLAC, MP3, WAV, BWF (Broadcast WAVE Format) and AAC. Most of these flavours come with embedded metadata.

IRIS isn’t a perfect system, and it’s not an instant drop-in system; I don’t have the time to maintain it as such. What it is, though, is a flexible and powerful system that any average Linux user can install and have running in hours, and which can be used by any station looking to improve their import process.  The entire project is open source, and can be obtained here- there’s also a bugtracker and wiki with some documentation (unfinished) on it. If you’d like to contribute, feel free- as I’ve stepped out from student radio to work on student television, I’ve not got a huge amount of time to work on it at the moment.

I’m not moving any of my Ubuntu stuff forward to 11.04 just yet; probably won’t for a while, given the situation with desktop managers and stability particularly on older hardware like the stuff we use at Insanity. So, 10.04.02 LTS it is. It’s still supported for a while and has no realtime issues like 10.10.

However, there’s one snag; in my last guide we used a Rivendell package provided by the guys at Tryphon. No such package currently exists for 2.0.x, so we’ll be rolling our own from source. This is far from ideal (and I am working to get 2.0.x packages for Debian made) but it’s a good start, and will work fine- upgrades are just a bit of a pain.

Okay. Let’s start with dependencies. I’m assuming a virgin from-source system here.

sudo aptitude install libcdparanoia-dev libflac++-dev libflac-dev \ 
 libsamplerate0-dev libid3tag0-dev libid3-3.8.3-dev \
 libcurl4-gnutls-dev libsndfile1-dev libpam0g-dev \
 libsoundtouch1-dev alsa-source libtwolame-dev libmp3lame-dev \
 libmad0-dev libqt3-mt-mysql qt3-dev-tools qt3-dev-tools-compat \
 qt4-dev-tools libjack-dev libjack0.100.0-dev jackd qjackctl \
 jackeq jack-rack jack-capture libasound2-dev mysql-client \
 polymer qt3-qtconfig patchage

Stick that in your terminal and wait a few minutes. There’s a lot there. At some point, jackd will ask if you want to enable realtime priority – say yes. Once that’s done, download the Rivendell tarball and extract it. Then wander into that directory.

wget http://rivendellaudio.org/ftpdocs/rivendell/rivendell-2.0.2.tar.gz
tar zxf rivendell-2.0.2.tar.gz
cd rivendell-2.0.2

Next, configure it- the libexec path makes sure that the paths for some new RD2 services are set up properly (specifically Apache2). Note that if you need a HPI sound card to work in this box, now would be the time to go off and install the drivers from the AudioScience website (libhpi). Do that before this.

./configure --libexecdir=/usr/local/libexec

Now build it, and subsequently install all the binaries and libraries.

make
sudo make install

This is going to take a while, even on decent modern hardware, so let me give you a taste of what’s still to come- specifically we’ve got one more hurdle ahead of us on top of the setup we needed for 1.7.2, which is the Apache web server. Rivendell 2.0.x uses a web service approach to import and export of audio data, which is a great leap forward- as well as making security improvements, it means you can delegate the import/export jobs to specific machines. Say you’ve got a pile of low-end production PCs- you don’t really want those guys to be on the front line of audio ingest. The high-end server in the rack that doesn’t do a lot, however, is completely the tool for the job. So you can tell those production PCs to use the import/export services of that server instead of their own. The flipside is we get to configure an extra thing, but it’s really a piece of cake. Don’t sweat it. And like MySQL, you don’t have to install Apache on every single machine you want to run Rivendell on, only the ones that will be using the import/export service.

Right. Chances are, make is probably still chugging away despite my attempt to distract you. Come back when it finishes.

All done? Installed? Okay, we’re about halfway there. You should now have a complete build of Rivendell with support for every format under the sun (that Rivendell supports, anyway- AAC/MP4 one day, I swear!) with support for ALSA or JACK or (if you installed libhpi before configuring) HPI sound cards. Next, let’s get the cosmetic things done- installing icons and menu items, ensuring bitmap fonts are enabled, that sort of thing.

sudo cp debian/*.desktop /usr/share/applications/
sudo cp debian/*.xpm /usr/share/pixmaps/
sudo rm -rf /etc/fonts/conf.d/70-no-bitmaps.conf
sudo ln -s /etc/fonts/conf.avail/70-force-bitmaps.conf /etc/fonts/conf.d
sudo fc-cache -f -v

That’s that out of the way. Next, set up a Rivendell user and group, add the current user to the audio group (for realtime permissions) and we can get on with the rest.

sudo adduser --system --no-create-home --disabled-login rivendell
sudo addgroup rivendell
sudo adduser rivendell rivendell
sudo adduser `whoami` audio

Okay, we’re getting somewhere. Now, rd.conf. Either copy up conf/rd.conf-sample to /etc/rd.conf, or make a new one that looks a bit like this:

[Identity]
AudioOwner=rivendell
AudioGroup=rivendell
Password=letmein

[mySQL]
Hostname=localhost
Loginname=rduser
Password=letmein
Database=Rivendell
Driver=QMYSQL3

[Logs]
Facility=Syslog

[Alsa]
PeriodQuantity=4
PeriodSize=1024

[Tuning]
UseRealtime=Yes
RealtimePriority=9

[Format]
Channels=2

Bam! Obviously you’ll want to use different passwords to the default and that sort of thing. We’re going with ALSA for now; using jackd is a simple (trivial, even) change to rd.conf. Check the conf/rd.conf-sample file in the source to find out how. Let’s get one other bit of housekeeping done now- we need to configure what sound cards we use for ALSA, which is set in /etc/asound.conf. But easier still, Rivendell provides a program to make this file for us. Pull up a terminal and run this to get this up and follow the instructions:

sudo rdalsaconfig

Once that’s done, let’s bop Apache on the head quickly. Note you only need this is if you’re running a single machine setup or if you’re installing on the machine you want to run import/export jobs on. There’s no harm in having apache/rdxport available, but it’s unneeded overhead if you’re going to delegate to other boxes anyway. Still, say we wanted to install it…

sudo apt-get install apache2
sudo cp conf/rd-bin.conf /etc/apache2/conf.d/
sudo service apache2 restart

Simple, right? Right. Let’s move on and get MySQL sorted out. This is for the main server only if you’re doing client-server stuff; you’ve already got the MySQL client installed if you’re building a client to connect to another existing MySQL server. Run this and when prompted, give it a nice secure password.

sudo apt-get install mysql-server

Let’s get the DB and sound path created and check all is well. Run this and when prompted enter your MySQL root password and the username ‘root’ (if this is your first DB – otherwise, it should just open. Close it, ignore any errors).

sudo mkdir /var/snd
sudo chown rivendell:rivendell -R /var/snd
rdadmin

Okay. We’ve only got to sort one little snag out before we’re good to go- making sure Rivendell services come up cleanly on startup of the machine. There’s complications here, specifically /var/run which gets purged on startup by Ubuntu. Rivendell expects /var/run/rivendell to exist and the rivendell user/group to be able to write there- PIDs are stored for various processes here. Let’s fix that by tweaking our initscript, then set the initscript to start Rivendell on boot.

sudo nano -w /etc/init.d/rivendell

In that script, add the following line before the line that reads ‘function StartDaemons’.

mkdir /var/run/rivendell 2>/dev/null

Okay, now run this- this will make Ubuntu start the daemons on boot.

sudo update-rc.d rivendell defaults

One last thing – edit /etc/pulse/client.conf and uncomment the autospawn=yes line, and set it to no. This stops pulseaudio taking over the sound card so ALSA can get at it.

sudo nano -w /etc/pulse/client.conf # (then do the edit)
sudo killall pulseaudio

And now we reboot and if everything went to plan, you should now have a working Rivendell installation on your system! Check for ripcd/caed/rdcatchd running, fire up rdadmin and friends, and get to configuring your perfect station.

A few observations

Currently there’s a segfault in Rivendell 2.0.2 with some ALSA cards. If caed doesn’t start up and /var/log/messages has a pile of caed output and then a segfault and you’ve got multiple cards available to you in rdalsaconfig, try disabling/enabling each one till you find a working one. Use ‘sudo service rivendell restart’ to restart the daemons without rebooting.