Mongoid lets us build (much as ARel lets us do in ActiveRecord) criteria up, which consist of a few things, but mostly a selector (“Which records?”) and options (“How do you want them?”). So the answer is actually pretty straightforward – we serialize these objects and then use them to build our base criteria, on which we can then do pagination. Neat, right?

So, how do we do this? Easy, actually.

Essentially we need a method for setting scope, and one for loading it. So let’s define those methods and set it up.

Okay – now we’ve got a scope key stored in our @list_scope instance variable. This refers to an entry in our cache store (in our case, Redis) which holds info on how to reconstruct that criteria. We’re tagging along a few special parameters, too, not just the essentials.

Now that we’ve got this we can refer to that scope where we need to – for instance, in links to other controllers, or in data- attributes to specify regions to automatically reload on the page.

Pulling out a criteria (and any other data we might want) is trivial at this stage. This technique can let you throw arbitrary-complexity queries around in userspace, allowing for clever client-side stuff and a smoother workflow for many applications and users. Give it a shot and have a play with what you can do!

So, lately I’ve been moving out from the broadcast area and getting back into webapp development, but some of the things I’ve been working on touch quite heavily on deduplication, of images and music. This is quite an interesting topic, so let’s have a look at what we can do now.

Doing exact deduplication – stopping someone uploading the same file twice to a website – is pretty easy. You just hash the uploaded file (or de-encapsulate the data and hash that if you want to be a little more resilient) with something like SHA256 or SHA512. It’s fast, effective and easy. Lookups are as fast as your RDBMS is. This works with images, audio, videos, you name it.

What’s much harder is doing perceptual deduplication, or content deduplication. If I upload two files which are the same except one’s a PNG and one’s a JPEG, I want to be able to say “Hang on, you’ve already uploaded that!” when you upload the second file. Similarly, what if we resize an image? We want something resistant to that sort of attack.

There are already perceptual hashing libraries like pHash out there which work by generating hashes which look very similar to your average md5sum, but are actually generated perceptually – the Hamming distance between two hashes is a measure of the similarity of the images represented by those two hashes. This is a great thing, but it’s pretty useless on large datasets without some specialist software to manage databases of hashes and querying based on Hamming distance. The pHash guys will of course sell you this solution, but there’s the problem – there’s quite a bit of money to be made with this sort of product, and useful open source implementations seem to be quite rare if not non-existent.

pHash is also a bit of an odd thing in that it works on multiple media types – audio, video and images. More specifically for audio are techniques for audio fingerprinting, like AcoustID which aim to generate a specific fingerprint for each recording of a song. Distance between songs isn’t something we’re very interested in, because audio releases are typically few in number, and rarely are we looking for a song which sort of sounds like X – if it sounds different, it’s a different recording of the song, or has been mastered.

Images are very different because people often make small changes, or change the formatting or file type of an image, and these circulate and get thrown around all over the place. We want to be able to accept things that are legitimately changed, but flag up things that are almost perfectly identical to something we already have in a database.

So, what can we do? Well, we can use simpler techniques to reduce the number of images to compare down to a small number – say, 50 – and then compare the Hamming distance on full pHashes using that technique. But do we actually need this? Certainly for some databases, merely using those simpler techniques may well be sufficient.

I’ve been building an image board, lately, which I’m using Dragonfly for, an awesome on-the-fly image serving system. We’re storing everything in MongoDB, with GridFS for file storage. Here’s what we’re doing.

First, we need some analysis of the image contents. I’m using a roughly perceptually weighted average intensity metric. Here’s a tiny little Python script using Python-OpenCV’s SWIG bindings to perform that analysis rapidly.

Now we need to get Dragonfly in on this, and register a function we’ll use to handle GIFs. This goes in our Dragonfly initializer.

And finally we need to store all this and actually do the find-duplicates step. Note the aspect ratio stuff – with the ImageMagick analyser, Dragonfly will handle storing this for you, which makes life easier.

In our find_duplicates function we just look for images that have both a similar intensity and a similar aspect ratio. If both of these things are very close we’ve got ourselves a potential duplicate. We’re not doing exact matching on intensity/aspect ratio, more fuzzy matching, because compression changes and resizes can often affect both slightly.

Of course, the proof is in the pudding- this site only has a hundred images in it now, and we may need to adjust the distance variable, but so far it’s working very well and isn’t bringing up any false positives while correctly identifying duplicates. The way we’re doing this from a workflow perspective is important, too – when we do find duplicates, we ask the uploader of the image to check and make sure they’re not uploading anything we already have by presenting the duplicate images to them. They can then mark the duplicate (which deletes the image they just uploaded and leaves a redirect to the image their upload duplicates) or confirm it’s not a duplicate. The statistics from those actions will be interesting to see over time as a metric of success for this approach!

So, that’s how I’ve done image dedupe in a Rails app – what approach are you using?

The internet has flourished and works so very well because it is decentralized, based on open protocols, and systems working together to let people communicate. Let’s just compare standard email with the BlackBerry flavour for a moment.If you send an email, your email client talks to a server in a protocol called SMTP, the Simple Mail Transfer Protocol. The server will then take that SMTP request and talk to the server which the target domain has nominated to be the mail server for that domain. This is found out using the open Domain Name System, and supports simple load balancing and failover by virtue of multiple MX records and priorities. The servers then have a quick conversation via SMTP, and at some point (assuming all goes to plan), your email is now sat in someone’s mailbox. Their client will talk to the server regularly (using POP3 or IMAP) and will pick up on the new email that just came in, and tell them about it.

These protocols are all completely standard, simple and free to implement. They run over the standard TCP/IP network layer that makes up the internet, and there’s no centralization (except in the case of DNS, which is still massively decentralized compared to BBM). Everything supports full end-to-end encryption using the secure socket layer, a much more robust system for securing transmissions than BlackBerry’s unencrypted transports and symmetric keying.

Now let’s look at a typical BlackBerry user and compare them to an Android user (or iPhone, or Windows Phone, or even Palm Pilots – the protocols existed back then). Both are trying to send and receive emails. Most people rely on a free provider for their email, like Gmail or Hotmail, but many people run their own mail servers or use organizational servers (university mail servers, your office’s mail server, and so on). There’s a vast web of servers, all cooperating together to make things work. Enter the BlackBerry- you now remove those protocols from the device and instead put them into a mail server at the other end. This server talks SNMP to the rest of the world, and even speaks IMAP/POP to your mail server, behind the scenes- but access to your mails from your device requires the BlackBerry go-between to be functional. If that single point of failure fails, it’s all over- no connectivity for you.

So why would, on paper, you want to use the BlackBerry service? Well, BBM is an additional feature- group messaging based on… the BlackBerry equivalent of MAC addresses? Huh. Okay, so we’ve got a comms methodology from the 90s which basically implements a dumbed-down version of IRC or Jabber for your phone. Not like we’ve had that on other platforms for as long as I can remember – I was using Jabber on my phone years ago to talk to people on the other side of the world about EVE Online fleet operations instead of paying attention in maths lessons. Again, things like IRC and Jabber are open standard protocols, free to implement, and implemented in a manner that is widely distributed with little to no single points of failure. IRC was designed for the days of acoustic couplers, and is still used widely by millions of people. The BBM protocols aren’t even secure- they’re encrypted, but the key used for encryption is shared amongst all phones. It’s a bog standard symmetric crypto cipher. This means if you can convince a phone that it’s actually another phone, you can read any other person’s messages and send messages as them. Contrast with IRC – the network is still assumed secure (just as with BBM) and unencrypted, but links between servers and links between client and server can all use the secure socket layer- a very secure protocol.

On top of this we’ve got the push email feature – which lets you get emails sort-of-instantly (actually delayed unless your organization runs a BlackBerry Messaging Server, as I understand it) pushed to your phone instead of having to have the phone fetch new emails. Granted, that’s quite a nice feature. But it’s not a game-changer to the extent where I’d be willing to sacrifice my email connectivity on my device to have it- for days at a time, no less. In addition, there’s open tools that will do this for you on Android and iPhone. I remember back when I was first using a Windows Mobile PDA, I set up push email in the course of a few days of hacking around with an off-the-shelf product. This was back in the days of the XDA, when push email was just kicking off, and it wasn’t too hard to make work. These days it’s even easier.

BlackBerry falls over because they are a quite heavily centralized company and their infrastructure is not designed for the modern age. Note that the BlackBerry announcement about their recent outages referred to “Europe, the Middle East and Africa”. One problem can hit all those users. That’s not a well-designed system, it’s a legacy system shoehorned in from the days when BlackBerry Messaging Server was the sort of thing you ran if you were a big company. It is bound to fail and when it fails it pisses off many millions of users.

The question of why people decided to go with BlackBerry remains a mystery to me- I honestly don’t know why people would choose a BlackBerry over any other phone. I have spent countless hours debugging and fixing broken BlackBerries for friends, and many more hours still trying to explain how to use them to other friends (who inevitably end up sending them back and getting a ‘droid). Perhaps there’s some secret feature I’m not seeing, but it just seems to me that the hardware’s mediocre, the software’s awful at best, and the infrastructure is flaky. Why choose that over solid and varied hardware choice, great software with flexibility, and open infrastructure with no reliance on single providers or services? I know which one of these combinations has given me years of flawless service with not one day without email.