No, this is a post about why you should ignore Facebook. Turn a blind eye and let it pass. It will, in time, fade away, like Yahoo, MSN and others before those. It may have a huge number of users, but then so did MySpace. People will move on, and Facebook is already worrying about growth figures. But that’s not why you should be ignoring it.

You should be ignoring it because all of the things that you try and promote can be just as easily undone, and using Facebook as a solution to problems that your organization faces will only lead to more problems in the long run- not least of all when Facebook finally goes the way of MySpace and people move on. But that’s not all.

First off, some disclosure about myself. I don’t have a Facebook account. I nuked my account after my exams ended, before the holidays started, months ago. I’m an active member of the Royal Holloway Students’ Union. I attend numerous events and work with various media outlets on campus. At no point in doing all of this have I actually needed to visit Facebook.com, much less log in. The last stored login details in my browser aren’t even mine- they’re a friend’s left over from when she borrowed my PC to check some messages. I’ve written apps for Facebook before, and have worked with the APIs and tools that Facebook gives developers to let them access what they describe as ‘the social graph’- that is, your data. And I spend a lot of time working on computer security and, specifically, web security and analysis. I wouldn’t call myself an expert or even a specialist, but I have half a clue about this stuff.

So- your organization is organizing an event. You’ve got a website, and you’ve got someone (or a company) maintaining it. What do you do- put the event information on your website, or on Facebook? Unfortunately, lots of people will answer “Facebook” to this question. The correct answer is your website first, Facebook to raise awareness, but the information and (if appropriate) booking/RSVP information should be on your website. The Student Radio Awards are the latest in a long line of offenders to reach my inbox, informing me gleefully of a page on their website which promptly takes you to Facebook so you can RSVP to their nomination parties. As a person who chooses not to use Facebook, I now cannot interact with these events, and if I want to RSVP I’d have to reactivate my account, which I’m not doing for an SRA party. I emailed back, and got the response “But you can still see the information without joining Facebook”. True enough- but here’s the rub.

• Your organization does not control access to that data any more. Facebook does.
• Your organization is specifically endorsing Facebook at this stage. Do you really want to do that, with all the privacy and security issues flaring up right now?
• Your organization is specifically blocking people who do not want to use Facebook from using your services.
• To view your data, users are forced to interact with Facebook servers. Users with privacy or security concerns can now not use your service.
• Facebook is not your server. It’s unlikely that your server is blocked from any workplace networks- how about Facebook?

Some of my other brethren in the webdevel community (SEO and marketing experts particularly) will be flaunting the positive aspects – reaching more people, less operational costs for your own site and infrastructure, being seen to be ‘social’ (which is a big deal for some organizations who have talked themselves into it thanks to overzealous marketing gurus wanting to be seen to keep up with emergent trends- come on, guys, it’s just a fancy way of sending email newsletters). Do they balance up? Actually, no. The enhanced reach that comes with getting on Facebook’s timeline/front page for people is a big deal. There’s no hiding from that, especially in a university environment where people just don’t seem to understand that Facebook is optional. There’s no harm in using Facebook to link to things on your site, of course- there’s no such thing as bad publicity. But the second you start putting information on Facebook that isn’t on your own infrastructure, you’re damaging yourself. In severe cases you can piss users off in droves.

Facebook is not infrastructure for your organization. Build your own infrastructure- everyone on the web will be better off for it, and your users will thank you.

The privacy issues flaring up at Facebook are serious, and in some states in Germany, sites are being fined for using the Like button as it violates German law. Do you really want to be supporting, encouraging and endorsing the sort of company that supports this degree of invasion of privacy?

If the pile of (in my mind very compelling) reasons why you shouldn’t be using Facebook as infrastructure listed above don’t get you, okay, well let’s look at this another way. Facebook is a company that exists to make money off your data. You, as a company, help them, as a company, get more money (through advertising) in return for some page views. But it wasn’t always this way. In the past, people built interesting and engaging websites, and people actually visited them of their own volition. You didn’t need social engagement to get page hits, because people would visit your site anyway. And you know what? They still do, Facebook or no. And you own it. What you make, you own, you can control precisely as you desire. Facebook you can control however they decide you can control it all one week to the next- everything changes so often that forming solid policies or organizational structures on how to use Facebook is next to impossible. The page views to your site that are effectively the sole reason why organizations started using Facebook meanwhile dwindle as you put more on Facebook and less on your site. In extreme cases you stop updating your site, and get confused as to why people ask why your site’s entirely empty and has no information on it- “oh, it’s on the Facebook page”. More damage to your organization’s reputation and web presence.

Organizations should be proud of their web presence, and shouldn’t just say “Okay, our website’s rubbish, let’s just use Facebook instead”. Fix your website. Build your own infrastructure. If you want to be social, at least use Twitter, which is simple, straightforward, open and easy. It’s hugely popular, has far fewer privacy and data retention concerns and issues inherent to its nature, and is massively more powerful in terms of engagement and potential page hits.

It’s not your infrastructure, though, just as much as Google Plus or Diaspora aren’t. Look at NASA’s tweets- they all link to a NASA website. The Guardian doesn’t link to Facebook. You shouldn’t either.

The aim of the project is this: To build a tool to do real-time analysis of Tweets for any event in terms of the sentiment of those tweets towards the various subjects of an event

I am fairly good at doing simple apps quickly. I had all but one component of this app done by the first Leader’s Debate here in the UK (allowing me to collect my data set for future development- around 185,000 tweets from 35,000 users). I’ve thrown in a handy diagram which details the data collection portion of the app as it stands. But here’s the quick overview:

• Streamer – Uses the Twitter streaming API to receive new tweets and for each tweet, throws them onto the appropriate AMQP exchanges
• Parser – Receives a tweet and loads it into the database. Doesn’t actually do any parsing as such yet, but could be extended to do so (extracting URIs and hashtags are the things I’m thinking of)
• Classifier – Receives a tweet and does clever stuff on it to determine all subjects and associated sentiments, passing the results back to AMQP
• ClassificationLoader – Receives the results from the Classifier and loads them into the database

Now, for starters this app isn’t done yet, so this is all strictly subject to change. For instance, I’d like to have the DB loader pass the tweet on to the classifier instead of the streamer since that’ll let the classifier store with reference to a DB object, and a few things like that. However, this distributed component structure means that I can run multiple copies of every component in parallel to cope with demand, across any number of computers. EC2 included, of course, but I can also use my compute cluster at home where network speed/latency isn’t a huge issue. Right now I don’t need that, but it’s nice to have and doesn’t involve a lot more work. It also lets me be language-agnostic between components, which leads me to…

CL/NLP. Short for computational linguistics/natural language processing, this is a seriously badass area of computer science. It’s still a developing field and a lot of great work is being done in it. As a result, the documentation barely exists, there are no tutorials, no how-to manuals, and what help you have assumes innate knowledge of the field. And I know nothing (Well, I know a fair bit now) about linguistics or computational linguistics or NLP. So, getting started was hard work. I ran into knowtheory, a chap in the Datamapper IRC channel of all places who happened to be a linguist interested in CL and whom has helped out substantially with my methods here.

I’ve gone through about 5 distinct versions and methods for my classifier. The first three were written in Python using the NLTK toolkit, which is great for some stuff but hard to use, especially to get results. That, and using NLTK was giving me very good results but at the cost of speed- several seconds to determine the subjects of a tweet, let alone do sentiment analysis or work out grammatical polarity and all that. Now, getting perfect results at the cost of speed was one way to go, and for all I know it might still be the way to go, but I decided to try a different plan of attack for my fifth attempt. I started fresh in Ruby using the raingrams gem for n-gram analysis, and the classifier gem to perform latent semantic indexing on the tweets.

I boiled this down to a really, really simple proof of concept (It’s worth noting that I spent _days_ on the NLTK approach. Those of you who know me will know that days are very, very rarely used to describe the amount of time I’d spend on one component of an app to get it to a barely-working stage). I figured I could train two trigram models (using sets of three words) for positive and negative sentiment respectively, then use the total probabilistic chance of a given tweet’s words (split into trigrams) appearing in either model as a measure of distance. Positive tweets should have a higher probability in the positively trained model, and a lower probability in the negatively trained one. Neat thing is, this technique sort of worked. I trained LSI to pick up on party names etc, and added common words into an unknown category so that any positive categorization would be quite certain. This doesn’t take into account grammatical polarity or anything like that, but still. Then, using the classifications, I can work out over my initial dataset what the end result was; and here it is:

# Frequencies
Total: 183518 tweets
Labour: 30871, Tory: 35216, LibDem: 25124
# Average Sentiment
#  calculated by sum of (positive_prob - negative_prob)
#  divided by number of tweets for the party
Labour: -0.000217104050691102
Tory: -0.000247080522382047
LibDem: 0.000394512163310021
# Total time for data loading, training and computation
# I could speed this up with rb-gsl but didn't have it installed
real    13m5.759s
user    12m35.800s
sys     0m12.170s

So according to my algorithm, the liberal democrats did very well while labour and especially tories didn’t do so well. Which, if you read the papers, actually fits pretty well. However, algorithmically speaking the individual results on some tweets can be pretty far out, and so there’s lots of room for improvement. And my final approach I think has to consider part-of-speech tagging and chunking, but I need to work out a way to do that faster to be able to integrate it into a realtime app.

All in all, working on Backchat has so far been hugely rewarding and I’ve learned a lot. I’m looking further into CL/NLP and looking at the fields of neural network classifiers for potentially improved results, all of which is great fun to learn about and implement. And hopefully before next Thursday I’ll have a brand new app ready to go for the second Leader’s Debate!

Anyway. We got a lot of fluff, this year. Aside from further elaboration on stuff already announced, there were actually no major announcements made at fanfest. We did have some interesting info about New Eden, CCP’s EVE-Online-Online website. And there was some evidence (gasp!) that CCP were listening to third party developer suggestions at the API roundtable.

There was almost enough minor stuff announced to make it worthwhile. We did get a release date for Dominion – 1st December 2009. But no New Eden with the launch. And knowing CCP we’ll probably not get API changes till a bit after that. What’s really awesome though is that we will be getting new APIs. I’m just hoping they’re useful APIs…

Anyway, while I was sitting at home being mostly bored, I decided I’d had enough pressing F5 on the Twitter search page, and put together a website (ff.mmmetrics.co.uk – it’s down now) to grab EVE fanfest feeds from Twitter and Flickr. This became popular enough within a few hours that we had to rip it off the server and give it it’s own Amazon EC2 virtual server, as it was in danger of crashing ISKsense and EVE Metrics. Doh. A wild success, in any case, for a simple but handy website. What the website did make us realise is how little headroom we have on our current server. We kinda knew that already but it did make the point quite well.

EVE Metrics 2.1 has launched mostly well but we’re still having issues with the API processing code. Makurid has been working hard to pin down the cause of the problems and destroy it while I’ve been fixing up servers and moving sites around, and we’re getting a bit closer to having a complete fix. We’re not there yet, but we will be soon with any luck.