The series continues. (previously: intro and part 1)

Part 2: Analysing a trust-aware internet transaction: del.icio.us network

In Part 1, we analysed a typical transaction on reddit, an article aggregator with the principal function of recommending you interesting articles to read when you are bored or just in need of some news.

Today, we look at a service with a very similar premise - del.icio.us network. While the premise is exactly the same (give you a list of articles which might be interesting), and while the basic notion is similar (a disconnected set of people basically ‘vote’ on stories), the actual implementation is completely different. Specifically, the way trust is interweaved into the the network feature compared to reddit’s system is entirely different.

Where reddit seems to actively try to eliminate trust as a factor (it is for example impossible to see who votes for what, only comments and submissions can be found, though not easily) - del.icio.us network works solely on trust relationships.

Let’s revisit the same transaction of part 1, but this time with del.icio.us network.

del.icio.us recommending me something to read

I go to my del.icio.us/network page. I will need to trust the operators of del.icio.us, which can be problematic, as del.icio.us is owned by Yahoo, a business. Businesses, in theory, have no morals. Fortunately in practice I can take off my paranoia hat and trust healthy competition - google does not point me to any convincing evidence that Yahoo is trying to surreptiously hawk political views or allow unmarked advertising. I’ll trust this site - enough, at least, to let it recommend articles to me.

The network page is a lot like any reddit page - a bunch of articles, some with very obvious descriptions, some less so. There’s some extra fluff (total number of del.icio.us users who bookmarked a given article, and a tag list). While potentially interesting, from a trust point of view this information is just as useless as reddit’s article score.

The next issue of trust is, for each article that appears here, if I can actually trust that I should give it my due attention. This is where del.icio.us/network differs from reddit and digg: An article is on that page ONLY because one of my direct connections thought it was sufficiently cool to bookmark it. I trust those people I manually add to my delicious network. Thus I can directly trust the articles that show up on my network page. The exact mechanism of trust is left to the user; I may trust one of my network contacts because they are my friend. I may trust someone else because I like his blog and the articles he links to there seem interesting. Regardless of why I trust my network contacts - the point is that I personally trust them.

The final step of trust is - once I decide to read an article, can I trust that the operators of the server that hosts the article are trustworthy? This step is also much more adequately addressed: One of my personally trusted contacts saw fit to go through the trouble of bookmarking it. At least a modicum of due diligence has probably been applied.

From a trust point of view then, del.icio.us/network is on the up and up. There is no problem here - trust-wise, this system will not collapse under the weight of its own popularity. Of some schmoe manages to sign up for a del.icio.us account and starts bookmarking spam, tripe, and drivel, I don’t even notice.

Basically, my network is a wheel: I’m at the center, with all my connections arranged around me, feeding article recommendations to me.

There’s even a responsibility system built in: If one of the users in my network keeps bookmarking crappy articles, I can remove them. One common problem with responsibility (a.k.a. karma systems - scores for users) is that the trust issue isn’t addressed at all: The karma of any given user is again determined by untrustable, unaccountable masses. Removing someone from recommending articles to you completely is much more effective from a trust point of view.

Trust is neccessary… but not sufficient

Unfortunately, though, just because you built a system that maintains trust in the transaction, doesn’t mean your idea is any good.

Some problems with del.icio.us:

• Traffic - once you run out of articles, there are no more. On reddit and digg, there are always more stories to read because the pool of submitters is much larger.
• GroupThink - If all the users in your network read the same blogs, work in the same area, and have the same thoughts, your network is very unlikely to bring you new ideas in new topics, or well written arguments for viewpoints you do not hold. In practice large communities suffer just as much (Digg and Reddit have of late sported front pages where every single article is either extolling the virtues of one Ron Paul, presidential candidate for the 2008 elections in the United States of America, or taking the mickey out of George W. Bush).
• Rating - While on reddit each article has a score and thus you can sort them, on del.icio.us an article is either on your network page, or it isn’t. Once your network produces more articles than you can handle, there is no way to prioritize them usefully.

Fortunately, trust can help us out here, if you apply some more of it to del.icio.us network. None of the steps I’m going to explain here have been implemented by del.icio.us yet. It would make for a much better experience if they would.

A wheel does not a network make!

By acknowledging that a network is more than just a wheel with spokes, these problems can be addressed!

In the ‘wheel’ view of a del.icio.us/network, I can actually check out the networks of friends, check out people THEY have deemed fit to add to their network, check out what those people have been posting, and if I like it, add it to my network. That’s one way of solving a dearth of articles: Just add more people to the network.

So, instead of a wheel, I can treat delicious as a connected network:

There’s really no reason why this can’t be done automatically. Anytime I’m out of articles, so to speak, it should be possible to just say: Go to the ‘next layer’ - give me articles recommended by friends of my friends. Trust is more or less multiplicative, after all: If I trust Jack, and Jack trusts Joe (I don’t know Joe), I can trust Joe to some extent. Once 2 layers no longer give me enough articles, I can go to a third layer, ad nauseam.

We can solve the other problems in a similar fashion, but a more holistic approach solves them all.

First, we establish a scoring system on a per-article basis, dependent on the network. The network of del.icio.us basically consists of users, connected to each other (each connection represents someone being in the network of someone else). Now add the articles themselves to this network: Anytime I bookmark an article, I am connected to the article directly. Anytime a friend of mine bookmarks it, I’m connected to it through my friend.

It is of course possible that I’m connected to an article in a number of ways. A friend of a friend bookmarked it, a colleague’s brother’s girlfriend’s classmate bookmarked it, and one of the bloggers read by someone whose opinions I admire bookmarked it, for example. In the network this is represented by the network by having 3 different ‘paths’ I can take to arrive at the article.

These paths can be distilled into one final personalized score. Each connection takes a chunk of 80% out of the total score - so a friend’s friend’s friend, 3 steps, is .8 * .8 * .8 = 0.512 in total score. For multiple different paths, you can’t just sum them up (or you could end up with a score above 100%), but there are a number of algorithms (naively: of all paths, take the highest scoring, divide by 2. Take the next highest scoring, divide by 4. Take the third highest scoring, divide it by 8, ad nauseam, then add them all up. This number can never exceed 100%. Another way of doing this is to consider each link in the network as a resistor in an electric circuit. Multiple resistors placed in a series multiply their resisting effects and thus reduce current. However, multiple resistors placed in parallel lessen the effect, but, whatever you do, you can never get more power out than you put in. Now replace resistors with links on the network and you have an algorithm!)

This scoring/recommendation algorithm can even be extended to del.icio.us users: The score of a user is then entirely dependent on how well he’s connected to your own network (though relying too much on this can lead to GroupThink!).

Such a system solves all 3 problems. To wit:

Traffic

Research in social networks finds that usually social networks are virtually completely connected. There’s a path from any one person to any other. Thus, it’s possible to derive a score for every article and you can just keep reading indefinitely, though, of course, as you keep reading, each further article has a lower score.

There’s some excellent research by GustavoG on the social network of Flickr (also a web app that allows you to set up a network of friends). Very pretty pictures of tightly interwoven networks, such as this one:

Rating

As already explained, any given article is no longer a simple yes/no proposal: Articles recommended by a number of your direct friends rate highly. Articles only recommended by one distant link (A friend’s friend’s friend, and that’s it) rate lowly.

GroupThink

This is where it gets very interesting. Because everyone builds their own unique community, GroupThink is no longer a virtual guarantee. For example, on digg or reddit, if a well written article that happends to put a ‘taboo’ topic in a good light (like Java, Microsoft, George Bush, traditional media, and a few others), or a ‘holy’ topic in a bad light (Ruby on Rails, web2.0, digg/reddit itself, Apple, Linux, and a few others), chances are very high it gets drowned out in the noise of the crowd. Even if all the people whose judgement I actually trust did vote it up, I never see it. Contrast this your own unique community, where articles at least have a chance.

There are two forms of GroupThink: Accidental and intentional. On both Digg and Reddit, you occasionally see a post imploring to put an end to the flood of the latest meme-of-the-day posts. Ironically these also get voted up with some frequence. Clearly then not all GroupThink is actually desired by those experiencing it. In a social network this GroupThink is eliminated; you can simply hunt down which elements in your network are fielding the majority of an onslaught of a certain meme, and toss them from your network or at least lower your level of trust in them.

The other type is intentional: Where a reader actually wants to read more about the same topic over and over again. There’s not all that much to be done; trying to force reading other things onto such a person is tantamount to censure and very hard to distinguish from forced propaganda.

In practice, in real life, GroupThink is somewhat rare, because you have friends from many places. Colleagues, family, old school buddies - friends of people you’ve dated that you kept in contact with, etcetera. If these real life bonds also exist in your del.icio.us network, ostensibly the chance of GroupThink is much reduced.

I could be wrong, but a system like that sounds like the ultimate source of articles. As much or as little as you want to read, resilient to GroupThink, nearly impossible to spam, and ever evolving to your tastes. Unfortunately, as far as I know, nothing quite like it exists just yet.

… or does it? The remarkable quality of the early phase

A version of this ultimate article recommendation engine did exist, briefly.

reddit itself, meets this system! At least, it did, in the first few months after the launch. The users of reddit back then amounted to a single connected social network. A number of important features weren’t there (all votes are equal instead of being attenuated by the distance in ‘friend links’ from you, for example), but on the whole this was it. If you happend to use reddit in those days, or you know someone who has (I fortunately managed to catch the tail end of those days), you may hear about or remember the amazing quality of articles.

This idea actually can be observed in many budding social networks. For a little while, Orkut (google’s ‘myspace’) was a trove of excellent networking opportunities. This was back when Orkut required very scarce invites.

Invites are an excellent way to keep the size of a social network into the efficient phase as long as you can, but of course it does restrict growth - by its very definition that’s how it manages to keep the efficiency of the social network high. In fact, a number of more or less ’secret’ smaller social networks that work on invites and a strong sense of responsibility (a misbehaving user gets kicked, and the one who invited the abuser also gets kicked!) have been running strong for years. The one problem with that tactic is that it can’t scale.

A trust network can!

The final part 3 will be posted the day after tomorrow (Friday evening). In it, expanding this idea to other walks of the web and of life in general, the importance of identity in such a trust-bound world, and how Identity 2.0 and open APIs are the beginning of a brave new world. As an encore, part 3 will also briefly discuss a problem I’ve so far omitted: Doing all these scoring calculations is computationally speaking extremely difficult. spoiler: There’s a way out of it, more or less!

To continue reading, go to part 3.

This entry was posted on Thursday, June 14th, 2007 at 2:51 and is filed under fundamentals, social, startup, trust, webapplications.