Article Navigator

Navigate By : 
[Article Index]

Subscribe by Email

Your email:

My Website Grade

Dharmesh Shah's Blog

Current Articles | RSS Feed RSS Feed

The Reddit Recommendation Engine: Does It Work At All?

Digg digg it | Reddit reddit | del.icio.us del.icio.us | StumbleUpon StumbleUpon 

I’ve been a user of reddit.com for a little while now.  Part of the value of reddit.com is its recommendation engine.  Based on what kinds of links you vote up/down, reddit comes up with a list of recommended links in its library.

 

However, in my experience, the system just doesn’t seem to work very well.  I’ve done my best to “train” the system (as suggested by the reddit folks) by signaling that I like articles about software, technology, startups, etc. and don’t like articles about politics, goldfish(?) and other weird stuff.  However, the system continues to recommend articles to me that are of absolutely no interest.  And, I’m a reasonably simple guy and not that hard to figure out.

 

This got me to thinking a bit:  How does the engine actually work?  Recognizing that the algorithm is proprietary and private, I still have some high-level guesses.  I think the algorithm is probably one (or a combination) of the following approaches:

 

  1. The engine tries to match me up with others that voted similarly.  Then, it finds articles that the other person liked and recommends them to me.
  2. The engine does some sort of “content matching” using keywords in the articles to figure out what I like or don’t like based on prior votes.

 

In either of the above scenarios, the engine probably also factors in the popularity of articles overall.

 

Now, here’s my issue:  Without having some idea of how the engine works, how do we trust it?  For weeks, I’ve been struggling with figuring out why reddit.com recommends what it does.  When it sends me strange recommendations, I simply write it off to:  “Its probably just me – its got to be working for others…”.  So, that’s my question:  Is it working for you?

 

Though it’s a bit of a stretch, I relate this back to the days when encryption algorithms were kept “secret”.  Now, the common convention is to disclose the encryption algorithm as a way for the community to ensure that the approach is valid and legitimate.  This would quickly expose anyone that was using a simple obfuscation algorithm and passing it off as strong encryption.  Similarly, how do we know that the reddit recommendation engine isn’t flawed – or worse, mostly random?  I’m not saying that it is, but as an individual user, how do you know it isn’t?

 

In any case, if your experiences vary greatly from mine (and you think the reddit engine is the best thing since sliced bread), post a comment.  Would love to hear about positive experiences.  I know that reddit can’t divulge their “super-secret recommendation algorithm”, but I’m curious to know if its just me that is having the problem, or others are too.

 

 

Comments

reddit just flat out sucks.
Posted @ Tuesday, April 11, 2006 12:00 PM by a
Why would you need to "trust" it? Either the results are usable or they aren't. For all I care, they have a bunch of monkeys looking at my votes pick what other links I might enjoy.
Posted @ Tuesday, April 11, 2006 12:02 PM by a2800276
I agree. I really like reddit, and have been using it as my main "find-interesting-stuff-on-the-net" site, but I use the "hot" and "new" sections almost exclusively. The "recommended" section does a terrible job of finding links of particular interest to me. In fact, a couple of times, a link that I had previously voted down was #1 on my "recommended" section. What's that all about?
Posted @ Tuesday, April 11, 2006 12:30 PM by Thomas Anderson
Just chiming in on this... I love reddit, but the recommendations have been way off for me. I, also, only use hot and new.
Posted @ Tuesday, April 11, 2006 1:12 PM by bevoreo
and where did "rising" just go?
Posted @ Tuesday, April 11, 2006 1:19 PM by coward
"Rising" is now under the "new" page as a viewing option. They wanted to clean up the top bar, so they moved several buttons into the "browse" page. (ex: "top")
Posted @ Tuesday, April 11, 2006 1:39 PM by Austin
Rising is now and option on the "new" tab. And it seems the recommended section has given up on me -- it keeps giving me a list of very old links, all of which I've already seen (and voted on), I never get a new link.
Posted @ Tuesday, April 11, 2006 1:45 PM by pdw
I'm having an opposite reaction this week. It seems like it suddenly got a lot better at picking things. I'm finding that I read nearly everything on the recommended page, and vote up about half of it.

In fact, I was just about to blog about it in a positive light. Maybe I'm just momentarily lucky.
Posted @ Tuesday, April 11, 2006 1:50 PM by Thayne
The system is greatly biased. You should only trust it if the most popular content matches your own interests. And just because your interests overlap with what is popular, your interests will not necessarily be popular. The system is incredibly narrow-minded and incestual this way. It's not really concerned with true information and certainly discourages critical thought and insight.
Posted @ Tuesday, April 11, 2006 1:54 PM by john swankadillo
I'd like to have a way to mod down blog-jacked articles without affecting the algorithm's perception of the types of news I'd like to read. I don't do it very often, but I have occasionally voted down several articles by overt blog-jackers, and as a result of this (partially, at least), my recommended page is pretty random.
Posted @ Tuesday, April 11, 2006 2:17 PM by Lo
Reddit's recommendation engine is just a load of ...marketing.
Posted @ Tuesday, April 11, 2006 8:06 PM by Bob
I read it because it has a lot of links to good articles that change almost every day. Plus I like to see what other people consider interesting...
Posted @ Tuesday, April 11, 2006 8:23 PM by Tim
I'm working on a different way to do personalized recommendations. The method could be used in a site like reddit.
The prototype (which should be ready in a week or two) gives the submitter of a URL control over who sees it. In other words, it is the submitter who is responsible for the quality of recommendations for his/her submission.

There's a catch of course. As a submitter, you need to be careful about who will see your submission. If you spam many people with your submission, they may just vote it down thus reducing its score.

Although the submitter has control over who sees the submission, the score of the URL determines its ranking within the list of recommendations for a user.

So it's important that you target your submission appropriately to maximize its score, so that it will rank more highly as a recommendation.

We now have a real-time game. You make a submission and target it. Targeting is based on previous submissions. You could say something like target people who liked this earlier submission, avoid people who disliked this submission, etc. You could have many such targeting rules. These targeting rules combine to give users a "targeting score" for that submission. If the targeting score is above a certain threshold, then the user will see the submission as a recommendation.

You may change your targeting rules frequently after you have made a submission. If your initial targeting does not appear to be working well, you can try to tweak it. Speed matters. If you tweak it too slowly, the score of your submission might go down quickly.

If you would like to help me test the prototype, please contact me at amichail@gmail.com. I should have something ready for testing within a week and ready for release within 2 weeks
Posted @ Tuesday, April 11, 2006 8:54 PM by Amir Michail
Doesn't work for me too. I think it's better to remove "Recommended" and replace it with "Rising". K.I.S.S.!
Posted @ Wednesday, April 12, 2006 1:18 AM by almkglor
You can try an early prototype of the idea I described here:

http://targetyournews.com/
Posted @ Wednesday, April 12, 2006 2:06 AM by Amir Michail
While I like what the reddit guys have attempted to do with their system, I am of the opinion that their design choices ultimately lead them down a path where really good recommendations aren't possible. To me the problem is that reddit is really based on the concept of popularity, and recommendations are just bolted on as an after thought. If you are into the whole popularity concept *and* you think approximately like the crowd on reddit, it's a great site. If on the other hand you have a bit more unique tastes...well reddit/digg/slashdot aren't really designed with you in mind.

I was really hoping reddit would be the news system I have been envisioning for a few years, unfortunately after using reddit extensively it really is more designed around the concept of popularity than anything. The problem is that a simple up/down vote isn't really a good mechanism for personal relevance, but it's a pretty good method for a group popularity filter. Unfortunately that means you really are just a small blip in the overall collective, and your voice doesn't really matter.

I've also tried very hard to train the reddit recommendation engine, and unfortunately it just doesn't work well at all for me.

So with all that said, I've built a system from the ground up with the concept that you have totally unique tastes. Rather than letting everyone vote equally on an article, my system learns who you agree with, how strongly you agree, and what topics you agree on. Additionally there facilities for viewing articles within tags, and a really powerful feature called channels that lets you aggregate posts by tags, users, comments, etc.

The whole point is that Zifus (my system) is designed to make the news on the web personally relevant to you, instead of turning you into some sub-optimal group filter for the rest of the site.

If you are interested in testing out zifus, we just moved into beta testing and we are bringing on more testers. You can drop me a line at: betatest.zifus@recursor.net, sometimes the spammers burnout my spamgourmet accounts very fast, so if I don't respond within a day or two, send me another e-mail.

Posted @ Wednesday, April 12, 2006 2:19 AM by Beowulf
Nah, the reddit recommendation engine has never worked very well for me either. Right now, I have "DRM key to Linux's consumer success? " as my #15 recommended link, when I've never voted up any Linux link or even anything closely related (other than Paul Graham and Joel Spolsky articles). Do I need to vote down these types of links in order to remove them from my recommendation screen?
Posted @ Wednesday, April 12, 2006 11:36 AM by SweetP
Nope - doesn't work - flatout sucks. I doubt the first variant ( matching people with similar votes ) without any popularity modification wouldn't work, so I don't think they use that one.
Posted @ Thursday, April 13, 2006 4:39 AM by zack
The site is really fast, so I'm guessing they use "item-based" collaborative filtering algorithm, where they can pre-computed item-item relationship to speed up the page response. Collaborative filteringn is a big research topic and I'm "suprised" that reddit's recommendation engine is so weak...
Posted @ Friday, April 14, 2006 7:29 PM by guest
You should look at the proposed scheme at shunya.in. We claim to be building a completely transparent voting scheme to maintain popular community stories/new. Maybe we are naive, but hey, we love being naive.
Posted @ Sunday, April 30, 2006 6:51 AM by Shunya
How to distinguish hight quality Supplier of Quality Swiss Rolex Replicas - Swiss Replica Watch - wholesale replica - fake rolex - replica handbag - replica bags, lv, coach, chanel...
at http://www.replica-supplier.com
Posted @ Sunday, April 15, 2007 3:16 AM by Mandy
We are glad to offer best watch,replica watches,fake watch at
http://www.replica-watch-source.com or http://www.designer-replica-watch.com
Posted @ Sunday, April 15, 2007 3:17 AM by James
http://www.nextstudent.com/scholarship_search/scholarship_search.asp

Wedding coordinators and day planners that are based in Phoenix, Scottsdale, and Sedona. Serving all areas of Arizona.
Posted @ Thursday, June 07, 2007 3:28 AM by Lisa Tucker
Reddit doesn't appear to have a personalized recommender system. It appears to simply take an aggregate rating for articles to support the masses. You did successfully describe the k-nn collaborative filtering algorithm, but I'm sure they aren't using it. So long as there is less than 5% coordinated shilling k-nn is something like 98% accurate at predicting what you will like.
Posted @ Sunday, October 21, 2007 1:01 PM by GLNoob
I should mention that k-nn is also expensive to run and the volatility of reddit may make it inappropriate to try to run real time. Most update their matricies over night (which takes hours). There are constant time algorithms, but they have caveats galore and wouldn't help much in the case of reddit. Generally recommendation algorithms are not patented (since they are mathematical equations) and are open to public use.
Posted @ Sunday, October 21, 2007 1:04 PM by GLNoob
It used to be only semi-relevant, but of late it seems as if it has tossed out my votes and simply shows me *almost* the same page as the hot page.
Posted @ Monday, October 22, 2007 1:59 AM by Juliet
Allow me to add my vote to the 'recommendations don't work' bank.
Posted @ Monday, October 22, 2007 8:38 AM by Ben
Reddit's recomendation engine sucks.

What sucks even *more* are the noobs/ex-VB dickheads using it like it was the Usenet.
Posted @ Monday, October 22, 2007 3:18 PM by JT
Post Comment
Name
 *
Email
 *
Website (optional)
Comment
 *

Allowed tags: <a> link, <b> bold, <i> italics

Receive email when someone replies.