OMG Automatic Reddit

[Wolf Tivy]

BACKGROUND

So I was just thinking about the demise of RSS, the philosophy that names should not matter, the fact that names do matter, and machine learning.

Why is an RSS reader interesting? Why is it a good idea to watch specific blogs? What would be the alternative? Why not just look at the reddit or HN frontpage?

The quality of output on a blog is self-correlated relative to the overall quality of the internet. In other words, a historically good blog is likely to stay good, which means finding good authors and just watching their output allows you to read better stuff without scouring the whole internet. Hence, RSS readers.

Ok, so it seems like sites like reddit and HN should serve the same purpose or better by actually scouring the whole internet and then rating everything for quality. A problem is that not everyone is interested in exactly the same stuff as you, so the you::quality of the reddit front page is not as good as it could be.

Hence subreddits. Things get categorized, and you pick a few interesting categories and watch them. This is essentially back to the RSS reader.

I think these systems (RSS readers, reddit) were not designed with a solid theoretical understanding of the actual problem they are solving, and are not implemented in a way that robustly solves the actual problem. I think this is an inefficiency to be captured.

A DIFFERENT IDEA:

Gather all the articles in the internet, updating as they are published. Do this by grabbing all the rss feeds, properly incentivised crowdsourcing, whatever. Ignore quality for this part. You can bootstrap manually with a few interesting sources and no community; it doesn't literally have to be the whole internet.

Present to your users a page of recommended links, possibly with short summaries (or long titles). They can remove articles from the list because they sucked, mark them as "more like this", and maybe save them for later reference in a "favorites" or something. Whatever. The page of links is articles that the user has not read, that are estimated to be high value (indicated by the user clicking "more like this"). They need not be current news; there is already plenty of good stuff on the internet.

Extract features from the articles, like terms used, author, blog, google ranking, user votes, basically every piece of evidence you can get your digital hands on. Build a model that predicts the value of a given article to a given user from these features. Use the model to feed that user's page. You should be able to do at least as well as "anything from blogs the user likes are good for the user", and probably a lot better.

User's tastes are not completely independent, so people who like X may also like Y, from which you can infer that user A who likes X will also like Y. You can get rather sophisticated with the modelling here. Think clustering, etc.

Putting this all together, we get a content aggregator that automatically adjusts to the users preferences, and is able to predict accurately what the user will find valuable, possibly to the point of optimally solving the content aggregation/filtering problem that reddit/HN/RSS/etc only hint at. The idea is to deliver as much value to the user as possible given a limited amount of time that the user has to read stuff.

IMPLEMENTATION NOTES

Current content rating systems (upvoting, etc) don't generally use good machine learning practices to actually solve the problem. Jsalvatier and I investigated using bayesian hierarchical models instead of dumb upvotes and downvotes to rate content. It looks pretty promising, actually. I'd recommend using real machine learning for the above. Being dogmatic about it is probably a bad idea though. The point is to make the best possible front page for the user, not push the envelope in bayesian models or whatever.

Human curators can vastly outperform current AI technology when available, so especially in the early days of building such a system, it would be wise to use lots of human intervention to power the model.

Might be fun to use redirects or whatever to track which links the user has read, so that the site can bug them later with questions (you read article X yesterday, how did you find it?). Users don't reliably do things unless prompted, so this could allow the model to get more information when the VOI is high. Be careful with annoying popups.

The site should be useful for busy professionals with little time who want to read the best of the internet. Count words to estimate reading time.

The monetization model is obvious. Have sponsors pay to get links uprated. Better make this useful to the user, or the model will learn that ads suck and simply route around. If you're clever and lucky, you may even be able to get users to pay for something.

I don't know if commenting or any form of community is useful at all. If so, only to gather users to feed the rating system. Should explicitly make a point of linking to the HN/reddit/etc discussion.

How to gather users is troublesome. What causes a user to create more users?

Secure the fuck out of the database as you will be building highly specific models of the person's interests.

CONCLUSION

If this exists, pls link me. If not, please build it. Does anyone need a programming/startup project?

I've spent an hour on this. Make it worth it.

[Andrew McKnight]

I'm not sure what algorithms StumbleUpon uses, but it seems to be very similar on the surface.

--
 
---
You received this message because you are subscribed to the Google Groups "Vancouver Rationalists" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vancouver-rationa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Wolf Tivy]

So it does. I'll check them out and see if it meets the needs.

[evan]

I cannot code for shit. Nor can I generate the digital machinery for
Bayesian or machine learning models for shit, either. That being said, for
what little it's worth, I like the sound of this idea, and I would be
willing to volunteer my menial labor to see this project come to fruition.

Have you considered posting this idea to a wider forum, such as the LW Facebook group, or the LW Discussion or forums?

[Wolf Tivy]

No point volunteering if nobody is headlining.

Ideas are cheap, no one cares. Posting it around doesn't get me anything. I just wondered if any of you wanted a cool project.

--

[Kenneth Bruskiewicz]

> That being said, for what little it's worth, I like the sound of this idea, and I would be willing to volunteer my menial labor to see this project come to fruition.

Learn to program!

--

 
---
You received this message because you are subscribed to the Google Groups "Vancouver Rationalists" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vancouver-rationa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

"It's immoral for a sucker to keep his money." - Canada Jones

"There's a sucker born every minute." - Attributed to P.T. Barnum.

Litany against Barnum-Jones:
If the fair market value of snow is X, I wish to pay X for snow. If the fair market value of snow is not X, I do not wish to pay X for snow. Unless the auctioneer is a sucker.

Ignore the following. It is a nonsense sentence that disables Google ads from displaying next to my emails by triggering sensitive keywords. 

I enjoy the massacre of ads. This sentence will slaughter ads without a messy bloodbath.

[Kenneth Bruskiewicz]

> Putting this all together, we get a content aggregator that automatically adjusts to the users preferences, and is able to predict accurately what the user will find valuable, possibly to the point of optimally solving the content aggregation/filtering problem that reddit/HN/RSS/etc only hint at. The idea is to deliver as much value to the user as possible given a limited amount of time that the user has to read stuff.

After doing some research into Search Engine Optimization, I realized that this is exactly what modern search engines are, implicitly. Someone who wants this kind of service just has to make a Google Plus account and start following people who think like them, even if you aren't going to be followed back.

I'm not sure how willing you are to read articles on SEO but I can dig some up if anyone wants to look into this further.

[Wolf Tivy]

>just

The connotation attached to this word implicitly underestimates how much money/utility there is to be made by smashing trivial inconveniences.

I don't know if you are making this mistake, but there is a trend in technology where someone says "here's this cool thing X that solves problem Y" and another person says "Why do I need your thing? In framework Z, I can just W to solve Y".

[Ramon]

reddit looked into doing this and found it was infeasible I believe

http://www.reddit.com/r/redditdev/comments/dtg4j/want_to_help_reddit_build_a_recommender_a_public/
https://news.ycombinator.com/item?id=3719003
http://www.reddit.com/r/redditdev/comments/lowwf/attempt_2_want_to_help_reddit_build_a_recommender/
https://groups.google.com/forum/?fromgroups#!forum/rrecommender

[Wolf Tivy]

Wow thanks for that link.

Sweet find.

--