Simulating Hacker News Homepage (again)

[Michael Mahemoff]

This was raised earlier [1], but the URL mentioned [2] is returning results days-weeks old. I've tried tweaking the "72" figure all the way down to ~1, but it returns results 2 days old.

I'm aware there's the RSS feeds and the new bigrss [3 (thanks Andre)], but they're not so useful for my Reader app [4], because they don't include points or comments count. I had previously used the unofficial iHackerNews API, but found it was timing out more often than ideal.

1. http://groups.google.com/group/hnsearch/browse_thread/thread/a9092b76eb34aeb6

2. http://www.google.com/url?sa=D&q=http://api.thriftdb.com/api.hnsearch.com/items/_search%3Flimit%3D30%26sortby%3Dproduct(points,pow(2,div(div(ms(create_ts,NOW),3600000),72)))%2520desc%26pretty_print%3Dtrue&usg=AFQjCNHhwO4QH2S-RfWAZN_QgIOaepwULw

3. http://ycombinator.com/newsnews.html

4. http://project.mahemoff.com/hn

[lightyrs]

Agreed, this is a major problem.  Andres has been great in that he's very responsive, however, this needs to addressed.

[lightyrs]

I think the HNSearch homepage should always have an up-to-date one-liner of the exact query needed to reproduce the HN homepage.  It's been this weird cat-and-mouse game thus far and I don't think it serves anyones best interests.  If PG really wants the API to replace scrapers, we need this information to be easily accessible.  Andres, please consider this.  That being said, I do really appreciate all of the time and effort that you have put into the API thus far.

Thanks,

Harris

[Harish Agarwal]

Hey all,

We've been making major infrastructure changes lately and the indexer
has been going down frequently. Sorry about that.

Moving forward we are going to focus on making the indexer more
robust, hopefully this will help prevent these sorts of issues in the
future.

Harish

[lightyrs]

Thanks Harish.  Noticing improvements already.

[Andres Morey]

I added methods to insert HNSearch item ids into the HN feeds here:

http://www.hnsearch.com/rss

http://www.hnsearch.com/bigrss

By parsing the feeds you can obtain item ids and then fetch them using the get_multi method:

http://api.thriftdb.com/api.hnsearch.com/items/_bulk/get_multi?ids=3145237-337e2,3145118-2e3b9,3144628-86545

I'm not sure if this is the best long term solution so keep track of this thread for any changes. Let me know if that helps.

Andres

[Michael Mahemoff]

Many thanks Andres.

I'm just wondering if you'd consider adding the auxiliary fields, since it's fairly small as feeds go (containing only a title, not the content). author and create_ts are standard RSS fields anyway, so the only non-standard data would be num_comments and points.

I could certainly get by with a batch fetch, but I suspect those few extra fields will save a lot of round-tripping, making users happier. (In this case, I'd be using Google's Ajax Feed API, so it would also cut the load from HNSearch's servers altogether.)

Either way, thanks for adding the IDs.

[Andres Morey]

My only hesitation in adding metadata to the rss feed is that the HNSearch server has to make an extra request to ThriftDB and, in terms of latency, it's no faster for HNSearch to make the call than any other machine running on AWS.

Also, HNSearch data will always lag behind HN (by at least 5 minutes) so this doesn't seem like a good long term solution. I have some ideas on better solutions but I'll need to think about it some more.

Anyways, I added <username>, <create_ts>, <num_comments>, <points> to the rss feeds (where available):

http://www.hnsearch.com/rss

http://www.hnsearch.com/bigrss

Andres

[Michael Mahemoff]

Wow, that's awesome, thanks! Looks like this pretty much enough to replicate the homepage.

At least for my app, I can certainly deal with a lag and would be perfectly fine with the RSS was cached for a few minutes. Maybe you could just cache the RSS and longer-term, build in XMPP/PSHB for the (much fewer, I suspect) apps that really need real-time info.

[Michael Mahemoff]

Got it working now against the new RSS - you can see it in use here:
https://github.com/mahemoff/hackernews/blob/2a258f90fb91543ca6ce1aa2162e88b19a9264f4/index.coffee#L48

And live at:

http://project.mahemoff.com/hn/

If anyone else is using the Google Ajax Feed API, one gotcha is it caches aggressively, which is why the above code includes a random nocache param.

[Andres Morey]

Hmm - I thought the HN RSS feed returned the stories on the front page. Are you trying to recreate the HN homepage?

Andres

[Michael Mahemoff]

Not sure what you mean? I'm trying to show the same stories as the homepage, but with a different presentation. Also, the stories are sorted by created_ts on my app.

[Andres Morey]

Nevermind - I made a request to the rss url and got some cached data.

Andres