Re: SIOC auto-crwaling

[Alexandre Passant]

Hi

Unless you have access to real time updates on these different sources (using a publish-subscribe system like pubsubhubbub) you probably need to monitor the RSS feeds regularly to find new items to crawl.

Hope that helps,

Alex.

--
Dr. Alexandre Passant - @terraces
Founder, CEO - seevl.net - @seevl
Sent with Sparrow

On Monday 12 November 2012 at 07:01, Doanh Duong wrote:

Hi SIOC-Dev Team,

Currently, I'm building a Semantic Web data storage to hold all social web resources from Blog, Wiki, Facebook, Twitter, etc and represent it by using SIOC, FOAF ontology. My problem is how I can crawl my social post, comment even once it was created or updated immediately. I have list of users, each user has many accounts and also Url for these social sites.

Can you give me any advice?

Thanks in advanced.

Doanh

--
You received this message because you are subscribed to the Google Groups "SIOC-Dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/sioc-dev/-/IkR-xsbaCucJ.
To post to this group, send email to sioc...@googlegroups.com.
To unsubscribe from this group, send email to sioc-dev+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sioc-dev?hl=en.

[Doanh Duong]

Hi Alex,

In fact, I could not access real time there sites because it comes from over the world so that limit me at least. RSS feed, I have idea about it, similar a scanning service and it scan these sites by scheduled and find out a new or updated one. However, as you know scanning might become so hard and heavier while we only need some of them in amount. I continue to research on it and update you here.

Thanks your advice.

Doanh

[Alexandre Passant]

Can't you just restrict the feeds to the ones you need (eg by hashtag)

Also, Facebook has a real Tom update system that you may consider depending on your needs:

https://developers.facebook.com/docs/reference/api/realtime/

--
Dr. Alexandre Passant - @terraces
Founder, CEO - seevl.net - @seevl
Sent with Sparrow

To view this discussion on the web visit https://groups.google.com/d/msg/sioc-dev/-/OhcHoIM1rvgJ.