Myqron: pull-to-push web services adapter
Neil Garb
I've put together a little web services that takes as input a pull-
type web service. It then polls the service at customizable intervals
and pushes updates (if found) via XML POST to an endpoint of your
choice.
At the moment, it only supports RSS, but if there is demand I can add
support for other pull services such as IMAP, POP, atom, etc.
I developed it quite rapidly, and released it just to gauge demand.
Any feedback (to ne...@myqron.com) would be thoroughly appreciated.
http://myqron.com
Thanks.
Neil Garb
http://codecaine.co.za
Pedro Melo
On Apr 28, 2009, at 12:39 PM, Neil Garb wrote:
> I've put together a little web services that takes as input a pull-
> type web service. It then polls the service at customizable intervals
> and pushes updates (if found) via XML POST to an endpoint of your
> choice.
>
> At the moment, it only supports RSS, but if there is demand I can add
> support for other pull services such as IMAP, POP, atom, etc.
This seems a lot like pubsubhubbub:
http://code.google.com/p/pubsubhubbub/
At least one of the scenarios that pubsubhubbub covers seems exactly
the same.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: me...@simplicidade.org
Use XMPP!
Jeff Lindsay
-jeff
Jeff Lindsay
http://webhooks.org -- HTTP callbacks are the future
http://shdh.org -- A party for hackers and thinkers
http://tigdb.com -- Discover indie games
Brett Slatkin
I agree this is good stuff. Similarly, PubSubHubbub hubs may automatically turn any Atom feed subscription into push by polling. Our demo hub does this right now. This is important because it provides a way to bootstrap adoption of feed pushing without the participation of feed publishers.
I'd say the big difference here is Myqron works only for RSS and Hubbub only works for Atom.
-Brett
On Apr 28, 2009 11:15 AM, "Jeff Lindsay" <prog...@gmail.com> wrote:
Nice work! Simple alternative to Gnip and pubsubhubbub ... I'll be checking it out. I've been waiting for somebody to do this btw.
-jeff
On Tue, Apr 28, 2009 at 4:39 AM, Neil Garb <neil...@gmail.com> wrote: > > > Hi all > > I've put ...
Jeff Lindsay
http://webhooks.org -- HTTP callbacks are the future
http://shdh.org -- A party for hackers and thinkers
http://tigdb.com -- Discover indie games
--~--~---------~--~----~------------~-------~--~----~ You received this message because you are su...
Jeff Lindsay
Hey Brett, I haven't said hi yet. Nice work on hubbub. I've noticed there's been lots of activity recently. Have you seen FETHR? Also, I'm talking with the Identi.ca / Open microblogging guys and they're talking about using hubbub in their system. I looked at the code a while back for hubbub, but I really haven't played with it still ... you should try and come to a SuperHappyDevHouse so we can talk!
-jeff
Neil Garb
This is exactly how this project arose -- there was a requirement for
web hooks which couldn't be satisifed by the provider's resources.
The idea of Myqron, if not its current implementation, empowers the
consumer without the provider supporting web hooks explicitly.
The pubsubhubbub methodology is clearly more formal than my approach,
but I think both approaches are beneficial to the progression of web
hooks as a concept. I wasn't aware of 'Hubbub' until these replies --
I'll certainly be checking it out as soon as I have a chance.
Thanks!
- Neil
On Apr 28, 8:59 pm, Brett Slatkin <bslat...@gmail.com> wrote:
> I agree this is good stuff. Similarly, PubSubHubbub hubs may automatically
> turn any Atom feed subscription into push by polling. Our demo hub does this
> right now. This is important because it provides a way to bootstrap adoption
> of feed pushing without the participation of feed publishers.
>
> I'd say the big difference here is Myqron works only for RSS and Hubbub only
> works for Atom.
>
> -Brett
>
>
> Nice work! Simple alternative to Gnip and pubsubhubbub ... I'll be checking
> it out. I've been waiting for somebody to do this btw.
>
> -jeff
>
> > Hi all > > I've put ...
>
Brett Slatkin
> How convenient!
>
> Hey Brett, I haven't said hi yet. Nice work on hubbub. I've noticed there's
> been lots of activity recently. Have you seen FETHR? Also, I'm talking with
> the Identi.ca / Open microblogging guys and they're talking about using
> hubbub in their system. I looked at the code a while back for hubbub, but I
> really haven't played with it still ... you should try and come to a
> SuperHappyDevHouse so we can talk!
Thanks (re: nice work). And thanks for that post on the WebHooks blog!
Cool stuff.
I met Evan (from identi.ca) just the other day here in SF and we
talked a bit about how Hubbub could fit with OMB. I've also read about
FETHR and I think their model makes sense. I need to dig into their
source more, but it'd be great if they used something like Hubbub for
the message transport.
One thing related to FETHR and OMB that I'd like to see is a mapping
of these microblogging semantics (follow, ack follow, send message,
unfollow) to simple Atom feeds. This would provide the basis for doing
microblogging federation purely through something like Hubbub. I think
such a system could be simpler than other proposals because it
requires nothing more than Atom feeds from the publishers. However,
there are still questions about authentication and discovery that need
to be addressed that aren't covered by the PubSubHubbub draft spec.
I plan to come to the next SHDH if I can. termie's been telling me I
should go for a while. Hopefully I'll see you then.
> This is exactly how this project arose -- there was a requirement for
> web hooks which couldn't be satisifed by the provider's resources.
> The idea of Myqron, if not its current implementation, empowers the
> consumer without the provider supporting web hooks explicitly.
>
> The pubsubhubbub methodology is clearly more formal than my approach,
> but I think both approaches are beneficial to the progression of web
> hooks as a concept. I wasn't aware of 'Hubbub' until these replies --
> I'll certainly be checking it out as soon as I have a chance.
Cool man! I like the tag-line for this idea is: "Turn poll into push".
One distinct choice we've made so far is to support Atom and not RSS.
Maybe there could be an RSS to Atom bridge on the hub (and
vice-versa)? Then consumers can always consume Atom, regardless of the
publisher's format, and be done with it. Otherwise, do you have any
plans to opensource Myqron, or do you plan to run it as a service?
-Brett
Neil Garb
> Otherwise, do you have any plans to opensource Myqron, or do you plan to run it as a service?
I definitely think it'll always remain a service -- I can't imagine
any scenario in which it would be useful otherwise. My plan for the
moment is to refine the user experience and fine-tune the polling. If
the demand arises, I'd be happy to open it up. One definite intention
of mine to allow users to write their own adapters, potentially in a
language of their choice, but I haven't fully thought out the
mechanics.
- Neil
Jeff Lindsay
You can use webhooks to let people write adhoc adapters (see DabbleDB plugin system), and use that infrastructure to write an adapter handler service that's open source. Provide a little DSL to let people write adapter handlers and you're set. See GitHub's service integration method.
I think you should run the service as a community webservice. I've been working out a way this can be done for several years (about as long as webhooks). My current idea involves it being a) open source, b) on a cloud platform like App Engine to abstract infrastructure (who's server, access to server, etc), c) some automated financial stuff tied into the community. I can get into more detail if you'd like. I might write a blog post about it.
Brett:
The discovery and authentication is something that I've been talking with the folks working around GetPingd with. I went to them because GetPingd is a spec for discovery and subscription to content updates. Unfortunately, I'm not a fan of pubsub semantics because I'm coming at this from a callback/hook perspective ... but they seemed to be on the right start despite language specifics. In fact, we're talking about authentication right now on the getpingd group list: http://groups.google.com/group/getpingd/browse_thread/thread/f2c5e8c89e0772d4/be7720dc1d302076#be7720dc1d302076
-jeff
Jeff Lindsay
http://webhooks.org -- Make the web more programmablehttp://progrium.com -- More interesting things
Neil Garb
> working out a way this can be done for several years (about as long as
> webhooks). My current idea involves it being a) open source, b) on a cloud
> platform like App Engine to abstract infrastructure (who's server, access to
> server, etc), c) some automated financial stuff tied into the community. I
> can get into more detail if you'd like. I might write a blog post about it.
Yes please! I remember watching a video of the App Engine launch: I
understood it was only Python (and now I see Java). My python-foo is
not what it probably should be. What are the alternatives? My only
experience with cloud-* is Amazon's S3.
- Neil
Jeff Lindsay
Myqron is ... Ruby?
-jeff
Neil Garb
On Wed, Apr 29, 2009 at 8:39 AM, Jeff Lindsay <prog...@gmail.com> wrote:
> My preference for this "community webservice" model is something that
> abstracts even the sysadmin, which means App Engine is really the only
> viable solution. I considered Amazon's EC2, but it still makes you deal with
> sysadmin. App Engine is even better now with Java because the JVM means it
> can run any language that runs on the JVM (PHP, Ruby, Javascript, etc). This
> made it possible for me to write Scriptlets.org (should be useful for
> anybody interested in web hooks).
>
> Myqron is ... Ruby?
>
Myqron's written in PHP using Zend Framework.
- Neil
Jeff Lindsay
Neil Garb
I'm more interested in gauging demand -- if things pick up then I'll
take the project to the next level.
Thanks for all your advice!
- Neil
Daniel Parker
If I get the idea of "community webservice" right, might Heroku be an
option? It would be ruby/rails, which has a large, benevolent
community of supporters - many of whom are the type of people
interested in moving this sort of thing forward. Deploying a website
on Heroku is as simple as pushing to your heroku git repo, and we all
know git has become very good for community-driven projects, thanks to
Github.
- daniel parker -
On Apr 29, 5:24 am, Neil Garb <neil.g...@gmail.com> wrote:
> I'll definitely investigate this as an option moving forward. For now
> I'm more interested in gauging demand -- if things pick up then I'll
> take the project to the next level.
>
> Thanks for all your advice!
>
> - Neil
>
>
>
> On Wed, Apr 29, 2009 at 8:49 AM, Jeff Lindsay <progr...@gmail.com> wrote:
> > Well it can run on App Engine if you can port the datastore.
>
> >> > My preference for this "community webservice" model is something that
> >> > abstracts even the sysadmin, which means App Engine is really the only
> >> > viable solution. I considered Amazon's EC2, but it still makes you deal
> >> > with
> >> > sysadmin. App Engine is even better now with Java because the JVM means
> >> > it
> >> > can run any language that runs on the JVM (PHP, Ruby, Javascript, etc).
> >> > This
> >> > made it possible for me to write Scriptlets.org (should be useful for
> >> > anybody interested in web hooks).
>
> >> > Myqron is ... Ruby?
>
> >> Myqron's written in PHP using Zend Framework.
>
> >> - Neil
>
> > --
> > Jeff Lindsay
> >http://shdh.org-- A party for hackers and thinkers
Jeff Lindsay
Jeff Lindsay
http://webhooks.org -- Make the web more programmable
http://shdh.org -- A party for hackers and thinkers