[mule-user] Asynchronous services using http

[Mohit Anchlia]

Does mule support some kind of callback method when using asynchronous http endpoints? Or is there a way to design this in mule?

[David Dossot]

If your outbound HTTP endpoint is in-only, you can not retrieve its response.

[David Dossot]

If your outbound HTTP endpoint is in-only, you can not retrieve its response. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

Then how would one go about implemented HTTP publish subscribe using mule? Is it even possible over http when using mule?

[David Dossot]

Oh, that is quite a different question :)

Do you want Mule to expose a publish-subscribe mechanism over HTTP for others to use or Mule to subscribe to some sort of external pub-sub engine over HTTP?

- If the former, I suggest you learn about PubSubHubbub: http://blogs.mulesoft.org/push-web-in... This could be a nice (and almost standard) way to offer this. Otherwise, if you have something else in mind please detail further.

- If the latter, more information is needed about this external pub-sub engine. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

Sorry, there are the 2 things:

 

1. Publish/Subscribe like Pubhubsubpub from google. I looked at that link earlier but it just gives example of Hub. Is there a complete example of publisher, Hub and subscriber that I can look at?

2. Way for clients asynchronously send the request and then later get the response. One way is to use piggybacking on next request. But is there a better way for this? Any thoughts or suggestions.

[Andrew Perepelytsya]

HTML5 websockets. Support for them us still far and between, but this is the future. Today you must resort to some variant of polling, if only hiding the fact from user API.

HTH,
Andrew

[David Dossot]

1.a) A publisher is a combination of an atom (or RSS) feed and a simple HTTP outbound call to register with the Hub. The atom module of Mule should help you here: http://www.mulesoft.org/documentation...

1.b) A subscriber is a combination of a standard HTTP inbound endpoint to receive new content and a simple HTTP outbound call to register with the Hub.

2) Yes. Make your client a PubSubHubBub subscriber so it doesn't need to poll but gets called back when there is new data. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

Can I use PubSubHubBub without using atom or RSS? I just want to use regular xml api messages that we support, JMS queue and http calls.

 

Regarding point 2 below I am not sure how it will work, in this scenario client for eg sends some work request asynchronously and later point needs to know that it was "completed". I am not sure how this would work unless it first sends the work, then we send the subscription channel name and client then subscribes to it.

[David Dossot]

Wait did you say JMS queue? Why don't you expose some sort of HTTP -> JMS bridge?

I did this in the past using Mule: client was HTTP POSTing work, which was dispatched to a JMS work queue, and was receiving back the Mule correlation ID as a token. Another HTTP endpoint was used to regularly poll for the result, using the correlation ID in a JMS selector on a result queue.

In your case, you do not want polling, right? In that case let the client submit a callback URL alongside his job data to Mule and have Mule call this URL back with the result when done. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

This is one option but this would require clients to poll, which is what I am trying to avoid in this first case where we are trying to push content from the server when available to subscribers. Your suggestion would work well for my second question in my previous email.

 

I am looking more of a push model. And I am trying to understand step by step how this would work as described in the seconds presentation http://code.google.com/p/pubsubhubbub/ but that mule blog is really meaningless to me.

 

I need to see these basic steps, could you help me understand how that would work with mule or a better place to see an example:

 

1. How clients subscribe and get the hub url

2. How publisher notify hub when message arrives in the queue for instance

3. how hub distributes content to clients

[David Dossot]

I thought Mohit was talking about *outbound* interactions, but the conversation makes it clear he's concerned with *inbound*. I misunderstood the initial request :( --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[David Dossot]

In my experience with PuSH, the easiest is to read the spec: http://pubsubhubbub.googlecode.com/sv...

Each step for each actor is very well explained. Here are the direct links for your three questions:

1. http://pubsubhubbub.googlecode.com/sv...
2. http://pubsubhubbub.googlecode.com/sv...
3. http://pubsubhubbub.googlecode.com/sv... --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[David Dossot]

But PuSH won't leverage your JMS queues, which I thought is something you wanted? --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

I think I am not able to correlate mule components with the following:

 

1. How to subscribe

2. How then clients get hub url

3. How publishers notifies hub url

4. How hub fetches and send the content

I am trying to see to implement this flow what my config would look like and how clients would use it.

 

Once I understand the above I can ask question about why JMS can't be used since JMS also supports notifications.

 

Thanks for your time.

[David Dossot]

1. That would be client side. Do you plan to use Mule client side? If yes, that would be an HTTP outbound endpoint.
2. The Hub URL is well known in advance.
3. With an HTTP POST.
4. This is done by the hub element of the Mule PubSubHubbub module. The publisher would use the Mule Atom module to expose data over an HTTP inbound endpoint.

This said, the difficulty is in mapping PubSubHubbub to your requirement:

> client for eg sends some work request asynchronously and later point needs
to know that it was "completed"

I guess the atom feed would be for a single topic, ie. work results. But clients subscribed to this topic would receive all results not only theirs. This may create too much overhead.

PS. I never said JMS can't be used, I meant the Mule PubSubHubbub module won't use it. I actually think my proposal to use HTTP+JMS would work best for your use case. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

There are 2 requirements. The one I am currently discussing is the one where all subscribers subscribed to a topic get notified. What I am unable to understand from this example is:

 

1. Where and how subscription occurs?

1. Where is subscriber's callback_url being stored and used?

2. Where and how is hub getting notifications from publisher?

3. How does client know hub url, but it looks like it's something that known in advance and hard coded on the client side like you mentioned.

 

----

<spring:beans>

  <spring:bean id="hubObjectStore" class="org.mule.util.store.PartitionedInMemoryObjectStore" />

</spring:beans>

<pubsubhubbub:config objectStore-ref="hubObjectStore" />

<flow name="myHub">

  <http:inbound-endpoint address="http://localhost:8080/hub" />

  <pubsubhubbub:hub />

</flow

[David Dossot]

1. The client sends a subscription request to the Hub, over HTTP, specifying the topic it's interested in. See: http://pubsubhubbub.googlecode.com/sv...

1. The callback_url is past to the Hub at subscription time. It is used by the Hub to send messages like subscription request verification (see: http://pubsubhubbub.googlecode.com/sv...) and new publisher content (see: http://pubsubhubbub.googlecode.com/sv....

2. The publisher sends new content notification to the Hub over HTTP. See: http://pubsubhubbub.googlecode.com/sv...

3. Yes, that is correct, the Hub URL is known in advance. It is known by both the subscribers and the publishers. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

I think I am getting little better clarity. It looks like the presentation is little different than the draft, which made things very confusing.

Is it correct that all the communication including subscription occur using hub URL. Google presentation threw me here as it looks as if it contacts publisher first and then calls hub url. Even other books that I read also shows that subscription occurs directly with the publisher url seprately and then with hub url seprately.

 

But I am still stuck:

 

1. In mule where is hub storing callback_url

2. In mule how is hub getting notifications, content fetch from publisher and content distribution (complete section 7 from the link)? http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.3.html#publishing and

[David Dossot]

1. The Hub uses an Object Store as it's backing storage. This is from the blog I linked above:

"If you wonder about the object store, any implementation of org.mule.util.store.PartitionableObjectStore can be used as a back-end for the hub. You???ll want to use a persistent one, like the Redis-backed Object Store, to have your hub subscriptions survive a Mule restart!"

There is also now a MongoDB object store. Otherwise for testing an InMemory object store can get you started.

2. The publisher notify the Hub with HTTP POST requests. The Hub then performs HTTP GET requests on the publisher's feed to retrieve its content, finds out what is new only, then HTTP POST the new entries (Atom or RSS) to all the client subscribed to the topic of this publisher. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

thanks! now remaining question how does point 2 implemented in mule, if I break your point 2 how is this configured in mule?

 

--

1. The publisher notify the Hub with HTTP POST requests.

2 The Hub then performs HTTP GET requests on the publisher's feed to retrieve its content,

3. finds out what is new only, then HTTP POST the new entries (Atom or RSS) to all the client subscribed to the topic of this publisher

--

Thanks so much for helping me out here.

[David Dossot]

So you mean behind the scene, inside the Hub, how is point 2 done?

Well, no big secret, this is open source :)

If you look in:

https://github.com/mulesoft/mule-modu...

you'll see the Hub uses Rome's FeedFetcher for that operation. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

Not all of that can be behind the scenes, I think. For eg: publisher calling/pinging hub, and then hub fetching ("GET") the content should be configurable? These 2 things should be visible as I would assume publisher can be any provider that supports pub/sub. In that example I don't see how publisher is pinging hub and how is hub fetching the content from publisher.

 

Also, in that example of mule who is the publisher? Is it Pojo, redis, jms ?

[David Dossot]

I don't follow you here, at least I don't see what you would want to make configurable in the Hub offered by Mule. Rome's FeedFetcher is doing a pretty good job at fetching RSS/Atom data so really I don't know what you would want to change here.

The publisher itself is not covered by this particular Mule module (nor the subscriber by the way) so you can use whatever you want to implement them, including Mule. I suggested using the Atom Module and the HTTP Transport to build the publisher, in case you decide to build it with Mule. But this is up to you. You could use any Atom (or RSS) library and HTTP server and client. --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

Is FeedFetcher integrated inside the hub? Do you know of a link that has a complete example using hub, publisher. Not so much the client side but atleast the server side.

 

Thanks for the advice. I'll take a look at the atom module.

[David Dossot]

Yes FeedFetcher is integrated inside the hub.

Regarding Publisher code, the official PuSH test suite has a mock that gives a good idea how to do it: http://code.google.com/p/pubsubhubbub...

Otherwise there are implementations in different languages: http://code.google.com/p/pubsubhubbub...

Including Java: http://code.google.com/p/pubsubhubbub... --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email

[Mohit Anchlia]

Thanks for the information. I'll take a look. Thanks again