Re: Faye and Salesforce

[James Coglan]

On 25 February 2012 22:16, Doug Friedman <dfri...@gmail.com> wrote:

I've been working this past week on a Node.js implementation your Faye project, and I have to say its fascinating.  What I'd like to be able to do is subscribe to a feed published by Salesforce.com's streaming API.  I noticed there was a fork due to some tweaking that needed to be done to successfully connect to Salesforce, and I've been using said fork, but I haven't quite gotten it to work yet.  As I'm new to Node and to Faye and the Bayeux Protocol in general, I'm not even sure what to look for.  Do you think you could provide me with some pointers or 'Common Problems' that I might be able to look for when debugging my app?

Which fork are you using, and what's the endpoint you need to connect to on the Salesforce servers? I accepted some patches a while ago to support cookies in the server-side client, and the reason given for the patches was to support the Salesforce API. Apparently they now seem to be using OAuth, although it's very hard to understand from their docs, with no prior knowledge of what Salesforce is, how to get authorized and which endpoint to connect to. That's what I'd look for before debugging the HTTP traffic being sent.

 

Ideally, though, I'd like to be able to do all of this in Ruby, but I chose Node because it seems there have already been some attempts made on the web.  Have you ever tried to connect to the SFDC Streaming API via Ruby?

I'm afraid I haven't. I just signed up for a Salesforce developer account and have been reading http://wiki.developerforce.com/page/Getting_Started_with_the_Force.com_Streaming_API, but am still very confused -- there's a lot of domain-specific knowledge assumed and I can't see any clear explanation of where their Bayeux server is hosted and how to connect to it.

I might be that Faye needs to add the ability to set headers on HTTP requests to make this work, but I'm in the middle of trying to get the long-postponed 0.8 release out so would be great if you can respond quickly so if know if new features are needed. Either way, the Node and Ruby versions of Faye should be feature-equivalent and you should be able to use either version to do what you want.

By the way, I'm forwarding this to the mailing list and I encourage you to follow up there since others may have experience with this: http://groups.google.com/group/faye-users

Cheers,

James

[James Coglan]

On 25 February 2012 23:40, James Coglan <jco...@gmail.com> wrote:

Ideally, though, I'd like to be able to do all of this in Ruby, but I chose Node because it seems there have already been some attempts made on the web.  Have you ever tried to connect to the SFDC Streaming API via Ruby?

Further to this I have now tried this and I've got stuck for now. Here's what I've done, following the example given here: http://www.salesforce.com/us/developer/docs/api_streaming/index_Left.htm#StartTopic=Content/code_sample_java_add_source.htm

Create an XML file that looks like this:

<soapenv:Envelope xmlns:soapenv='http://schemas.xmlsoap.org/soap/envelope/' 

    xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' 

    xmlns:urn='urn:partner.soap.sforce.com'>

  <soapenv:Body>

    <urn:login>

      <urn:username>F...@EXAMPLE.com</urn:username>

      <urn:password>USE_YOUR_OWN_PASSWORD</urn:password>

    </urn:login>

  </soapenv:Body>

</soapenv:Envelope>

Then make a Node program that does this:

var https = require('https')

var content = require('fs').readFileSync('sf.xml').toString()

var request = https.request({

  method: 'POST',

  host:   'login.salesforce.com',

  port:   443,

  path:   '/services/Soap/u/22.0/',

  headers: {

    'Content-Type': 'text/xml',

    'SOAPAction': "''",

    'PrettyPrint': 'Yes',

    'Content-Length': content.length

  }

})

var body = ''

request.on('response', function(response) {

  response.on('data', function(c) { body += c.toString() })

  response.on('end', function() {

    console.log(body)

  })

})

request.write(content)

request.end()

This will print out a bunch of XML for the authorization response. Of interest is the sessionId, which I added the following lines to Faye::Transport::Http#request to add the cookies as given in the Java example:

      @client.cookies.set_cookie(@endpoint, "com.salesforce.LocaleInfo=us")

      @client.cookies.set_cookie(@endpoint, "login=F...@EXAMPLE.com")

      @client.cookies.set_cookie(@endpoint, "sid=THE_SESSIONID_FROM_THE_NODE_OUTPUT")

      @client.cookies.set_cookie(@endpoint, "language=en_US")

I then try to connect the Ruby Faye client to https://na14-api.salesforce.com/cometd (this address depends on your account) and I get this:

<html>

<head><title>401 Request requires authentication</title></head>

<body>

<h1>401 Request requires authentication</h1>

<p /><hr />

<small>

</small>

</body></html>

So, no dice so far. Would be great if someone could enlighten me, so we can find out whether I need to expose the ability to add headers/cookies when using the Faye client so that this works correctly.

[James Coglan]

On 26 February 2012 00:47, Doug Friedman <dfri...@gmail.com> wrote:

I see, thanks for the advice.  I'll have to take a look at the end points for the requests.  I'll follow up tomorrow with some more info on salesforce.

I found it -- documented in http://www.salesforce.com/us/developer/docs/api_streaming/api_streaming.pdf, you authenticate by adding the header 'Authorization: OAuth $sessionId' to the request. If I do that, my Ruby client handshakes successfully and I can send subscriptions to listen for messages.

This suggests I need to add a setHeader() method to the Faye client to allow this to work. All you'd have to do then would be (after getting a session ID from their SOAP service):

    client = Faye::Client.new('https://na14-api.salesforce.com/cometd/24.0')

    client.set_header 'Authorization', 'OAuth ...'

    

    client.subscribe '/some/topic' do |message|

      # ...

    end

[Doug Friedman]

Very nice, you are 100% correct. Do you mean that the current master
Faye branch supports headers, or that it will need to be amended to
include them?

I couldn't tell you exactly what changes need to be made to Faye, but
everything that you've stated above is correct about salesforce. For
those who are interested, the following two examples illustrate how to
connect to salesforce via OAuth in Ruby and Node respectively:

https://github.com/richardvanhook/omniauth-salesforce
https://github.com/gidzone/nodejs-sfdc-sample

You'll go through the OAuth dance, and Salesforce will respond with
three important pieces of information: Access Token, Refresh Token
and Instance URL. In each request, you'll need to include the Access
Token (i.e. SessionID) in the header as you've described. The Access
Token has a finite lifespan, so the simplest way to maintain a
persistent connection is to catch the 400 error when it expires, and
send in a refresh token to obtain a new session -- logic that probably
can be implemented in the application that uses Faye.

To your question about which fork I was using: https://github.com/aashay/faye.

Thanks for the help,
Doug

On Feb 25, 8:11 pm, James Coglan <jcog...@gmail.com> wrote:

> On 26 February 2012 00:47, Doug Friedman <dfrie...@gmail.com> wrote:
>
> > I see, thanks for the advice.  I'll have to take a look at the end points
> > for the requests.  I'll follow up tomorrow with some more info on
> > salesforce.
>

> I found it -- documented inhttp://www.salesforce.com/us/developer/docs/api_streaming/api_streami...,

[Doug Friedman]

James-

Ah, just also saw your comment re: the 0.8 release. Yes, if you could
get a 'set header' method in there before the 0.8 release that would
be huge. Please let me know if you need help setting up a Salesforce
environment to test in. I'd be glad to pitch in.

Doug

On Feb 26, 10:09 am, Doug Friedman <dfrie...@gmail.com> wrote:
> Very nice, you are 100% correct.  Do you mean that the current master
> Faye branch supports headers, or that it will need to be amended to
> include them?
>
> I couldn't tell you exactly what changes need to be made to Faye, but
> everything that you've stated above is correct about salesforce.  For
> those who are interested, the following two examples illustrate how to
> connect to salesforce via OAuth in Ruby and Node respectively:
>

> https://github.com/richardvanhook/omniauth-salesforcehttps://github.com/gidzone/nodejs-sfdc-sample

[James Coglan]

On 26 February 2012 15:24, Doug Friedman <dfri...@gmail.com> wrote:

Ah, just also saw your comment re: the 0.8 release.  Yes, if you could
get a 'set header' method in there before the 0.8 release that would
be huge.  Please let me know if you need help setting up a Salesforce
environment to test in.  I'd be glad to pitch in.

It's in 0.8, which has actually just shipped. I will post more tomorrow, just finished pushing all the packages out and it's late here. The short version is you can do this to connect to Salesforce (I've actually done this, but not created any PushTopics, just made sure I can connect):

    client = Faye::Client.new('https://na14-api.salesforce.com/cometd/24.0')

    token = get_salesforce_token

    client.set_header 'Authorization', "OAuth #{token}"

    

    client.bind 'transport:down' do

      token = get_salesforce_token

      client.set_header 'Authorization', "OAuth #{token}"

    end

This ought to do what you want, in that it will catch when the client disconnects and reset the token. It is not ideal, for reasons I'll get into, but it's a start and the 0.8 release was already plenty big enough and I've been trying to ship it for weeks. If this doesn't work we'll do bug fix releases.

The reason I'm not entirely happy with this is that it doesn't give you specific enough access to what Salesforce does with HTTP to catch the error robustly. I *ought* to work, by dint of the fact that a request that does not return JSON will trigger transport:down, but that's kind of a blunt instrument in this case. When you're not authenticated, Salesforce send a 4xx response containing HTML.

This is problematic for several reasons that would impact any attempt to provide sensible APIs in Faye for dealing with this.

First, their OAuth only accepts tokens in the header (rather than a query string, in violation of the OAuth spec), and ability to set headers is transport-dependent (e.g. callback-polling and websocket transports cannot do this; fortunately Salesforce only supports long-polling. I already don't like the Faye setHeader() method for this reason: it makes transport details leak out of the client interface.

Second, in the case of en error they do not respond in a way that a Bayeux client knows how to deal with: instead of sending a /meta/handshake response with an error field, they send a 4xx status code containing HTML. This makes the response literally unreadable by certain transports, and even for transports that can read it, they don't understand arbitrary HTML that is clearly meant for human consumption.

So what I'm saying is, this raises interesting questions about what parts of Faye would need to be made pluggable in order to handle this robustly. Does the long-polling transport need to understand non-200 responses, and emit errors containing the response text? Does the user need to be able to completely replace the long-polling transport? Should setHeader() throw an error if the underlying transport does not support it? Does transport selection need to accept user-defined predicates (for example because header-setting cannot be feature detected, the knowledge of that requirement must be encoded in the library or left to users to plug in). This is probably not an exhaustive list.

I hope what we have in 0.8 will fix your problem, and like I said bug fixes will be released to best handle this with the current feature set. The question then remains: are all the questions above important enough that additional work is needed to support servers with slightly esoteric requirements?

[Doug Friedman]

James-




First off -- I created a PushTopic in a salesforce org and yes it
works, thank you.  This is huge.  I also tested the Refresh Token case
and it works (I'm sure you just hard coded a fake token and then hard
coded the real thing in the below block, but hey that works).  FYI,
you can obtain an access token here at richard van hook's OAuth
salesforce example: https://omniauth-salesforce-example.herokuapp.com/
 and revoke it with a get request to the following endpoint:
https://login.salesforce.com/services/oauth2/revoke?token=<your
token> .




Second of all, Ha!  I won't venture to guess why you are getting that
HTML message back from Salesforce but I can tell you that according to
the SFDC docs (and my previous experience) you should be getting a
JSON encoded response in the case of an invalid Access Token:
https://login.salesforce.com/help/doc/en/remoteaccess_oauth_refresh_token_flow.htm
.   Either way, you are probably right that it would be useful to have
some way of bubbling up an error in case of a broken connection.  I
wonder how the Java Cometd client handles this.




In short, if your question is whether your project is going to have to
accomodate "slightly esoteric requirements" I'm sure the answer is
ultimately, yes if it is going to be the 'go to' sub/pub library in
Ruby and Node.




If you've got any questions, feel free to reach out and thanks again.

On Feb 26, 7:28 pm, James Coglan <jcog...@gmail.com> wrote:

[James Coglan]

On 27 February 2012 06:02, Doug Friedman <dfri...@gmail.com> wrote:

Second of all, Ha!  I won't venture to guess why you are getting that
HTML message back from Salesforce but I can tell you that according to
the SFDC docs (and my previous experience) you should be getting a
JSON encoded response in the case of an invalid Access Token:
https://login.salesforce.com/help/doc/en/remoteaccess_oauth_refresh_token_flow.htm

I got this when trying to initiate a Bayeux connection without sending an access token. To get a token, I used the Node script from my previous email to get a sessionId. So the HTML response comes from their Bayeux server, not their OAuth service.

In short, if your question is whether your project is going to have to
accomodate "slightly esoteric requirements" I'm sure the answer is
ultimately, yes if it is going to be the 'go to' sub/pub library in
Ruby and Node.

I meant specifically does it need to handle anything particular about this service and how it requires certain transport properties and signals errors, or can we stick with the current strategy of treating any response that is non-200 or does not contain JSON as a transport error?

[Doug Friedman]

As I am not a Salesforce employee, I wouldn't be able to tell you
that. I'll try to get someone from Salesforce's attention to this
question so they can advise you appropriately.

On Feb 27, 5:21 am, James Coglan <jcog...@gmail.com> wrote:

> On 27 February 2012 06:02, Doug Friedman <dfrie...@gmail.com> wrote:
>
> > Second of all, Ha!  I won't venture to guess why you are getting that
> > HTML message back from Salesforce but I can tell you that according to
> > the SFDC docs (and my previous experience) you should be getting a
> > JSON encoded response in the case of an invalid Access Token:
>

> >https://login.salesforce.com/help/doc/en/remoteaccess_oauth_refresh_t...