push faye data to browser and browser reloads ?

[Rubybox Rubybox]

Say you push data from faye to a browser, on a member based profile website.

The user is known and gets data from faye, but WHAT IF the user just on the moment the data is send ( when called from background tasks ) is reloading the page so the data is not received?

Is there a good workaround for this?

Im using a background task that on a certain time in near future sends data to the user

but i have no way of checking if the data is received and resend it if necessary

so i need a way to solidify this,

any suggestions? thx!

[James Coglan]

On 17 April 2012 08:47, Rubybox Rubybox <voor...@gmail.com> wrote:

Say you push data from faye to a browser, on a member based profile website.

The user is known and gets data from faye, but WHAT IF the user just on the moment the data is send ( when called from background tasks ) is reloading the page so the data is not received?

Im using a background task that on a certain time in near future sends data to the user

but i have no way of checking if the data is received and resend it if necessary

Faye does not have a built-in acknowledgement or delivery-query scheme, you would need to build one yourself. The simples thing to do is give each pushed message an ID and store it in a table, and have the receiving client send a message back on another channel that the background process listens to, to say it received the message ID. When an ID is acknowledged, you can delete it from the table, or delete and re-send after a timeout.

It might be worth looking at my Terminus project, which does slightly similar stuff to script browsers from Ruby: http://github.com/jcoglan/terminus

[Jordan Lance]

Hi James, Thanks for your replay I have been planned this feature and now working on it.
I have been looking at your app but couden't really figure out the relevant parts. Your idea on using a notifications table and sending message ID is clear to me and I will be implementing something like you suggested. Im using private pub at the moment I think that has a message ID in itself I could use.


Thx a lot in advanche!

[Roman Gaufman]

Lets say you do manually implement sending acknowledging that the browser received the message, what if then the user reloads the browser while that acknowledgement is being sent?

I don't really see how you would workaround for this entire scenario. When you load the page, you have a snapshot of the data the user wants, right? - say the last 50 message of a chat room, or the list of recent articles, etc. Then the browser subscribes to Faye and incrementally adds any changes/additions.

Why do you care if a message aknowledgement is sent or not if the user is reloading the browser which shows the latest data anyway?

[James Coglan]

On 19 June 2012 14:07, Roman Gaufman <hack...@gmail.com> wrote:

Lets say you do manually implement sending acknowledging that the browser received the message, what if then the user reloads the browser while that acknowledgement is being sent?

I don't really see how you would workaround for this entire scenario. When you load the page, you have a snapshot of the data the user wants, right? - say the last 50 message of a chat room, or the list of recent articles, etc. Then the browser subscribes to Faye and incrementally adds any changes/additions.

Why do you care if a message aknowledgement is sent or not if the user is reloading the browser which shows the latest data anyway?

This is a good point, and shows that you're better off focusing on making sure your client-side handles any incoming message idempotently so no matter what state it's in it can do something sensible with each message. The fewer assumptions to make about the reliability of the network the better off you'll be. 

[Jordan Lance]

Thanks guys good poitns, this is indeed a very complex problem. My problem is I'm using faye to submit data and have "Realtime " redirects in my app.

@Roman:

Thx for contribution to this discussion interesting view you have and haven't thought off it that way; the problem is my realtime need as desired and described below. 

Consider below simplified example:

* current_time is 11:00PM

* logged in user signs up to join Group X wich starts at 12:00 PM

* redis background scheduler sends faye data to web app at 12:00PM to subscribed faye data channel  "/start" 

* the client receives data at subscribed channel "/start" and reloads the page to /group/whatever/start

I have been solidifying the rest of my app but this is now a *serious issue* I would love to discuss in more detail with you, on approaches to solve ( if possible at all ) 

My goals are:

* Make sure the user is redirected at 12:00PM no matter what ! ( they should be logged in, reloading there page or not, they should be redirect immediate when 12.00pm has passed )

+ Im thinking of building some safety in say 30 seconds before the actual time to fail save any browser reloads

So im in the need of a solid "location changer" that runs from the background in my app and can force the client to a certain location at time X

Perhaps I have to switch away from the location changing idea but thats the implementation currently, any thoughts ideas on this?

One thought is to place a DIV inside every page and loads its contents with AJAX when data is received on a subscribed channel, but same problem persist how to be sure the data is received and if, have it executed in realtime.

I might now be able to solve this solid with faye but anyone a solution on how to make this solid perhaps with other technique?

[James Coglan]

On 6 July 2012 01:18, Jordan Lance <voor...@gmail.com> wrote:

My goals are:

* Make sure the user is redirected at 12:00PM no matter what ! ( they should be logged in, reloading there page or not, they should be redirect immediate when 12.00pm has passed )

+ Im thinking of building some safety in say 30 seconds before the actual time to fail save any browser reloads

So im in the need of a solid "location changer" that runs from the background in my app and can force the client to a certain location at time X

Seems like there's some overlap with this thread: https://groups.google.com/group/faye-users/browse_thread/thread/611fb183fc956b7d

The way Terminus (http://terminus.jcoglan.com) does this is to try sending a message to the client telling it to visit a certain URL. It expects to receive a message back from the client within a few seconds confirming that it's now at the new URL. Each client is issued with a random ID that forms its channel name, and it sends this ID back in the confirmation message. If the server does not get a confirmation within a certain amount of time, it tries again. If the client does not send a heartbeat for a certain length of time, it is dropped from the registry and no more redirect messages are sent to it.

[Jordan Lance]

Thanks James for this clarification, I have now taken serious look at implementing this since rest of project parts are near completion.

You wrote the terminus does this but digging the code I don't instantly see any way of this, could you perhaps be so kind to faccilitate an example on how a terminus message delived check could be implemented?

Thx for the great writeup wich I have seen before but forgot to reply on my bad

[Jordan Lance]

Studied the code and tested Terminus which is great btw!

on the Terminus Github page you note: 

" Support IE, which has no built-in XPath engine for querying the DOM. I’m working on Pathology (seegithub.com/jcoglan/pathology) to try and fix this but it’s currently not fast enough. "

I see terminus.js checks for IE, so its not possible to send back the URL to faye to check if its redirected properly? 

Not that much of a jquery guru but trying hard to understand the flow in the terminus code.

Adding to the complexity is im using Private Pub by Ryan Bates to handle the communication safely. 

Below is my custom JS to communicate with faye true the privatepub gem. 

function buildPrivatePub(doc) {

  var self = {

    connecting: false,

    fayeClient: null,

    fayeCallbacks: [],

    subscriptions: {},

    subscriptionCallbacks: {},

    faye: function(callback) {

      if (self.fayeClient) {

        callback(self.fayeClient);

      } else {

        self.fayeCallbacks.push(callback);

        if (self.subscriptions.server && !self.connecting) {

          self.connecting = true;

          var script = doc.createElement("script");

          script.type = "text/javascript";

          script.src = self.subscriptions.server + ".js";

          script.onload = self.connectToFaye;

          doc.documentElement.appendChild(script);

        }

      }

    },

    connectToFaye: function() {

      self.fayeClient = new Faye.Client(self.subscriptions.server);

      //self.fayeClient.disable('websocket');

      //Faye.Logging.logLevel = 'debug';

      self.fayeClient.addExtension(self.fayeExtension);

      for (var i=0; i < self.fayeCallbacks.length; i++) {

        self.fayeCallbacks[i](self.fayeClient);

      };

    },

    fayeExtension: {

      outgoing: function(message, callback) {

        if (message.channel == "/meta/subscribe") {

          // Attach the signature and timestamp to subscription messages

          var subscription = self.subscriptions[message.subscription];

          if (!message.ext) message.ext = {};

          message.ext.private_pub_signature = subscription.signature;

          message.ext.private_pub_timestamp = subscription.timestamp;

        }

        callback(message);

      }

    },

    sign: function(options) {

      if (!self.subscriptions.server) {

        self.subscriptions.server = options.server;

      }

      self.subscriptions[options.channel] = options;

      self.faye(function(faye) {

        faye.subscribe(options.channel, self.handleResponse);

      });

    },

    handleResponse: function(message) {

      if (message.eval) {

        eval(message.eval);

      }

      if (callback = self.subscriptionCallbacks[message.channel]) {

        callback(message.data, message.channel);

      }

    },

    subscribe: function(channel, callback) {

      self.subscriptionCallbacks[channel] = callback;

    }

  };

  return self;

}

var PrivatePub = buildPrivatePub(document);

* IE support is a must have, is there any way to hook into above code to check if a page has been redirect and send this back or any thoughts on how to approach this? 

[James Coglan]

On 17 September 2012 16:14, Jordan Lance <voor...@gmail.com> wrote:

Studied the code and tested Terminus which is great btw!

on the Terminus Github page you note: 

" Support IE, which has no built-in XPath engine for querying the DOM. I’m working on Pathology (seegithub.com/jcoglan/pathology) to try and fix this but it’s currently not fast enough. "

I see terminus.js checks for IE, so its not possible to send back the URL to faye to check if its redirected properly? 

Not that much of a jquery guru but trying hard to understand the flow in the terminus code.

The big problem with IE is it doesn't support XPath. I'm working on http://github.com/jcoglan/pathology to try to fix this. I've got it mostly working in the Android browser (which doesn't support XPath), so it's looking promising but progress is slow. It doesn't use jQuery -- I didn't want to inject that into pages since there's too large a chance of collisions with user code.

 

* IE support is a must have, is there any way to hook into above code to check if a page has been redirect and send this back or any thoughts on how to approach this? 

I'm not sure what you mean about this redirection stuff. Can you clarify what it is you're trying to implement? Sorry about my shoddy memory, got a lot on my plate right now. 

[Jordan Lance]

Hi James,

Thanks for the quick reply. I shall clarify as short as possible:

* My app subscribes to certain channels 

* I send data to this channel ( reloading the users browser ) 

* This works all fine but the problem is , if data is send, and browser reloads, the connection is broken and the reload never happens.

Im now trying to make this solid, implementing a check in the private_pub.js I posted above to check IF a browser has indeed been redirect, if not, it should send again a redirect datastream to the subscribed channel.

And Im looking to make this work for Internet Explorer as well.

Im now trying to enhanche the js methods from privatepub to implement this but have a bit of hard time getting it all together, so any suggestions to get in the right direction would be very welcome.

Guess its quite a must have to check if data has been delivered to a faye channel correctly else you end up with a breakable, non solid solution. 

[James Coglan]

On 17 September 2012 17:10, Jordan Lance <voor...@gmail.com> wrote:

* My app subscribes to certain channels 

* I send data to this channel ( reloading the users browser ) 

* This works all fine but the problem is , if data is send, and browser reloads, the connection is broken and the reload never happens.

Im now trying to make this solid, implementing a check in the private_pub.js I posted above to check IF a browser has indeed been redirect, if not, it should send again a redirect datastream to the subscribed channel.

And Im looking to make this work for Internet Explorer as well.

Okay, here's how Terminus does it. This should work in IE since it's got nothing to do with the DOM:

* The Terminus client picks an ID (random 128-bit string) and caches it in window.name

* The Terminus client pings the server every 3 seconds, meaning it sends a message to /ping with its ID, URL, user agent, details

* The Terminus client listens on the /clients/$ID channel for its ID

* When the Terminus driver tells a browser to go to another page, it sends a redirect instruction to /clients/$ID

* The client receives the message, and changes window.location

* On the new page, the Terminus client is again loaded, picks its ID out of window.name, and begins pinging again

* On the server side, the visit() method blocks until it receives another ping from that browser -- that's how you know the new page is loaded

So basically you need some way to identify the browser across page loads, bearing in mind it will have a different Faye ID on each page. If everything you're doing is on the same domain, you might be able to stash the ID in a cookie.

[Jordan Lance]

Its all on the same domain, I could also store the ID as part of my Rails application perhaps. Your description is clear and Im reading the terminus.js script, but have some problem linking it to the right places in the private_pub.js script.  

since its wrapping the faye functions a bit, I will try to hack the private pub JS I posted above, hope my JS skills allow me to figure it out.  If you could hook me up with a rough setup of how to implement this in the private_pub.js that would be awsome, this is quite danting my JS skills a.t.m,

thank you for the support so far! 

[James Coglan]

On 17 September 2012 17:34, Jordan Lance <voor...@gmail.com> wrote:

Its all on the same domain, I could also store the ID as part of my Rails application perhaps. Your description is clear and Im reading the terminus.js script, but have some problem linking it to the right places in the private_pub.js script.  

since its wrapping the faye functions a bit, I will try to hack the private pub JS I posted above, hope my JS skills allow me to figure it out.  If you could hook me up with a rough setup of how to implement this in the private_pub.js that would be awsome, this is quite danting my JS skills a.t.m,

I'm not actually that familiar with PrivatePub, but what it basically does is require a password on published messages to verify their authenticity. This effectively turns your Faye server into a 'push only' server unless the browser clients can also get the password somehow -- possibly through an Ajax call using session-based authentication (this must be protected against cross-domain access). I would ping Ryan Bates about this if I were you.