Bug with single quote - client side Core.js

[Nicolas]

Hi,

I found a bug when sending a string that includes a single quote. 

To reproduce the bug, simply try sending a string that has a single quote like this:

pipe_user.sendRaw("name_raw",{"msg"," ' ");

Then you will get a bug on Core.js on the client side (I use mootools client)

Does anybody know why the bug happens? Is there an easy way to correct this?

See below for the exact line of the bug.

Thanks.

Nicolas.

/***

* Parse received data from Server

*/

parseResponse: function(raws, callback) {

if (raws) {

if (this.status < 0 ) {

this.failCounter = 0;

this.status = 1;

this.startPoller();

this.fireEvent('apeReconnect');

}

}

var check = false;

var chlCallback;//Callback on challenge

if (raws) {

raws = JSON.parse(raws); <<<<<================= This is where the error is raised. Why?

Uncaught SyntaxError: Unexpected token ' (repeated 4 times)

if (!raws){ // Something went wrong, json decode failed

this.check();

return;

}

for (var i = 0; i < raws.length; i++){ //Read all raw

var raw = raws[i];

if (callback && $type(callback) == 'function') {

callback.run(raw);

}

this.callRaw(raw);

//Last request is finished and it's not an error

if (!this.transport.running()) {

if (!raw.data.code || (raw.data.code != '006' && raw.data.code != '007' && raw.data.code != '005' && raw.data.code!= '001' && raw.data.code != '004' && raw.data.code != '003')) {

check = true;

}

} else {

//Invalidate check if something went wrong with other raw or a new request have been launched

check = false;

}

}

} else if (!this.transport.running()) check = true; //No request running, request didn't respond correct JSON, something went wrong

if (check) this.check();

},

[Louis Charette]

I would suggest using decodeURIComponent  and encodeURIComponent on your string before and after sending it.

I don't have a vanilla APE setup on hand, but when I send  "J'aime les pommés" to a custom command I get back "J'aime%20les%20pomm%C3%A9s". decodeURIComponent bring back the string clean again. 

--
--
You received this message because you are subscribed to the Google
Groups "APE Project" group.
To post to this group, send email to ape-p...@googlegroups.com
To unsubscribe from this group, send email to
ape-project...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ape-project?hl=en
---
APE Project (Ajax Push Engine)
Official website : http://www.ape-project.org/
Git Hub : http://github.com/APE-Project/
 
---
You received this message because you are subscribed to the Google Groups "APE Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ape-project...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nicolas Guibert]

Hi Louis,

I already use decodeURIComponent but not encodeURIComponent.

I'll add it and let you know if this helps.

Thanks for your help.

Nicolas.

2013/3/14 Louis Charette <charett...@gmail.com>

--

Nicolas Guibert

Games in Mind Ltd

Managing director

7 Spencer Walk

Tel: +44 (0)207 435 0315

UK Mobile: +44 (0)7638 657 537

French Mobile: +33 (0)7 53 17 20 13

Email: con...@happymeeple.com

[Nicolas]

No that does not help. I did not use encodeURIComponent in the first place because it was messing things up. I think APE already does the encoding itself when sending. So one needs to decode but not encode (I  remember pointing out this weird behaviour at the time, probably more than one year ago).

Here is an example of string that causes a problem. As you can see there is a single quote (several actually here).

oui%20ergonomie%20diff%C3%A9rente%20mais%20j'y%20%C3%A9tais%20pas%20all%C3%A9%20depuis%20plus%20d'une%20semaine

I don't really know how to cope with this...

[Pablo Tejada]

Hi guys,

I came accross this problem when building my own client framework. The problem comes from the server failing to properly encode/scape single quotes in the JSON response.

Take a look at this file, the problem is addressed at the very beginning. If the first attempt to decode the json string fails a second attempt is made after single quotes are properly scaped, if it fails again the the whole proccess/connection dies.

https://github.com/ptejada/ApePubSub/blob/master/js/src/onMessage.js

Hope you guys can properly port this temporary solution i came up with,

Cheers

[Nicolas]

Great help Pablo. I'll have a look at this now. Looks good.

Thanks!!!

[Louis Charette]

Make sense. I don't encode before sending also.

Pablo, any idea where the encoding fail in the first place? Server C code or in the JSF?

Envoyé de mon iPhone

[Pablo Tejada]

Most likely in the server code as the problem also occurs while using websocket as the transport method.

[Nicolas]

It seems to be working for the moment.

For others' reference, I have copied your solution on Core.js like this:

I have replaced this bit:

if (raws) {

raws = JSON.parse(raws); 

if (!raws){ // Something went wrong, json decode failed

this.check();

return;

}

by this piece of code:

        if (raws)

        {

                        try

                        {

                            raws = JSON.parse(raws);

                        }

                        catch(e)

                        {

                            //Temporary FIX for malformed JSON with escaped single quotes

                            raws = raws.replace(/\\'/g, "'");

                            try 

                            {

                                    raws = JSON.parse(raws);

                            }

                            catch(e)

                            {

                                this.check();

return;

[Louis Charette]

Are you using the latest source from GitHub? Would be nice to find the real issue with that. I remember something changed in the server source at one point regarding encoding. The latest version (before the MySQL fix) was giving me problem with french accentuated character. I didn't updated my prod server yet so I can't check right now.

  - Louis

[Pablo Tejada]

I tested on both builds, before and after the MySql fix. Single quotes wont go by without errors.

[Peter Reijnders]

A few weeks ago i encountered the same problem. At the time I was in a hurry and did a blablabla.replace(/\'/g, '') before sending the RAW back to the clients.

I just took some time to investigate this.

There are 3 transport types:
0    Long Polling: default transport method
1    JSONP: for cross domain push
2    XHRStreaming: for better performance and low latency


The problem has partially been tackeled for JSONP
    https://github.com/APE-Project/APE_Server/commit/88a75bddab6695e5c4d2da4bb43fb70c2cb019de
    JSONP fix: APE_Server does not escape single-qoutes in Ape.transport....
Unfortunately that breaks Longpolling


                        disabled the lines 229-232 in json.c
APE.Config.transport = 0;    works perfect
APE.Config.transport = 1;     not tested
APE.Config.transport = 2;    SyntaxError: missing ) after argument list

                        enabled the lines 229-232 in json.c
APE.Config.transport = 0;    SyntaxError: JSON.parse: bad escaped character
APE.Config.transport = 1;     not tested
APE.Config.transport = 2;    works


What I do not understand (yet) is how transport can effect the serialisation, a data structure that is beeing transformed into json should always render the same. I have to think about that / investigate further.

Peter

[Louis Charette]

I believe you have reversed the transport identifier:

So it should be:

There are 3 transport types:

0    Long Polling: default transport method

1    XHRStreaming: for better performance and low latency 
2    JSONP: for cross domain push

6    WebSocket

I confirm that, Lon polling and websocket does work without lines 229-232 in json.c, but JSONP doesn't (And XHR just doesn't work at all on my setup...) It doesn't work with JSONP because the string is sent back as a function argument:

Ape.transport.read('[{"time":"1364065440","raw":"92","data":{"data":{"test":"j'aime%20les%20pomm%C3%A9es"},"from":{...}},"pipe":{...}}]')

That explains the fix in the first place. But escaping the characters brakes the other 3 transports...

  - Louis

[Nicolas]

Hi guys,

I don't know if this is related to the above at all, but here is what I get in parseResponse sometimes (but not always) under IE8, IE9, IE10:

raws:[{""time"":""1365798932"",""raw"":""IDENT"",""data"":{""user"":{""casttype"":""uni"",""pubid"":""aecee0a501f71ee7cc70fbec0e08934a""}}},{""time"":""1365798932"",""raw"":""SESSIONS"",""data"":{""sessions"":{""uniPipe"":""null""},""chl"":1}}]

As you can see, there are double " everywhere.

Although I have not logged what happens after that, it is clear to me that these double " are the reason why I never reach the onRaw function. 

So why on earth does APE sometimes get these double quotes under IE?

Any clue?

Nicolas.

[Louis Charette]

Which transport are you using?

  - Louis

[Nicolas Guibert]

I believe it is the default one. Long polling I guess. The one that opens connection and close them if nothing happened in 25 seconds.

2013/4/13 Louis Charette <charett...@gmail.com>

[Nicolas]

After enquirying a bit more, I realized that the double double quotes is probably not the issue. I don't know why and when it occurs, but 1) it also occurs sometimes with FF, 2), my application does not break on the first double double quotes but later.

Here is an example of where it breaks:

[{""time"":""1365799173"",""raw"":""game_started"",""data"":{""match_id"":403525,""food_cost"":""4"",""opponent_language"":""fr"",""opponent_first_name"":""Andr?""startup_conditions"":[99,[99,10,7,5,0,0,0,0,0,0,0,0,0],[99,11,1,7,0,0,0,0,0,0,0,0,0]],""raw_id"":1,""page_counter"":9}},{""time"":""1365799173"",""raw"":""manager_infos"",""data"":{""match_id"":403525,""manager_infos"":{""die_rolled"":14},""raw_id"":2,""page_counter"":9}}] 

I think the "é" from "André" is most likely the problem. Look: it is here changed into ?. More importantly, the JSON is broken. Instead of having:

""opponent_first_name"":""Andr?"",""startup_conditions"":[99,[99,10,7,5,0,0,0,0,0,0,0,0,0],[99,11,1,7,0,0,0,0,0,0,0,0,0]]

I receive this:

""opponent_first_name"":""Andr?""startup_conditions"":[99,[99,10,7,5,0,0,0,0,0,0,0,0,0],[99,11,1,7,0,0,0,0,0,0,0,0,0]]

See: "","" was replaced by "" only.

Any idea where this comes from? Server, Client?

Thanks.

Nicolas.

[Nicolas]

Good. I am now able to reproduce the bug here. the é of "André" placed in last position is causing the problem. If somewhere else in the string it does not.

Now why is the é causing a problem?

Nicolas.

[Pablo Tejada]

Probably for the same reason you can't register a André.com domain. Try to encodeurlcomponents before sending this string and then decodeurlcomponent when received in the client. You should also test it with other accentuated characters and see if it also breaks.

[Nicolas]

Hi Pablo,

The "André" string is coming from a Mysql query.

Although I did try a while ago (see https://groups.google.com/forum/?fromgroups=#!topic/ape-project/pQybF6IYquY) to call SET NAMES 'utf8' before so that the queries are encoded in UTF8 (and I have tried again today), I am not sure they really are. The logs can't print the right character é and this may be because they are unable to do it or because my characters are wrong. I have no way to know. John Kettlekey  mentioned in this thread that he had troubles with logs too.

Anyway, now I know where the problem comes from. I will stay clear of any string coming from Mysql. That just does not seem reliable.

Thanks for your help all.

Nicolas.

PS: please note that "Dédé" or "Andréfdfdq" were not causing any problem. It just seems that IE wrongly interprets something. It is a very strong possibility that there is nothing wrong with what I am doing, just IE losing track in a very specific case, a very specific sequence of characters...