illegal character in json feed

[miramardesign]

All browsers throw error "illegal character" in twitter jsonp feed

http://jsfiddle.net/NBCGf/34/

Please advise,
//mh

[Taylor Singletary]

Do you have more details about the specific illegal character(s) you're receiving? How frequently are you polling the API? Have you tried reproducing your scenario outside of a browser-based environment?

@episod - Taylor Singletary

--
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk

[michael hazzard]

Taylor();

I took a pic of firebug here:
https://docs.google.com/leaf?id=0B1lcI5OlKM2xNDczZDdmYTMtMjFjZi00Y2E2LWE4MTMtM2YyM2I5ZTM4Njhl&sort=name&layout=list&num=50


It is similar to this case (I believe the same chars)
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/7567404004266db5?pli=1

It is reproducible on the jsfiddle url if you look at the error console usually 10-20 iterations.
http://jsfiddle.net/NBCGf/34/

(Note: I recently added window.onerror to get around it for now and continue loading feeds.)

Thanks,
//mh

--
Thank You,

Michael Hazzard

(954) 247-4461

[michael hazzard]

Oh answers to #2 and #3

2) I am polling the jsonp api every second in this test but it will be changed for production to like 6 seconds and cached after all 30 feeds are loaded

3) No, I haven't tried outside the browser since this is strictly a jsonp feed and I chose jsonp over server-side xml but now regretting a bit.

//mh

[Taylor Singletary]

It would be very helpful if you could provide a specific Tweet ID that is exhibiting this behavior -- the test case you're presenting is a bit artificial. Do you know if the environment you're doing this consumption is fully capable of understanding the UTF-8 character set? Tweets can contain just about any UTF-8 character and your parsing should be prepared for that. 

On a side note, executing a specific query against the Search API every six seconds isn't the most efficient use of the API. Have you considered increasing the time you wait to poll and using additional Search API feature like since_id?

@episod - Taylor Singletary

[michael hazzard]

Taylor,

Firefox 4.01 on Windows Server 2003 and IE8 on Windows Server 2003
 (I believe they are both fully utf-8 compatible)
(using jQuery 1.3.2 js library, I would upgrade but I am stuck at it w/ other dependencies )

Can you not reproduce the error on the jsFiddle? (I will test on my ubuntu laptop when I get home)

As for the tweetid/url it is happening on multiple but I have
http://search.twitter.com/search.json?q=from:LansdowneResort&rpp=1&callback=?
hard-coded in the jsfiddle example and its showing error after a couple iterations.

In any event, if you cannot, I am fine w/ the window.onerror error handling I have implemented and have it working in both browsers now and have implemented dom caching so as not to make too many requests.

Update:
I just tested in chrome and it threw
search.json:1Uncaught SyntaxError: Unexpected token ILLEGAL
http://search.twitter.com/search.json?q=from:TrumpSoHo&rpp=1&callback=jsonp1308002393614&_=1308002425423

So I imagine maybe it is an OS/charset thing.

Thanks
//mh

[michael hazzard]

Update:

I tested on my Ubuntu Laptop (11.04 ) + Firefox 4.01 and got the "illegal character" error, albeit much later (291 tries and 21 tries after 2 tests)

What's interesting is the whole feed seems to be scrambled only on that 1 response(see bottom of image) I had previously thought it was just the beginning. What would cause it to change charset after 120 responses? 





Error:



Thanks,


//mh

[Mike McNeil]

Same thing's happening to me.  Check out: http://search.twitter.com/search.json?q=%23jimromeisburning&callback=dog

Should look like this:

dog(

{

• -
  results: [
  • -
    {
    • from_user_id_str: "257702817"
    • profile_image_url: "http://a1.twimg.com/profile_images/1364757456/2011-05-22_13.20.33_normal.jpg"
    • created_at: "Tue, 14 Jun 2011 20:36:33 +0000"
    • from_user: "ATLienSince1987"
    • id_str: "80735397449183232"
    • -
      metadata: {
      • result_type: "recent"
      }
    • to_user_id: null
    • text: "#JimRomeIsBurning"
    • id: 80735397449183230
    • from_user_id: 257702817
    • geo: null
    • iso_language_code: "is"
    • to_user_id_str: null
    • source: "<a href="http://levelupstudio.com" rel="nofollow">Plume  </a>"
    } ......
  ]
• max_id: 80735397449183230
• since_id: 0
• refresh_url: "?since_id=80735397449183232&q=%23jimromeisburning"
• results_per_page: 15
• page: 1
• completed_in: 0.185672
• since_id_str: "0"
• max_id_str: "80735397449183232"
• query: "%23jimromeisburning"

}

)

But 3/5 times it doesn't, it looks like this:

dog( �  ݘYo�F ǿʚF݇Z��<  ��&��85b�)�  M�č�]�� 7�w��.ےl� ҇�A�����o�3�oN%��hj���7�_�2nkY�*���r�  A�iH �� U��
 �2 ȸ�
hϛf������X� 75�w�_� ��@ \� ńt��P  �R�2 kS�I�~ 
`���I#�8i`�^+  ����Ȏ��.󻌡_0|���`�}�{���P:�$
���M�8`�E � 	 e  K�$Y�$N��� qs=�л��ԍ���i�� NW�E �ȯֲ�SU~4�<�_��Vz0Y���,txǧNw��Cg �|VU��H���~MMf�P�sۀ�N��k�V��sP4� �W����Kk�g�h �J ��n�L  �i+�d1��M� � O  �g�E[ʿZ� <���{�m W�g�q ���]؈ � ���9  ���֏A� ] v9_��͔wi�CF0 a�qHE�+
Ǧ�P#� Um!�=��Y�`�T��  K+��1�� �c< �2�  =�� t��v߆	A� $�� }�	* !<b›�|  �v9��h- gj( ��.܇  �2  G���4,� � 5�DcS�
�s3FF������bɄU*f>�H��O� ��4��Dd ���X &�<5ʍ�[0қN���B�|2h#.D�� ��7�� � A@s#  ��  O��  �Ę��>@љ�@ ��g� Z��"py\ Iw)bA�q(" ; �S$E  � F/e=��"O� �%� d����G@*A ����� m���Թ� K�J~B'?�hTɦ�Fu[I��(;c��; � �Gpʇ�E�
dQ�IH�/ �.�p �E$ ��<�&��Q�L��< � ���*��Gb{4 �Y� �S��.�/$���۪ ���	��υ��P �f��8�OP- ���>���Z͏�� `ky�'�� D��Lt���D�����I� A~ 'M�[ �Od E'5�#:sڥj��� ����>��Dz9.Z��/��]Q�Y�~�x d3+���f{���,� JIP�M �=� % h$���圜���m���>%|���_M�  ���Q�#,�*����Osj�V\ �� 7B���C`�p��  r  odҡ���M n�Z1� 	N	�p(v)�|p (
�Xd���fȵP�!�@�A �p E��hW �d<��*iP�fPY%5J��  �ٯ���џ�R����ܶj �����eW P  'ۡ� �a
�   c ��3����b�kc� �p��v i�O����� z����B�u��K���r��.r�����Do[����������uq��zf�ؤ��oSu�� �����T����M�  ۥ${��R u1 �� ���>�E�H 5�-(=�7���ZA�	���ei8*�Tvd�TY���L�� yf�t	 �� = ��Z�w1�H�@� p�*v� ~Ru�N�ܶ �C5���!jJmVC��\��	s�ptk������[ w�^2�){.��q �}���j�wI����
�K�XնY��3m�Am,/o��7x8� �������&�p�`  @>#�ż��ʸ�/� �l����v�#�� vi�-���	`2娐 '�aE�����Ɗ % ����� �/����m�!�� �
D x );

Stackoverflow question:

http://stackoverflow.com/questions/6348065/sencha-touch-jsonp-error

[Matt]

Can confirm this is happening to me too, for this request:

http://search.twitter.com/search.json?&q=from%3Atfr%20OR%20from%3ASOIANYC&rpp=5&callback=jQuery16106527990615355083_1308086280457&_=1308086281400

(I'm fetching tweets from two users - "tfr" and "SOIANYC" - if I
remove SOIANYC from the query, it returns properly-formatted data.
maybe this helps?)

On Jun 14, 9:51 pm, Mike McNeil <michael.r.mcn...@gmail.com> wrote:
> Same thing's happening to me.  Check out:

>  http://search.twitter.com/search.json?q=%23jimromeisburning&callback=dog<http://search.twitter.com/search.json?q=%23jimromeisburning&callback=dog>

>
> Should look like this:
> dog(
> {
>

>    - -
>    results: [
>       - -
>       {
>          - from_user_id_str: "257702817"
>          - profile_image_url: "
>          http://a1.twimg.com/profile_images/1364757456/2011-05-22_13.20.33_nor...
>          "
>          - created_at: "Tue, 14 Jun 2011 20:36:33 +0000"
>          - from_user: "ATLienSince1987"
>          - id_str: "80735397449183232"
>          - -
>          metadata: {
>             - result_type: "recent"
>          }
>          - to_user_id: null
>          - text: "#JimRomeIsBurning"
>          - id: 80735397449183230
>          - from_user_id: 257702817
>          - geo: null
>          - iso_language_code: "is"
>          - to_user_id_str: null
>          - source: "&lt;a href=&quot;http://levelupstudio.com"

>          rel=&quot;nofollow&quot;&gt;Plume  &lt;/a&gt;"
>       } ......
>    ]

>    - max_id: 80735397449183230
>    - since_id: 0
>    - refresh_url: "?since_id=80735397449183232&q=%23jimromeisburning"
>    - results_per_page: 15
>    - page: 1
>    - completed_in: 0.185672
>    - since_id_str: "0"
>    - max_id_str: "80735397449183232"
>    - query: "%23jimromeisburning"

[Taylor Singletary]

Thanks for all the extra info, folks -- this is very useful and I've now got a reproducible case. We'll look into this.

Thanks!
@episod - Taylor Singletary

[Paul Bailey]

We are seeing the same problem on our latest site: http://grb.culturemap.com/senior-games/

It is intermittent; sometimes good JSON is returned and sometimes not.

[michael hazzard]

Fyi Guys,

The way I worked around this since a try-catch was useless is that I used a window.oneerror = function(){ to ignore the offending tweet and count errors etc.

//mh

On Tue, Jun 14, 2011 at 6:17 PM, Paul Bailey <paul.m...@gmail.com> wrote:

We are seeing the same problem on our latest site: http://grb.culturemap.com/senior-games/

It is intermittent; sometimes good JSON is returned and sometimes not.

--

Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk

[Romica]

I have the same problem (bad JSON most of the times). The weird thing
is that if I access the URL direct from the browser it works (good
JSON every time). I'm using jQuery.getJSON and I think that this (or
something connected to this) might be the problem.

Unfortunately I cannot use window.onerror (as Michael suggested)
because I have multiple asynchronous requests at the same time (it
would be tricky and expensive to find out which one is bad - at least
I suppose so).

Good luck in solving this

[marc fawzi]

Re: error handling the jsonp "script error"

I started using the previously mentioned "window.onerror" mechanism
wherein I reload the iframe from which I'm initiating the jsonp call,
up to 10 times, 3 seconds apart (the average delay in getting the
script back from twitter, which triggers the error which in turn
starts another jsonp request and so on, up to 10 times)

I'm using this approach in addition to my pre-existing "5 retries on
general error, 10 seconds apart" non-blocking error handler which was
able to survive the "illegal character" error on its own and recover
from it, but sometimes it ended up with "Sorry, try again
later" (twitter recovers after a while on its own and my app continues
from there but sometimes the error persists with twitter.) The pre-
existing handler was using more time on average (about 20-50 seconds)
and the newly added window.onerror mechanism, which runs before the
self-terminating, non-blocking handler loop, is taking less time on
average (6-15 seconds) to go thru the same process for the jsonp
"script error" reported by the browser. However, sometimes, this error
persists for too many retries and falls out of the window.onerror but
99% of the time so far it's getting caught by my pre-existing general
error handler (now, the second layer of defense)

I never thought I had to double up! :)

Re: illegal character

Wow. Really? you can have the unicode character for "delete" \u0008 in
the response? I have a full dump of the jsonp response in firebug and
that's the character that firebug terminates at when reporting the
error so I assume that's the culprit, but could be others as well. On
a more relevant note, I don't think the illegal character coming back
in jsonp have anything to do with the tweet's content. It looks as if
some tweets contain some combination of characters that cause twitter
to return a binary dump of some sort (in casual terms, it causes it to
hallucinate.)

Anyway.... intriguing. :)

Marc

[michael hazzard]

Marc,
Nice theory but I'm not so sure since I'm reloading the same exact tweet (I think)over and over again and its getting corrupted.
Although I've heard of type of thing before where an application (notepad) tries to do charset encoding detection but fails and outputs a corrupted file upon reopening.

[themattharris]

Hey everyone,

Thanks for all the information you provided. We were able to reproduce
the problem and the engineers have now released a fix.
You shouldn't receive any garbled API responses anymore, if you do
please let us know.

Best,
@themattharris

On Jun 14, 7:59 pm, michael hazzard <miramardes...@gmail.com> wrote:
> Marc,
> Nice theory but I'm not so sure since I'm reloading the same exact tweet (I
> think)over and over again and its getting corrupted.
> Although I've heard of type of thing before where an application (notepad)
> tries to do charset encoding detection but fails and outputs a corrupted
> file upon reopening.

[michael hazzard]

Much better!

I let http://jsfiddle.net/NBCGf/34/ run 1000 tries successfully before taking mercy on your servers and shutting it off.

Good job and thanks,

//mh