Entity encoding issue (only effects XML responses)

[Alex Payne]

A change on our end resulted in some entities not being properly
encoded for XML. If you're seeing ampersands encoded along with the
entities themselves, that's the bug in question.

We're identified the problem and are working on a fix, and will deploy
one as soon as possible. Don't update your applications; the XML
responses we're currently providing aren't encoded as intended. Also
note that this does not effect API responses in JSON format.

--
Alex Payne - API Lead, Twitter, Inc.
http://twitter.com/al3x

[Mario Menti]

On Tue, Sep 23, 2008 at 9:04 PM, Alex Payne <al...@twitter.com> wrote:

A change on our end resulted in some entities not being properly
encoded for XML.  If you're seeing ampersands encoded along with the
entities themselves, that's the bug in question.

We're identified the problem and are working on a fix, and will deploy
one as soon as possible.  Don't update your applications; the XML
responses we're currently providing aren't encoded as intended.  Also
note that this does not effect API responses in JSON format.

Hi Alex,

is it possible that this somehow also affected the posting of messages? I've suddenly seen changes in how some characters are posted, coming from twitterfeed, and am certain I haven't made any changes during this period at my end:

Compare this recent post: http://twitter.com/jemimakiss/statuses/934245954

... with this from 2 days ago: http://twitter.com/jemimakiss/statuses/931898377

Thanks for any help/insight - just wondering if this is something you will fix (as you suggested above), or if I should modify my code..

Cheers,

Mario.

[Alex Payne]

It shouldn't have changed posting. We'll check that out.

--

[Mario Menti]

On Thu, Sep 25, 2008 at 5:01 PM, Alex Payne <al...@twitter.com> wrote:

It shouldn't have changed posting.  We'll check that out.

Hi Alex - the issue appears to have been fixed about 10 hours ago, the encodings are now correct, so someone at your end must have made it work again..

Cheers,

Mario. 

[Alex Payne]

Yes, we sorted it out.

--

[David Håsäther]

* Alex Payne:

> Yes, we sorted it out.

I'm still experiencing issues with entity references used wrongly.
I've tried analyzing a bit:

I posted a tweet[1] looking like this:

< & > &lt; &amp; &gt;

This comes through in the web interface and the JSON API format as

&lt; & &gt; &lt; &amp; &gt;

This is clearly wrong since a) The first "&" should be &amp; (at least
for consistency), and b) the second "&lt;", "&amp;" and "&gt;" should
really be "&amp;lt;", "&amp;amp;" and "&amp;gt;" respectively. Because
of this, in the web interface, this currently renders as

< & > < & >

which is not what I posted.

I.e., I'd expect

< & > &lt; &amp; &gt;

to be "escaped" as

&lt; &amp; &gt; &amp;lt; &amp;amp; &amp;gt;


Moving on to the XML API format, for the same tweet, I get

&amp;lt; &amp; &amp;gt; &amp;lt; &amp;amp; &amp;gt;

which, apart from the double "escaping" on the first "&amp;lt;" and
"&amp;gt;" is what I would expect. I suspect that this may be the
result of "escaping" "<" and ">" before "&". An example (using
JavaScript syntax) shows the difference:

This

"< & > &lt; &amp; &gt;".replace(/</g, "&lt;").replace(/>/g,
"&gt;").replace(/&/g, "&amp;")

results in

&amp;lt; &amp; &amp;gt; &amp;lt; &amp;amp; &amp;gt;

while

"< & > &lt; &amp; &gt;".replace(/&/g, "&amp;").replace(/</g,
"&lt;").replace(/>/g, "&gt;")

results in

&lt; &amp; &gt; &amp;lt; &amp;amp; &amp;gt;

The latter being exactly what I would expect (in all formats, XML,
JSON and the web interface), and what would render correct in the
browser, namely as

< & > &lt; &amp; &gt;

[1] http://twitter.com/testcases/statuses/936394085

[Alex Payne]

Thanks for the detailed report. We'll re-fix it :)

--

[ehmo]

Hey guys,
i've one idea. If you following someone but he is not following you
and he'll write you PM, then will be very nice from system, if i can
reply for example just to this message.

Better will be opportunity to send PM to anyone who i want to and
this recipient can allow msges from me or not. Because this is very
stupid to ask everybody to follow me because i can't react on his PM.

Have nice day,
ehmo