Problem sending tweets with nbsp chars

[Roy Leban]

Twitter is rejecting tweets as too long when the nbsp character is
used. Here is an example tweet in plain text

Clue 5 of 15: R _ C _ _ _ N _ N _ _ E _ _ _ T E R _ _ _
_ N E E R _ N _ _ R _ C U L T U R E http://www.puzzazz.com/s348
[140 chars]

And, as I'm sending it with the nbsp characters:

Clue 5 of 15: R _ C _  
 _ _ N _  
 N _ _ E _  
 _ _ T E R    _  
 _ _ _ N E E R  
 _ N  
 _ _ R _ C U L T U R E
http://www.puzzazz.com/s348

If the nbsp's are each counted as 6 characters, this would be 400
chars, but Twitter accepts tweets like this. For example, this tweet:

http://twitter.com/Puzzazz/status/9781320047

is 114 chars but I send 304 chars with the nbsp's.

I have a guess that this only happens when the resulting tweet is
exactly 140 chars. To test this theory, I just modified the site to
shorten that tweet below 140. Sure enough, it works:

http://twitter.com/Puzzazz/status/9963348931

[Taylor Singletary]

Hi Roy,

You shouldn't be sending spaces as " " -- that's HTML entity encoding. It's best to send space characters as "%20" instead.

For example:

You'd set your POST body to:

status=There%20is%20%20%20%20%20space%20for%20love%20in%20%20%20the%20universe

If you were trying to set the status

"There is    space for love in   the universe"

In your signature base string for OAuth you'd have to encode those spaces one more time:

POST&http%3A%2F%2Fapi.twitter.com%2F1%2Fstatuses%2Fupdate.xml&oauth_consumer_key%3Dri8JxYK2ddwSV5xIUfNNvQ%26oauth_nonce%3DulF0XQetLMOm5Sr9Yrp027Hzu2mPoTuTqFgshncHBo%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1267717205%26oauth_token%3D819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw%26oauth_version%3D1.0%26status%3DThere%2520is%2520%2520%2520%2520%2520space%2520for%2520love%2520in%2520%2520%2520the%2520universe

Not all Twitter API clients will choose to preserve multiple spaces on display though.

Taylor

[Roy Leban]

Hi Taylor,

I tried %20 along with a lot of other things and   was the only thing that worked in all places -- the web, Twitter clients, and SMS messages to cell phones. Other than this problem, it has worked great for nine months. If Twitter has made changes such that %20 will now work where it didn't before, I'd be happy to switch. But, my guess is that this bug would apply equally well to the %20s.

I also believe that this is a new bug, as I've been using the same code which makes sure I'm under the limit for a long time. I would have to dig through my database of previous puzzles to find out for sure.

I might be able to work around this by setting my internal limit to lower than of 140, but I don't know the exact number. Tweets with as few as 136 characters are being rejected. As a *temporary* workaround, I modified it to check if the tweet is >=136 and chop off the URL in this case. That pushed it quite a bit under 140 and it works fine:

http://twitter.com/Puzzazz/status/9985983801

,I vote for fixing the bug. If that can't happen, can I at least find out what the true limit is? That way, I don't have to figure it out via trial and error.

/Roy

[Charles A. Lopez]

URL Encoding? 

--
Charles A. Lopez
charle...@gmail.com

What's your vision for your organization?
What's your biggest challenge?

Let's talk.
(IBM Partner)

[Roy Leban]

Thread of a discussion:

twitterapi: @RoyLeban nbsps will be literally encoded as nbsps (and not spaces) ^RK

RoyLeban: .@twitterapi Yup, nbsp is what I need. See @Puzzazz to see what I'm doing. Any chance bug'll get fixed? If not, what's the real tweet limit?

twitterapi: @RoyLeban it's not a bug. &nbsp are handled like that on purpose. ^RK

RoyLeban: @twitterapi What does "like that" mean? You count each nbsp as 1 char until you get close to 140 then count them as more? Sounds like a bug

____

Here's the deal: non-breaking spaces are the only thing which work as non-breaking spaces. I know that sounds like a tautology, but look at the @Puzzazz feed and you'll see what I need. I tried unicode non-breaking spaces, I tried UTF8 non-breaking spaces, I tried %20's, I tried multiple spaces. The only thing that worked properly are   entities and they work perfectly. Except...

There's a bug in how they're counted. If I don't send very many, I do not have problems and each &nbsp; is counted as 1 character. I can send a tweet of >300 chars that resolves to <140 without problems. However, if the resulting tweet is close to 140 after the nbsp's are resolved, Twitter erroneously claims that the tweet is too long. It is not an off-by-one bug as a tweet of 136 fails. I do not know what the lower bound is.

I don't know for sure, but I think this bug was recently introduced. I have not had this problem in the past. However, there was one 3-day span in which no tweets went out, then they suddenly started again. By the time I noticed, it was too late to do an analysis as I wasn't logging the error message from Twitter (I am now). It could have been a similar temporary problem.

I'd love to see this bug fixed. If you can't fix it, is it possible to give me enough information about the bug so that I can at least know what the true tweet limit is?

Thanks very much!

/Roy

On Thu, Mar 4, 2010 at 7:45 AM, Taylor Singletary <taylorsi...@twitter.com> wrote:

[Brian Smith]

> Hi Taylor,
>
> I tried %20 along with a lot of other things and &nbsp; was the only
> thing that worked in all places -- the web, Twitter clients, and SMS
> messages to cell phones. Other than this problem, it has worked great
> for nine months. If Twitter has made changes such that %20 will now
> work where it didn't before, I'd be happy to switch. But, my guess is
> that this bug would apply equally well to the %20s.

First of all, %20 is a regular space, URL-encoded. The non-breaking space is Unicode codepoint 160 (A0h). You can encode the non-breaking space in XML as "&#xa0;" and in JSON as "\u00a0".

Secondly, I don't know what you are doing with the non-breaking spaces, but they're unlikely to work as you expect them to. My Twitter client replaces all runs of consecutive whitespace (including newlines, tabs,non-breaking spaces, and more exotic whitespace) as a single normal space, and I suspect that many apps will do the same. Attempts to control the layout of a tweet are futile at worst and annoying at best. I am sure you are trying to format tweets in a helpful, non-annoying, cool way, but your formatting is unlikely to survive given the anti-annoying-people countermeasures that good clients have in place.

Cheers,
Brian :)

[Scott Wilcox]

Agreed, I strip out too.

[Roy Leban]

To Scott and Brian, please take a look at the tweets on @Puzzazz. There is no other way to have them appear correctly. What I am doing is not annoying in the least and it may be futile in your particular Twitter clients but I tested what I'm doing in many clients and it worked fine.

I'm open to suggestions but "you're screwed" is not a suggestion. And changing what I'm doing doesn't address the fact that Twitter has a bug.

For the record \u00a0 did not work. I didn't try &#xa0; as I got to &nbsp; first. I would guess it would work but have the same bug with Twitter.

/Roy

[M. Edward (Ed) Borasky]

I looked at the @puzzazz stream. It's interesting, but you've already
got a mix of links to puzzles and embedded puzzles and clues. So I'm
not sure what you'd lose by making your tweet stream 100 percent links
with searchable US-ASCII text, other than getting filtered out by
algorithms that say, "Don't follow people whose tweets are 90 percent
links or more." ;-) This looks like a fun tweet stream, and I followed
it. ;-)

And by having every tweet contain a link, you *gain* click tracking,
using, say, the free bit.ly pro tracking capability! I assume you're
trying to "market" something, for some definition of "market". Why not
make it easy for Google and Bing to find you via Twitter, and for
yourself to monitor, by having your tweets be a mix of natural human
language and trackable links?
--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/

"A mathematician is a device for turning coffee into theorems." ~ Paul Erdos

[@epc]

On Mar 8, 4:06 pm, Roy Leban <r...@royleban.com> wrote:
> I'm open to suggestions but "you're screwed" is not a suggestion. And
> changing what I'm doing doesn't address the fact that Twitter has a bug.

What code are you using to post these tweets? Is it possible that it
is transcoding &nbsp; to something before transmitting to twitter?
Because what appears in your older tweets is the (correct) &#160; not
&nbsp; AFAIK, twitter does not decode HTML entities.
--
-ed costello

[Julio Biason]

On Mon, Mar 8, 2010 at 5:15 AM, Roy Leban <r...@royleban.com> wrote:
> RoyLeban: @twitterapi What does "like that" mean? You count each nbsp as 1
> char until you get close to 140 then count them as more? Sounds like a bug

Don't get me wrong, but I'm guessing that your problem is that you are
thinking Twitter counts chars, when it counts bytes, actually. Raffi
posted an URL with the proper way to count characters:
http://apiwiki.twitter.com/Counting-Characters

--
Julio Biason <julio....@gmail.com>
Twitter: http://twitter.com/juliobiason

[Cameron Kaiser]

> > RoyLeban:__@twitterapi What does "like that" mean? You count each nbsp as 1

> > char until you get close to 140 then count them as more? Sounds like a bug
>
> Don't get me wrong, but I'm guessing that your problem is that you are
> thinking Twitter counts chars, when it counts bytes, actually. Raffi
> posted an URL with the proper way to count characters:
> http://apiwiki.twitter.com/Counting-Characters

No, that document just makes it clear that no matter how a character is
composed, Twitter will always accept 140 characters. Raffi has made it plain
(because I bluntly asked) that 140 character -- not just byte -- tweets are
kosher. Even if they're not in Hebrew.

I think the OP's problem is in handling entities, not UTF-8.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Po-Ching Lives! ------------------------------------------------------------