Invisible Character Mystery

590 views
Skip to first unread message

Bull

unread,
Oct 20, 2011, 9:56:47 AM10/20/11
to BBEdit Talk
Good morning,

From time to time a mysterious invisible character shows up in my
BBEdit text. It appears on the screen as just a space, but when I Show
Invisibles + Show Spaces, the diamond symbol is not there. I
highlighted the space and opened the ASCII Table, but no ASCII code
was indicated. I tried all sorts of different text codings but the
character remains invisible. When I Zap Gremlins it does indeed get
zapped. When I try to save a file that contains the character with
Western encoding I'm told the character is unmappable, and that I have
to save it as UTF-8.

The problem comes if I copy a string with the character in BBEdit and
paste it in Terminal (or almost anywhere else). If the character is in
the middle of a string, the two adjacent words are catenated into one.
If the character is at the end of a string, then when I paste that
string in Terminal it appears as a space but is treated as part of the
word it abuts.

I don't really care what this character is, but I would love to know
what I'm doing that creates it. I just spent an hour trying to debug a
perfectly good Perl script when I fed the script a string with the
mystery character at the end.

Thank y'all,
Wayne Durham
Blacksburg, Virginia

Rich Siegel

unread,
Oct 20, 2011, 9:59:28 AM10/20/11
to bbe...@googlegroups.com
On Thursday, October 20, 2011, Bull <dur...@vt.edu> wrote:

> From time to time a mysterious invisible character shows up in my
> BBEdit text. It appears on the screen as just a space, but when I Show
> Invisibles + Show Spaces, the diamond symbol is not there.

It's probably a nonbreaking space, which you get by typing Option-space.

R.
--
Rich Siegel Bare Bones Software, Inc.
<sie...@barebones.com> <http://www.barebones.com/>

Someday I'll look back on all this and laugh... until they sedate me.

Bull

unread,
Oct 20, 2011, 10:13:07 AM10/20/11
to BBEdit Talk
No, the nonbreaking space shows up as a small filled circle with Show
Invisibles. This thing doesn't appear at all.

Bull

unread,
Oct 20, 2011, 11:08:41 AM10/20/11
to BBEdit Talk
After fooling around with the mystery character a little more, it does
more strange things in Terminal than I thought. In the middle of a
string it appears on the screen to be removed and the surrounding
parts catenated, but when processed (by my perl cryptogram solver) it
is still there, and is treated as three distinct letters.

On Oct 20, 9:59 am, Rich Siegel <sie...@barebones.com> wrote:

Steve Kalkwarf

unread,
Oct 20, 2011, 11:13:36 AM10/20/11
to bbe...@googlegroups.com
On Oct 20, 2011, at 11:08 AM, Bull wrote:

> After fooling around with the mystery character a little more, it does
> more strange things in Terminal than I thought. In the middle of a
> string it appears on the screen to be removed and the surrounding
> parts catenated, but when processed (by my perl cryptogram solver) it
> is still there, and is treated as three distinct letters.

File->Hex Dump Front Document…

What are the bytes?

Steve

Bull

unread,
Oct 20, 2011, 11:21:40 AM10/20/11
to BBEdit Talk
In BBEdit, this line:
b b
(there's a mystery character before the first b, between the two bs,
and after the second b) gets Hex Dumped as
00 62 FE FF 00 62 FE FF 00 0D .b...b....

Bull

unread,
Oct 20, 2011, 11:30:15 AM10/20/11
to BBEdit Talk
Sorry, I left off a bit. There's a FE FF at the beginning of the line
too. In other places I saw two mystery characters in a row show up as
FE FF FE FF 00.

Bull

unread,
Oct 20, 2011, 11:49:38 AM10/20/11
to BBEdit Talk
It's a BOM isn't it? Where did it come from?

Bull

unread,
Oct 20, 2011, 7:19:58 PM10/20/11
to BBEdit Talk
This thread has come down to this question: How did I generate a BOM
pair of bytes while simply typing text in BBEdit?

François Schiettecatte

unread,
Oct 20, 2011, 7:55:44 PM10/20/11
to bbe...@googlegroups.com
Two possible options come to mind:

- your text encoding default is set to include BOM

- the encoding pulldown at the bottom left of the text window was set to include BOM (note that you can change the encoding of a document on the fly, pretty useful)

François

> --
> You received this message because you are subscribed to the
> "BBEdit Talk" discussion group on Google Groups.
> To post to this group, send email to bbe...@googlegroups.com
> To unsubscribe from this group, send email to
> bbedit+un...@googlegroups.com
> For more options, visit this group at
> <http://groups.google.com/group/bbedit?hl=en>
> If you have a feature request or would like to report a problem,
> please email "sup...@barebones.com" rather than posting to the group.
> Follow @bbedit on Twitter: <http://www.twitter.com/bbedit>

Bull

unread,
Oct 21, 2011, 4:56:54 AM10/21/11
to BBEdit Talk
Coding is and always has been set to Western (Mac OS Roman).

On Oct 20, 7:55 pm, François Schiettecatte <fschietteca...@gmail.com>
wrote:
> Two possible options come to mind:
>
> - your text encoding default is set to include BOM
>
> - the encoding pulldown at the bottom left of the text window was set to include BOM (note that you can change the encoding of a document on the fly, pretty useful)
>
> François
>
> On Oct 20, 2011, at 7:19 PM, Bull wrote:
>
> > This thread has come down to this question: How did I generate a BOM
> > pair of bytes while simply typing text in BBEdit?
>
> > On Oct 20, 11:49 am, Bull <dur...@vt.edu> wrote:
> >> It's a BOM isn't it? Where did it come from?
>
> > --
> > You received this message because you are subscribed to the
> > "BBEdit Talk" discussion group on Google Groups.
> > To post to this group, send email to bbe...@googlegroups.com
> > To unsubscribe from this group, send email to
> > bbedit+un...@googlegroups.com
> > For more options, visit this group at
> > <http://groups.google.com/group/bbedit?hl=en>
> > If you have a feature request or would like to report a problem,
> > please email "supp...@barebones.com" rather than posting to the group.

François Schiettecatte

unread,
Oct 21, 2011, 7:22:54 AM10/21/11
to bbe...@googlegroups.com
Maybe it got copy-pasted in, or you duplicated the file from another one, at this point this is speculation.

F.

> please email "sup...@barebones.com" rather than posting to the group.

Doug McNutt

unread,
Oct 21, 2011, 4:15:14 PM10/21/11
to bbe...@googlegroups.com
At 07:22 -0400 10/21/11, Fran�ois Schiettecatte wrote:
>Maybe it got copy-pasted in, or you duplicated the file from another one, at this point this is speculation.
>

It's possible that the behavior depends on the format in which the file is actually saved on disk. Most of use UTf-8 which would look like bytes of old.

But what if the OP has storage set to one of the UTF-16 options? Would things be different?
--

--> From the U S of A, the only socialist country that refuses to admit it. <--

Bull

unread,
Oct 21, 2011, 7:08:52 PM10/21/11
to BBEdit Talk
That sounds good, but the file with the problematic text was never
saved to disk, as the text originated in the Scratchpad.

Maybe the mystery character comes from something about the
Scratchpad's ambiguous nature: looks like a file, acts like a file,
but is not a file.

Does the Scratchpad have some snippet of code that generates a BOM
under some circumstances?
Reply all
Reply to author
Forward
0 new messages