Re: [tw] TiddlyFox & encoding

[Jeremy Ruston]

Hi Chuck

There have been some reports of character encoding issues when running under Ubuntu, but nothing I've been able to pin down until now.

Do you get the mis-encoding when using the latest TiddlyWiki downloaded from tiddlywiki.com? What about if you use a copy of 2.6.5 from tiddlywiki.com/archive?

Many thanks

Jeremy

--

Jeremy Ruston

jeremy...@gmail.com

On 29 Nov 2012, at 19:40, Chuck Bearden <cfbe...@gmail.com> wrote:

Hi all,

Many thanks to Jeremy Ruston for the TiddlyFox extension. I have a question about it.

I run Ubuntu 11.10, and of course FF gets upgraded out from under me. The latest upgrade to 17 left me unable to save changes to TiddlyWikis, even with the special edits in prefs.js. I installed TiddlyFox and I can save changes again.

However, it appears that the encoding of the TiddlyWiki has been changed after saving from UTF-8 to Latin-1. It could be due to some change in FF, but my initial suspicion is that it is a TiddlyFox thing.

Any thoughts?

Thanks,
Chuck

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To view this discussion on the web visit https://groups.google.com/d/msg/tiddlywiki/-/2P9PR2UaErMJ.
To post to this group, send email to tiddl...@googlegroups.com.
To unsubscribe from this group, send email to tiddlywiki+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/tiddlywiki?hl=en.

[Chuck Bearden]

Hi Jeremy,

The TiddlyWiki I'm using is a 2.6.5 (October 6, 2011) that I've been editing through several FF versions.

I've tested with an empty 2.6.5 and an empty 2.6.6, and the problem seems to affect only the 2.6.5 TiddlyWiki. I pasted identical text (including 'ü' '…', and some polytonic Greek) into each and saved them, then read the files into Python variables as strings. When I try to decode the 2.6.5 file as UTF-8, it throws an exception. Encoding the 2.6.5 as Latin-1 as well as the 2.6.6 as Latin-1 or UTF-8 works fine. Those are tiddlers that have only ever been edited under FF17 with TiddlyFox.

I just discovered the Upgrade option Backstage. The Upgrade area is still visible with the "Loading core code" notice. It has created what appears to be a backup file, and there are differences between the empty.html and the empty.pre-core*.html files, but the Upgrade screen never goes away and the version info remains the same. Is this expected?

Thanks for all your work!
Chuck

[Helge]

Same here, a TW file that was (to my knowledge) only edited with FF17/TiddlyFox has had its (formerly UTF) encoding corrupted during one of the recent edits. I am using 2.7.0b2 with Firefox 17.0 on a Mac.

Helge.

[Chuck Bearden]

I was able to upgrade my TiddlyWiki to 2.6.6 using the instructions in this post:

  https://groups.google.com/d/msg/tiddlywiki/vxwzWg8GV6o/0zE582IUXncJ

The file now seems to be UTF-8.

Chuck

[Chris]

Hi, I should have been watching this thread as I've been solving the self same problem over the same timeframe. I've had the same problem with my iOS app because I read the tiddlywiki as UTF8 - and iOS is fussy enough to reject the file offhand if it is not pure UTF8. Had not realized that the problem was hitting other platforms too.

Here's what I found:

1. There are indeed non-UTF8 characters in 2.6.5. They are in the jQuery script. They go away in 2.6.6 I believe because 2.6.6 a uses a newer version of the jQuery script without the offending characters. 

2. The problem characters are "\xA0" at character 15050 of the jQuery library script. The problem persists with Firefox because when Tiddlyfox saves it reads from the original file (including the offending bytes), transcludes the tiddler section, and writes back out - but neither the read nor the write includes a conversion to UTF8. What this means is that you end up with a potentially non UTF8 compliant file each time you save from FF17 with TiddlyFox. I think it should be possible to fix with a change the write statement in TiddlyFox - but  I've been sluggish in looking at this in any more detail.

3. You can fix the problem for 2.6.5 tiddlywikis by running them through a e.g. iconv on Mac or Linux - to create a true UTF8 file. So far this has worked for me - including tiddlywikis with heavy jQuery use by plugins.

4. Of course you can also fix the problem by upgrading to 2.6.6.

Hope this helps.

Chris.

On Sunday, December 16, 2012 7:19:54 AM UTC-7, Michał Bagiński wrote:

Hi,
I had the same problem with wrong encoding when using this addon and TW 2.6.5. I have investigated the issue and wrote my own addon from scratch using Mozilla Add-on Builder. You can download it from http://ul.to/q7mrb1j4. Note that you need to uninstall TiddlyFox before testing mine because both addons don't work together properly.

I'll let you test it and if it works fine, either I'll try to fix Jeremy's addon or agree with him to take mine because two simple addons doing the same thing certainly don't make sense; this is why I haven't published mine on Mozilla Add-ons website.

Some technical remarks: jQuery embedded in TW 2.6.5 contains one two-byte UTF-8 character, it's just after the &&(e.browser.safari=!0),j.test(" fragment of the file. This character isn't encoded correctly by TiddlyFox while my addon handles it properly. The easiest test is to download empty TW 2.6.5, modify a tiddler, enter some unicode characters like ąę, save, and compare with non-edited empty TW 2.6.5 in encoding-aware diff tool. If both files have UTF-8 encoding, the relevant bit in jQuery doesn't differ, and your edited tiddler is correct as well, you have a confirmation that the addon works fine.

Michał

[Helge]

Thanks Michał,

so far it seems to work. No changed encoding or shredded Umlauts with 2.7.0b2. :-)

Helge.

[Jeremy Ruston]

Hi Michał

Sorry for the delayed reply, I've been only slowly catching up with a huge email backlog. And thank you to you and Chris for figuring out the jQuery issue.

Can you publish the source code of your add-on? I'd be interested to understand what it does differently. Can it be structured as a pull request to TiddlyFox, or does it work in a different way?

Many thanks,

Jeremy

On Sun, Dec 16, 2012 at 2:19 PM, <mic...@8aginski.com> wrote:

Hi,
I had the same problem with wrong encoding when using this addon and TW 2.6.5. I have investigated the issue and wrote my own addon from scratch using Mozilla Add-on Builder. You can download it from http://ul.to/q7mrb1j4. Note that you need to uninstall TiddlyFox before testing mine because both addons don't work together properly.

I'll let you test it and if it works fine, either I'll try to fix Jeremy's addon or agree with him to take mine because two simple addons doing the same thing certainly don't make sense; this is why I haven't published mine on Mozilla Add-ons website.

Some technical remarks: jQuery embedded in TW 2.6.5 contains one two-byte UTF-8 character, it's just after the &&(e.browser.safari=!0),j.test(" fragment of the file. This character isn't encoded correctly by TiddlyFox while my addon handles it properly. The easiest test is to download empty TW 2.6.5, modify a tiddler, enter some unicode characters like ąę, save, and compare with non-edited empty TW 2.6.5 in encoding-aware diff tool. If both files have UTF-8 encoding, the relevant bit in jQuery doesn't differ, and your edited tiddler is correct as well, you have a confirmation that the addon works fine.

Michał

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To view this discussion on the web visit https://groups.google.com/d/msg/tiddlywiki/-/pWZD2bTuRyAJ.

To post to this group, send email to tiddl...@googlegroups.com.
To unsubscribe from this group, send email to tiddlywiki+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/tiddlywiki?hl=en.

--
Jeremy Ruston
mailto:jeremy...@gmail.com