Upgrading the upgrading engine: testers needed, mostly among _those who use old FireFox_

[Yakov]

Hi everyone,

while testing/debugging the upgrading mechanism of TW 2.9.1, I've encountered a severe issue for FireFox users: non-ASCII characters are corrupted in updated TW. I've found the source of the problem and it seems to be a patch which fixed old FireFox' bad behaviour about UTF-8 files.

Now I've come up with 2 versions of fixing this. The first one just removes that patch and probably introduces some trouble for those who still use old FF or FF-based browsers (I have almost no idea how old are those those FF which are affected). The second one deals with it in a more educated manner, borrowing a similar fix from the TWc core itself, although that fix has no comments (including in git history) so I have no idea if it separates FF versions correctly.

On the other hand, I only care about old FF versions which are actually used by TWc users (I don't know how many such users are out there). So I invite those who do use old FF versions to test the updated upgrade engine (both with TiddlyFox and through the old FF saving capabilities). I also want to know how many such users are out there because Okido recently tested jQuery 3.3.1 and it is quite tempting to update it.

If you are using up-to-date FF or even another browser, you can try this too, but it's better to post in another thread because issues that you may encounter are not related to the encoding conversion (it is done for window.netscape !== undefined only).

Now, how to test:

1. download the archive and unzip into some folder (better to start from a folder which has only ASCII symbols in the full path)
   
2. open both "fake 2.9.2" TWs in your FF, allow TiddlyFox to save if you use it
3. have a look at contents of "Non-ASCII stuff", you'll need to check whether the those get corrupted after the upgrade
   
4. everything is already pre-set, go on to the sidebar → upgrade → upgrade → start
5. don't mind the error message ("It appears that this TiddlyWiki has been incorrectly saved. Please see http://www.tiddlywiki.com/#Download for details"), it's just wrong; and cancel the extra download dialog after the second autoreload
6. once TW is updated, check that you see 2.9.3 in MainMenu, the contents of "Non-ASCII stuff" are the same (if they are not, it's likely they are obviously not, like ⌫ is turned into a hieroglyph or something like that)
7. let me know how it went for you
   

I've tested this in FF 54 which is obviously not that old, but I saw some users still using FF 3, so.. I'll try to take this into account.

Best regards,
Yakov.

[Alfonso Arciniega]

Yakov,

I'll give it a test and get back to you.

Cheers,

Alfonso

[Alfonso Arciniega]

Hi Yakov,

1) Test on FF 52.9.0 ESR (64-bit), using TiddlyWiki for Firefox 2.0.2-prerelease extension, on Windows 10: Non-ASCII stuff tiddler remains the same

2) Test on FF 3.6.28 (32-bit), no extension required, on Windows 10: Non-ASCII stuff tiddler got corrupted: the tiddler content was replaced with non-ASCII stuff

Additional comments:

a) FF 3.6.28 is my preferred version for TWC because all extensions and macros work seamlessly. Some extensions and macros, especially old ones, don't work on FF 52.9.0 ESR. Examples of extension and macros that don't work are:
PrettyDatesPlugin
SaveAndReloadMacro
LessBackupPlugin
<<ticklerAlert>> macro

b) Some TWC v2.9.1 documents cannot be saved on TiddlyDesktop v0.0.13. I upgraded some mGSD documents from TWC v2.8.1 to TWC v2.9.1 and then they could not be saved. I reversed the documents version to v2.8.1 and they work fine now. Simple basic TWC v2.9.1 documents work fine though.

See http://mgsd.tiddlyspot.com/
and
http://mgsd1.tiddlyspot.com/ <-- This one contains the extension and macros listed above that only work on very old versions of FF.

Regards,

Alfonso

[Yakov]

Hi Alfonso,

thanks for testing! Did I get you correctly: these results are the same for the "_ fake 2.9.2 path1.html" and "_ fake 2.9.2 path2.html" files? What I was hoping is that upgrading "_ fake 2.9.2 path2.html" will work in FF 3.6.

Could you open console in both FF versions and check out what config.browser.firefoxDate[1] returns (just copy-paste into console and hit enter)? (presumably console is opened by ctrl+shift+i or ctrl+shift+k but it may be different for the old versions)

Thanks for the additional notes, let's clarify a couple of details:

• I've only managed to find
  http://mptw.tiddlyspot.com/#PrettyDatesPlugin
  http://mgsd1.tiddlyspot.com/#SaveAndReloadMacro
  <<ticklerAlert>> (hidden from search): http://mgsd1.tiddlyspot.com/#MgtdList
  where can I see LessBackupPlugin?
  
  

• Some TWC v2.9.1 documents cannot be saved on TiddlyDesktop v0.0.13
  

  
  

• I wonder if you can provide a TW so that I can reproduce the issue and find out the source. May be you can remove personal data from a problematic TW (and making sure the problem is still there) or send one privately?
  
  This is an unpleasant issue and I'd like to fix this
  

Best regards,
Yakov.

[okido]

HI Yakov,

All tests are done on Linux.

Firefox 14: the core is upgraded and the non-ascii text is OK, for both versions
Firefox 50: the core is upgraded in the file that is saved, but all content is lost, for both versions
Firefox 60: the core is upgraded in the file that is saved, but all content is lost, for both versions

The LessBackupsPlugin comes from Simon Baird @ HTTP://mptw.tiddlyspot.com/#LessBackupsPlugin
I modified the plugin slighty so it is working under FF14 ~ FF60, the saveBackup function is overwritten to do so.
I will drop the code in a different post.

Have a nice day, Okido

[Alfonso Arciniega]

Hi Yakov,

Both files show the same results.

I couldn't get the console returns on any browser.

The LessBackupPlugin is in the mgsd1.tiddlyspot.com TWc

The TWc's that cannot be saved in TiddlyDesktop are relatively large in terms of data and plugins. If I start stripping them from data and plugins until TiddlyDesktop is able to save them, this process could take a very long time. I will try it sometime in the not to far future.

Regards,

Alfonso

[Yakov]

Hi Alfonso and Okido,

I've tried to install elder FireFoxes from PortableApps but that failed for me (for some reason running the exe of the installed FF 3.6.28 actually opens my current FF 54), so it would be lovely if you help me with couple more things to diagnose the problems.

Alfonso, in the updated zip there's another pair of TWs, could you try upgrading _ fake 2.9.2 path3.html ? Its pair (empty fake 2.9.3 3 no fix.html) has "unfixed" encoding and what I need to know is whether this one doesn't corrupt non-ASCII characters during upgrading in FF 3.6.28. And also please copy the line in the "debug info" textarea.

Okido, could you glance the value of config.browser.firefoxDate[1] in the FF versions that you used? Knowing that FF 14 works correctly is nice (although I have no idea if that conversion glitch was the same for Windows and Unix-like OSes), so that timestamp will be particularly useful.

I wonder what saving method did you use, especially for FF 60? It's weird that in FF 50 & 60 the content was lost, though I'm aware of one bug [I've already prepared a fix] and I need to know whether URL hash and query of TWs were empty at the beginning of upgrading. If they were not and if it's possible to clear them with the setup you use, please retest upgrading (in FF 50 & 60) with empty hash and query. Also, if that fails, check if backups for the TWs were created during the upgrading (in normal upgrading process, they are copies of the upgrading TWs with names like _ fake 2.9.2 path1.pre.core.upgrade.20181201.1109071070.html). And one more thing: when upgrading fails, does it stop after the 1st refresh (does page URL contain added ?time=... query and #upgrade:... hash?) and does console show the utf is null error? [note for myself: fail of upgradeFrom → manualConvertUTF8ToUnicode, possible to "miss" when using debugger]

By the way, what Linux do you use? I'm considering using Virtual Box to text various FF versions, may be you can give an advice about the distribution to use for testing? Ideally, I should find the FF version where the "conversion" was fixed and figure out a descriminative expression (using config.browser.firefoxDate[1] or something else), although it's ok to fix things only for those old versions that are actually used.

Best regards,
Yakov.

[Yakov]

Alfonso, regarding plugins that work in FF 3.6.28 but don't work in 52.9.0 ESR:

• http://mptw.tiddlyspot.com/#PrettyDatesPlugin – what exactly doesn't work for you? I've tested it in FF 54 it seems to work as expected. The plugin uses a dangerous practice of changing Date.prototype, though, so may interfere with other extensions doing so
  
• http://mgsd1.tiddlyspot.com/#SaveAndReloadMacro – doesn't it reload? or reloads before saving (doesn't save)? something else?
• <<ticklerAlert>> (requires http://mgsd1.tiddlyspot.com/#MgtdList and also MgtdConf + MgtdSettings + MgtdIndexedTags + MgtdAppFramework) – what params does it use? After I've installed all the mentioned tiddlers it doesn't show me any error but doesn't show anything else as well and I haven't found docs for it yet
  

These plugins seem to be easily adaptable for newer FireFox if something doesn't work. As for http://mptw.tiddlyspot.com/#LessBackupsPlugin (= http://mgsd1.tiddlyspot.com/#LessBackupsPlugin), it uses some extended I/O functionality so indeed server like James' node.js one or MainTiddlyServer and the plugin should be updated to work with newer FF or other browsers (I'm developing smart backuping funcionality for MTS anyway).

By the way, any news regarding TiddlyDesktop v0.0.13 and TWC v2.9.1?
Let me bump some info for myself:

Some TWC v2.9.1 documents cannot be saved on TiddlyDesktop v0.0.13. I upgraded some mGSD documents from TWC v2.8.1 to TWC v2.9.1 and then they could not be saved. I reversed the documents version to v2.8.1 and they work fine now. Simple basic TWC v2.9.1 documents work fine though.

[...]

[okido]

Hello Yakov,

Saver methods:

Firefox 14: TiddlyFox 1.0 alpha 17
Firefox 44: TiddlyFox 1.0 alpha 17
Firefox 50: TiddlyWiki for FireFox 2.0.1
Firefox 60: SaveTiddlers 0.8

Results of upgrading:
Firefox 14: config.browser.firefoxDate[1] "[\"gecko/20100101\", \"20100101\"]"

Firefox 44: _ fake 2.9.2 path1.html, upgrade is OK, config.browser.firefoxDate[1] "["gecko/20100101", "20100101"]"
_ fake 2.9.2 path2.html, upgrade is OK
_ fake 2.9.3 path3.html, upgrade is OK, ASCII chars are corrupted,
_ fake 2.9.3 path3.pre.core.upgrade.20181202.1017389480.html ASCII OK
_ fake 2.9.3 path3.20181202.1017408630.html, Tiddlers are lost, version is 2.9.3

Firefox 50: _ fake 2.9.2 path1.html, does not update at first, alert message: "It appears that this TiddlyWiki has been incorrectly saved. Please see http://www.tiddlywiki.com/#Download for details", if you click OK and continue the result is an upgraded TWc. config.browser.firefoxDate[1] "["gecko/20100101", "20100101"]"
_ fake 2.9.2 path2.html, same as _ fake 2.9.2 path1.html
_ fake 2.9.3 path3.html, does upgrade but content is lost, /_ fake 2.9.2 path3.pre.core.upgrade.20181202.1041221460.html is unchanged

Firefox 60: _ fake 2.9.2 path1.html, does not upgrade, config.browser.firefoxDate[1] "[\"gecko/20100101\", \"20100101\"]", TypeError: utf is null
_ fake 2.9.2 path2.html, same as _ fake 2.9.2 path1.html, TypeError: utf is null
_ fake 2.9.3 path3.html, same as _ fake 2.9.2 path1.html, TypeError: utf is null

I use linux Mint, Firefox is installed using different profiles for every version, you can run several versions at the same time, use -p to start every FireFox with its profile name.

Regarding http://mgsd1.tiddlyspot.com/#SaveAndReloadMacro, it does not save, it only reloads on FF50.
It is reloading before saving is finished, I will try if it works with a promise, it should only load after saving is finished.

[Alfonso Arciniega]

Yakov,

Upgraded _fake 2.9.2 path3.html with no issues (FF 3.6.28).
The debug info shows:
config.browser.firefoxDate[1] = "20120306"

Cheers,

Alfonso

[Alfonso Arciniega]

Yakov,

PrettyDatesPlugin: the issue is simply: the dates are not pretty as the intention of the plugin. This one works on TWc 2.8.1 on old versions of FF, but it doesn't on TWc 2.9.1. As you mentioned before, there should be a conflict with some other plugin though I still have to identify which one. It definitively does not work on FF 52.5.0 ESR.
SaveAndReloadMacro: works on some TWc 2.9.1 though not on all, especially heavy plugin loaded. Similar solution to the previous plugin.
LessBackupsPlugin: it does not work for TWc 2.9.1 on older FF version. However, it does work with Okido's new plugin. On the other hand, Okido's plugin does not work on older versions of FF.

Everything is a bit confusing. Thank you for trying. Again, maybe we should leave this plugin issues as they are.

Regards,

Alfonso

[Yakov]

Hi guys,

thanks for all the tests. I think I should report my progress here.

First, I've took my time, installed a virtual machine and tested saving, backuping, upgrading in Win XP if FF 4-15. In short, the conclusion is: what should be distingiushed is not some FF version upgrade, but rather usage of old FF I/O interfaces which allow to save and do other stuff without any extension (prior to FF 15) – or usage of TiddlyFox or other contemporary methods.

Second, the current problem with the core is – the conversion helpers are placed in random places of the core like convertUTF8ToUnicode in the upgradeFrom function (which deals with a somewhat high-level logic) instead of being put inside the I/O methods itself (mozillaLoadFile in this example).

Third, in the case of loading the only needed fix is exactly moving convertUTF8ToUnicode from upgradeFrom to mozillaLoadFile. Unfortunately, this isn't enough to fix the whole thing because backups are created in a manner different from saving main TW and for correct functioning not only loading methods should be fixed, but saving as well. Like it was mentioned previously, the inconsistent usage of conversion methods in the core also forces to introduce additional hacks in server solutions (which ~fix the incorrect js).

Finally, I have an idea of how the saving methods should be fixed, too, but this will require more thorough testing than loading because of greater area of usage of those helpers and some browser-specific code. In the best scenario I'll get the working fix this week, but some post-testing will be required because of usage of those methods in some plugins and the need of browser-specific tests.

In conclusion, I'm quite happy with the results and enthusiastic to finish this fix of the core, the situation is much clearer than it was in the beginning, but some work still has to be done. There are some confusing things, though, including Okido's results with FF50 + TiddlyWiki for FireFox 2.0.1 (let me call it TiddlyFox 2.0.1 for simplicity; it worked for me in FF54 and for Alfonso for FF52.9, so that's disturbing) and less confusing with FF60 + SaveTiddlers 0.8 (well, this is a new thing to be tested separately).

***

Alfonso, I'll take a look at the extensions later (I don't think anybody should be left behind and everybody should have a chance to upgrade smoothly to the newer environments); I've almost finished the update for LessBackupsPlugin which allows not to overwrite yearly backup once it is created etc, but now I'm focused on fixing the core.

Best regards,
Yakov.

[Yakov]

Right, here's the new fix assembly. This one is supposed (with high level of confidense :) to work with both FF "native" saving and TiddlyFox (and also fix some other encoding issues). I'll be thankful if you test upgrading in this one too.

For now I've only tested the assembly in FF 14 – saving (with non-ASCII content), import (including non-ASCII content), creating backups, upgrading – all works fine both with and without TiddlyFox. I will test this also in IE (since the encoding changes may affect it too) and with some other I/O extensions (SaveAsPlugin, SnapshotPlugin, SharedTiddlersPlugin and may be others) because the conversion change will cover most of I/O functionality (well, saveFile and loadFile methods and some others).

Then I'll finalize the fix and commit it into the core. May be I'll even release 2.9.2 with almost only upgrading fixes to celebrate the progress since after it new releases will be reasonably easier to adapt.

Best regards,
Yakov.

[Alfonso Arciniega]

Hi Yakov,

Happy Holidays!

Just in case, I tested the "conversion scope fix.zip" on FF 3.6.28 and the upgrade test works fine.

Alfonso

[Yakov]

Hi Alfonso,

great, you too!

I've made some more tests and adjustments and finally made a pull request, but then found that download saving got affected, so I have to explore and fix it too before merging. Yeah, these encoding and I/O things are quite complicated, that's why they were not fixed in a consistent manner before. Pity that I haven't managed to make a release before the new year, but, well.. this work requires accuracy and research and hence time.

Best regards,
Yakov.