Non-UTF8 characters corrupting TW 2.7.0 // possible solution

[Chris]

Hi,

Please does anyone else have thoughts on this topic. The problem has been bugging me since around 2.6.5. The empty TW has had non UTF-8 character \xFF buried in the jQuery script. Saving with TiddlyFox transforms this to \xFD - but still the non-UTF-8 character causes problems anyplace where strict adherence to UTF-8 is mandated. In my case the impact is in trying to save on iOS as part of TWEdit - my objective-C code is unable to read the original file as a UTF-8 file in prep for inserting the STOREAREA and reserving.

You can see the specific file problem by running the empty 2.7.0 file through the w3c validator at http://validator.w3.org

I did initial experiments by brute force making the TiddlyFox output UTF-8 - it works but it is not really a good solution - it does not get to root cause.

Since then I dug in and traced the fileSave execution using FireBug in FireFox. The best I can see is that the problem occurs in "updateOriginal" where "original.substr(posDiv[1])" is appended on the back end of the STOREAREA.

var revised = original.substr(0,posDiv[0] + startSaveArea.length) + "\n" +

convertUnicodeToFileFormat(store.allTiddlersAsHtml()) + "\n" +

original.substr(posDiv[1]);

I've made the change below to my version of empty.html - ensuring that "original.substr(posDiv[1])" gets converted to UTF8 before appending. 

var revised = original.substr(0,posDiv[0] + startSaveArea.length) + "\n" +

convertUnicodeToFileFormat(store.allTiddlersAsHtml()) + "\n" +

convertUnicodeToFileFormat(original.substr(posDiv[1]));

It works - no need for my brute force version of TiddlyFox anymore - and I'm happy for now. But would really like to know if others have opinions and if others consider this is a solid solution.

Cheers - Chris.

function updateOriginal(original,posDiv,localPath)

{

if(!posDiv)

posDiv = locateStoreArea(original);

if(!posDiv) {

alert(config.messages.invalidFileError.format([localPath]));

return null;

}

var revised = original.substr(0,posDiv[0] + startSaveArea.length) + "\n" +

convertUnicodeToFileFormat(store.allTiddlersAsHtml()) + "\n" +

convertUnicodeToFileFormat(original.substr(posDiv[1]));

var newSiteTitle = convertUnicodeToFileFormat(getPageTitle()).htmlEncode();

revised = revised.replaceChunk("<title"+">","</title"+">"," " + newSiteTitle + " ");

revised = updateLanguageAttribute(revised);

revised = updateMarkupBlock(revised,"PRE-HEAD","MarkupPreHead");

revised = updateMarkupBlock(revised,"POST-HEAD","MarkupPostHead");

revised = updateMarkupBlock(revised,"PRE-BODY","MarkupPreBody");

revised = updateMarkupBlock(revised,"POST-SCRIPT","MarkupPostBody");

return revised;

}

[Chris]

Further to my original post - this version of the fix might be better:

function updateOriginal(original,posDiv,localPath)

{

if(!posDiv)

posDiv = locateStoreArea(original);

if(!posDiv) {

alert(config.messages.invalidFileError.format([localPath]));

return null;

}

var revised = original.substr(0,posDiv[0] + startSaveArea.length) + "\n" +

store.allTiddlersAsHtml() + "\n" +

original.substr(posDiv[1]);

var revised = convertUnicodeToFileFormat(revised);

var newSiteTitle = convertUnicodeToFileFormat(getPageTitle()).htmlEncode();

revised = revised.replaceChunk("<title"+">","</title"+">"," " + newSiteTitle + " ");

revised = updateLanguageAttribute(revised);

revised = updateMarkupBlock(revised,"PRE-HEAD","MarkupPreHead");

revised = updateMarkupBlock(revised,"POST-HEAD","MarkupPostHead");

revised = updateMarkupBlock(revised,"PRE-BODY","MarkupPreBody");

revised = updateMarkupBlock(revised,"POST-SCRIPT","MarkupPostBody");

return revised;

}

[PMario]

Hi chris,

Can you create a ticket?

at: https://github.com/TiddlyWiki/tiddlywiki/issues?state=open

-m