[BUG] Tortoise Diff silently not showing encoding changes as changes

[Struzyk Rafal]

We've recently run into an issue where one of our resource files got broken.

This was because one of the team members must have re-encoded an .rc file to UTF-8 and it went unnoticed.

In such a situation (comparing an ASCI file with UTF-8) TortoiseMerge reports the difference *only* if the files are identical other than the encoding change:

If there's any other difference the change is not highlighted in any way when looking at the changed lines.

Total Commander comparison (the difference is detected and explicitly visualized):

TortoiseMerge comparison - the line difference is not shown and not detected:

The reason is probably that TortoiseMerge chooses the "right" encoding to display each file and considers characters to be the same if they decode to the same character )int UTF-8?).

As least for this use case I think the expectation would be to compare the files in the same "base" encoding (so if comparing ASCII with UTF-8 during a merge I would prefer to compare them in ASCII by default with an option to "ignore encoding changes" similar to the "ignore whitespace changes" that would perform the comparison like it's being done today.

I have not found a workaround. It might be possible using the lower pane but we're not able to change the left view to ASCII.

Tight view can be switched to any encoding needed:

But the left view is locked:

[Stefan]

On Friday, May 7, 2021 at 4:55:39 PM UTC+2 Rafal S wrote:

We've recently run into an issue where one of our resource files got broken.

This was because one of the team members must have re-encoded an .rc file to UTF-8 and it went unnoticed.

In such a situation (comparing an ASCI file with UTF-8) TortoiseMerge reports the difference *only* if the files are identical other than the encoding change:

If there's any other difference the change is not highlighted in any way when looking at the changed lines.

Total Commander comparison (the difference is detected and explicitly visualized):

that only works for changes ANSI<->UTF8. If the change is e.g. from/to utf-16, then every single line would be different and therefore the diff would be completely useless. 

TortoiseMerge comparison - the line difference is not shown and not detected:

As your screenshot shows: it is shown and it is detected. You just have to look.

 

The reason is probably that TortoiseMerge chooses the "right" encoding to display each file and considers characters to be the same if they decode to the same character )int UTF-8?).

As least for this use case I think the expectation would be to compare the files in the same "base" encoding (so if comparing ASCII with UTF-8 during a merge I would prefer to compare them in ASCII by default with an option to "ignore encoding changes" similar to the "ignore whitespace changes" that would perform the comparison like it's being done today.

I have not found a workaround. It might be possible using the lower pane but we're not able to change the left view to ASCII.

But the left view is locked:

left view is unlocked once you click the left view and then the "enable edit" button (top left). 

[Rafal S]

Thanks - unfortunately even after switching the left view to ascii this line is displayed without changes (I suspect the file is re-encoded to ascii "correctly"?).

Attached are to files showing the problem - we can easily compare them with Tortoise-SVN->Diff explorer context menu to reproduce the problem - please let me know if there is some other workaround that would allow us to "spot the difference".

ps. When comparing with TotalCommander the "encoding" comparison options are explicitly exposed offering the ability to perform the desired comparison (ie. either after conversion or in raw form). I think the lower pane would be a great to offer a similar feature in TortoiseMerge/Diff (ie. what's probably missing/causes trouble is the translation from UTF-8 to ASCII is probably done "nicely" vs just showing us the raw characters?)

Thanks