[Patch] Stitching projects with non-ascii path fails

[Guy Rutenberg]

Hi,

Using Hugin 2017.0 (and prior versions), if one tries to stitch a project where there are non-ascii characters in the project's file path the stitching fails. This is related to bug https://bugs.launchpad.net/hugin/+bug/678808 and its duplicates.
After some debugging it turns out that the culprit is the call to hugin_stich_project using wxExecute. wxExcute does a poor (buggy) handling of converting the internal Unicode representation to UTF-8 and drops any arguments containing non-ascii characters. The attached bug works around it, and solves the issue.

I've tested it on Debian Stretch. It should probably be tested on other platforms as well, or alternatively, I can change the patch to only affect wxGTK.

Thanks,
Guy
https://www.guyrutenberg.com/

[T. Modes]

Am Samstag, 23. September 2017 18:20:47 UTC+2 schrieb Guy Rutenberg:

Hi,

Using Hugin 2017.0 (and prior versions), if one tries to stitch a project where there are non-ascii characters in the project's file path the stitching fails.

Sorry, but I can't reproduce the issue. It works fine here also with non-ascii characters (tested on Fedora).

This is related to bug https://bugs.launchpad.net/hugin/+bug/678808 and its duplicates.

Sorry, but this bug is Windows only and has nothing to do with wxExecute.

After some debugging it turns out that the culprit is the call to hugin_stich_project using wxExecute. wxExcute does a poor (buggy) handling of converting the internal Unicode representation to UTF-8 and drops any arguments containing non-ascii characters. The attached bug works around it, and solves the issue.

If you think that this is a wxWidgets bug, then it should be reported and fixed upstreams in wxWidgets.

But until now nobody else on Linux reported this bug so far. So this is really strange and maybe related to some special configuration on your side.

You have also only fixed the call to hugin_stitch_project but left the call of PTBatcherGUI, which is also called by wxExecute in a similar way.

[Guy Rutenberg]

Hi,

Thanks for replying.

Sorry, but I can't reproduce the issue. It works fine here also with non-ascii characters (tested on Fedora).

I'm using Debian 9, with the en_IL.utf8 locale. The steps I've taken to reproduce the bug are:
1. Start a new project where the pto is in some non-ascii path and set the output prefix to be in the same directory (with non-acii name).
2. Attempt to stitch.
3. hugin_stitch_project is executed and fails. I've attached the log it shows.

If you strace hugin using an ascii path

$ strace -s 1000 -f -t -v -e trace=process -o huing_good.strace hugin /tmp/hugin/abcd/IMG_0421\ -\ IMG_0421.pto

you can see the execve syscall that executes hugin_stitch_project:

execve("/usr/bin/hugin_stitch_project", ["hugin_stitch_project", "--delete", "-o", "/tmp/hugin/abcd/IMG_0421 - IMG_0421", "/tmp/huginpto_WrUC2G"],...

 However, doing the same but with a non-ascii path

strace -s 1000 -f -t -v -e trace=process -o huing.strace hugin /tmp/hugin/אבגד/IMG_0421\ -\ IMG_0421.pto

results in the following:

execve("/usr/bin/hugin_stitch_project", ["hugin_stitch_project", "--delete", "-o", "", "/tmp/huginpto_c4sLVF"],...

The empty argument to "-o" is than concatenated later on in the call to 'nona' which results in a missing parameter.

execve("/usr/bin/nona", ["/usr/bin/nona", "-v", "-z", "LZW", "-r", "ldr", "-m", "TIFF_m", "-o", "/tmp/huginpto_w9GzzF"],

This is the same call but when the bug does not manifest itself.

execve("/usr/bin/nona", ["/usr/bin/nona", "-v", "-z", "LZW", "-r", "ldr", "-m", "TIFF_m", "-o", "IMG_0421 - IMG_0421", "/tmp/huginpto_HoUdv9"],

As can be seen, the output prefix parameter is missing in the non-ascii case.

This is related to bug https://bugs.launchpad.net/hugin/+bug/678808 and its duplicates.

Sorry, but this bug is Windows only and has nothing to do with wxExecute.

You are right, my mistake. I'll open a new bug report, sorry for the inconvenience.

After some debugging it turns out that the culprit is the call to hugin_stich_project using wxExecute. wxExcute does a poor (buggy) handling of converting the internal Unicode representation to UTF-8 and drops any arguments containing non-ascii characters. The attached bug works around it, and solves the issue.

If you think that this is a wxWidgets bug, then it should be reported and fixed upstreams in wxWidgets.

I do intend to report it and try to fix it there as well. But I think there are two benefits for making a workaround in hugin:
1. You don't have to wait for a new wxWidgets release and then wait until the new version becomes the default in different distros and hugin is built against it.
2. It avoids unnecessary parsing and handling of the command line arguments. The current method works by constructing a single string containing all the arguments, requiring quoting filenames, and then letting wxExecute break it up to separate arguments using wxCmdLineParser::ConvertStringToArgs(), only to call the version of wxExecute that gets an array of arguments. By directly using that version of wxExecute, you both avoid unnecessary parsing and fix the non-ascii issue.

 

But until now nobody else on Linux reported this bug so far. So this is really strange and maybe related to some special configuration on your side.

It might indeed be the case, but does it make it not worthwhile to fix if the patch doesn't cause any other problems? :-)

 

You have also only fixed the call to hugin_stitch_project but left the call of PTBatcherGUI, which is also called by wxExecute in a similar way.

You are right, I should probably fix all other calls to wxExecute which get user input. I simply prefer to see if the proposed workaround will be accepted before start working on every instance of wxExecute requiring attention :-)
But I prefer to see if the intentions are to fix the issue before committing more time to make the patches :-)

Thanks for you feedback,
Guy

[T. Modes]

Am Sonntag, 24. September 2017 21:09:47 UTC+2 schrieb Guy Rutenberg:

I'm using Debian 9, with the en_IL.utf8 locale.

More interesting would be the wxWidgets version and maybe wxWidgets configuration (e.g. it is compiled as Unicode version or is Unicode disabled).

The steps I've taken to reproduce the bug are:

1. Start a new project where the pto is in some non-ascii path and set the output prefix to be in the same directory (with non-acii name).
2. Attempt to stitch.
3. hugin_stitch_project is executed and fails. I've attached the log it shows.

By default the stitching processor is PTBatcherGUI and not hugin_stitch_project. So you have already changed some settings.

But again it works fine here on Fedora. I can't reproduce it here.

Here the output from strace:

execve("/usr/local/bin/hugin_stitch_project", ["hugin_stitch_project", "--delete", "-o", "/home/xxx/\320\257\320...<snip>/pano", "/tmp/huginpto_v9oFXG"],

You see the non-ascii name is correctly passed to hugin_stitch_project.

So this is not a principal bug, it seems to depend on a special configuration or an older wxWidgets version.

PS: Information about wxWigets in Fedora
wxWidgets Library (wxGTK port)
Version 3.0.2 (Unicode: wchar_t, debug level: 1),
compiled at Oct  5 2016 02:40:16

Runtime version of toolkit used is 3.18.
Compile-time GTK+ version is 3.18.9.

[Bruno Postle]

On 24 September 2017 20:09:46 BST, Guy Rutenberg wrote:

> However, doing the same but with a non-ascii path
>strace -s 1000 -f -t -v -e trace=process -o huing.strace hugin
>/tmp/hugin
>/אבגד/IMG_0421\ -\ IMG_0421.pto

Could this be a problem with right-to-left text? Do you see the same error with non-ascii European characters?: øéßæç…

--
Bruno

[Gunter Königsmann]

I seem to remember that wxWidgets 3.1.x is much better in respect to uncode support. But as it is officially "development" it is not used everywhere...

...and the problem wxMaxima always has can be relevant here, too: what if wxWidgets uses Unicode, but the file system doesn't?

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/0FDA9D23-03D8-4350-BE1D-49F125BCC6B0%40postle.net.
For more options, visit https://groups.google.com/d/optout.

[Guy Rutenberg]

On Sunday, September 24, 2017 at 11:04:23 PM UTC+3, T. Modes wrote:

Am Sonntag, 24. September 2017 21:09:47 UTC+2 schrieb Guy Rutenberg:

I'm using Debian 9, with the en_IL.utf8 locale.

More interesting would be the wxWidgets version and maybe wxWidgets configuration (e.g. it is compiled as Unicode version or is Unicode disabled).

The steps I've taken to reproduce the bug are:

1. Start a new project where the pto is in some non-ascii path and set the output prefix to be in the same directory (with non-acii name).
2. Attempt to stitch.
3. hugin_stitch_project is executed and fails. I've attached the log it shows.

By default the stitching processor is PTBatcherGUI and not hugin_stitch_project. So you have already changed some settings.

But again it works fine here on Fedora. I can't reproduce it here.

Here the output from strace:

execve("/usr/local/bin/hugin_stitch_project", ["hugin_stitch_project", "--delete", "-o", "/home/xxx/\320\257\320...<snip>/pano", "/tmp/huginpto_v9oFXG"],

You see the non-ascii name is correctly passed to hugin_stitch_project.

I've removed my ~/.hugin file and tried again and the bug persists. With ascii-only characters

execve("/usr/bin/PTBatcherGUI", ["/usr/bin/PTBatcherGUI", "-b", "-v", "/tmp/hugin/abcd/IMG_0421 - IMG_0421.pto", "/tmp/hugin/abcd/IMG_0421 - IMG_0421"],...

But with non-ascii

execve("/usr/bin/PTBatcherGUI", ["/usr/bin/PTBatcherGUI", "-b", "-v", "", ""],...

and PTBatcherGUI gives "Could not open project file:/tmp/" when stitching.

 

So this is not a principal bug, it seems to depend on a special configuration or an older wxWidgets version.

PS: Information about wxWigets in Fedora
wxWidgets Library (wxGTK port)
Version 3.0.2 (Unicode: wchar_t, debug level: 1),
compiled at Oct  5 2016 02:40:16

Runtime version of toolkit used is 3.18.
Compile-time GTK+ version is 3.18.9.

This might be the problem. The configuration on my part is
wxWidgets: wxWidgets 3.0.2

wxWidgets Library (wxGTK port)
Version 3.0.2 (Unicode: wchar_t, debug level: 1),

Runtime version of toolkit used is 2.24.
Compile-time GTK+ version is 2.24.31.

These are the default libraries when compiling in Debian Stretch. I'll try to comile against more recent libraries to see if it persists.
Is there anybody else using Debian with the same libraries that doesn't have this bug? If so it would help to rule out libraries version as the problem.

On Monday, September 25, 2017 at 1:24:19 AM UTC+3, Bruno Postle wrote:

Could this be a problem with right-to-left text? Do you see the same error with non-ascii European characters?: øéßæç…

This causes the bug as well.

On Monday, September 25, 2017 at 7:20:22 AM UTC+3, Gunter Königsmann wrote:

I seem to remember that wxWidgets 3.1.x is much better in respect to uncode support. But as it is officially "development" it is not used everywhere...

...and the problem wxMaxima always has can be relevant here, too: what if wxWidgets uses Unicode, but the file system doesn't?

If it is indeed only solved in wxWidgets 3.1.x, I think it's better to apply a workaround, as no linux distro uses wxWidgets 3.1.0, and it would take years now for the bug to be actually fixed on users' machines.
Both wxWidgets and my file-system (ext4 with a utf8 locale) re unicode aware.

Thanks,
Guy

[T. Modes]

Am Dienstag, 26. September 2017 06:56:04 UTC+2 schrieb Guy Rutenberg:

But with non-ascii

execve("/usr/bin/PTBatcherGUI", ["/usr/bin/PTBatcherGUI", "-b", "-v", "", ""],...

and PTBatcherGUI gives "Could not open project file:/tmp/" when stitching.

Okay, I found the ticket: http://trac.wxwidgets.org/ticket/16206

It is indeed fixed upstreams in the 3.1 series, but not in there 3.0.x series

Maybe Fedora has applied this fix to their packages (but I could not find it. Maybe somebody else can check this).

The best way would be to ask the Debian packages to apply this patch to their wxGTK package (instead of fixing each wxExecute in Hugin. This would be a big work.)

[Guy Rutenberg]

Hi,

On 26 September 2017 at 18:28, T. Modes <Thomas...@gmx.de> wrote:

Okay, I found the ticket: http://trac.wxwidgets.org/ticket/16206

It is indeed fixed upstreams in the 3.1 series, but not in there 3.0.x series

Maybe Fedora has applied this fix to their packages (but I could not find it. Maybe somebody else can check this).

The best way would be to ask the Debian packages to apply this patch to their wxGTK package (instead of fixing each wxExecute in Hugin. This would be a big work.)

Thanks! I've sent a bug report to Debian, and I hope they would be able to apply this patch.

Thanks for your help,

Guy

 

[Guy Rutenberg]

Hi,

On 29 September 2017 at 17:07, Guy Rutenberg <guyrut...@gmail.com> wrote:

Hi,

On 26 September 2017 at 18:28, T. Modes <Thomas...@gmx.de> wrote:

Okay, I found the ticket: http://trac.wxwidgets.org/ticket/16206

It is indeed fixed upstreams in the 3.1 series, but not in there 3.0.x series

Maybe Fedora has applied this fix to their packages (but I could not find it. Maybe somebody else can check this).

The best way would be to ask the Debian packages to apply this patch to their wxGTK package (instead of fixing each wxExecute in Hugin. This would be a big work.)

Looking again at wxWidgets' ticket, I saw that hugin does not call wxSetlocale, which means that it doesn't properly inherit the locale from the environment. I've applied the following patch

--- a/src/hugin1/hugin/huginApp.cpp
+++ b/src/hugin1/hugin/huginApp.cpp
@@ -250,6 +250,7 @@ bool huginApp::OnInit()
     m_xrcPrefix = wxT(INSTALL_XRC_DIR);
     m_DataDir = wxT(INSTALL_DATA_DIR);
     locale.AddCatalogLookupPathPrefix(wxT(INSTALL_LOCALE_DIR));
+    wxSetlocale(LC_ALL, "");
 #endif
 
     if ( ! wxFile::Exists(m_xrcPrefix + wxT("/main_frame.xrc")) ) {

and it now wxExecute behaves properly. According to libc's docs this should be done in every localized program.

A C program inherits its locale environment variables when it starts up. This happens automatically. However, these variables do not automatically control the locale used by the library functions, because ISO C says that all programs start by default in the standard ‘C’ locale. To use the locales specified by the environment, you must call setlocale. Call it as follows:

setlocale (LC_ALL, "");

to select a locale based on the user choice of the appropriate environment variables.

(https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html)

Thanks! I've sent a bug report to Debian, and I hope they would be able to apply this patch.

Meanwhile, the Debian maintainer asked the wxWidgets team to backport their patch to the official 3.0 branch.

Thanks,

Guy

[T. Modes]

Hi,

Am Samstag, 30. September 2017 11:57:46 UTC+2 schrieb Guy Rutenberg:

Looking again at wxWidgets' ticket, I saw that hugin does not call wxSetlocale, which means that it doesn't properly inherit the locale from the environment. I've applied the following patch

Calling wxSetlocale on Windows have unwanted side effects. After it the numbers are not correctly formatted.

and it now wxExecute behaves properly. According to libc's docs this should be done in every localized program.

A C program inherits its locale environment variables when it starts up. This happens automatically. However, these variables do not automatically control the locale used by the library functions, because ISO C says that all programs start by default in the standard ‘C’ locale. To use the locales specified by the environment, you must call setlocale. Call it as follows:

setlocale (LC_ALL, "");

to select a locale based on the user choice of the appropriate environment variables.

Does also setlocale work, or have it to be wxSetLocale? (setlocale works here on Windows without the side effects of wxSetlocale).

[Guy Rutenberg]

Hi

On 30 September 2017 at 16:50, T. Modes <Thomas...@gmx.de> wrote:

Hi,

Calling wxSetlocale on Windows have unwanted side effects. After it the numbers are not correctly formatted.

 

Does also setlocale work, or have it to be wxSetLocale? (setlocale works here on Windows without the side effects of wxSetlocale).

`setlocale` works just as well. If I understand the #if's correctly, the line I've edited would not affect Windows at all (as __WXMSW__ is not defined).

Thanks,

Guy

[T. Modes]

Hi,

Am Mittwoch, 4. Oktober 2017 10:05:34 UTC+2 schrieb Guy Rutenberg:

Does also setlocale work, or have it to be wxSetLocale? (setlocale works here on Windows without the side effects of wxSetlocale).

`setlocale` works just as well. If I understand the #if's correctly, the line I've edited would not affect Windows at all (as __WXMSW__ is not defined).

I know. But there is already for Windows

#if defined _MSC_VER
    setlocale(LC_NUMERIC, "");
#endif

in the code. If one call is sufficient for both OS we can get rid of the conditional compilation and only maintain a single call.

I committed the fix to the repository.