spatialite_gui is unable to open file whose path contains characters other than ANSI codes

[Paul]

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes. In the meanwhile, Firefox Add-on, SQLite Manager, can successfully open the sqlite3 database files. I am on Windows 10, which shall be using Unicode.

I have a little Windows application which is having the same trouble if it is compiled with ANSI (or MBCS) code. However when it is compiled with UNICODE, it have no problem to open the same sqlite3 database files.

Could the problem spatialite_gui 2.0.0-devel is having result from the file path being not properly encoded?

Thanks a lot for your attention.

[Paul]

If I removed those non-ANSI characters along the file path, spatialite_gui can open the file without any issue. It is just kind of inconvenience.

[mj10777]

Could you send a zip files which has an 'offending' path included in it together with the file (database)?

Does this happen if the database file-name also have 'offending' characters?

- this would also be interesting to test

- I cannot test this for windows, but can for linux and android

-- which could help in narrowing down the problem

(just saw your second note)

I know that wxWindows has problems with unicode in some versions (wxWidgets version 3.0.2)

- it brings a warning the first time that be answered with 'ignore' or 'continue'

-- so the wxWindows version you are using would also be helpfull

Does this happen when you open the file directly with spatialite (sqlite3) with the path?

- spatialite some_offending_path/some_offending_database.db

Mark

[mj10777]

On Wednesday, 14 September 2016 06:51:17 UTC+2, mj10777 wrote:

On Wednesday, 14 September 2016 06:35:32 UTC+2, Paul wrote:

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes. In the meanwhile, Firefox Add-on, SQLite Manager, can successfully open the sqlite3 database files. I am on Windows 10, which shall be using Unicode.

I have a little Windows application which is having the same trouble if it is compiled with ANSI (or MBCS) code. However when it is compiled with UNICODE, it have no problem to open the same sqlite3 database files.

Could the problem spatialite_gui 2.0.0-devel is having result from the file path being not properly encoded?

Could you send a zip files which has an 'offending' path included in it together with the file (database)?

Does this happen if the database file-name also have 'offending' characters?

- this would also be interesting to test

- I cannot test this for windows, but can for linux and android

-- which could help in narrowing down the problem

(just saw your second note)

I know that wxWindows has problems with unicode in some versions (wxWidgets version 3.0.2)

- it brings a warning the first time that be answered with 'ignore' or 'continue'

-- so the wxWindows version you are using would also be helpfull

You can see that with spatialite_gui with

- Files/About menu 

[mj10777]

On Wednesday, 14 September 2016 06:51:17 UTC+2, mj10777 wrote:

On Wednesday, 14 September 2016 06:35:32 UTC+2, Paul wrote:

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes. In the meanwhile, Firefox Add-on, SQLite Manager, can successfully open the sqlite3 database files. I am on Windows 10, which shall be using Unicode.

I have a little Windows application which is having the same trouble if it is compiled with ANSI (or MBCS) code. However when it is compiled with UNICODE, it have no problem to open the same sqlite3 database files.

Could the problem spatialite_gui 2.0.0-devel is having result from the file path being not properly encoded?

Could you send a zip files which has an 'offending' path included in it together with the file (database)?

Does this happen if the database file-name also have 'offending' characters?

- this would also be interesting to test

- I cannot test this for windows, but can for linux and android

-- which could help in narrowing down the problem

(just saw your second note)

I know that wxWindows has problems with unicode in some versions (wxWidgets version 3.0.2)

- it brings a warning the first time that be answered with 'ignore' or 'continue'

You can also try the following to see if this a wxWidget problem

- start spatialite_gui 

-- type á into the sql editor

--> do you receive a warning? (An assert error)

If you do, click on 'Do not show this again' and continue

- retype the á

-- there should then be no warning

Then try to open the file with the  'offending' characters

- if it opens correctly, then this is most likely a wxWidgets problem

[Goyo]

El miércoles, 14 de septiembre de 2016, 6:35:32 (UTC+2), Paul escribió:

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes.

I can replicate this in Windows 7. Just rename any working database and put a "ñ" anywhere in the name and try to connect to it from spatialite_gui, after clicking "Open" in the file dialog I get an error message:

-------------------------------------------
Unable to connect:
c:\path\to\dbñ.sqlite

Not existing file or forbidden access
-------------------------------------------

And the program does not connect to the database.

spatialite_gui 2.0.0-devel
wxWidgets version 3.0.2
Spatialite version 4.3.0a

Did not try with other versions, tools or operating systems.

Goyo

[mj10777]

On Thursday, 15 September 2016 08:46:09 UTC+2, Goyo wrote:

El miércoles, 14 de septiembre de 2016, 6:35:32 (UTC+2), Paul escribió:

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes.

I can replicate this in Windows 7. Just rename any working database and put a "ñ" anywhere in the name and try to connect to it from spatialite_gui, after clicking "Open" in the file dialog I get an error message:

-------------------------------------------
Unable to connect:
c:\path\to\dbñ.sqlite

and what happens if you call it directly with sqlite3 of spatialite?

sqlite3 test_ogr_kml/dbñ.sqlite

SQLite version 3.8.5 2014-06-04 14:06:34

Enter ".help" for usage hints.

sqlite> .tables

ElementaryGeometries                                   

HK_berlin_3068                                         

SpatialIndex                                           

berlin_admin                                           

berlin_admin_changed_belongs  
... many, many, more

virts_geometry_columns_statistics                      

spatialite> 

both work correctly for me in linux.

Also 'spatialite_gui test_ogr_kml/dbñ.sqlite'

- works correctly.

I would be important to know if the non-gui programs have the same problem or not.

Mark

[mj10777]

On Thursday, 15 September 2016 08:59:59 UTC+2, mj10777 wrote:

On Thursday, 15 September 2016 08:46:09 UTC+2, Goyo wrote:

El miércoles, 14 de septiembre de 2016, 6:35:32 (UTC+2), Paul escribió:

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes.

I can replicate this in Windows 7. Just rename any working database and put a "ñ" anywhere in the name and try to connect to it from spatialite_gui, after clicking "Open" in the file dialog I get an error message:

-------------------------------------------
Unable to connect:
c:\path\to\dbñ.sqlite

and what happens if you call it directly with sqlite3 of spatialite?

sqlite3 test_ogr_kml/dbñ.sqlite

SQLite version 3.8.5 2014-06-04 14:06:34

Enter ".help" for usage hints.

sqlite> .tables

ElementaryGeometries                                   

HK_berlin_3068                                         

SpatialIndex                                           

berlin_admin                                           

berlin_admin_changed_belongs  
... many, many, more

virts_geometry_columns_statistics                      

spatialite> 

both work correctly for me in linux.

Also 'spatialite_gui test_ogr_kml/dbñ.sqlite'

- also using the menu Open, works correctly 

[mj10777]

On Thursday, 15 September 2016 08:46:09 UTC+2, Goyo wrote:

El miércoles, 14 de septiembre de 2016, 6:35:32 (UTC+2), Paul escribió:

I have seen that spatialite_gui 2.0.0-devel with either Spatialite 4.3.0a or Spatialite 4.4.0-RC was unable to open sqlite3 database files whose full path contains characters, such as Chinese, other than ANSI codes.

I can replicate this in Windows 7. Just rename any working database and put a "ñ" anywhere in the name and try to connect to it from spatialite_gui, after clicking "Open" in the file dialog I get an error message:

-------------------------------------------
Unable to connect:
c:\path\to\dbñ.sqlite

Not existing file or forbidden access
-------------------------------------------

And the program does not connect to the database.

the next question (after a quick look at MS-Documentation)

- how is the Hard-Disk formatted?

-- NTFS support Unicode (which hopefull UTF-8), but FAT12, FAT16 and FAT32 do not (use the OEM character set) 

[Goyo]

El jueves, 15 de septiembre de 2016, 8:59:59 (UTC+2), mj10777 escribió:

 

and what happens if you call it directly with sqlite3 of spatialite?

Actually sqlite3 also has their own problems with this in Windows 7: If a file name with non-ASCII characters is provided in the command line, a new file with the non-ASCII characters replaced by "?" is created and open:

d:\users\goyo\test>dir
 El volumen de la unidad D es Data
 El número de serie del volumen es: F44A-C483

 Directorio de d:\users\goyo\test

20/09/2016  09:53    <DIR>          .
20/09/2016  09:53    <DIR>          ..
29/10/2009  01:55         2.128.896 regions.sqlite
12/09/2016  21:12           682.496 sqlite3.exe
               2 archivos      2.811.392 bytes
               2 dirs  13.309.505.536 bytes libres

d:\users\goyo\test>copy regions.sqlite regioñs.sqlite
        1 archivo(s) copiado(s).

d:\users\goyo\test>dir
 El volumen de la unidad D es Data
 El número de serie del volumen es: F44A-C483

 Directorio de d:\users\goyo\test

20/09/2016  09:53    <DIR>          .
20/09/2016  09:53    <DIR>          ..
29/10/2009  01:55         2.128.896 regions.sqlite
29/10/2009  01:55         2.128.896 regioñs.sqlite
12/09/2016  21:12           682.496 sqlite3.exe
               3 archivos      4.940.288 bytes
               2 dirs  13.307.310.080 bytes libres

d:\users\goyo\test>sqlite3 regions.sqlite
SQLite version 3.14.2 2016-09-12 18:50:49

Enter ".help" for usage hints.
sqlite> .tables

Aeroporti               layer_sub_classes       spatial_ref_sys
geom_cols_ref_sys       layer_table_layout      symbol_bitmaps
geometry_columns        pattern_bitmaps         views_geometry_columns
geometry_columns_auth   project_defs            virts_geometry_columns
layer_params            raster_pyramids
layer_statistics        reg2008_s
sqlite> .quit

d:\users\goyo\test>sqlite3 regioñs.sqlite
SQLite version 3.14.2 2016-09-12 18:50:49

Enter ".help" for usage hints.
sqlite> .

tables
sqlite> .quit

d:\users\goyo\test>dir
 El volumen de la unidad D es Data
 El número de serie del volumen es: F44A-C483

 Directorio de d:\users\goyo\test

20/09/2016  09:54    <DIR>          .
20/09/2016  09:54    <DIR>          ..
29/10/2009  01:55         2.128.896 regions.sqlite
29/10/2009  01:55         2.128.896 regioñs.sqlite
20/09/2016  09:54                 0 regio?s.sqlite
12/09/2016  21:12           682.496 sqlite3.exe
               4 archivos      4.940.288 bytes
               2 dirs  13.307.506.688 bytes libres

But it works if the file name is provided at the sqlite3 command prompt:

d:\users\goyo\test>sqlite3
SQLite version 3.14.2 2016-09-12 18:50:49

Enter ".help" for usage hints.

Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open regioñs.sqlite
sqlite> .tables
Aeroporti               layer_sub_classes       spatial_ref_sys
geom_cols_ref_sys       layer_table_layout      symbol_bitmaps
geometry_columns        pattern_bitmaps         views_geometry_columns
geometry_columns_auth   project_defs            virts_geometry_columns
layer_params            raster_pyramids
layer_statistics        reg2008_s
sqlite> .quit

spatialite_gui *does* work when the file name is passed in the command line:

d:\users\goyo\test>spatialite_gui regioñs.sqlite

But it fails when the file name is provided inside the program (Open menu) as I said before.

The file system is NTFS.

Regards
Goyo

[mj10777]

On Tuesday, 20 September 2016 10:25:14 UTC+2, Goyo wrote:

El jueves, 15 de septiembre de 2016, 8:59:59 (UTC+2), mj10777 escribió:

 

and what happens if you call it directly with sqlite3 of spatialite?

Actually sqlite3 also has their own problems with this in Windows 7: If a file name with non-ASCII characters is provided in the command line, a new file with the non-ASCII characters replaced by "?" is created and open:

Could you try this with spatialite?

spatialite regioñs.sqlite

I expect the same result, but would be nice to know.

Here we can look to see how the command arguments are treated

- so this is something that can be looked at

The file system is NTFS.

Do you have a FAT file-system

- if yes, please try the same combinations

Mark 

Regards
Goyo

[Goyo]

El martes, 20 de septiembre de 2016, 10:36:46 (UTC+2), mj10777 escribió:

Could you try this with spatialite?

spatialite regioñs.sqlite

I expect the same result, but would be nice to know.

Not sure what you mean, I thought the spatialite cli was abandoned and it was reccommended using sqlite3 and then loading the spatialite extension.

spatialite_gui works well that way as I said before, it only fails when the funny file name is selected in the open file dialog.

Do you have a FAT file-system

- if yes, please try the same combinations

Will try tomorrow with a FAT pendrive.

Goyo

[mj10777]

On Tuesday, 20 September 2016 10:55:05 UTC+2, Goyo wrote:

El martes, 20 de septiembre de 2016, 10:36:46 (UTC+2), mj10777 escribió:

Could you try this with spatialite?

spatialite regioñs.sqlite

I expect the same result, but would be nice to know.

Not sure what you mean, I thought the spatialite cli was abandoned and it was reccommended using sqlite3 and then loading the spatialite extension.

- it has not been abandoned, but will not be extended with spatial-specific macros

[Goyo]

El martes, 20 de septiembre de 2016, 10:58:41 (UTC+2), mj10777 escribió:

Not sure what you mean, I thought the spatialite cli was abandoned and it was reccommended using sqlite3 and then loading the spatialite extension.

- it has not been abandoned, but will not be extended with spatial-specific macros

Is there an x86 windows binary available? I could not find it.

Goyo

[a.fu...@lqt.it]

On Tue, 20 Sep 2016 12:04:54 -0700 (PDT), Goyo wrote:
> Is there an x86 windows binary available? I could not find it.
>

http://www.gaia-gis.it/gaia-sins/windows-bin-x86-test/spatialite-4.4.0-RC0-win-x86.7z
or
http://www.gaia-gis.it/gaia-sins/windows-bin-amd64-test/spatialite-4.4.0-RC0-win-amd64.7z

please look at the left bottom of this HTML page:
http://www.gaia-gis.it/gaia-sins/
you'll find a "MS Windows Binaries" panel.

bye Sandro

[Goyo]

Sorry this took me so long.

- Using a FAT or NTFS filesystem did not make any difference.
- Using spatialite 4.4.0 RC0 instead of 4.3.0a dis not make any difference either.
- .restore behaved the same in sqlite3 and spatialite (spatialite does not have .open)
- spatialite behaved as similar to sqlite3 as I could expect

In the following I only use dir when the file regio?s.sqlite is created and then I delete it before running the next command.

E:\test>sqlite3 regioñs.sqlite

SQLite version 3.14.2 2016-09-12 18:50:49
Enter ".help" for usage hints.
sqlite> .tables
sqlite> .

quit

E:\test>dir
 El volumen de la unidad E no tiene etiqueta.
 El número de serie del volumen es: EB6C-61CB

 Directorio de E:\test

27/09/2016  08:32    <DIR>          .
27/09/2016  08:32    <DIR>          ..

29/10/2009  01:55         2.128.896 regions.sqlite
29/10/2009  01:55         2.128.896 regioñs.sqlite

07/09/2015  17:18        32.584.206 spatialite_gui.exe

12/09/2016  21:12           682.496 sqlite3.exe

30/09/2016  12:10        13.012.494 spatialite.exe
30/09/2016  12:13                 0 regio?s.sqlite
               6 archivos     50.536.988 bytes

E:\test>sqlite3

SQLite version 3.14.2 2016-09-12 18:50:49
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open regioñs.sqlite
sqlite> .tables
Aeroporti

              layer_sub_classes       spatial_ref_sys

geom_cols_ref_sys       layer_table_layout      symbol_bitmaps
geometry_columns        pattern_bitmaps         views_geometry_columns
geometry_columns_auth   project_defs            virts_geometry_columns
layer_params            raster_pyramids
layer_statistics        reg2008_s

E:\test>spatialite regioñs.sqlite
SpatiaLite version ..: 4.3.0a   Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct DBF access]
        - 'VirtualXL'           [direct XLS access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualSpatialIndex' [R*Tree metahandler]
        - 'VirtualElementary'   [ElemGeoms metahandler]
        - 'VirtualXPath'        [XML Path Language - XPath]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'VirtualGPKG' [OGC GeoPackage interoperability]
        - 'VirtualBBox'         [BoundingBox tables]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.9.1, 04 March 2015
GEOS version ........: 3.5.0-CAPI-1.9.0 r4084
LWGEOM version ......: 2.1.7
TARGET CPU ..........: mingw32
the SPATIAL_REF_SYS table already contains some row(s)
SQLite version ......: 3.8.11.1
Enter ".help" for instructions
SQLite version 3.8.11.1 2015-07-29 20:00:57
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
spatialite> .tables
ElementaryGeometries                vector_layers
SpatialIndex                        vector_layers_auth
geom_cols_ref_sys                   vector_layers_field_infos
geometry_columns                    vector_layers_statistics
geometry_columns_auth               views_geometry_columns
geometry_columns_field_infos        views_geometry_columns_auth
geometry_columns_statistics         views_geometry_columns_field_infos
geometry_columns_time               views_geometry_columns_statistics
spatial_ref_sys                     virts_geometry_columns
spatial_ref_sys_all                 virts_geometry_columns_auth
spatial_ref_sys_aux                 virts_geometry_columns_field_infos
spatialite_history                  virts_geometry_columns_statistics
sql_statements_log
spatialite> .quit

E:\test>dir
 El volumen de la unidad E no tiene etiqueta.
 El número de serie del volumen es: EB6C-61CB

 Directorio de E:\test

27/09/2016  08:32    <DIR>          .
27/09/2016  08:32    <DIR>          ..

29/10/2009  01:55         2.128.896 regions.sqlite
29/10/2009  01:55         2.128.896 regioñs.sqlite

07/09/2015  17:18        32.584.206 spatialite_gui.exe

12/09/2016  21:12           682.496 sqlite3.exe

30/09/2016  12:10        13.012.494 spatialite.exe
30/09/2016  12:16         5.808.128 regio?s.sqlite
               6 archivos     56.345.116 bytes
               2 dirs   2.536.984.576 bytes libres

E:\test>spatialite
SpatiaLite version ..: 4.3.0a   Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct DBF access]
        - 'VirtualXL'           [direct XLS access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualSpatialIndex' [R*Tree metahandler]
        - 'VirtualElementary'   [ElemGeoms metahandler]
        - 'VirtualXPath'        [XML Path Language - XPath]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'VirtualGPKG' [OGC GeoPackage interoperability]
        - 'VirtualBBox'         [BoundingBox tables]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.9.1, 04 March 2015
GEOS version ........: 3.5.0-CAPI-1.9.0 r4084
LWGEOM version ......: 2.1.7
TARGET CPU ..........: mingw32
the SPATIAL_REF_SYS table already contains some row(s)
SQLite version ......: 3.8.11.1
Enter ".help" for instructions
SQLite version 3.8.11.1 2015-07-29 20:00:57
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
spatialite> .restore regioñs.sqlite
spatialite> .tables
spatialite> .quit

E:\test>dir
 El volumen de la unidad E no tiene etiqueta.
 El número de serie del volumen es: EB6C-61CB

 Directorio de E:\test

27/09/2016  08:32    <DIR>          .
27/09/2016  08:32    <DIR>          ..

29/10/2009  01:55         2.128.896 regions.sqlite
29/10/2009  01:55         2.128.896 regioñs.sqlite

07/09/2015  17:18        32.584.206 spatialite_gui.exe

12/09/2016  21:12           682.496 sqlite3.exe

30/09/2016  12:10        13.012.494 spatialite.exe
30/09/2016  12:24                 0 regio?s.sqlite
               6 archivos     50.536.988 bytes
               2 dirs   2.542.792.704 bytes libres

E:\test>spatialite_gui regioñs.sqlite

REM Just works, but it does not if I use the open file dialog.

[mj10777]

This looks as if WxWidget's is getting confused on windows systems

- not sure how it resolves the problem internally

From the MS documentation I saw (which may have changed with different Windows Version)

- UTF-8 should be used on NTFS Hard-Disks

- ASCII on FAT* Hard-Disks

So this is probably not a problem that can be solved by any changes in spatialite_gui.

sorry.

Mark

[Jukka Rahkonen]

I believe that this deals with the same issue:
 
Try to import shapefile c:\data\järvi.shp with spatialite-gui.  You can select the .shp with the file browser, but import fails with error
"cannot open shapefile cause: unable to open c:\data\järvi.shx for reading: No such file or directory"

-Jukka Rahkonen-

[a.fu...@lqt.it]

reality check
-------------
I've prepared a small testcase containing several
shapefiles and text files based on "exotic" paths
(chinese, japanese, hebrew, greek, cyrillic and so on);
you'll find the sample attached to this post.

objective findings: spatialite_gui can correctly open any
file identified by an "exotic" path.
at least, this is true if you use a sane minded operating
system based on the UTF-8 universal charset (Linux, Android,
Mac OS X and many others *nix derivatives).

unhappily not all operating systems are well designed in
a clear and rational way; for historical reasons Windows is
based on a chaotic mix of many different charsets, and just
to simplify things command line tools and desktop (GUI) tools
behave in different ways.

please note: the so called ANSI is just a misnomer vaguely
identifying many different 8-bit codepages; it usually
corresponds to CP1252 in Europe and US, but about twenty
alternatives exist.

in subsequent times (after releasing Windows 95) Microsoft
switched from this ANSI botch to UNICODE, and so was
finally able to properly support an universal encoding
not being limited to just a single specific cultural
context.
unhappily (once again) MS UNICODE does not corresponds
to the universally spread UTF-8; it actually corresponds
to UTF-16LE (little endian); and this introduced many
further issues.

UTF-8 was carefully designed so to never break compatibility
with the already existing and well established C runtime API
supporting text strings and paths manipulation (strcpy,
strlen, strcmp, fopen and alike)
UTF-16 causes a lot of really unpleasing API incompatibilities,
and necessarily requires using its own non-standard and
MS-specific "wide character" API (wcscpy, wcslen, wcscmp,
_wfopen etc).

if you feel a little bit confused, that's not all: the
NTFS filesystem stores all paths using UNICODE (aka
UTF16-LE), but the FAT filesystem still adopts ANSI.
desktop (GUI) applications are expected to always use
UNICODE, but command line tools are expected to use
ANSI ... OMG, what a messy chaos !!!


conclusions:
------------
1. spatialite CLI and spatialite_gui already support
"exotic" paths on Linux and friends.
2. spatialite_gui on Windows (any version) can just
support ASCII paths.
3. patching spatialite_gui so to correctly support
"exotic" paths on Windows too is surely possible,
but will certainly require a long time and a very
careful full refactoring of all internal
functions reading or writing from an external
file.

bye Sandro

[a.fu...@lqt.it]

just a short follow-up adding more fun :-D

I've just discovered that if you attempt to unzip the "exotic"
sample by using the standard Windows ZIP tools all paths
will be irremediably corrupted.
anyway, if you use a wisest tool such as 7-zip then the
zipfile will be decompressed and a folder containing
the correct paths will be created.

it seems rather clear that Windows has many serious
issues related to "exotic" paths, and they don't
affect spatialite_gui alone, they plague many other
system components in a more general way.

bye Sandro

[mj10777]

To continue the never ending story...

Another major problem is the 'translation' / converting to and from 'utf-8' 

- in many cases it is not possible to determine if a conversion is needed, since the character-set being used is not stored

Converting a utf-8 string that is already a utf-8 string will produce rubbish.

A typical sample:

The TIFF Specification (6.0) states that a TIFF-TAG is:

8-bit byte that contains a 7-bit ASCII code

Many software product enforce this (outdated) specification, really believing that TIGGTAG's containing

- documentation

- copyright information

will NEVER contain a non-latin character

Sample: GIMP:

TIFF image Message

The TIFF format only supports comments in

7bit ASCII encoding. No comment is saved.

Other software is/are more realistic  ...

Gdal will accept a proper UTF-8 string:

gdal_edit.py -mo 'TIFFTAG_COPYRIGHT=Senatsverwaltung für Stadtentwicklung und Umwelt Berlin' 2010.Berlin.5000.25833.FAX4_MINISWHITE.600.tif

and will show it properly with gdalinfo:

Metadata:

  AREA_OR_POINT=Area

  TIFFTAG_ARTIST=mj10777.de.eu

  TIFFTAG_COPYRIGHT=2010 Senatsverwaltung für Stadtentwicklung und Umwelt

  TIFFTAG_DATETIME=2010

  TIFFTAG_DOCUMENTNAME=2010 Berlin K5 1:5000

  TIFFTAG_IMAGEDESCRIPTION=2010 Landeskartenwerk Karte von Berlin 1:5000 (K5)

In QGIS (in linux) this will be 'converted' to UTF-8 

- I assume that something in QT determines: being a TIFFTAG it must therefore be a 7-bit ASCII code because the specification says so

TIFFTAG_COPYRIGHT=2010 Senatsverwaltung für Stadtentwicklung und Umwelt

This is a no-win situation.

---

For the Windows world this means

- that this matter will probably never be totally resolved

Microsoft has always been reluctant to break older applications

- Windows 3.* and DOS applications were required to work on Windows NT (XP etc.)

For this reason, the Parent/Child processes used in unix (now linux), was not adapted.

That is why viruses thrive under Windows, but are practicality non existent on unix/linux based systems.

Mark

bye Sandro