Creating variable names in Devanagari

[Sharad Lele]

Hi folks, I am using QGIS and I want to create variable names in Devanagari (for instance जिला instead of District). If I create a new column in the attribute table and enter the column name in Devanagari using Microsoft Indic Language Tool on my computer, it seems to work (जिला is shown as the column name). But the moment I save the edits and reopen the attribute table, this gets truncated into something like जिल?. The last character is not really a question mark, something more complex. 

Any suggestions for making the column names appear and save properly in Devanagari? (The same would be true for Kannada or other scripts, I am sure).

Sharad

[Dilawar Singh]

The answer is not going to be simple.

Make sure that every time you save or open the file, the application uses encoding utf-8. Microsoft uses another weird encoding called utf-16 (https://en.wikipedia.org/wiki/UTF-16).  Stay away from utf-16 it if you want to copy-paste anything from the web or web-based tools.

Long answer: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

best,

   Dilawar 

---- On Mon, 04 Dec 2023 13:24:28 +0530 Sharad Lele <shara...@gmail.com> wrote ---

Hi folks, I am using QGIS and I want to create variable names in Devanagari (for instance जिला instead of District). If I create a new column in the attribute table and enter the column name in Devanagari using Microsoft Indic Language Tool on my computer, it seems to work (जिला is shown as the column name). But the moment I save the edits and reopen the attribute table, this gets truncated into something like जिल?. The last character is not really a question mark, something more complex. 

Any suggestions for making the column names appear and save properly in Devanagari? (The same would be true for Kannada or other scripts, I am sure).

Sharad

--

Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org

---

You received this message because you are subscribed to the Google Groups "datameet" group.

To unsubscribe from this group and stop receiving emails from it, send an email to datameet+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/b7074482-2d88-4762-8765-eb8b55bee3b0n%40googlegroups.com.

[Sharad Lele]

Thanks, Dilawar. The article was very interesting/useful. But for a simpleton like me, the question then is: How do I ensure that QGIS3 in particular is using UTF-8 encoding every time?

Sharad

[Sharad Lele]

Just to clarify/muddy the situation: I am able to enter long Devanagari text in the District/Jila column without any problems. It is the column/field name itself that I run into problems, which may have to do with the 8 characteri limit?

[Dilawar Singh]

I found this https://gis.stackexchange.com/questions/381308/how-to-change-qgis-3-default-encoding.

The file you are loading may have a different encoding that QGIS3. You have to verify this manually and make sure that QGIS3 and file encoding is the same. 

I am guessing the while QGIS3 is using utf-8, the file that you are loading may be utf-16 encoded. Not sure if there are automated tools that can convert UTF-16 files to UTF8 file while preserving unicode (https://superuser.com/questions/1786434/convert-utf-16-le-to-utf-8-in-windows-via-command-line).

best,

   Dilawar 

---- On Mon, 04 Dec 2023 14:05:01 +0530 Sharad Lele <shara...@gmail.com> wrote ---

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/29ffbab3-117a-4a67-8988-ae253db7e2cbn%40googlegroups.com.

[Sharad Lele]

The shapefile encoding is set to UTF-8.  But any edits to variable names only last as long as I don't save the file. Save and reopening results in going back to the same truncated/funny display as above 

[Ujaval Gandhi]

Use the geopackage format. It's a much better modern format without the limitations of shapefiles. The default encoding would be UTF-8 and it should work. Tested with a new geopackage layer and it is able to create column names and values in Devanagri script https://i.imgur.com/2prELf6.png

---

Ujaval Gandhi

Spatial Thoughts

www.spatialthoughts.com

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/bbcbdfe6-6f43-47a9-a84d-c1fd27f65af4n%40googlegroups.com.

[Sharad Lele]

Thanks, Ujaval. Yes saving as .geopkg works (saves the field names in full and displays properly upon reopening). This is the only solution that has worked so far.

But I am not sure this is going to work for us in this particular situation where we are then using the file in postGIS to display in a webGIS. Let me ask our webGIS person.

Sharad

[Nikhil VJ]

Hi Sharad,

If you have to import the data into a web-based backend, then your programmers will be most comfortable with: GeoJSON format.

Because : It's a simple text/json format which can even be opened as raw text, and so there are many different ways to load it in a program and use data from it.

And this will also support Devnagri columns (which become property keys here) same as geopackage.

Another thing to look out for: Ensure that the database being used by the backend has the same UTF-8 encoding set.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/7a1d4771-a606-451b-8f1d-3d706fcbb360n%40googlegroups.com.

[Sharad Lele (शरच्चंद्र लेले)]

Ok, thanks, Nikhil. Let me see what works at our backend.

Sharad

You received this message because you are subscribed to a topic in the Google Groups "datameet" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/datameet/HAOzs0y-lLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to datameet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuMp_SN3Oy9Xagycx8yqAUC9RFE_hoNiSkfR16FWfbmnbg%40mail.gmail.com.

[Sharad Lele]

Hi Nikhil, Ujaval, and others who responded:

Shiv, my colleague who handles our webgis, tried importing the geopkg version (which was storing and displaying Devanagari variable names in QGIS) into postGIS. He says "I used the normal pguploader, shape2posrgis, and ogr2ogr method". But reports that postGIS does not accept those variables. 

Any workarounds? 

Sharad

[Shiv Hastawala]

Hi Sharad

I'm sorry, I haven't ever worked with shapefiles or geopkg files so I'm afraid I don't have expertise in this area. I just had access to these data files, so I passed them on. 

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAPuYznFbmTB7s4J5uWxpALhEmH1r3otpjzNHp_9FkYbhGqip-w%40mail.gmail.com.