Unicode problem

[Oyuka Ganbaatar]

Hello everyone,

I have problem with unicode on PxWin. I created classification value set on the PxWin. 

And I added value to aggregation files. After clicked save btn, text was fine (Pic 1)

Then I opened this file again, in that time my text was like this -> ?????? (Pic 2)

And Also I displayed px file on my web page. It was same. (Pic 3)

Did anyone encounter this problem of Unicode?

Help me to solve this problem :)

[Hans Baumgartner]

Hi,

Sending some agg, vs and px- files would be helpful.

Check your vs and agg files with a text editor to see what they really contain.
Our vs and agg files are all in ANSI


Hans

 

Lähettäjä: pca...@googlegroups.com <pca...@googlegroups.com> Puolesta Oyuka Ganbaatar
Lähetetty: tiistai 6. elokuuta 2024 12.31
Vastaanottaja: pcaxis <pca...@googlegroups.com>
Aihe: Unicode problem

--
You received this message because you are subscribed to the Google Groups "pcaxis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pcaxis+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pcaxis/be2f22ac-5d43-49b9-84c2-1a92a50b6f54n%40googlegroups.com.

[Oyuka Ganbaatar]

 Thank you for your reply.

Here are my files.

My text - Өлгий сум

sender notified by Mail Track for Gmail

To view this discussion on the web visit https://groups.google.com/d/msgid/pcaxis/AS4PR10MB604029845B7E2ADA4390567791BF2%40AS4PR10MB6040.EURPRD10.PROD.OUTLOOK.COM.

[Hans Baumgartner]

Hi,

I know producing vs and agg files for the first time is a pain. The first time is usually not successful.

So if someone thinks this is easy … think again.

Please make your first vs and agg files for the variable gender (men, women) and sum these up in a agg file
to a total and list also men, women. It is simple but, boy it takes some doing the first time.

Troubleshooting these things is also a pain, but if Statistics Finland can help someone using PxWeb … why not.  

In px-file:
…
VALUES("Аймаг")="Завхан","Говь-Алтай","Баян-Өлгий","Ховд","Увс","Орхон","Өвөрхангай","Булган",

"Баянхонгор","Архангай","Хөвсгөл","Төв","Говьсүмбэр","Сэлэнгэ","Дорноговь","Дархан-Уул","Өмнөговь",

"Дундговь","Дорнод","Сүхбаатар","Хэнтий","Улаанбаатар";
DOMAIN("Аймаг")="aimag";                      (this links to the vs.-file)
…

In vs file:
…
[Domain]

1=Aimag

 

[Valuecode]

1=181

2=182

3=183

4=184

 

[Valuetext]

1=Zavkhan

2=Govi-Altai

3=Bayan-Ulgii

4=Khovd
…
This is not the same classification as in the variable in the px-file
his classification should be the same as in the px-file!

The agg-file
…
Aggreg]

Name=Zavkhan

Valueset=Aimag

1=181

2=182

3=183

 

[Aggtext]

1=????? ???

2=Govi-Altai

3=Bayan-Ulgii

 

[181]

 

[182]

 

[183]
…
this is  not a valid agg file …

So read the manual for PxWin to learn how to create a vs and the agg files for it:
https://www.scb.se/globalassets/vara-tjanster/px-programmen/tutorial-pxwin_1.2_v1.pdf
If I remember correctly Åsa has produced a tutorial for this.

Here is the specification for the vs and agg files:
https://pxweb2.stat.fi/aggregation.doc

We have over 10 000 vs and aggregation files in our PxWeb services.
I know of no other user with these kinds amounts of aggregation files.
I have tested PxWin and I know it is possible to produce a working vs and agg files using a px-file as the source for the classification.

This table is a heavy duty aggregation user:
https://pxweb2.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/statfin_vaerak_pxt_11re.px/
Aggregation files used in area and age …

 

hans.bau...@stat.fi

 

Lähettäjä: pca...@googlegroups.com <pca...@googlegroups.com> Puolesta Oyuka Ganbaatar

Lähetetty: tiistai 6. elokuuta 2024 13.49
Vastaanottaja: pca...@googlegroups.com
Aihe: Re: Unicode problem

To view this discussion on the web visit https://groups.google.com/d/msgid/pcaxis/CAOkSFrJNG3EgJFjMP3Upzmqz9Os8FZ-xtLWXDKJFwVpub890Bw%40mail.gmail.com.

[Oyuka Ganbaatar]

Hello

Thank you for your help. 

Agg files type is only V in Px Win. This option is not selected, just disabled. Is it okay ? You can see from pics.

[Hans Baumgartner]

Hi,

> Agg files type is only V in Px Win. This option is not selected, just disabled. Is it okay ? You can see from pics.

We use 100% Type=V
The others are related to regions and don’t suit our purposes.
This makes it possible to aggregate (sum) and/or just display values from the original classification.

 

 

Hans

 

Lähettäjä: pca...@googlegroups.com <pca...@googlegroups.com> Puolesta Oyuka Ganbaatar

Lähetetty: torstai 8. elokuuta 2024 12.09
Vastaanottaja: pcaxis <pca...@googlegroups.com>
Aihe: Re: Unicode problem

sender notified by Mail Track for Gmail

To view this discussion on the web visit https://groups.google.com/d/msgid/pcaxis/5d4a84a4-c2d1-4c56-af75-81f85e15d6d5n%40googlegroups.com.

[Hans Baumgartner]

Hi,

Just made a demo/test of agg px, vs an agg files.
https://pxweb2.stat.fi/agg_test_demo.zip

There is a minor bug in PxWin when you load a agg-file:
It only shows the last word in the name field, but all is in the field if you study it closer


but it is all there if you scroll the line …

Hans

Lähettäjä: pca...@googlegroups.com <pca...@googlegroups.com> Puolesta Oyuka Ganbaatar

Lähetetty: torstai 8. elokuuta 2024 12.09

Vastaanottaja: pcaxis <pca...@googlegroups.com>
Aihe: Re: Unicode problem

To view this discussion on the web visit https://groups.google.com/d/msgid/pcaxis/5d4a84a4-c2d1-4c56-af75-81f85e15d6d5n%40googlegroups.com.

[Oyuka Ganbaatar]

Hi, 

Thank you for sending the demo. I will see what I can do.

Oyunaa

sender notified by Mail Track for Gmail

You received this message because you are subscribed to a topic in the Google Groups "pcaxis" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pcaxis/voNfa1fk0oY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pcaxis+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pcaxis/AS4PR10MB6040F67EDFCC7F795E7BA71D91BA2%40AS4PR10MB6040.EURPRD10.PROD.OUTLOOK.COM.

[Petros Likidis]

Hi,

I looked at the code and it uses the System.Text.Encoding.Default both when reading and writing the file. 

My best guess is that you input in PxWin are in Unicode but that the system default encoding is probably some ANSI variant that can not properly save the character and replaces it with question marks when saving it to a file.

I would consider this a bug in PxWin. It should always save the vs and agg files in some Unicode friendly encoding like UTF8 or UTF16. Although I am not sure how the rest of the Px applications handles the encoding when reading vs and agg files to it may actually be a big thing to fix.

Have you manually tried to edit the vs and agg file in e.g. notepad (make sure to save it with UTF8) and try to open it in PxWin?

Kind regards

Petros