ExportToTextFile encoding

[J.REZZOUK]

Hello,

How can I manage encoding of text file generated by ExportToTextFile ?
Thanks
Jamal

[Gokhan Soydan]

Hello Jamal,

You cannot control the encoding of text file generated by
ExportToTextFile. Do you have issues with the encoding? What are you
trying to export? If you have a XML variable, I would suggest using
ExportToXML module as it keeps the encoding, because converting an XML
variable into a text variable might lose some of the original encoding.

Gokhan

> --
> You received this message because you are subscribed to the Google
> Group "TopBraid Suite Users", the topics of which include TopBraid
> Composer,
> TopBraid Live, TopBraid Ensemble, SPARQLMotion and SPIN.
> To post to this group, send email to
> topbrai...@googlegroups.com
> To unsubscribe from this group, send email to
> topbraid-user...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/topbraid-users?hl=en

[J.REZZOUK]

Hello Gokhan,

I am in the context of the following use-case : I am using SKOS-XL to manage a terminology and associated lexical information. I would like to export that as linguistic resources for a text-mining solution. But input format has some specific constraints : text-file+specific syntax + UTF-8 encoding. Today, when I open the text file it is ANSI encoded,
Do you think I have to call a function to encode string in UTF8 in sml:text property of the sml:ExportToTextFile module ?

Jamal

[Gokhan Soydan]

Jamal,

There is no direct way of specifying encoding in sml:ExportToTextFile or
in a SPIN/SPARQL function. However, you could set it up at the VM
arguments at TopBraid Composer startup.

Here are the instructions:
1) Exit TopBraid Composer, if it is running.
2) At the installation folder of TopBraid Composer, open "TopBraid
Composer.ini" in any text editor.
3) Add the following line at the very end of the file (don't forget the
"-" character):
-Dfile.encoding=UTF-8

4) Start TopBraid Composer.
5) Now all your files in your workspace are in UTF-8 encoding including
the ones that you create.

I couldn't find out how to check for or set the text encoding outside
TopBraid Composer. I believe the application that you mentioned would
figure out if it is UTF-8 or not once you do the export.

Let me know if this works.

Gokhan

[Gokhan Soydan]

Jamal,

I was just looking further and saw that after setting that VM argument,
and running the script, opening the generated text file in UltraEdit,
the file was shown to be in UTF-8 encoding.

Gokhan

[J.REZZOUK]

Thanks Gokhan,
It works fine :-)

Jamal

[J.REZZOUK]

Oups...now, with this modification, when I import an Excel spreadsheet (my understanding about Excel is that it is UTF-16LE encoded) I have "strange" characters  for diacritics.
Jamal

[Gokhan Soydan]

Jamal,

In that case, you have two choices:

Choice 1: Change -Dfile.encoding=UTF-8 value to -Dfile.encoding=UTF-16LE

Choice 2: You can also manage the encoding at Project level. Here are
the instructions:
- Right click on a project (top-level folder) in the Navigator, and
select Properties
- In Properties dialog box, click on Resource.
- When Resource appears in the box, find "Text file encoding". Select
"Other", and "UTF-16LE" from the drop-down list.

One of the two above may work.

Gokhan

[Gokhan Soydan]

Jamal,

Another suggestion by my colleague, Gavin, is that if you are on Linux,
you can use iconv tool or some advanced text editor in another OS to
convert from one encoding to another. But of course, this would be a
solution outside TopBraid Composer, so it may be uncomfortable.

Gokhan

[J.REZZOUK]

Thanks for all your suggestions. But I am a little bit lost with encoding...

I have in fact a process with the following tasks :
- import data from Excel files (UTF-16LE encoded ?)
- apply some transformations based on TBCME modules + javascript functions using regexp
- export as text file (I would like do get UTF-8 encoded text file)

Depending on the JVM encoding option I use, one of the task doesn't work :
-no specific encoding : export text file is ANSI encoded and javascript regex works except with diacritics
-Dfile.encoding=UTF-8 : Excel import doesn't import diacritics
-Dfile.encoding=UTF-16LE : javascript functions don't work

Jamal

[Gokhan Soydan]

Jamal,

Unfortunately, there is no easy solution to this at the moment. Except
for one module, SPARQLMotion import and export modules at the moment are
lacking support for encoding. On Monday, my colleagues and I have
decided to review and put custom encoding support to the relevant
SPARQLMotion modules in one of the next TopBraid Suite releases.

I assume the setting Project properties as I described you in my earlier
email didn't work. The reason might be that since SPARQLMotion
implementation is at Java level, only the JVM-level encoding setting
would be in effect.

So, currently, you might use this solution for your use case:
1) Exit TopBraid Composer, if already running

2) Set the JVM parameter:
-Dfile.encoding=UTF-16LE

3) Start TopBraid Composer

4) "import data from Excel files" and export it to a temporary ontology.
This ontology should be exported in the platform default encoding
independent of the JVM encoding. ( While in UTF-16LE, don't edit and
save the SPARQLMotion ontology or any other ontology, as it might
corrupt the ontology file!!! )

5) Exit TopBraid Composer.

6) Set the JVM parameter:
-Dfile.encoding=UTF-8

7) Start TopBraid Composer

8) import the temporary ontology and "apply some transformations based
on TBCME modules + javascript functions using regexp" and "export as
text file"

I would suggest trying above or variations of above to make it work.

Gokhan