Changing csv encoding

[bnoeson]

Hi,

I am trying to create CSV files with French characters using BeanIO.

When I use the "default" BeanIO writer :

Writer out = factory.createWriter(builderName, new File(filePath);

I get "??" when where there are French characters like é, â, ô etc.

I tried a second method, changing the encoding of writers : 

Writer writer = null;
try {
 writer = new OutputStreamWriter(new FileOutputStream(new File(filePath)), StandardCharsets.UTF_8);
}
catch (final IOException e) {
 throw new RuntimeException("Error during Export file creation : " + filePath);
}
 
Writer out = factory.createWriter(builderName, writer);

But it is even worse than the first method : I get characters like ï¿½ï¿½..

Do you have an idea how I should do to make sure to output the right characters?

Thanks in advance,

Boris

[Nico Schlebusch]

Hi Boris,

The writing (Writer) part seems to be correct as far as I can tell. Is the problem not coming from the reading/source side of things? From whatever the source data is, it is converted to Java's Unicode representation when you store that as a String. So I would make sure that if I read UTF-8/ASCII/Latin-1 it is actually what I'm getting from the source. This is just a guess.

Please tell us where the data is coming from? DB / Web services / JMS etc in which encoding? ASCII / Latin-1 / UTF-8

If possible also provide us with some sample data and/or test case and expected outcome to help you figure this out.

Kind regards,
Nico Schlebusch

[bnoeson]

Thanks for your quick reply.

You are right, I checked and there seem to be a problem with the data source. 

It comes from a .properties file, and it is sent via REST service to another application. 

I see when debugging that the data is already not right.

I will send you a message if I still have a problem after solving this issue.

[bnoeson]

Hi,

I have to update this issue because now the data provided seems ok but the csv file has still not the right characters.

When I create the csv file, with 

Writer out = factory.createWriter(builderName, new File(filePath);

it puts "?" at the place of the special character.

And when I use the other method : 

Writer writer = null;
try {
 writer = new OutputStreamWriter(new FileOutputStream(new File(filePath)), StandardCharsets.UTF_8);
}
catch (final IOException e) {

 throw new RuntimeException("Error during file creation : " + filePath);

}
 
Writer out = factory.createWriter(builderName, writer);

It displays Ã´ for ô, Ã© for é.

The context : I send the data UTF-8 encoded by json (I use "application/json;charset=UTF-8") to the application which will use BeanIO. When I debug, the data is well formatted when transferred to the BeanIO object.

Sample data you could use : 

- whole words : hôpital léger planète prêt

- simple characters : é è à ô î û ù

 

Boris

[bnoeson]

Ok I finally found a working solution: I set the encoding to ISO-8859-1 and now it works.

But still, I don't understand why it doesn't work in UTF-8.

For the posterity : the working solution

Writer writer = null;
try {

 writer = new OutputStreamWriter(new FileOutputStream(new File(filePath)), StandardCharsets.ISO_8859_1);
}
catch (final IOException e) {