How to make non ascii characters to output nicely in the console

[Per Nyfelt]

In the REPL and also using the ScriptEngine i fail to get nonascii characters tp be printed properly.

Console is now running the Renjin REPL, type quit() to exit

Renjin 3.5-beta64
Copyright (C) 2019 The R Foundation for Statistical Computing
Copyright (C) 2019 BeDataDriven
> print("åöä")
[1] "\u2020\u201d\u201e"

Using the scriptengine i see basically the same thing using

@Test
public void testUnicode() throws ScriptException {
  RenjinScriptEngine engine = new RenjinScriptEngine();
  engine.eval("print('åäö')");
}

[1] "\u00c3\u00a5\u00c3\u00a4\u00c3\u00b6"

I tried to create an outputstream to address it i.e. with a write method as follows:

class AppenderOutputStream extends OutputStream {
  StringBuffer str = null;

  AppenderOutputStream(StringBuffer stringBuf) {
    str = stringBuf;
  }

  @Override
  public void write(int b) {
    char[] chars = Character.toChars(b);
    str.append(chars);
  }
}

and then set the stdout of the session to use this AppenderOutputStream:

@Test
public void testUnicodeWithOutputStream() throws IOException, ScriptException {

  StringBuffer outSb = new StringBuffer();
  try(
      OutputStream out = new AppenderOutputStream(outSb);
      PrintWriter outputWriter = new PrintWriter(out)
  ) {
    RenjinScriptEngine engine = new RenjinScriptEngine();
    engine.getSession().setStdOut(outputWriter);
    engine.eval("print('åäö')");
    assertEquals("[1] åäö", outSb.toString());
  }
}

But no joy :(

If course this works:
@Test
public void testUnicodeWithVar() throws ScriptException {
  RenjinScriptEngine engine = new RenjinScriptEngine();
  engine.eval("charVar <- 'åäö'");
  StringVector vec = (StringVector)engine.eval("charVar");
  assertEquals("åäö", vec.asString());
}

But it would be nice to get the console/scriptengine output to behave the same.

Does anyone have ideas how to do this i.e. how to get the output from renjin to print non ascii characters nicely?

[Bertram, Alexander]

You will need to change the logic here: https://github.com/bedatadriven/renjin/blob/master/core/src/main/java/org/renjin/parser/StringLiterals.java

Some characters should be escaped (like newlines, control characters etc) but the logic is currently way too conservative.

Best,

Alex

--
You received this message because you are subscribed to the Google Groups "Renjin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to renjin-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/renjin-dev/d7c510e8-70cc-4acd-93cf-adcb931e9323%40googlegroups.com.

[Per Nyfelt]

Thanks Alex!

I will give it a shot!

Btw when I tried to build master i got a failure when building grDevices:

....

devPS.c:54:18: fatal error: zlib.h: No such file or directory

compilation terminated.

/home/ubuntu/renjin/tools/gnur-installation/src/main/resources/etc/Makeconf:169: recipe for target 'devPS.o' failed

make: *** [devPS.o] Error 1

> Task :packages:grDevices:make FAILED

.....

turns out libz-dev is not part of the Vagrantfile.

After doing

sudo apt-get install libz-dev

I managed to build everything just fine. 

Regards,

Per

On Friday, August 9, 2019 at 4:23:15 PM UTC+2, Alexander Bertram wrote:

You will need to change the logic here: https://github.com/bedatadriven/renjin/blob/master/core/src/main/java/org/renjin/parser/StringLiterals.java

Some characters should be escaped (like newlines, control characters etc) but the logic is currently way too conservative.

Best,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to renji...@googlegroups.com.