JLine, Jansi and Windows - A fun new game for the family.

[Josh Suereth]

So, for those who may not be aware, there's a conflation of Jline/Jansi versions out there that are causing issues for windows users.  In particular, it seems solving these contraints all at the same time to be nigh impossible currently:

• Autocomplete/jline features working in cygwin
• Autocomplete/jline features working in cmd.exe
• ANSI colors working in cygwin
• ANSI colors working in cmd.exe

SO far, it seems we can pick at best 3 of 4 here.  IN particular, here's my findings after nearly going mad:

• ANSI coloring in cmd.exe requires JANSI to install its hooks before any output.  Due to the way classloading, native libraries + sbt reboot work, we need this to happen in a classloader *below* the sbt classloader (like the launcher) where we can instantiate jline (but only when running in cmd.exe), and ensure that anyone using jline has to use this version of JANSI (a potential dependency conflict that seems to not have arisen yet).
• ANSI coloring anywhere else is broken if you ask JANSI to install its native hooks, for some reason.
• While jline's unix terminal works for cygwin, you have to manually run STTY calls external to the JVM, because java.exe is unable to fork off cygwin processes, and does not see its PATH correctly.
• Scala 2.10.x and previous "shade" the jline library, so that we cannot specify "unix" terminal to both the scala console *and* to the sbt 0.12.x (and previous) jline library at the same time.  This means cygwin users need to either go without (why would you use cygwin if not for good terminal)?  For sbt 0.13, we can just use -Djline.terminal=unix and everything is gravy.

For SBT 0.13 we'd like to find a way to get all the above features working without using too many custom script hacks.  Right now sbt.msi is using a script hack so that when you call the .bat file, you're using an alternative launcher that helps JANSI work correctly in cmd.exe.   I'm writing to elicit some conversation about possible alternatives here that could help us resolve the situation in 0.13.

Thoughts welcome!   I'm at the stage I feel we should just abandon coloring in cmd.exe until we find a better alternative.

- Josh

[Paul Phillips]

On Tue, Jun 25, 2013 at 11:58 AM, Josh Suereth <joshua....@gmail.com> wrote:

Thoughts welcome!   I'm at the stage I feel we should just abandon coloring in cmd.exe until we find a better alternative.

If something has to give, I hope we'd all agree "colors in cmd.exe" is first against the wall.

[Mark Harrah]

On Tue, 25 Jun 2013 14:58:51 -0400
Josh Suereth <joshua....@gmail.com> wrote:

> So, for those who may not be aware, there's a conflation of Jline/Jansi
> versions out there that are causing issues for windows users. In
> particular, it seems solving these contraints all at the same time to be
> nigh impossible currently:
>
>

> - Autocomplete/jline features working in cygwin
> - Autocomplete/jline features working in cmd.exe
> - ANSI colors working in cygwin
> - ANSI colors working in cmd.exe

>
>
> SO far, it seems we can pick at best 3 of 4 here. IN particular, here's my
> findings after nearly going mad:
>
>

> - ANSI coloring in cmd.exe requires JANSI to install its hooks before

> any output. Due to the way classloading, native libraries + sbt reboot
> work, we need this to happen in a classloader *below* the sbt classloader
> (like the launcher) where we can instantiate jline (but only when running
> in cmd.exe), and ensure that anyone using jline has to use this version of
> JANSI (a potential dependency conflict that seems to not have arisen yet).

> - ANSI coloring anywhere else is broken if you ask JANSI to install its

> native hooks, for some reason.

> - While jline's unix terminal works for cygwin, you have to manually run

> STTY calls external to the JVM, because java.exe is unable to fork off
> cygwin processes, and does not see its PATH correctly.

> - Scala 2.10.x and previous "shade" the jline library, so that we cannot

> specify "unix" terminal to both the scala console *and* to the sbt 0.12.x
> (and previous) jline library at the same time. This means cygwin users
> need to either go without (why would you use cygwin if not for good
> terminal)? For sbt 0.13, we can just use -Djline.terminal=unix and
> everything is gravy.

So, it sounds like for 0.13:

* on non-cygwin Windows,
+ need to have JAnsi on the same classpath as sbt and not the launcher. The situation here is not clear to me.
+ need to call systemInstall. This changes System.out to something that supports ansi codes. Presumably, systemUninstall needs to be called before 'reboot'.
* due to JAnsi using a native library and presumably loading it in a non-standard way (not via System.loadLibrary), it is not possible to address issues arising from it being used in multiple locations, such as after a 'reboot'.

* on cygwin
+ need to wrap the call to 'java' in stty commands to properly set up the terminal. This can't be done in sbt itself because being a Java process, it can't launch cygwin commands and 'stty' is a cygwin command.
+ set -Djline.terminal=unix in order to configure both Scala (<2.11)'s JLine and normal JLine to use the right terminal, which is not properly auto-detected by jline

* on unix
+ no native libraries are involved
+ stty is on the standard path

So, unix is fine. cygwin has a plausible solution, although users will have to change their startup scripts and perhaps sbt should warn if jline.terminal=jline.UnixTerminal and/or possibly correct it.

non-cygwin Windows is the problem that requires further consideration. It seems to be independent of the others, though. There aren't tradeoffs that affect other platforms as long as cygwin can be detected. (From other discussions, detection might not be straightforward.). What I don't understand are the requirements surrounding JAnsi and its native libraries.

1. Where in the class loader hierarchy does JAnsi need to exist and why?
2. How does JAnsi load its native library?
3. Does JLine interact with JAnsi directly or does it just need JAnsi to wrap System.out?
4. If JLine interacts directly with JAnsi, how and why does it do so?

> For SBT 0.13 we'd like to find a way to get all the above features working
> without using too many custom script hacks. Right now sbt.msi is using a
> script hack so that when you call the .bat file, you're using an
> alternative launcher that helps JANSI work correctly in cmd.exe. I'm
> writing to elicit some conversation about possible alternatives here that
> could help us resolve the situation in 0.13.
>
> Thoughts welcome! I'm at the stage I feel we should just abandon coloring
> in cmd.exe until we find a better alternative.

I'm not there yet, probably because you've absorbed most of the madness (thanks!). I'll look into the questions I have before conceding. What is the status quo? Does 0.12 handle both coloring and tab completion for both sbt and Scala prompts on cmd.exe (independent of the quality of hacks that achieve this!)?

-Mark

> - Josh
>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Mark Harrah]

My understanding is that cmd.exe problems can be isolated to cmd.exe and so unix users will be unaffected by whatever cmd.exe-related tradeoff is made.

-Mark

[Josh Suereth]

not quite:  JANSI gets loaded via jline, and possibly directly, by a few places.  What we want is for JANSI to load on an *early* classloader, so it can install its native hooks *once* for cmd.exe.  If we have multiple instances vying for world domination (or at least to load JANSI's native windows hooks), then the first one works, and all others fail.   If we have it loaded on an outer class loader, then the behavior breaks at first reboot.   JAnsi must either be loaded by the launcher, or some classloader *before* an application (like the scala provider).  This, of course, just "happens" to work because JANSI is pretty stable across versions, binary-wise, so we haven't run into explosion yet.  I'm hoping the JANSI project remains pretty stagnant in terms of new methods, otherwise this assumption begins to harm us.

 

 + need to call systemInstall.  This changes System.out to something that supports ansi codes.  Presumably, systemUninstall needs to be called before 'reboot'.

Right. If you read: http://jansi.fusesource.org/, you can see that we *could* provide our own wrapper around System.out that calls AnsiConsole.out instead, but I think the native-hook business is still there.   The question, in my mind, is if JANSI support needs to move down as an optionally enabled sbt-launcher feature.  Right now there's a dead-hook in it:  https://github.com/sbt/sbt/blob/0.13/launch/src/main/scala/xsbt/boot/Boot.scala#L51

This turns out to cause issues on *NON* cmd.exe platforms.   I think, the easiest path out would be to have this initialization be an optional setting you can pass in, and include un-proguarded JANSI right in the launcher, with a flag to enable this installation when running in cmd.exe.  that way, we can detect via a script, and have the launcher do the right thing, as well as update the docs for folks who write their own launch scripts.

  

 * due to JAnsi using a native library and presumably loading it in a non-standard way (not via System.loadLibrary), it is not possible to address issues arising from it being used in multiple locations, such as after a 'reboot'.

While this says JNA, they migrated to HAWT -> https://github.com/fusesource/jansi/blob/master/jansi/src/main/java/org/fusesource/jansi/WindowsAnsiOutputStream.java

You can see the HAWT load call here: https://github.com/fusesource/jansi-native/blob/master/src/main/java/org/fusesource/jansi/internal/Kernel32.java#L37

Which appears to eventually use regular load-library calls: https://github.com/fusesource/hawtjni/blob/master/hawtjni-runtime/src/main/java/org/fusesource/hawtjni/runtime/Library.java#L313

 

* on cygwin
 + need to wrap the call to 'java' in stty commands to properly set up the terminal.  This can't be done in sbt itself because being a Java process, it can't launch cygwin commands and 'stty' is a cygwin command.
 + set -Djline.terminal=unix in order to configure both Scala (<2.11)'s JLine and normal JLine to use the right terminal, which is not properly auto-detected by jline

Correct on both accounts.  Note: leineingein (sp?) does the same hack.  I'd bet dollars to donuts most other build tools using jline are as well.

 

* on unix
 + no native libraries are involved
 + stty is on the standard path

So, unix is fine.  cygwin has a plausible solution, although users will have to change their startup scripts and perhaps sbt should warn if jline.terminal=jline.UnixTerminal and/or possibly correct it.

Possibly.  jline.terminal=jline.UnixTerminal works fine for sbt itself, it's the scala REPL (console tasks) which are affected by Scala's shading.   Like I said, I'd like to unify the sbt launcher scripts to all use the same mechanism, preferably the jline 2.x style "unix".   Scripts meant to span multiple sbt versions will be unable to do so though.  for these, I see no solution for cygwin sbt 0.12-  and scala 2.9+

 

non-cygwin Windows is the problem that requires further consideration.  It seems to be independent of the others, though.  There aren't tradeoffs that affect other platforms as long as cygwin can be detected.  (From other discussions, detection might not be straightforward.).  What I don't understand are the requirements surrounding JAnsi and its native libraries.

1. Where in the class loader hierarchy does JAnsi need to exist and why?

So, IMHO - JANSI needs to exist either at the system classloader, or *before* loaded applications of the launcher.  This is because:

  * It loads native DLLS

  * It rewrites System.err + System.out

  * It needs to survive "reboot" of a launched application

If it was only 2 of those 3, we could use DLL loading trickery.  However, since it *assumes* it can take over System.out, and none of our tools actually delegate into a JAnsiConsole, this is the best scenario currently.  I honestly don't even know if we could load more than one JANSI library and have it work correctly on windows.

 

2. How does JAnsi load its native library?

Links above, but duplicated here:

 [1] https://github.com/fusesource/jansi/blob/master/jansi/src/main/java/org/fusesource/jansi/WindowsAnsiOutputStream.java

 [2] https://github.com/fusesource/jansi-native/blob/master/src/main/java/org/fusesource/jansi/internal/Kernel32.java#L37

 [3] https://github.com/fusesource/hawtjni/blob/master/hawtjni-runtime/src/main/java/org/fusesource/hawtjni/runtime/Library.java#L313

 

3. Does JLine interact with JAnsi directly or does it just need JAnsi to wrap System.out?

I'm not positive what JLine is using.  I only know it embeds some of JAnsi.

 

4. If JLine interacts directly with JAnsi, how and why does it do so?

Again, I'm not sure.  Perhaps someone else knows jline a bit better?  Paul?

 

> For SBT 0.13 we'd like to find a way to get all the above features working
> without using too many custom script hacks.  Right now sbt.msi is using a
> script hack so that when you call the .bat file, you're using an
> alternative launcher that helps JANSI work correctly in cmd.exe.   I'm
> writing to elicit some conversation about possible alternatives here that
> could help us resolve the situation in 0.13.
>
> Thoughts welcome!   I'm at the stage I feel we should just abandon coloring
> in cmd.exe until we find a better alternative.

I'm not there yet, probably because you've absorbed most of the madness (thanks!).  I'll look into the questions I have before conceding.  What is the status quo?  Does 0.12 handle both coloring and tab completion for both sbt and Scala prompts on cmd.exe (independent of the quality of hacks that achieve this!)?

Yeah, so sbt.msi has a set of hacks such that the BASH script (cygwin) and BAT file (cmd.exe) have *different* launcher main methods.   We're essentially loading JANSI up in the launcher and then farming down to the regular launcher in the BAT file.   In cygwin, we just do the stty hack and go to the regular launcher.

I'd love to be able to minimize to just tweaking a few flags here and there, and the stty hack, if we can do it.   

[Mark Harrah]

On Tue, 25 Jun 2013 20:53:39 -0400

Sorry, those were questions I was going to investigate, which I did but hadn't posted the results yet. Some of this overlaps with what you said, but:

1. JAnsi should be probably shared among all class loaders in a jvm to be safe because the terminal configuration is of course global and with JAnsi in multiple class loaders, the separate, non-authoritative state would get out of sync with the authoritative state (the actual terminal). I didn't find concrete evidence in the Java code that it maintains state separate from the OS, but there could be some in native code. JLine might maintain some as well.

I see no reason why JAnsi would need to be at the launcher level, however, elaborated in #2. It should be sufficient to be a dependency of sbt via jline. The systemInstall method should be called on startup in order to swap System.out for a stream that understands ansi codes. systemUninstall should be called before 'reboot'. It is entirely under user control, though, so it shouldn't be a problem.

2. JAnsi loads its native library via hawtjni's Library, which extracts native libraries from jars on the classpath to random temporary files. The random temporary file is a recent change and is unlikely to be present in older jline/jansi versions used by Scala. As long as JAnsi is available from a parent loader, however, the newer version should be used instead. Because of the random jar name, a duplicate loading error should not happen after an sbt 'reboot'.

3/4. JLine both directly calls JAnsi as well as needs it to wrap System.out. JLine's WindowsTerminal uses JAnsi native bindings to get/set attributes like console mode (for echo, for example) and terminal width and height or to read the next byte directly. JAnsi usage is not limited to windows, however. It is also used to strip ANSI codes from a string to compute its displayed length. This does not use native code. If necessary, this usage could be replaced by a short block of code to specifically filter escape codes, but I don't see a need for this.

The new output stream is provided by JAnsi and translates escape codes into native calls. JLine manipulation appears to be mainly via escape codes on all platforms and not special cased to directly call JAnsi on Windows. So, if colors are displayed, JAnsi is wrapping System.out and the native library is successfully installed. If shortcuts aren't working, it is probably a problem with JLine reading the next byte via JAnsi (which does it via native code).

More details:

* JLine bundles JAnsi, except for a few classes to save a meaningless amount of space. The reason for this isn't clear to me.
* JAnsi will silently swallow any issues with loading the JNI library
* A call to AnsiConsole.systemInstall is necessary to swap in custom System.out and System.err implementations. Even with this explicit call:
+ jansi will only install the ansi code handler if the os.name system propery starts with "Windows"
+ -Djansi.passthrough=true will disable JAnsi output rewrapping, preserving the original streams
+ -Djansi.strip=true will filter ansi codes without interpreting them
* Reading the next byte directly is how special input characters (shortcuts/navigation) are detected
* JLine's WindowsTerminal will use JAnsi to read the next byte if two conditions hold: this feature isn't disabled (via -Djline.WindowsTerminal.directConsole=false) and if System.in is the standard input (detected as it being a FileInputStream on FileDescriptor.in).
* Swapping System.out is how colors are handled as well as navigation and such.

> > > For SBT 0.13 we'd like to find a way to get all the above features
> > working
> > > without using too many custom script hacks. Right now sbt.msi is using a
> > > script hack so that when you call the .bat file, you're using an
> > > alternative launcher that helps JANSI work correctly in cmd.exe. I'm
> > > writing to elicit some conversation about possible alternatives here that
> > > could help us resolve the situation in 0.13.
> > >
> > > Thoughts welcome! I'm at the stage I feel we should just abandon
> > coloring
> > > in cmd.exe until we find a better alternative.
> >
> > I'm not there yet, probably because you've absorbed most of the madness
> > (thanks!). I'll look into the questions I have before conceding. What is
> > the status quo? Does 0.12 handle both coloring and tab completion for both
> > sbt and Scala prompts on cmd.exe (independent of the quality of hacks that
> > achieve this!)?
> >
> >
> Yeah, so sbt.msi has a set of hacks such that the BASH script (cygwin) and
> BAT file (cmd.exe) have *different* launcher main methods. We're
> essentially loading JANSI up in the launcher and then farming down to the
> regular launcher in the BAT file. In cygwin, we just do the stty hack and
> go to the regular launcher.
>
> I'd love to be able to minimize to just tweaking a few flags here and
> there, and the stty hack, if we can do it.

It seems to me that for cmd.exe,

* share JAnsi bin a class loader common to sbt and Scala (not currently done)
* call AnsiConsole.systemInstall and AnsiConsole.systemUninstall

I think that's really it.

-Mark

[Josh Suereth]

Right.  Sounds like something worth trying out.   seems we need a way to share JANSI between sbt and scala.  Do you have a hook for that currently?

[Mark Harrah]

On Tue, 25 Jun 2013 21:56:39 -0400

A possibility is where the launcher pulls down JNA, it could do JAnsi instead. JNA was in there before because WindowsTerminal is documented to use it, but that's obviously not the case now and I don't know if or when it was ever true.

I will look into this more tomorrow.

-Mark

[Josh Suereth]

It was true. It stopped using JNA because that's an LGPL dependency, whereas JAnsi is not, so more business are happier about that.

-Mark

[nafg]

Incidentally, what about:

• Powershell (interestingly Atomic Scala suggests using it, so they can give the same "ls" command for all OSes.)
  
• Console2 (cmd.exe interpreter but a much better --- though not perfect --- terminal emulator).

[Paul Phillips]

On Wed, Jun 26, 2013 at 8:22 PM, nafg <nafto...@gmail.com> wrote:

• Powershell (interestingly Atomic Scala suggests using it, so they can give the same "ls" command for all OSes.)

We've investigated powershell in the past. At the time, installing powershell on the latest windows xp required the following "optional" updates, i.e. they would not generally be present, and these things were not small:

"Microsoft .NET Framework 3.5 Service Pack 1"

".NET Framework 3.5 Family Update (KB951847) x86" 

"Windows PowerShell 2.0 and WinRM 2.0 for Windows XP and Windows Embedded (KB968930)"

That is a skyscraper of dependency for backspace and some color codes. Even if we wanted to pursue it, it could only supplement cmd.exe, not replace it, which means it would be moving us in the wrong direction.

But maybe things have changed. For all I know, Windows XP has all the relevance of Tamagotchi and powershell is standard in windows whatever it is now. Do you know?

[Naftoli Gugenheim]

I don't know firsthand (I haven't gotten around to fixing the windows partition's boot sector since I broke it quite a while ago... :) ), although those .NET dependencies are surely safe to expect. If the book may hold any clues, this is what it says:

Once the Windows Explorer is running, move through the folders on
your computer by double-clicking on them with the mouse. Navigate
to the desired folder. Now click in the address bar at the top of the
Explorer window, type ‚powershell‛ and press the ‚Enter‛ key. This
will open a shell in the destination directory. (If Powershell doesn’t
start, go to the Microsoft website and install it from there).
In order to execute scripts in Powershell (which you must do to test
the book examples), you must first change the Powershell execution
policy.
On Windows 7, go to the ‚Control Panel‛ ... ‚System and Security‛ ...
‚Administrative Tools.‛ Right click on ‚Windows Powershell Modules‛
and select ‚Run as Administrator.‛
On Windows 8, use Windows+W to bring up ‚Settings.‛ Select ‚Apps‛
and then type ‚power‛ in the edit box. Click on ‚Windows PowerShell‛
and then choose ‚Run as administrator.‛
At the Powershell prompt, run the following command:
Set-ExecutionPolicy RemoteSigned

If asked, confirm that you want to change the execution policy by
entering ‚Y‛ for Yes.
From now on, in any new Powershells you open, you can run
Powershell scripts (files that end with ‚.ps1‛) by typing ./ followed by
the script’s file name at the Powershell prompt.

 

I'm sure you can deduce whatever there is to deduce from that as well as I can...

I don't remember if powershell is, in addition to a shell language, a different console implementation. I would assume that Windows' coloring deficiency has to do with console UI. On the other hand, console2 replaces only the latter (and it has no dependencies). Actually come to think of it I did actually try console2 on someone's computer recently. I think I may have run into a JLine issue...

--
You received this message because you are subscribed to a topic in the Google Groups "simple-build-tool" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/simple-build-tool/629FH51VaGM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to simple-build-t...@googlegroups.com.

[OlegYch]

fwiw i recently replaced console2 with conemu (which does support ansi color sequences, tested in sbt 0.12) and never looked back

[OlegYch]

i.e. with
java -Djline.terminal= -Dsbt.log.format=true -jar sbt-launcher.0.12.3.jar
with tcc/le, cmd and cygwin bash in conemu i get colors, tab completion, non-borked text editing and console task for scala 2.9 and 2.10 on windows

[Josh Suereth]

I'm testing on powershell.exe + cmd.exe.  So far JANSI works ok on powershell.  Cygwin is the hellish fun.

--

You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.