Support for multiple-line input in jaql shell command history

[Jing Guo Yao]

------------------------
Shell Investigation
------------------------

HBase Shell
-----------
HBase shell uses JRuby. It does not support multiple-line input in
command history.

PIG Grunt
-------------
PIG Grunt uses Jline. It does not support multiple-line input in
command history.


Python and Ruby (Not Jython and JRuby)
------------------------------------------------------
Python shell and IRB support multiple-line input in command history.
They use the command history facility provided by OS shells (Windows
command line console and Linux shell console such as Bash).

Conclusion
---------------
Among the shells investigated by me, I find that shells using command
history provided by OS shells support multiple-line input in command
history. If the shells use some other mechanism, they don't support
multiple-line input in command history.

-----------------------
Jline Investigation
-----------------------
Jline does not use command history provided by OS shells. Jline
implements command history by itself. The basic mechanism is as
this.
Jline stores the previous commands in a text file. When UP arrow key
is pressed, it will print out the previous command.

I have read the Jline source code and do some experiment with it. Now
I can store multiple-line input as one entry in command history. The
new line and carriage return characters are replaced with blanks
before the multiple-line input is stored. When UP arrow key is
pressed, the formatted whole input will be displayed instead of a
single line of the original input. For example, the following
multiple-line input is issued in jline:

$array = [
1,
2,
3];

When UP arrow key is pressed, the following text will be displayed:

$array = [ 1, 2, 3];

But there is a problem with my approach. Suppose that the total
length
of the lines in a input is bigger than the console width. For
example,
the following input is issued in jline:

$employees = [
{name: "Jon Doe", income: 20000, mgr: false},
{name: "Vince Wayne", income: 32500, mgr: false},
{name: "Jane Dean", income: 72000, mgr: true},
{name: "Alex Smith", income: 25000, mgr: false}
];

When UP arrow is pressed, the following text will be displayed:

$employees = [ {name: "Jon Doe", income: 20000, mgr: false}, {name:
"Vince Wayne", income: 32500, mgr: false}, {name: "Jane Dean",
income: 72000, mgr: true}, {name: "Alex Smith", income: 25000, mgr:
false} ];

So far so good, I've got the previous input. But if I want to go
further in the command history, I am in trouble. For example, before
this command, the command is "$numbers = [1, 2, 3]". When UP arrow
key
is hit again, the following text is shown in jline:

$employees = [ {name: "Jon Doe", income: 20000, mgr: false}, {name:
"Vince Wayne", income: 32500, mgr: false}, {name: "Jane Dean",
income: 72000, mgr: true}, {name: "Alex Smith", income: 25000, mgr:
$numbers = [1, 2, 3]

Two previous commands are mixed. This is not what I want.

Here is the cause of this problem. If Java prints one line to the
console, it can move freely and erase existing characters using
some control characters (carriage return, backspace, etc). But if
Java
prints two lines to the console, I can't find a way to erase
characters in the second line and move the cursor to the first line.

So I must use command history provided by OS shells to support
multiple-line input in command history. The following simple Java
code
supports multiple-line input in command history:

public class Test {
public static void main(String[] args) throws Exception {
StringBuffer line = new StringBuffer();
System.out.print("=>");
char ch = (char) System.in.read();
while (ch != -1 && ch != '&') {
if (ch == '\n') {
System.out.print(line + "\n");
System.out.print("=>");
} else {
line.append(ch);
}
ch = (char) System.in.read();
}
}
}

But if the JVM is launched from a shell script in Cygwin, the command
history provided by OS shell does not work. The following simple
shell
script can show the effect.

#!/usr/bin/env bash
java Test

Maybe there are some options of stty which can make command history
works for this script. Any ideas?

---------------
Next Step
---------------
In my mind, there are 2 ways to do this job.
- Use windows command console to launch jaql shell. But jaql depends
to Hadoop and Hadoop must run in a Cygwin console.
- Do further Investigation of Cygwin console and make command history
provided by OS shells work even the JVM is launched with a shell
script in Cygwin.

Any suggestions will be highly appreciated.

[Vuk Erecegovac]

I may be wrong on this point, but I think hadoop requires various binaries that are present down cygwin's bin (e.g., "du"). Its primary dependence on cygwin is for its start-up scripts. For the jaql script, its primary dependence on those scripts is for the hadoop-env.sh scripts. Otherwise, it too depends on cygwin more out of convenience (don't have to maintain a .bat and .sh version of the same script) than necessity.

 

There are various pointers out there regarding cygwin and jline, here's one experience:

 

http://groovy.codehaus.org/Groovy+Shell

 

that mentions both terminal setting and stty options. However, if your proposals work for linux and not cygwin, I would be OK with stating that the multi-line history feature works for linux but not cygwin (and we can consider other cygwin specific work-arounds in the future).

Vuk

[Rainer Gemulla]

Good work! Personally, I am fine with multi-line support for just Unix
shells but you might want to try Vuk's suggestions first.

I do not quite understand why the Java code you gave supports multiline
input. Could you explain this a little bit more? The basic question is
should UP give the previous Jaql command or the previous set of lines?

VARIANT 1
---------

jaql> [1,2]; [3,4];
[UP] jaql> [1,2];
[UP] jaql> [3,4];

VARIANT 2
---------

jaql> [1,2]; [3,4];

UP
jaql> [1,2]; [3,4];

Another thing we have to deal with is when we have partial commands;
what would happen with an input like this

jaql> [1,2]; [3,
...
[UP] jaql> ???
jaql> 4];
[UP] jaql> ???

Best solution here is to not execute [1,2]; until "[3," has been finished.

Rainer

[Jing Guo Yao]

Hi, Rainer

Sorry for the confusion. What I am talking about is not multiple-line
input support. Jaql shell already supports it. I am talking about
multiple-line input support in command history. The basic idea is as
this:

1 The following multiple-line input is fed in jaql shell:

jaql> $employees = [

{name: "Jon Doe", income: 20000, mgr: false},
{name: "Vince Wayne", income: 32500, mgr: false},
{name: "Jane Dean", income: 72000, mgr: true},
{name: "Alex Smith", income: 25000, mgr: false}
];

2. When UP arrow key is pressed, the current jaql shell will show
only
one line of the previous input:

jaql> $employees = [

{name: "Jon Doe", income: 20000, mgr: false},
{name: "Vince Wayne", income: 32500, mgr: false},
{name: "Jane Dean", income: 72000, mgr: true},
{name: "Alex Smith", income: 25000, mgr: false}
];

jaql> ];

Instead of one line of the previous input, we want all the lines of
the previous
input.

The java code take inputs from System.in and echoes the input. If &
is
inputted, the program will exit. If it is launched by issuing "java
Test" in windows command line console. And two inputs are fed. One
input is one line. The other input is longer than the windows console
width. So it is wrapped.

D:\>java Test
=>abc
abc
=>1111111111111111111111111111111111111111111111111111111111
22222222222222222
111111111111111111111111111111111111111111111111111111111122
222222222222222
=>

At this moment, if UP arrow key is pressed, the console is like this:

D:\>java Test
=>abc
abc
=>1111111111111111111111111111111111111111111111111111111111
22222222222222222
111111111111111111111111111111111111111111111111111111111122
222222222222222
=>1111111111111111111111111111111111111111111111111111111111
22222222222222222

If UP arrow key is pressed again, the console is like this:

D:\>java Test
=>abc
abc
=>1111111111111111111111111111111111111111111111111111111111
22222222222222222
111111111111111111111111111111111111111111111111111111111122
222222222222222
=>abc

You can see that the two-line history entry is replaced by the one-
lie
history entry. But if the JVM is launched by a shell script in
Cygwin.
For this little java program, the command history does not work. And
I
have tried the solution provided in
http://groovy.codehaus.org/Groovy+Shell. It also does not solve this
problem.

By multiline input, you mean to input multiple jaql expressions in
one
line. Among the two variants, I choose VARIANT 2. jaql shell already
supports this. For partial commands, I agree with you. The first
expression
should not be executed unitl the second expression is finished.

jaql> [1,2];[3,
4];
[UP]jaql>[1,2];[3,4]


I have done some experiments on top of jline command history
facility. If the
total length of the multiple line of one input is not longer than
the console
width, my approach works. Here is one example.

jaql>[1,
2,
3];

[UP]
jaql>[1, 2, 3]

My original post gives an example where this approach does not work
if
the total length of the multiple lines of one input is longer than
the
console width.

[Jing Guo Yao]

Hi, Vuk

I agree with you. We don't want to maintain a .bat and .sh versions
of
the same script.

Simple Java Shell on Linux
------------------------------------
I have tried my little Java program on a linux box (Redhat EL5). On
linux box, this Java program does not inherit the command history of
the bash shell. This is different to windows. On windows box, this
Java program inherits the command history facility of a windows
command console. The following text shows the effect after UP arrow
key is pressed.

[root@risetest2 code]# java Test
=>abc
abc
=>^[[A

And I have tried to change stty options. But I can't get it work.

GNU Readline
-----------------
I have also checked the implementation of IRB (ruby shell). IRB uses
GNU readline library. And there is Java Readline
(http://java-readline.sourceforge.net/) which is JNI wrapper to
Readline. Jython uses it. But there is license problem. GNU readline
is GPL licensed. So we can't use it together with Java Readline. But
Java Readline also supports Editline (http://
libedit.sourceforge.net/)
which has a NON-GPL license.

Summary of Shells
-----------------------
Among the shells I have investigated, all shells based on Java
(JRuby,
groovsy shell, HBase shell and Pig Grunt) do not support multiple-
line
input in command history. Only python and IRB work well with history
entries which are longer than the console width. Other shells have
the
problem similar to my approach described in my original post. All of
the above shells display a single line of a multiple-line input when
UP arrow is pressed. Only bash shell has the feature we wanted.

jingguo@jinglin:~$ echo "abc";\
> echo "def"
abc
def
[UP]jingguo@jinglin:~$ echo "abc";echo "def"


Next Steps
-----------------------
The conclusion is that Jline command history can't provide this
feature. The root cause is that the cursor can't move to the first
line if an command history entry which is longer than the console
width is displayed. If we want to proceed , we need do more work on
Java Readline and Editline.

Any suggestions are welcomed.

> have tried the solution provided inhttp://groovy.codehaus.org/Groovy+Shell. It also does not solve this

> ...
>
> read more »

[Rainer Gemulla]

That's what I was talking about as well. The main point I wanted to
raise is that we should be clear about the semantics of pressing UP. It
can be 1) previous statement or 2) previous set of lines (with the last
of those lines having a statement delimiter).

The problem you are referring seems to be that it is not possible to
change any but the last line in the terminal. The initial discussion was
focused around cygwin; how did you overcome that problem in the Linux shell?

Rainer

>>> while (ch != -1&& ch != '&') {

[Jing Guo Yao]

Yes, we should be clear about the following 2 UP semantics.

1) previous statement (a single line of the previous multiple-line
input)
2) previous set of lines as one input.

Among the shells I have investigated, only bash shell uses the 1) UP
semantics. All the other shells (HBase Shell, PIG Grunt, , Groovy
Shell, JRuby, Ruby and Python) use 2) UP semantics.

If we display the whole set of lines of one input and the total
length
of these lines is longer than the console width, we have the problem
you mentioned. Even we choose 1) UP semantics, we also have this
problem. For example, if we have a previous input whose length is
bigger that the console width. When we issue such a input, the input
is automatically wrapped by the console. When we hit UP, this
previous
input will occupy more than one line. At this point, we also can't
change the lines other the last line in the console.

===================================
Investigation on Linux
===================================
Groovy Shell, HBase Shell, PIG Grunt and JRuby all have this problem
on Linux box. Only Python and IRB don't have this problem. IRB use
GNU
readline. GNU readline does not have the history entries clutter
problem. But it can't be used due to its GNU license. So if we want
to
proceed, we need to go with Java Readline and Editline.

On Sep 22, 12:43 am, Rainer Gemulla <Rainer.Gemulla.J...@gmx.de>

> ...
>
> read more »

[Jing Guo Yao]

On Sep 22, 12:43 am, Rainer Gemulla <Rainer.Gemulla.J...@gmx.de>

> ...
>
> read more »

[Rainer Gemulla]

Ok, that sounds hard to overcome due to licensing issues. Java readline
is LGPL but I don't know about editline. Not sure if we can use it.

Rainer

[Jing Guo Yao]

I googled Editline again and found that it is Berkeley-style
licensed (http://www.thrysoee.dk/editline/).

On Sep 22, 11:54 pm, Rainer Gemulla <Rainer.Gemulla.J...@gmx.de>
wrote:

> ...
>
> read more »