JLine3 conversion questions

1,042 views
Skip to first unread message

Brad Wood

unread,
Dec 20, 2017, 3:32:42 AM12/20/17
to jline-users
So I've finally begun my long-overdue conversion of CommandBox from Jline2 to JLine3. So far I've got the basic terminals and reader created and it's mostly working.  Here's a couple questions I've run across as I got through my code interaction with JLine.

---------------------------------------------------------

reader.setPrompt() looks like it became a protected method.  What is the correct way to set the prompt I want?  I see prompt can be passed to the all of the readLine() methods.  Is that the only way to do it now?

---------------------------------------------------------

I used to have a lot of this just for writing output to the screen 

reader.println( ... );

Which I was able to get working again like so:

reader.getTerminal().writer().println( ... );

Is there a more appropriate way for general output (possibly with ANSI escape characters mixed in) to the console?
Also, what is the difference between terminal.output() and terminal.writer()?

---------------------------------------------------------

Where do I look to find the equivalent of stuff like this:

reader.setHandleUserInterrupt( true );
reader.setExpandEvents( false );
reader.getCompletionHandler().setPrintSpaceAfterFullCompletion( false );

---------------------------------------------------------

That's it for now.  I'm sure I'll have more questions later.

Thanks!

~Brad

Guillaume Nodet

unread,
Dec 20, 2017, 3:50:28 AM12/20/17
to jline...@googlegroups.com
On Wed, Dec 20, 2017 at 9:32 AM, Brad Wood <bdw...@gmail.com> wrote:
So I've finally begun my long-overdue conversion of CommandBox from Jline2 to JLine3. So far I've got the basic terminals and reader created and it's mostly working.  Here's a couple questions I've run across as I got through my code interaction with JLine.

---------------------------------------------------------

reader.setPrompt() looks like it became a protected method.  What is the correct way to set the prompt I want?  I see prompt can be passed to the all of the readLine() methods.  Is that the only way to do it now?

Yes.
Though in the gogo demo for example, the prompt is held in a session variable:
 

---------------------------------------------------------

I used to have a lot of this just for writing output to the screen 

reader.println( ... );

Which I was able to get working again like so:

reader.getTerminal().writer().println( ... );

Is there a more appropriate way for general output (possibly with ANSI escape characters mixed in) to the console?
Also, what is the difference between terminal.output() and terminal.writer()?

The terminal output / writer will support ansi escape sequences.
To write them, you have multiple options:
  - raw writing,
  - through AttributedString#toAnsi / AttributedStringBuilder

You may end up with character encoding problems if you write unicode to the terminal.output()  directly instead of the writer.
 

---------------------------------------------------------

Where do I look to find the equivalent of stuff like this:

reader.setHandleUserInterrupt( true );

You need to set a signal handle for the INTERRUPT signal:
 
reader.setExpandEvents( false );

 
reader.getCompletionHandler().setPrintSpaceAfterFullCompletion( false );

This is not global anymore, you need to set it in the completer:
 

---------------------------------------------------------

That's it for now.  I'm sure I'll have more questions later.

np

Guillaume
 

Thanks!

~Brad

--
You received this message because you are subscribed to the Google Groups "jline-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jline-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Brad Wood

unread,
Dec 20, 2017, 4:56:53 PM12/20/17
to jline-users
Thanks for the replies.


> Though in the gogo demo for example, the prompt is held in a session variable:

Perfect, that's essentially what I had already done to work around this.

> To write them, you have multiple options:

Actually, I've already written my own DSL for dealing with ANSI escape sequences in CFML for CommandBox.  It allows me to just do stuff like this to apply formatting (gotta' love dynamic languages!):

print.line( 'Test!' );
print.greenOnRedLine( "Christmas?" );
print.underscoredLine( "Have I made my point?" );
print.boldRedOnBlueText( "Test dirt, don't wash." );
print.boldBlinkingUnderscoredBlueTextOnRedBackground( "That's just cruel" );

So all I need the the ability to take those formatted strings and write them out.   I'll continue to use the writer.

> You need to set a signal handle for the INTERRUPT signal:

Do you have any other information on that?  I see the interfaces and I see a very small section that mentions this in the wiki, but I'm not too familiar with the signals and what that means.  Do I need to write a class to handle the signals?  I'm not quite getting the pieces put together on this one.  


Are there any docs or example for setting options?  I see the code and I see a table of possible options in the wiki, but again I'm missing some mental links here on what code I need to use them.

> This is not global anymore, you need to set it in the completer:

Got it.  The wiki page for completions though seems to be empty.  Where can I learn about how to use completers in Jline3?

The other thing I need to convert is the history files.  The Wiki page is empty for this as well.  Does history work differently now, or has the class just been moved?

Brad Wood

unread,
Dec 20, 2017, 5:33:59 PM12/20/17
to jline...@googlegroups.com
A bit more on the history.  In Jline2, I could create more than one history object and provide a different physical file on disk for each one of them:

map( 'commandHistoryFile@java' ).toJava( "jline.console.history.FileHistory" )
.initWith( createObject( "java", "java.io.File" ).init( commandHistoryFile ) )
.asSingleton();

map( 'REPLScriptHistoryFile@java' ).toJava( "jline.console.history.FileHistory" )
.initWith( createObject( "java", "java.io.File" ).init( REPLScriptHistoryFile ) )
.asSingleton();

map( 'REPLTagHistoryFile@java' ).toJava( "jline.console.history.FileHistory" )
.initWith( createObject( "java", "java.io.File" ).init( REPLTagHistoryFile ) )
.asSingleton();

Then I could swap out the history file being used while the CLI was running.  For instance, when the user opened a REPL, the swapped out the REPL history object:

// Setup REPL history file
reader.setHistory( newHistory );

I'm looking through the code and it seems that the history object now loads the physical file location from a HISTORY_FILE variable in the reader.  Is there an example of setting this and how would I go about swapping it out at run time to have contextual histories like I did in Jline2?


Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


--

Brad Wood

unread,
Dec 20, 2017, 7:56:52 PM12/20/17
to jline...@googlegroups.com
Ok, here's some more questions.  I'm slowly figuring out completors by brute force, but I'm unable to get a few behaviors back.  I see that Jline3 tries to filter candidates for me, but that's causing some issues.  I do my own filtering in my custom completor so I can control it based on the rules I want.  In one instance, I'm sending back 3 candidates but JLine is ignoring them.  I assume because they don't match anything typed so far.  My completor is very complex and actually parses the input thus far (with my own parser, not JLine's) and determines possible values for the current option being typed based on the current context of the command in use, the style of parameters being used, etc etc.  So when I user types:

server start cfengine=

and hits tab, my completor parses the line, determines that the user is typing a "server start" command, is using named parameters, and is typing values for the cfengine param.  It then makes a web service call to determine the possible cfengine values and returns 3 candidates of "adobe", "lucee", and "railo" which need to be displayed to the user.  How do I get these to be displayed like Jline2 did?

And on the topic of completors, where can I find some better docs for the options passed to a candidate constructor?  I played with them for a bit, but not all of them made any sense.  Especially since the formatting of the candidates on the screen seemed to differ based on what I passed in.  Namely, how and why would I use the following and exactly how are they used?
  • displ
  • group
  • descr
  • suffix
  • key

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Brad Wood

unread,
Dec 20, 2017, 7:59:14 PM12/20/17
to jline...@googlegroups.com
Here's another completor question.  Where does the text "JLine" come from in this line?

JLine: do you wish to see to see all 114 possibilities (13 lines)?

Is that the app name of the terminal?  I can change that to be something like "CommandBox", but why does it need to be there at all?  I think it should be clear to my users that the question is coming from CommandBox, so I don't see any reason for a "Hello, my name is..." name tag.

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Brad Wood

unread,
Dec 21, 2017, 2:06:12 AM12/21/17
to jline...@googlegroups.com
Another completor question-- in JLine2, the complete() method returned an int which was "The index of the buffer for which the completion will be relative".  What is the equivalent to that in the Jline3?  

I think that may be related to some of the wonky behavior I'm seeing with my custom completor.  I can't figure out what's going on because I seem to get different behaviors in different situations which doesn't make a lot of sense.

For instance, when I type the following and hit tab (with the cursor where the udnerscore shows)
coldbox_
Then my completor adds a single candidate called "coldbox " (with a trailing space) and the prompt ends up like so:
coldbox _

This is fine.  I like it.

Now, if I go a step further and type this and hit tab
coldbox create_
My completor adds as single candidate called "coldbox create " but the prompt does this instead:
coldbox coldbox create _

Why does the first example overwrite the entire buffer, but the second example appends on to to?  I'm setting complete=false in all my candidates since my completor already has the logic to add the trailing spaces where necessary.  I also haven't modified the completor logic, this is as it was in JLine2, but I assume the issue is related to the fact that I'm no longer telling the completor the index of where the match starts.


Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Guillaume Nodet

unread,
Dec 21, 2017, 3:42:23 AM12/21/17
to jline...@googlegroups.com
The completer's task is to give back a list of candidates matching the *word* being completed.
So if your line is "coldbox cre_" and you hit tab, the completer should add a candidate with "create", not "coldbox create".
And yes, the LineReader will do its own selection among candidates.  The reason is that it allows typo fixing for example, which otherwise would not be easy to implement if it had to be done by each completer...

For the various fields of the Candidate, please have a look at the javadoc: 

The "JLine" in the "JLine: do you wish to see to see all 114 possibilities (13 lines)?" actually comes from the app name that can be set on the LineReaderBuilder.

For the history, I suppose the easiest way would be to have a facade implementation that delegates to another History implementation that can be switched.  Another possibility is to use the following to switch:
  reader.getHistory().save();
  reader.setVariable(LineReader.HISTORY_FILE, "the-new-file");
  reader.getHistory().load();


On the ^C handling, I think I mislead you. What you need to do is use the following to build the terminal:
TerminalBuilder.builder()
        .system(true)
        .nativeSignals(true)
        .signalHandler(Terminal.SignalHandler.SIG_IGN)
        .build();
This way, the native signals ^C, ^Z, etc... will be intercepted by JLine. 

Brad Wood

unread,
Dec 21, 2017, 11:09:20 AM12/21/17
to jline...@googlegroups.com
Thanks for the answers.  I'm still unclear on most of the items however.  (Would it be best if I started separate threads of each of these?)
 
The completer's task is to give back a list of candidates matching the *word* being completed.
So if your line is "coldbox cre_" and you hit tab, the completer should add a candidate with "create", not "coldbox create".

Can you show me if this is documented somewhere?  Is there a way to get the old behavior back?  I'd rather not have to rewrite all my custom completor logic.

And yes, the LineReader will do its own selection among candidates.  The reason is that it allows typo fixing for example, which otherwise would not be easy to implement if it had to be done by each completer..

This sounds like a problem then   CommandBox uses a complex custom completor that can predict what the user will type even before they begin typing the word based on the context of the command and the parameters they've typed thus far.  What can we do to get this regression fixed?  I need all candidates to be displayed to the user since they are all valid or I wouldn't have added them as candidates.  For instance, if a user types this and hits tab:

rm path=myFile.txt force=_

Then they should be prompted with true/false since those are the valid options for the Boolean "force" param.  What's odd is that the positional version works:

rm myFile.txt _

Hitting tab on the above will correctly prompt true/false to the user.  

For the various fields of the Candidate, please have a look at the javadoc: 

Perhaps I wasn't clear on this, but I've read the java docs and they don't provide enough information.  Take for instance, "Candidates which have the same key will be merged together."  What does that mean?  What is a key?  Where is an example?  What do you mean by "merged"-- like actually concatenated?  Why would I want that?  

I suppose the easiest way would be to have a facade implementation that delegates to another History implementation that can be switched

So this sounds like a regression then in JLine.  Shall I put in a ticket for getting the old behavior back that allows me to easily switch between histories?  That was very handy.

reader.setVariable(LineReader.HISTORY_FILE, "the-new-file");

So, I didn't mention this earlier, but I can't get the actual file variable to work either.  I'm basically doing what you have above but I'm setting it in the builder DSL.  The file just never gets created and my history is gone when I re-open the CLI.  What's the best way to debug this?

Also, I'm unclear on whether I actually need to be setting a history object in the read builder DSL.  It appears to use the DefaultHistory anyway.  is that only really needed if I have a custom history implementation?

reader = createObject( "java", "org.jline.reader.LineReaderBuilder" )
.builder()
.terminal( terminal )
.variables( {
LineReader.HISTORY_FILE : commandHistoryFile
} )
.history( DefaultHistory )
.completer( jCompletor )
.build();


.signalHandler(Terminal.SignalHandler.SIG_IGN)

Is there a doc of what all the different handlers are, and when/why I would want to set each one?  

And a new question on the LineRreaderBuilder.  Is there a way to provide options as part of the builder?  I have this line after I get done building:

reader.unsetOpt( LineReaderOption.INSERT_TAB );

Is there a way that can be done as part of the builder?


Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Guillaume Nodet

unread,
Dec 21, 2017, 12:15:42 PM12/21/17
to jline...@googlegroups.com
On Thu, Dec 21, 2017 at 5:08 PM, Brad Wood <bdw...@gmail.com> wrote:
Thanks for the answers.  I'm still unclear on most of the items however.  (Would it be best if I started separate threads of each of these?)
 
The completer's task is to give back a list of candidates matching the *word* being completed.
So if your line is "coldbox cre_" and you hit tab, the completer should add a candidate with "create", not "coldbox create".

Can you show me if this is documented somewhere?  Is there a way to get the old behavior back?  I'd rather not have to rewrite all my custom completor logic.

The only way to get the old behavior is to wrap your old completers into the new ones which should be quite easy.
 

And yes, the LineReader will do its own selection among candidates.  The reason is that it allows typo fixing for example, which otherwise would not be easy to implement if it had to be done by each completer..

This sounds like a problem then   CommandBox uses a complex custom completor that can predict what the user will type even before they begin typing the word based on the context of the command and the parameters they've typed thus far.  What can we do to get this regression fixed?  I need all candidates to be displayed to the user since they are all valid or I wouldn't have added them as candidates.  For instance, if a user types this and hits tab:

rm path=myFile.txt force=_

Then they should be prompted with true/false since those are the valid options for the Boolean "force" param.  What's odd is that the positional version works:

rm myFile.txt _

Hitting tab on the above will correctly prompt true/false to the user.  

What you want is achievable.  You just need to care about wrapping your completer in a new one with the correct behavior.
The new completer's candidate should always match the whole argument (and not line) being completed, that's why there's no index anymore.
So in the first case, the completer should return 2 candidates
  force=true
  force=false

The only case that would not be working right now is if your completer proposes candidates where the beginning does not match.
  


For the various fields of the Candidate, please have a look at the javadoc: 

Perhaps I wasn't clear on this, but I've read the java docs and they don't provide enough information.  Take for instance, "Candidates which have the same key will be merged together."  What does that mean?  What is a key?  Where is an example?  What do you mean by "merged"-- like actually concatenated?  Why would I want that?  

The important fields are "value" and "complete".  
The other fields (key, display, etc... are mainly used to slightly alter the way the candidates are presented to the user).  I suggest we deal with those once the main completion issue is over.
 

I suppose the easiest way would be to have a facade implementation that delegates to another History implementation that can be switched

So this sounds like a regression then in JLine.  Shall I put in a ticket for getting the old behavior back that allows me to easily switch between histories?  That was very handy.

Well, one can always call   ((LineReaderImpl) reader).setHistory(myHistory) whenever you want.
 

reader.setVariable(LineReader.HISTORY_FILE, "the-new-file");

So, I didn't mention this earlier, but I can't get the actual file variable to work either.  I'm basically doing what you have above but I'm setting it in the builder DSL.  The file just never gets created and my history is gone when I re-open the CLI.  What's the best way to debug this?

Also, I'm unclear on whether I actually need to be setting a history object in the read builder DSL.  It appears to use the DefaultHistory anyway.  is that only really needed if I have a custom history implementation?

reader = createObject( "java", "org.jline.reader.LineReaderBuilder" )
.builder()
.terminal( terminal )
.variables( {
LineReader.HISTORY_FILE : commandHistoryFile
} )
.history( DefaultHistory )
.completer( jCompletor )
.build();

It should work. The easiest way may be to put a breakpoint in the DefaultHistory#save method and see
  - if it's called after entering a line
  - what happens in it
 


.signalHandler(Terminal.SignalHandler.SIG_IGN)

Is there a doc of what all the different handlers are, and when/why I would want to set each one?  

Not really at this point.  Those are the same as the standard unix signals, but only the terminal related ones are defined.
 

And a new question on the LineRreaderBuilder.  Is there a way to provide options as part of the builder?  I have this line after I get done building:

reader.unsetOpt( LineReaderOption.INSERT_TAB );

Is there a way that can be done as part of the builder?

I just added it yesterday.  Have a look at version 3.5.4.

Brad Wood

unread,
Dec 21, 2017, 1:14:37 PM12/21/17
to jline...@googlegroups.com
Thanks for the additional answers.

Regarding the completors and only handing back the last "word" in the line, how should I determine what the last word is?  I'm not using JLine's parsing functionality at all.  CommandBox has its own parser based on our own rules for tokenizing strings and dealing with escape characters, etc.  I don't know that my definition of the last token in the line is the same as what JLine is expecting.  Is it safe to say that any chunks of text that don't have spaces such as force=true should be considered one "word" and always have the whole bit added as a candidate?

I suggest we deal with those once the main completion issue is over.

That's fine.  I really like the new completor stuff.  It's slick and a much better UI so good work on that.  I like where I think some of those extra options are going so I want to make sure I take advantage of everything I can.

So, about that pasing, I actually have some questions about the parser stuff built into JLine that I haven't had a chance to look at yet.  I want to make sure that JLine's parser doesn't get in the way and try to eat any escape characters or anything since I do all that on my own based on CommandBox's parsing rules  Is there a way to just turn off any internal parsng entirely to make sure I always get back the strings exactly as the user typed them?  From my small amount of testing, I've been seeing that JLine seems to be eating by backslashes which is messing up stuff once I try to do my own parsing on the line.

For instance:

CommandBox> echo brad\wood

should echo out exactly "brad\wood" since "\w" is not a CommandBox escape sequence.  And

CommandBox> echo brad\nwood

should echo out

brad
wood

since "\n" is the CommandBox escape for a new line.  

This parser stuff is cool and I wish it were here 4 years ago when I first started using JLine, but since I've already written all of it myself I need to just be able to turn it all off.

Well, one can always call   ((LineReaderImpl) reader).setHistory(myHistory) whenever you want.

But that wouldn't actually do any good since the actual path for the history file is stored in the reader and not the history object, right?  Honestly, in thinking about this, are you sure that was the right decision.   The file for history really seems like it should be instance data of the actual history object itself.  Especially since one might swap out a custom history object that doesn't even use a file!  If a reader can have 1 or more history instances, then it seems like each history instance should contain its own data regarding how it stores the history internally.

I just added it yesterday

Ooh, nice.  I'll check it out.  


Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Brad Wood

unread,
Dec 21, 2017, 2:01:08 PM12/21/17
to jline...@googlegroups.com
An update-- I was able to get the history file working.  Turns out I had a typo that was setting the variable Map key to the text "HISTORY_FILE" instead of the actual value of the constant.  My history stuff is back in business now.  Thanks!

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Guillaume Nodet

unread,
Dec 21, 2017, 3:07:24 PM12/21/17
to jline...@googlegroups.com
On Thu, Dec 21, 2017 at 7:14 PM, Brad Wood <bdw...@gmail.com> wrote:
Thanks for the additional answers.

Regarding the completors and only handing back the last "word" in the line, how should I determine what the last word is?  I'm not using JLine's parsing functionality at all.  CommandBox has its own parser based on our own rules for tokenizing strings and dealing with escape characters, etc.  I don't know that my definition of the last token in the line is the same as what JLine is expecting.  Is it safe to say that any chunks of text that don't have spaces such as force=true should be considered one "word" and always have the whole bit added as a candidate?

Not completely.  If you have a parser, I'd suggest you wrap it into the JLine parser and give it instead of the default jline's one.  This will also give you the possibility to hack a correct Highlighter.
If you provide your own Parser, the Completers have to be related somehow, i.e. the candidate must be the full replacement of the full word being completed.
 

I suggest we deal with those once the main completion issue is over.

That's fine.  I really like the new completor stuff.  It's slick and a much better UI so good work on that.  I like where I think some of those extra options are going so I want to make sure I take advantage of everything I can.

Ok, the value is the raw string that will be added to the buffer, the display is the optionaly colored string that will be displayed to the user, the help will be displayed put in parentheses (padded right).
The key is to actually display items in the same line.  For example, I use them to prefix commands, i.e. "break" and "gogo:break" if you run the jline demo.  Those are 2 candidates, but displayed on the same line, because they actually refer to the same command.
The group when candidates are displayed to the user.  The candidates are grouped by their "group", unless the "no-group" or "no-auto-group" option is set.  You can see it in the demo if you type "cr<tab>".  Given "cr" is no candidate beginning, the typo stuff will kick in and 2 groups will be displayed ("original" and "others").  
But you could define your own groups instead of those pseudo-generated ones to display candidates in a smarter way: for example you could extract previous completions from history and display them in a group and standard candidates in another one.
The "suffix" is used during completion of partial candidates, i.e. candidates where "complete" = false, which is the case for file/directories completers. Depending on the options, the suffix can be automatically added by default (when in a directory while completing a file name), and automatically removed if the user hits something else (for example a space).
I think I briefly covered them all...
If you plan to experiment, feel free to provide a javadoc PR ;-)
 

So, about that pasing, I actually have some questions about the parser stuff built into JLine that I haven't had a chance to look at yet.  I want to make sure that JLine's parser doesn't get in the way and try to eat any escape characters or anything since I do all that on my own based on CommandBox's parsing rules  Is there a way to just turn off any internal parsng entirely to make sure I always get back the strings exactly as the user typed them?  From my small amount of testing, I've been seeing that JLine seems to be eating by backslashes which is messing up stuff once I try to do my own parsing on the line. 

For instance:

CommandBox> echo brad\wood

should echo out exactly "brad\wood" since "\w" is not a CommandBox escape sequence.  And

CommandBox> echo brad\nwood

should echo out

brad
wood

since "\n" is the CommandBox escape for a new line.  

This parser stuff is cool and I wish it were here 4 years ago when I first started using JLine, but since I've already written all of it myself I need to just be able to turn it all off.

See above, just make sure JLine is using your own parser.
 

Well, one can always call   ((LineReaderImpl) reader).setHistory(myHistory) whenever you want.

But that wouldn't actually do any good since the actual path for the history file is stored in the reader and not the history object, right?  Honestly, in thinking about this, are you sure that was the right decision.   The file for history really seems like it should be instance data of the actual history object itself.  Especially since one might swap out a custom history object that doesn't even use a file!  If a reader can have 1 or more history instances, then it seems like each history instance should contain its own data regarding how it stores the history internally.

The History is pluggable, if you provide your own implementation to provide some contextuality, you don't have to use the HISTORY_FILE variable or any other built-in variable.  The idea was to have the default pluggable in the context of a shell, so using a session variable, so the fact that the variable name is defined in the LineReader interface is more for documentation than anything else.

Brad Wood

unread,
Dec 21, 2017, 3:46:20 PM12/21/17
to jline...@googlegroups.com
Does JLine apply the parsing at the time the line is read?  If so, I don't think I want to go down the road of trying to warp my parser into JLine's.  That would be a non-trivial rewrite of CommandBox at this point. The tokenizing and parsing of the line happens later in the flow and in a completely different part of the app than where the read line happens.  CommandBox already has its own implementation of variable expansions like

CommandBox> echo ${os.version}

Recursive parsing of nested commands like

CommandBox> echo `cat file.txt`

and special rules on piping data into a command

CommandBox> echo myFile.txt | cat | grep foo > out.log

I even have special scenarios where parsing needs to not even happen for special commands where I simply pass the input along to another source like where I allow users to run commands against their native OS's shell like so:

CommandBox> !java -version

What I really need is for the readLine() method to just give me exactly what the user typed and nothing else.  What would it take to get that behavior back?

If you plan to experiment, feel free to provide a javadoc PR

I will try.  I experimented with the group stuff last night, but it didn't seem to work at all.  For instance, if I type this and hit tab

CommandBox> rm
--force                 bar/                    cborm-shell/            lib/                    server-adobe.json       test.cfc
--recurse               box.exe                 coldbox-platform/       modules/                server.json             testbox/
.env                    box.jar                 commandbox.properties   myConfig.json           server.json.MD5
.git/                   box.json                foo/                    myFile.cfm              tailme.txt
JRE/                    box.l4j.ini             force=                  path=                   task.cfc
JRE_9/                  bradtest/               index.cfm               recurse=                test.bat

You can see the suggestions range from folder names, parameter names, boolean flags (like --force) and file names.  I'd love to be able to organize those in the completion output, but when I tried passing a group of "flags" for the flags and "non-flags" for everything else as an experiment, this is all I see:

CommandBox> rm _
flags
non-flags
--force                 bar/                    cborm-shell/            lib/                    server-adobe.json       test.cfc
--recurse               box.exe                 coldbox-platform/       modules/                server.json             testbox/
.env                    box.jar                 commandbox.properties   myConfig.json           server.json.MD5
.git/                   box.json                foo/                    myFile.cfm              tailme.txt
JRE/                    box.l4j.ini             force=                  path=                   task.cfc
JRE_9/                  bradtest/               index.cfm               recurse=                test.bat

You can see that the group names "flags" and "non-flags" appear at the start of the list but the actual candidates haven't changed at all, nor are they nested under their group like I thought they would be.  Am I misunderstanding how groups work?

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Brad Wood

unread,
Dec 26, 2017, 1:55:32 PM12/26/17
to jline-users
Any feedback on these items?

What I really need is for the readLine() method to just give me exactly what the user typed and nothing else.  What would it take to get that behavior back?

I've mostly worked around the issues above by adjusting the completor logic and setting the escape chars to null in the parser, but I'm still afraid the new parsing stuff is going to cause some issues down the road.  

and...

Guillaume Nodet

unread,
Dec 26, 2017, 2:55:59 PM12/26/17
to jline...@googlegroups.com
On Thu, Dec 21, 2017 at 9:45 PM, Brad Wood <bdw...@gmail.com> wrote:
Does JLine apply the parsing at the time the line is read?  If so, I don't think I want to go down the road of trying to warp my parser into JLine's.  That would be a non-trivial rewrite of CommandBox at this point. The tokenizing and parsing of the line happens later in the flow and in a completely different part of the app than where the read line happens.  CommandBox already has its own implementation of variable expansions like

CommandBox> echo ${os.version}

Recursive parsing of nested commands like

CommandBox> echo `cat file.txt`

and special rules on piping data into a command

CommandBox> echo myFile.txt | cat | grep foo > out.log

I even have special scenarios where parsing needs to not even happen for special commands where I simply pass the input along to another source like where I allow users to run commands against their native OS's shell like so:

CommandBox> !java -version

What I really need is for the readLine() method to just give me exactly what the user typed and nothing else.  What would it take to get that behavior back?

But you have the full line in ParsedLine#line()

Brad Wood

unread,
Dec 26, 2017, 3:07:22 PM12/26/17
to jline...@googlegroups.com
Sorry, I shouldn't have mentioned the completor there.  I'm just talking about what gets returned directly from the reader's readLine() method.  It appears to not be exactly what the user passed in, but a string that's already gone through some sort of parsing.  I had an example earlier where typing "foo\bar" at the prompt returns the string "foobar"  Interestingly enough, in addition to setting escape keys to null in the parser, setting DISABLE_EVENT_EXPANSION also seems to change this behavior.  I'm not sure why, but it underscores the point-- I don't want JLine touching my user input.  I just need it handed directly to me and I'll take care of all the parsing, expansion, escaping, etc that needs to happen.  

Also, feedback on the tab completion "group" functionality not seeming to work?

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Guillaume Nodet

unread,
Dec 26, 2017, 3:26:28 PM12/26/17
to jline...@googlegroups.com
I think you'd better implement a Parser for JLine, especially if you have already one.
JLine will behave "weird" when editing a command line where the Parsers do not match.
For example if you have a quote started but not finished, and if that's acceptable in your own parser, you will have a problem, as the LineReader's parser will ask for a closing quote.

You're right that DISABLE_EVENT_EXPANSION will prevent escaping.

To access the raw input, you can use 
   lineReader.getParsedLine().getLine()
Have a look at the Karaf code using this exact method:

Not sure about the "group" stuff.  Could you set up a small example maybe ?

Cheers,
Guillaume

Brad Wood

unread,
Dec 26, 2017, 4:39:16 PM12/26/17
to jline...@googlegroups.com
I think you'd better implement a Parser for JLine, especially if you have already one.

I think you'd have to agree that it's ironic that when I've asked for the ability for JLine to tell me when an up arrow has been pressed, you've said that's not something JLine should do and I need to implement it myself, yet in this case I just need JLine to read a line and you're pushing me to use an entire inbuilt parsing system in JLine that I don't need :)  

I can look into wiring up my parser, but I'm sure both of our implementations probably follow different lines of thought based on what we needed for our different projects.  The resolution in CommandBox is split up into the tokenization of the input string (turning a string into an array of "words" as you call them) and then there's the resolution of a command chain.

echo text="foo" > file.txt && cat file.txt | grep foo | sed s/foo/bar/ | more

That's a valid input to the commandbox parser.  It will generate 15 tokens, which will parse into a command chain of 6 separate commands, each with their own set of arguments, some of them positional and some of them named.  I'm thinking maybe I only need to worry about the first tokenization step, but as there's basically no javadocs on the parser classes, I'm just guessing as to how they are used.

Could you set up a small example maybe ?

Probably, but it would be in CFML.  Would that be useful to you?  


Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Guillaume Nodet

unread,
Dec 26, 2017, 6:06:03 PM12/26/17
to jline...@googlegroups.com
On Tue, Dec 26, 2017 at 10:38 PM, Brad Wood <bdw...@gmail.com> wrote:
I think you'd better implement a Parser for JLine, especially if you have already one.

I think you'd have to agree that it's ironic that when I've asked for the ability for JLine to tell me when an up arrow has been pressed, you've said that's not something JLine should do and I need to implement it myself, yet in this case I just need JLine to read a line and you're pushing me to use an entire inbuilt parsing system in JLine that I don't need :)  

Yeah, because JLine aims at providing support for terminals, not only windows console apps ;-)  One of the requirement is that it works on remote terminals, where there's no local keyboard, hence no concept of arrow keys...

You don't have to write your parser if you don't want/need to.  I don't you how you parser works, but for advanced integration, i'm just warning you that if the tokenizer is different, some behavior might be unexpected.  It was the same with JLine2, but you had no way to fix the tokenizer. If the JLine2 behavior was sufficient, and if you don't want to leverage more advanced stuff like the highlighter, you may be fine with the default parser.
 

I can look into wiring up my parser, but I'm sure both of our implementations probably follow different lines of thought based on what we needed for our different projects.  The resolution in CommandBox is split up into the tokenization of the input string (turning a string into an array of "words" as you call them) and then there's the resolution of a command chain.

echo text="foo" > file.txt && cat file.txt | grep foo | sed s/foo/bar/ | more

That's a valid input to the commandbox parser.  It will generate 15 tokens, which will parse into a command chain of 6 separate commands, each with their own set of arguments, some of them positional and some of them named.  I'm thinking maybe I only need to worry about the first tokenization step, but as there's basically no javadocs on the parser classes, I'm just guessing as to how they are used.

Yeah, I have the same kind of parser in the main jline demo.  That's the kind of parsing that may require advanced setup.  For example, jline validates the syntax.  I suppose "echo |" is not a valid command in your case.  If you write the Parser correctly, JLine will help the user and will not validate incomplete lines.   But again, you may not *need* those features.
 
Well, actually, just run the jline demo and cat your example (but from sed which is not implemented) and replacing more by less:

echo text="foo" > file.txt && cat file.txt | grep foo | less


The above is what is displayed to the user, so you have nice syntax highlighting, which does require some level of smart parsing in order to tokenize the input string correctly and some level of semantic analysis of each command.  The completers obviously have to complete the commands, even when not at the beginning of the line.   
To achieve that, you can add a lot of smart code in the completer.  What I'm proposing is slightly different: put some intelligence in the parser which is reused by the highlighter and completion system.  The completers thus do not need to understand the full line, just part of it.  For example, in the above, if you go to the "grep" token, remove the two end characters and type <tab>, it will complete correctly, even if the completer has no knowledge of the full line, because when called during completion, the parser will return a partial line "gre foo" instead of the full line.  This allows completer to only deal with a single command line and avoid redirections / pipes support.

But again, you don't have to do it that way.  You can use a dumb parser and put the intelligence in your completers.


Could you set up a small example maybe ?

Probably, but it would be in CFML.  Would that be useful to you?  

As long as you tell me how to run / debug your example, that could be ok.

Brad Wood

unread,
Dec 26, 2017, 7:23:31 PM12/26/17
to jline...@googlegroups.com
Ok, so it was a bit of a pain, but I have a parser in place that just delegates to my CommandBox tokenizer and uses the array of words to build an DefaultParser.ArgumentList instance which seems to be working.  At first, it totally screwed up my completor, but then I realized that the default parser parses a string with a trailing space like "foo " as TWO words.  [ "foo", "" ]  I didn't see that coming and it's certainly not how my tokenizer behaves but I was able to work around it.

Now, about this highlighting-- how do I tap into that?  I don't have anything that highlights as I type.  Do I need to enable something in the shell?  I didn't have highlighting that worked even before when I was using the defaultparser, even though I think it had rules that were fairly similar to my parser.  Looking at the default highlight, I'm not even sure what it's doing.  it appears to be looking at a search term and a region active thing, but neither of those sound like anything that I'm doing with CommandBox.  In your previous message you seemed to show that the names of commands were highlighted, but I'm not seeing how the defaulthighlighter class would do anything like that.

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Brad Wood

unread,
Dec 27, 2017, 12:43:36 AM12/27/17
to jline...@googlegroups.com
So I poked around at the highlighter stuff.  I don't have a clue what the default highlighter does, but I was able to write my own custom highlighter that works in tandem with my command parser to highlight the command portions of my input buffer.  This is actually pretty sweet. Adds a nice touch for sure.  



Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Reply all
Reply to author
Forward
0 new messages