Improvement: allow help and completion for nspace prefix

[Ingo Albrecht]

This patch series enables full completion and help for the prefix of an
nspace, another feature from my backlog.

It also happens to be another use of my previously submitted
clish_view_insert_command patch, so this has a new and fixed version of
that as well.

There also is a corresponding view_remove_command function now.

Please advise if this breaks anything subtle.

Greetings
prom

[Serj Kalichev]

Prefix of a namespace is not constant string. It's a regular expression.
For example we use "prefix" for numeric prefix for multiple commands
with the same name. The universal completion is not possible.

There are cases when prefix is really constant string ("do" for
example). In this cases we create special empty command "do" with help.
This command can't be executed because it's empty but it can be
completed and there is a help for it.


10.10.2020 18:12, Ingo Albrecht пишет:

[Ingo Albrecht]

On 10/12/20 12:34 PM, Serj Kalichev wrote:
> Prefix of a namespace is not constant string. It's a regular expression.
> For example we use "prefix" for numeric prefix for multiple commands
> with the same name. The universal completion is not possible.

Okay! I can see how that won't work.

> There are cases when prefix is really constant string ("do" for
> example). In this cases we create special empty command "do" with help.
> This command can't be executed because it's empty but it can be
> completed and there is a help for it.

Ah! Interesting. I did not notice before that this is done in the
examples. Will migrate my modules to this pattern.

This kind of definition is a bit hard to spot and somewhat subtle in its
semantics. Would have trouble explaining this to library users, but I
suppose that this can be left to eventual documentation and guides.

Greetings
prom

[Ingo Albrecht]

While we are talking about this kind of command ("prefixed alias"):

There is another type of definition that I have used in the past. I
would describe it as a "prefixed wrapper". The difference to a prefixed
alias is that this type of command has its own action and does not
execute the action of its inner command.

The use for this is command help. I would like to define a "help" prefix
command that will complete all available commands in the active view,
receiving the command as a parameter to the "help" action. This command
should not complete the parameters of the wrapped command however.

This is not a high-priority item for me, but I did have a partial
implementation of it based on clish_command_new_link. Not sure how well
it worked.

Imagine a view "main" with commands "show file" and "show environment":

> help
Available commands:
show file - Show information about files
show environment - Show information about environment
> help show
Available commands starting with "show":

show file - Show information about files
show environment - Show information about environment
> help show file
Command "show file":
<name> [file] File to show
>

(This output is freely invented and simplified. It supposes that "help"
is a command and can interpret its input using clish data structures.
This could possibly be implemented with a python action.)

Do you have an idea if this can be represented in clish at the moment?

[Serj Kalichev]

Firstly thanks for your work!

I have read your letters and I am very glad that some your thoughts
about klish improvements duplicate my thoughts about it or may be my
thoughts duplicate yours :). Probably it means that our way is a right way.

Let me to explain a current situation with klish development. Company
where I'm working uses klish in some projects so some time ago I could
spend much time to develop klish. Now company is interested in klish
improvements but this work has low priority and I'm working on another
project. It means that my answers and changes can be not so fast. But
I'm interested in klish and will work on it.

Some time ago I wanted to make total refactoring of klish. It's based on
"clish" project and has good and bad architecture features inherited
from this project. Many architecture decisions is good for rather simple
clish but not good for more complex klish. I begun the refactoring and
relalized that It will be better to fully rewrite klish instead
refactoring. So now master branch is for new klish version but 2.2
branch is for stable version.

I discussed new klish architecture with coworkers and writed a document.
But we use our native language. Firstly I rewrited a library (lub dir in
sourcecode tree). Now it's separate library named "faux". It's fully
rewritten "lub" library with additional modules.
http://libcode.org/projects/faux/. I'm telling about it because new
klish is better place for new features and changes than old one. But it
doesn't exist yet...

It's a bit of pity to waste energy to implement complex features with
old klish. So I'm not sure what the right way for klish development now
- change existing stable branch or delay new features and develop new klish.

Probably the right way is to apply simple features to stable klish and
leave complex features for future renewed klish.



13.10.2020 05:51, Ingo Albrecht пишет:

[Serj Kalichev]

I'm not sure but I think your way to implement help is too complex. I
think better way is to write special function in C to output this help.
Additionally it can use DETAIL tag. This function can be put to standard
plugin. Plugin has access to all klish internals so it will be rather
easy to implement it.

And I don't know another usefull task for "prefixed wrapper" excluding help.

13.10.2020 05:51, Ingo Albrecht пишет:

[Ingo Albrecht]

On 10/13/20 2:59 PM, Serj Kalichev wrote:
> I'm not sure but I think your way to implement help is too complex. I
> think better way is to write special function in C to output this help.
> Additionally it can use DETAIL tag. This function can be put to standard
> plugin. Plugin has access to all klish internals so it will be rather
> easy to implement it.

The thought behind this was to use this command with a python action
that uses binding access to clish data structures to resolve commands
and print verbose interactive help on them.

A baseline version of the same feature can for sure be implemented in C,
and I would welcome such an alternative. The goal of all this was to
have a "help" command that is more "helpful". I only have partial python
code to enable that.

Some inspiration for this comes from OpenVMS, which has a shell (DCL)
that can be extended using CDL (command definition language). These
command definitions are bound to a HELP library, enabling excellent
interactive help and manual access.

At some point I would like to emulate that behaviour. I might be able to
record a terminal video to show you exactly what I mean.

> And I don't know another usefull task for "prefixed wrapper" excluding
> help.
>

In theory (not in clish for various reasons) this kind of command
is used for many things on UNIX:

* nice
* sudo
* time
* runc
* xargs
* strace

All of these have subcommands that are worth completing. Most of them -
contrary to the "help" command - actually execute their inner command.

In practice I don't see a need for any of these things in clish since it
does not try to be a generic shell language.

[Serj Kalichev]

Patches 0001, 0002, 0003, 0005 were applied.
Thanks!

10.10.2020 18:12, Ingo Albrecht пишет:

[Ingo Albrecht]

On 10/13/20 2:50 PM, Serj Kalichev wrote:
> Firstly thanks for your work!
>
> I have read your letters and I am very glad that some your thoughts
> about klish improvements duplicate my thoughts about it or may be my
> thoughts duplicate yours :). Probably it means that our way is a right way.

It is good to finally get rid of my backlog.

I agree with your general direction such as the need for a thorough
refactoring. Most of the enhancement ideas also make sense to me. As an
open-source project klish could also use better documentation.

My overall goal is to make klish a little bit more flexible and general
so that I can use it for future work projects using platforms such as
OpenWRT or buildroot, which I have used before.

All I can promise for now is that I will finish my Python plugin and
make it useful with synthetic use-cases such as my website [1], which
currently runs clish 2.1 with some patches like pager support.

We will see about actual use-cases, but I will likely continue sending
the occasional patch.

> It's a bit of pity to waste energy to implement complex features with
> old klish. So I'm not sure what the right way for klish development now
> - change existing stable branch or delay new features and develop new
> klish.
>
> Probably the right way is to apply simple features to stable klish and
> leave complex features for future renewed klish.

It is good to know of your plans.

I'm mostly out of "crazy idea" patches that may be tricky to port. There
is still a patch for pager support that I want to get in somehow, but
that is useful enough to be worth porting forward.

Luckily my bigger work is all plugins which should be relatively easy to
port since they don't do anything esoteric.

I will take a look at libfaux and your branch when I find some time.

Greetings
prom


[1] https://promovicz.org/

[Serj Kalichev]

The pager is very interesting feature.


14.10.2020 13:22, Ingo Albrecht пишет:

[Ingo Albrecht]

On 10/14/20 11:05 AM, Ingo Albrecht wrote:
> Some inspiration for this comes from OpenVMS, which has a shell (DCL)
> that can be extended using CDL (command definition language). These
> command definitions are bound to a HELP library, enabling excellent
> interactive help and manual access.
>
> At some point I would like to emulate that behaviour. I might be able to
> record a terminal video to show you exactly what I mean.
>

Here you have a quick terminal recording of OpenVMS help in action:

https://asciinema.org/a/Bo6XYJCPo69R9XZm6QsfEt7AV

I think it's a very nice system and could fit well into clish. Maybe
something like this will be a possibility in the future.

Greetings
prom

[Ingo Albrecht]

On 10/15/20 2:45 PM, Ingo Albrecht wrote:
> On 10/14/20 11:05 AM, Ingo Albrecht wrote:
>> Some inspiration for this comes from OpenVMS, which has a shell (DCL)
>> that can be extended using CDL (command definition language). These
>> command definitions are bound to a HELP library, enabling excellent
>> interactive help and manual access.
>>
>> At some point I would like to emulate that behaviour. I might be able to
>> record a terminal video to show you exactly what I mean.
>>
>
> Here you have a quick terminal recording of OpenVMS help in action:
>

That was obviously the wrong link. Here is the right one:

https://asciinema.org/a/365597

[Serj Kalichev]

Good help. Good help levels and pager.

Do you like interactive prompt like this?
"
SHOW MEMORY subtopic?
SHOW subtopic?
Topic?
"
It's annoying for me. But it's not a problem because it can be an
optional feature.

I think that tags like DETAIL (or maybe "DOC") can help to organize such
structure. The COMMAND, PARAM can have DETAIL sub-tag with description.

About "prefixed wrapper". Probably now I undestand the idea of "prefixed
wrapper" but in my opinion it's not suitable way to implement
OpenVMS-like help. I'll try to explain... Help is based on command and
parameters. But wrapper allows to enter commands like:
$ help some-command 123 987 555
I mean user can enter values. Not parameter's names but values. What
does help show on such command? Does OpenVMS allow to enter strings like
this?  Does OpenVMS have unnamed parameters? I saw parameters like
"/ALL", "/FILES", "/FULL". But what about numeric parameters (for
example)? How help will be showed for it?


15.10.2020 15:45, Ingo Albrecht пишет:

[Ingo Albrecht]

On 10/15/20 7:07 PM, Serj Kalichev wrote:
> Good help. Good help levels and pager.
>
> Do you like interactive prompt like this?
> "
> SHOW MEMORY subtopic?
> SHOW subtopic?
> Topic?
> "
> It's annoying for me. But it's not a problem because it can be an
> optional feature.

Yeah, that feature is not very modern. It made more sense on teletypes.

Today I would just let the user quit fast and re-enter help fast. That
would achieve pretty much the same thing.

> I think that tags like DETAIL (or maybe "DOC") can help to organize such
> structure. The COMMAND, PARAM can have DETAIL sub-tag with description.

Yes, something like that. The existing help text might work for a
simpler version or as a fallback.

The new form of nested prefix commands might pose a bit of a formatting
challenge. I don't like the new syntax so much because it is a bit
verbose. But it's a matter of taste.

> About "prefixed wrapper". Probably now I undestand the idea of "prefixed
> wrapper" but in my opinion it's not suitable way to implement
> OpenVMS-like help. I'll try to explain... Help is based on command and
> parameters. But wrapper allows to enter commands like:
> $ help some-command 123 987 555

My implementation was ill-advised. It's an old patch. I just wanted to
point out the nice possibility so we can maybe have it in the future. It
would have to be done some other way.

> I mean user can enter values. Not parameter's names but values. What
> does help show on such command? Does OpenVMS allow to enter strings like
> this?  Does OpenVMS have unnamed parameters? I saw parameters like
> "/ALL", "/FILES", "/FULL". But what about numeric parameters (for
> example)? How help will be showed for it?

I think that help commands and help topics for every single parameter
aren't really necessary.

Sure, they would make for even more help content, but having just a
command description with a parameter listing would work for me since
they can also be discovered using the existing incremental completion.

Unprefixed parameters do not have help topics on OpenVMS. They are only
explained on the command page as "command-line pseudo-BNF". Such a
syntax can be generated and tabularized for nice display.

[Serj Kalichev]

Hello
Do you have any ideas about following topic?
On one hand I want to get rid of expanding in new klish. The reason is
security. On the other hand I like advanced VARs with PARAMs and ACTIONs
because them can be used as functions. For example completions are
targets for using "functions". I don't know how to implement usage of
such "functions" without unsecure expanding. I have some thoughts about
it but ideas are not good enough.

14.10.2020 13:22, Ingo Albrecht пишет:

[Ingo Albrecht]

On 10/16/20 9:54 AM, Serj Kalichev wrote:
> Hello
> Do you have any ideas about following topic?

I have various ideas about it, but I haven't decided on a final
perspective yet. Sorry for the long mail.

To me this boils down to what clish tries to be. It can either stay a
"command templating engine" or develop into more of a "programming
system" - which really is a programming language in disguise.

By myself I would prefer the "programming" approach. I even implemented
a sort-of clish clone a while back in a language called Dylan [1].
Programmatic solutions are powerful. Dylan has extensible syntax, so
there is no need for XML or escaping. Internally this library is more
flexible than clish and can handle any Chomsky Type-3 language, even if
the language is dynamically defined by code.

But clish currently isn't so flexible, and I believe that is a good
thing too. Being just a command templating engine aligns with UNIX
philosophy: do one thing only and do it well. It would be pragmatic to
keep it this way.

[1]
https://github.com/dylan-lang/command-interface/blob/master/examples/demo/command-interface-demo.dylan

> On one hand I want to get rid of expanding in new klish. The reason is
> security. On the other hand I like advanced VARs with PARAMs and ACTIONs
> because them can be used as functions. For example completions are
> targets for using "functions". I don't know how to implement usage of
> such "functions" without unsecure expanding. I have some thoughts about
> it but ideas are not good enough.
>

== Need for Expansion

The idea of eliminating expansion also crossed my mind. The main need
for it definitely comes from the shell plugin. Unfortunately expansion
like that is hard to avoid in shell. The only way I know of would be to
embed an actual shell - and even then the shell language itself brings
its own related problems (safe quoting, strange characters in filenames).

And then there are the "other uses" such as composing viewid and prompt
as well as certain configuration actions. Those are hard to avoid given
the current structure of clish.

It should be possible to leave expansion off for all but those things
that actually require it. Python and Lua certainly don't need it except
for some special cases (evaluation commands), and even those can be
implemented using PARAMs if we decide on that as a convention. XML
writers will hardly ever need to do such things, so it shouldn't be a
problem that its not discoverable. It can be documented instead.


== Need for Functions and Programming

Relatively recent extensions have introduced several features that tend
towards making "clish XML" a programming language. One is the use of
PTYPE actions for completion. Another is the concept of dynamic vars.

==== Actions as a programming construct

Actions already have several properties of functions. They take
parameters, and through oactions they can also produce output.

The only problem with all that is that clish is missing all the rest of
a programming language. It has neither sequences nor loops, and there is
no mechanism for composing actions. This makes for a bad programming
language.

==== Expansion as a programming language

Using expansion and extending it into a programming language has
happened before. A very prominent example is "make", which is composed
of a "rule language" and an "expression language" as well as the
embedded "shell language".

Some such systems work well and are perfectly fine, but in my experience
programming languages are better if they were intended as such from the
start.

==== Using embedded languages instead

My preferred approach for making clish more "broad" is to allow using
the embedded interpreters as the main programming environment.

These interpreters are maintained and somewhat well-designed, tested in
production and are well-supported as programming environments.

How to go about this is still a partly open question to me. I would
imagine some improved hooking mechanism so that plugins can insert their
own functions into completion and other such extension points.

This approach would give clish the power of a real programming language
for free.

[David Balme]

Sorry I did not understand.

Can you explain via examples?   I admit, it could be that I have not seen some previous conversation in this thread that explains.

--
You received this message because you are subscribed to the Google Groups "klish-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to klish-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/klish-dev/3877b1eb-5808-7752-6119-56bc979661cb%40gmail.com.

[Ingo Albrecht]

I thought about this a little more and have a proposal for how to do
this at an XML level instead of a string level.

Personally I am currently not after this feature, but I wanted to
contribute the idea nonetheless.

Actions are basically functions that take an optional set of params and
produce an output string. Interactive actions do not produce an output
string.

We could therefore allow using actions as string expansion functions
that replace the entire text content of the XML node that they are
contained in.

Let us call this "XML expansion" and the corresponding element "EXPAND".
This element is identical in definition to ACTION apart from the element
name.

This would require messing with the load order a bit, especially if one
wants to use a plugin in expansion. The new load order would have to be:

* load all XML documents (not just one)
* execute XSLT
* process STARTUP and PLUGIN
* execute STARTUP
* execute PLUGIN load
* process EXPAND
* perform XML expansion
* process everything else

All of this implies some refactoring because of how plugins are
currently implemented.

The load process would require a state structure that could be called
clish_load_t or similar.

Splitting the processing order or STARTUP and PLUGIN implies that it is
illegal to use XML expansion there.


An example:

<ACTION>
<EXPAND>
if [ -x `which htop` ]; then
echo htop
else
echo top
fi
</EXPAND>
</ACTION>

This would produce an error or warning:

<ACTION>
echo "Command not allowed here"
<EXPAND>
if [ -x `which htop` ]; then
echo htop
else
echo top
fi
</EXPAND>
</ACTION>

[Ingo Albrecht]

On 10/17/20 5:53 PM, Ingo Albrecht wrote:
>
> An example:
>
> <ACTION>
>   <EXPAND>
>     if [ -x `which htop` ]; then
>       echo htop
>     else
>       echo top
>     fi
>   </EXPAND>
> </ACTION>
>

There could also be a special form of this for the prompt:

<VIEW name="example">
<PROMPT>echo "$(hostname)> "</PROMPT>
</VIEW>

But my preference would be to do something more like this:

<VIEW name="example">
<ACTION builtin="python">
import socket
view.set_prompt("%s> " % socket.hostname())
</ACTION>
</VIEW>

(See my proposal for VIEW actions in upcoming feature list)

[Serj Kalichev]

Hello
We had a long discussion about expanding and shell script security with
my coworker and we have idea how to make shell a bit more secure. We
will use environment variables instead expanding to push parameters to
shell scripts. In this case we don't need a complex escaping. User can
use "$KLISH_PARAM". If user want to get value of VAR from klish he can
write something like that:
<ENV name="ENV_VAR_NAME" value=...
May be
<ENV><ACTION></ACTION></ENV>

About expanding...
The worst case for using expanding is shell script (or another
scripting). It's the case when result of expanding can be executed. When
result of expanding is string only - it's not so insecure. For example
command/param completion is not dangerous.

The possibility to switch off/on expanding is really good. The new klish
must have it.


16.10.2020 18:46, Ingo Albrecht пишет: