history, sourced script problem.

[R.Wieser]

Hello all,

I've got a problem with running the "history" command in a sourced
(preloaded) function:

- - - - - - - - - -
histtest() {
history -c
history -s "foobar"
}
histtest
- - - - - - - - - -

When I source this script the history gets cleared and the "foobar" entry is
added to it. IOW, it works as expected.

However, when I than afterwards just type "histtest" on the commandline the
history is cleared, but the "foobar" entry isn't added.

Question: what causes the difference and how do I fix it ?

Remark: the trouble seems to come from clearing the history. When I remove
that line the "foobar" entry is, in both cases, added as expected.

Regards,
Rudy Wieser

[Benjamin Esham]

R.Wieser wrote:

> Hello all,
>
> I've got a problem with running the "history" command in a sourced
> (preloaded) function:
>

> [snip]

Hi Rudy,

I think we'll need to know which shell you're using, and which version, in
order to give you a good answer.

If you're using Bash--what is $HISTCONTROL set to?

Benjamin

[Ben Bacarisse]

To more things along...

$ bash -norc -noprofile
bash-5.2$ cat x

histtest() {
history -c
history -s "foobar"
}
histtest

bash-5.2$ . ./x
bash-5.2$ history
1 foobar
2 history
bash-5.2$ histtest
bash-5.2$ history
1 history
bash-5.2$ bash --version
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
bash-5.2$ echo $HISTCONTROL

bash-5.2$

--
Ben.

[R.Wieser]

Benjamin,

> If you're using Bash-

Indeed I am.

> -what is $HISTCONTROL set to?

Its currently set to

"ignorespace:erasedups"

, Which is a setting I applied myself.

By the way, if its relevant (thanks ben),

$ bash --version

shows

"GNU bash, version 5.1.4(1)"

Regards,
Rudy Wieser

[R.Wieser]

Ben

> To more things along...
...

> bash-5.2$ . ./x
> bash-5.2$ history
> 1 foobar
> 2 history
> bash-5.2$ histtest
> bash-5.2$ history
> 1 history

Yes, thats what I'm getting too. And as far as I can tell, should not get.

Than again, my knowledge of Linux and bash is rather minimal. Perhaps there
is a good reason why it behaves like that, and there is a setting to gouvern
it ?

Regards,
Rudy Wieser

[marrgol]

I'm just guessing but I think it might be related to _when_ the clearing
is actually done and that history list stores command _lines_ but one
line can contain multiple commands. E.g.:

$ history -c
$ history -s foo
$ history
1 foo
2 history

is what you'd expect, but

$ echo zzz ; history -c ; history -s foo
zzz
$ history
1 history

probably is not, unless clearing history is by design performed after
the whole line is interpreted/executed.

I don't know if it can be fixed (or if there's anything to be fixed)
but since sourcing works as expected, why not just utilize it?

$ cat /path/file

history -c
history -s foobar

histtest () { . /path/file ; }
$ . /path/file

$ history
1 foobar
2 history

$ histtest

$ history
1 foobar
2 history

$

HTH

--
mrg

[R.Wieser]

marrgol,

> I'm just guessing but I think it might be related to _when_ the clearing
> is actually done

Yes, that was what I was thinking too. But thats the strange thing : why
does, in this case, clearing behave differently than adding ?

> $ echo zzz ; history -c ; history -s foo
> zzz
> $ history
> 1 history
>
> probably is not,

No, its not. Than again, I'm not sure if the above (sub) commands are
executed in sequence or if its done in parallel - in which case I could
imagine a race condition.

> unless clearing history is by design performed after the whole line is
> interpreted/executed.

The same went thru my mind too. But in that case, why ? Such out-of-order
behaviour causes the very, unexpected, problem I bumped into.

> I don't know if it can be fixed (or if there's anything to be fixed)

Currently I will settle for knowing /why/ it happens - so that I won't run
into it again when trying to use some other command/program instead of
"history".

> but since sourcing works as expected, why not just utilize it?

[snip]

I just tried it, and it does give the expected outcome. Thanks for that
(generic usable!) solution.

Its odd though : when running the script thru the "histtest" invocation the
script is /already/ running in the main shells environment, and sourcing it
there should not even work/make any difference (and I therefore didn't try
it). But here I am, looking at it and seeing that it /does/ make a
difference ...

... almost as if "sourcing" a script there makes it run in some special
environment/subshell.

Thanks for the response and solution. Its has been irking me for weeks now.
:-\

Regards,
Rudy Wieser

[Benjamin Esham]

R.Wieser wrote:

> Hello all,
>
> I've got a problem with running the "history" command in a sourced
> (preloaded) function:
>
> - - - - - - - - - -
> histtest() {
> history -c
> history -s "foobar"
> }
> histtest
> - - - - - - - - - -
>
> When I source this script the history gets cleared and the "foobar" entry
> is added to it. IOW, it works as expected.
>
> However, when I than afterwards just type "histtest" on the commandline
> the history is cleared, but the "foobar" entry isn't added.

The Bash manual seems misleading about how the "history" command works. It
claims that the -c option "may be combined with the other options to replace
the history list completely," but if you try the obvious thing,

history -c -s foobar

the history is simply cleared and no "foobar" entry is present afterward.

I did find a solution that seems to work with Bash 5.2.15(1)-release:

histtest() {
history -s foobar
history -d 1--2
}

This inverts the steps in your version of histtest: first we add the
"foobar" entry, and then we remove all of the entries except for the most
recent one. "1--2" may look like a typo, but it's a range of two history
entries: from entry 1 (the earliest entry in the list) to entry -2 (the
second to last entry in the list). The -d option accepts a hyphen-separated
range, and it also accepts negative numbers for counting back from the end
of the list, so in a case like this you end up with two consecutive hyphens.

I noticed that if the function only runs "history -s foobar", and does not
delete any history entries, the *invocation* of the function is nevertheless
excluded from the history listing. In other words, if you run the three
commands

histtest2() { history -s foobar2; }
histtest2
history

then the history listing will include "foobar2" but it will not show that
you ran "histtest2"! I'm not a Bash user, but I am inclined to agree with
you that Bash's handling of history within functions seems inconsistent with
how it works otherwise.

Hope this helps,

Benjamin

[R.Wieser]

Benjamin,

> The Bash manual seems misleading about how the "history" command works.
> It claims that the -c option "may be combined with the other options to
> replace the history list completely," but if you try the obvious thing,
>
> history -c -s foobar
>
> the history is simply cleared and no "foobar" entry is present afterward.

There is a (good) possibility that its not the history command itself thats
to blame, but the underlaying bash (or OS).

> I did find a solution that seems to work with Bash 5.2.15(1)-release:
>
> histtest() {
> history -s foobar
> history -d 1--2
> }

I was also thinking of that (did first try to replace the "-c" with a
full-range "-d" to see if it would make a difference), but didn't really
want to go for "circumvent the problem" hacks. Besides having to keep track
of how many "history -s" additions I would be doing to be able to remove the
rest.

In that regard I already I already found a "solution" to the problem by
echo-ing the "history -s" parts into a file, and than load that file after I
cleared the history. It works well enough, but I don't like it.

<following a brainfart and doing some checking>

Funny : that "history -d 1--2" causes a quite unexpected behaviour :

> - - - - - - - - - -
> histtest() {

> history -d 1--2
> history -s "foobar"
> history -s "barfoo"
> history -s "more stuff"

> }
> histtest
> - - - - - - - - - -

Sourcing this script causes the history to end up with four lines, the
oldest one being the command sourcing the script.

However, if I than afterwards execute "histstest" I get just the
"history -s" lines. <whut? why don't they disappear now ?>

If I would be a fan of hacks that won't break down when the underlaying
problem is fixed than I would probably use this one. :-)

Hmmm... Somehow I get the feeling that some lazy memory allocation (new
storage for the history) is involved.

> I noticed that if the function only runs "history -s foobar", and does
> not delete any history entries, the *invocation* of the function is
> nevertheless excluded from the history listing. In other words, if you
> run the three commands
>
> histtest2() { history -s foobar2; }
> histtest2
> history
>
> then the history listing will include "foobar2" but it will not show
> that you ran "histtest2"!

IIRC, that is defined behaviour : only commands started from the commandline
(typed by the user!) are remembered in the history.

And I don't think you really want to have it remember everything - just take
a peek at the output of "declare -F" (or "declare -f" if you want to see the
contentst too) to see how may scripts bash has predefined. Imagine yourself
having to wade thru a sh*tload of bash commands that you can't remember
having ever typed (because you didn't) ...

Regards,
Rudy Wieser

[R.Wieser]

Update :

I just tried the below and it works the same way, showing just the three
"-s" lines after the "-c", either by sourcing it or (afterwards) just
executing the function :

- - - - - - - - - -
histtest() {

history -s dummy
history -c

history -s foobar
history -s barfoo
history -s more stuff
}
histtest
- - - - - - - - - -

FWI : I started with "history 1--2", but that throws an error when the
history is empty - otherwise an "out of range" error is thrown and nothing
is added.

I'm back to being baffled, both in regard to the result as well as what
might be causing it.

Regards,
Rudy Wieser

[Benjamin Esham]

R.Wieser wrote:

> Benjamin,
>
>> The Bash manual seems misleading about how the "history" command works.
>> It claims that the -c option "may be combined with the other options to
>> replace the history list completely," but if you try the obvious thing,
>>
>> history -c -s foobar
>>
>> the history is simply cleared and no "foobar" entry is present afterward.
>
> There is a (good) possibility that its not the history command itself
> thats to blame, but the underlaying bash (or OS).

The history command is part of Bash, so I'm not sure that that distinction
is meaningful.

>> I did find a solution that seems to work with Bash 5.2.15(1)-release:
>>
>> histtest() {
>> history -s foobar
>> history -d 1--2
>> }
>
> I was also thinking of that (did first try to replace the "-c" with a
> full-range "-d" to see if it would make a difference), but didn't really
> want to go for "circumvent the problem" hacks. Besides having to keep
> track of how many "history -s" additions I would be doing to be able to
> remove the rest.

I don't know how complicated your real version of histtest is, but you might
consider creating an array of items that should be added to the history.
Then you can have Bash itself calculate how many history entries need to be
removed. You could even check the current size of the history so that you
don't try to remove any entries when there aren't any entries to remove.

> In that regard I already I already found a "solution" to the problem by
> echo-ing the "history -s" parts into a file, and than load that file after
> I cleared the history. It works well enough, but I don't like it.

I think you could make an alias that writes your commands to a (temporary)
history file and then starts a separate instance of Bash that uses that file
for its history. I'm not sure I'd consider that a hack or anything; it seems
like your use case is pretty different from the typical use of Bash's
history mechanism, so it shouldn't be surprising that not everything you
need is already built in.

>> I noticed that if the function only runs "history -s foobar", and does
>> not delete any history entries, the *invocation* of the function is
>> nevertheless excluded from the history listing. In other words, if you
>> run the three commands
>>
>> histtest2() { history -s foobar2; }
>> histtest2
>> history
>>
>> then the history listing will include "foobar2" but it will not show
>> that you ran "histtest2"!
>
> IIRC, that is defined behaviour : only commands started from the
> commandline (typed by the user!) are remembered in the history.

My point is that the user *did* type in "histtest2" and yet that command
does not appear in the history.

Benjamin

[Ben Bacarisse]

This is very odd. Just having one -s before the -c makes things work as
expected (for me):

bash-5.2$ cat x2

histtest() {
history -s dummy
history -c
history -s foobar
}

histtest

but commenting out that "history -s dummy" line and we are back to the
original odd behaviour. Replacing that line with a plain "history"
command does not help either.

--
Ben.

[R.Wieser]

Benjamin,

> I don't know how complicated your real version of histtest is,

It currently(!) isn't.

> but you might consider creating an array of items that should be
> added to the history.

Thats one of the possibilities, yes. Incrementing a counter every time that
"history -s" is executed would be another.

> My point is that the user *did* type in "histtest2" and yet that
> command does not appear in the history.

My apologies, I was assuming that those three lines where inside a script
too.

It might have something to do with "histtest2" being a preloaded function.

Regards,
Rudy Wieser

[R.Wieser]

Ben,

> This is very odd.

Indeed.

> Just having one -s before the -c makes things work as expected (for
> me):

One or more. Though having more that one is rather useless ofcourse.

> but commenting out that "history -s dummy" line and we are back to
> the original odd behaviour.

And back to the initial example. :-)

Regards,
Rudy Wieser