Am I infected with a rootkit?

[Jesper Dybdal]

I have a Debian pc functioning as router, firewall, file server, name
server, webserver, ...
It has very recently been upgraded to Bullseye.

On the internal network I have a Windows 10 pc.

A few days after the Debian upgrade, I had the following strange experience:

The windows machine had an ssh connection to the Debian machine (using
PuTTY), logged in as root on the Debian machine.
I then went for a walk with the dog, leaving the ssh session running.
When I came back, I wanted to re-issue some command to the ssh session,
so I pressed up-arrow a few times.

And there in the bash history were 4 lines that I had not written :-(

I am certain that nobody had been in my apartment while I was gone. And
even if they had, nobody with a key to my apartment would dream of
writing things like the 4 lines that I found in the history file.

The 4 lines were:
> md5users
> sp md5users
> sp /x/md5users
> ps /x/md5users
There is no file named "md5users" or directory named "/x" or command
named "sp" on the Debian machine.

I have scanned the Windows machine with two antivirus tools (Windows
defender and Malwarebytes).
I have run chkrootkit, rkhunter, and debsums on the Debian machine.
That did not find anything.

All of the above except chkrootkit were done on the running system, so
they might be influenced by a rootkit.

I have done a more manual check of the files belonging to the kernel
package, in the hope that a rootkit will not find it easy to fool that. 
There were 10 files in /lib/modules/5.10.0-21-amd64 that do not belong
in the current kernel package - I guess that they are leftovers from an
earlier version.  These 10 files do not seem dangerous to me; they are:
> modules.dep
> modules.devname
> modules.symbols.bin
> modules.symbols
> modules.builtin.bin
> modules.alias.bin
> modules.builtin.alias.bin
> modules.softdep
> modules.alias
> modules.dep.bin

Since this happened a couple of weeks ago, there has been no visible
sign of anything wrong.  I am taking care to mount backup disks only
when running from a booted rescue disk.  And I have for the time being
removed the ability of the Windows machine to log in as root on the
Debian machine.

I've tried logging all DNS requests from the Windows machine during a
power-on sequence.  I saw no clearly suspicious names among the
surprisingly many names being looked up.

What can I do?

* Is it probable that somebody can remote control one or both machines? 
Do those 4 lines ring a bell?  What are they all about?

* I would really like to know how this happened.  I consider myself to
be a careful person who does not get hit by viruses and other malware. 
I've had a Windows virus once - because I trusted an install program
from sourceforge.

* Is there a significant risk that the problem came with the Bullseye
upgrade?

* I really don't want to reinstall from scratch.  Not only because I
don't know whether there is a problem on one or both machines, but also
because I have no idea where any infection came from - it could easily
be from something that I would also reinstall.

* I could restore a backup of one or both of the machines.  But I have
no idea how long back I would have to go.  I would not like to go back
to before the Bullseye upgrade, since I would then have to repeat that
upgrade - and it was not quite trouble-free.

* Is there a place where I could download the correct checksums of all
installed files?  Some way to be able to run debsums from a booted
rescue disk, but checking the system on the hard disk against freshly
fetched checksums?

Any suggestions will be much appreciated.

Thanks,
Jesper

--
Jesper Dybdal
https://www.dybdal.dk

[Eduardo M KALINOWSKI]

On 16/04/2023 09:19, Jesper Dybdal wrote:
> And there in the bash history were 4 lines that I had not written :-(
>
> I am certain that nobody had been in my apartment while I was gone. And
> even if they had, nobody with a key to my apartment would dream of
> writing things like the 4 lines that I found in the history file.
>
> The 4 lines were:
>> md5users
>> sp md5users
>> sp /x/md5users
>> ps /x/md5users
> There is no file named "md5users" or directory named "/x" or command
> named "sp" on the Debian machine.

Which shell do you use, and how is it configured? Note that bash by
default does not share history between sessions, so even if someone
logged in as root (via other ssh session) and typed them, they would not
appear in your ssh session.

See https://mywiki.wooledge.org/BashFAQ/088, or wait for Greg to chime
in with details and corrections.


--
Eduardo M KALINOWSKI
edu...@kalinowski.com.br

[Michel Verdier]

Le 16 avril 2023 Jesper Dybdal a écrit :

> I have scanned the Windows machine with two antivirus tools (Windows defender
> and Malwarebytes).

Can you use clamav on windows ?

>> modules.dep
>> modules.devname
>> modules.symbols.bin
>> modules.symbols
>> modules.builtin.bin
>> modules.alias.bin
>> modules.builtin.alias.bin
>> modules.softdep
>> modules.alias
>> modules.dep.bin

These are generated during kernel install. And you can safely remove
/lib/modules/5.10.0-21-amd64 if these are the only files left.

> * Is it probable that somebody can remote control one or both machines?  Do
> those 4 lines ring a bell?  What are they all about?

Perhaps a bot trying to execute some commands. As they do not apply to
debian you debian machine should not be compromised.

> * I would really like to know how this happened.  I consider myself to be a
> careful person who does not get hit by viruses and other malware.  I've had a
> Windows virus once - because I trusted an install program from
> sourceforge.

Malware can be installed via web sites

> * Is there a significant risk that the problem came with the Bullseye upgrade?

no

> * I really don't want to reinstall from scratch.  Not only because I don't
> know whether there is a problem on one or both machines, but also because I
> have no idea where any infection came from - it could easily be from something
> that I would also reinstall.

I think you don't have to. For debian. For windows a full deinstall
without reinstall is the best :)

[Greg Wooledge]

On Sun, Apr 16, 2023 at 02:19:34PM +0200, Jesper Dybdal wrote:
> The windows machine had an ssh connection to the Debian machine (using
> PuTTY), logged in as root on the Debian machine.
> I then went for a walk with the dog, leaving the ssh session running.
> When I came back, I wanted to re-issue some command to the ssh session, so I
> pressed up-arrow a few times.
>
> And there in the bash history were 4 lines that I had not written :-(

I would initially ask "who else lives with you"....

> I am certain that nobody had been in my apartment while I was gone. And even
> if they had, nobody with a key to my apartment would dream of writing things
> like the 4 lines that I found in the history file.
>
> The 4 lines were:
> > md5users
> > sp md5users
> > sp /x/md5users
> > ps /x/md5users
> There is no file named "md5users" or directory named "/x" or command named
> "sp" on the Debian machine.

This certainly sounds like someone walking up to your machine and typing
the commands. Did you see the commands within the PuTTY session, or were
they *only* in the shell history?

> * Is it probable that somebody can remote control one or both machines?  Do
> those 4 lines ring a bell?  What are they all about?

If someone did this with remote access, it would have to be access to
the Windows machine. Nothing that could be done to the Debian machine
would affect the in-memory shell history of a running instance of bash.
Even writing to the /root/.bash_history file wouldn't cause the PuTTY
session's bash to read those lines into its in-memory history. At least,
not without a heavily altered bash history configuration.

(Have you altered root's bash history configuration on that Debian system?
If so, how?)

Those commands look like nonsense to me. However, the fact that they're
evolving from line to line looks like someone was trying to "get it
right", and failing to do so. Again, it looks like something that was
typed by an (ignorant) human. However, I can't even guess what the
intent was.

I tried googling "ps /x/md5users" and even "md5users" and got no useful
results. So, it doesn't look like a known malware worm. Also, one would
think that a malware worm would simply issue the desired command the first
time, and not have to fumble around trying to type it correctly.

[Michel Verdier]

Le 16 avril 2023 Eduardo M. KALINOWSKI a écrit :

> Which shell do you use, and how is it configured? Note that bash by default
> does not share history between sessions, so even if someone logged in as root
> (via other ssh session) and typed them, they would not appear in your ssh
> session.

I don't remember changing default for that and my bash shares between
sessions. But it is not important, someone can easily remove commands
from history. But if it was the case why not remove these remaining
commands ?

[Greg Wooledge]

On Sun, Apr 16, 2023 at 03:11:07PM +0200, Michel Verdier wrote:
> I don't remember changing default for that and my bash shares between
> sessions.

(NOTE: this is NOT the OP!) (Deletes a whole reply.)

OK, not-the-OP... your statement that bash "shares between sessions" is
extremely ambiguous.

Do you mean that if you open two simultaneous bash sessions, and type
a command into Session A, that it immediately appears in the history
of Session B? (Or, immediately after hitting Enter in Session B, maybe.)

If this is the case, you have MOST DEFINITELY altered the bash history
configuration, in an EXTREMELY significant way. Whether you remember it
or not.

What we need to know, however, is whether the OP has made a similar change.

[Jesper Dybdal]

On 2023-04-16 14:59, Michel Verdier wrote:
> Le 16 avril 2023 Jesper Dybdal a écrit :
>
>> I have scanned the Windows machine with two antivirus tools (Windows defender
>> and Malwarebytes).
> Can you use clamav on windows ?

I hadn't thought of that. I'll check.

>
>>> modules.dep
>>> modules.devname
>>> modules.symbols.bin
>>> modules.symbols
>>> modules.builtin.bin
>>> modules.alias.bin
>>> modules.builtin.alias.bin
>>> modules.softdep
>>> modules.alias
>>> modules.dep.bin
> These are generated during kernel install. And you can safely remove
> /lib/modules/5.10.0-21-amd64 if these are the only files left.

They are the only files on my harddisk that are not part of the .deb
file for the kernel.  There are lots of other files, but they match.

>> * Is it probable that somebody can remote control one or both machines?  Do
>> those 4 lines ring a bell?  What are they all about?
> Perhaps a bot trying to execute some commands. As they do not apply to
> debian you debian machine should not be compromised.

Unless the malware on the windows machine is smart enough to use my
secret key and decrypt it with a password retrieved from a key logger ...

> Malware can be installed via web sites

I tend to stay away from doubtful websites - but you are of course right.

> * Is there a significant risk that the problem came with the Bullseye
> upgrade?
> no
>
>> * I really don't want to reinstall from scratch.  Not only because I don't
>> know whether there is a problem on one or both machines, but also because I
>> have no idea where any infection came from - it could easily be from something
>> that I would also reinstall.
> I think you don't have to. For debian. For windows a full deinstall
> without reinstall is the best :)

In the long term, now that I'm retired, I hope to drop Windows
completely - but not quite today :-).

[Jesper Dybdal]

On 2023-04-16 14:40, Eduardo M KALINOWSKI wrote:
> On 16/04/2023 09:19, Jesper Dybdal wrote:
>> And there in the bash history were 4 lines that I had not written :-(
>>
>> I am certain that nobody had been in my apartment while I was gone.
>> And even if they had, nobody with a key to my apartment would dream
>> of writing things like the 4 lines that I found in the history file.
>>
>> The 4 lines were:
>>> md5users
>>> sp md5users
>>> sp /x/md5users
>>> ps /x/md5users
>> There is no file named "md5users" or directory named "/x" or command
>> named "sp" on the Debian machine.
>
> Which shell do you use, and how is it configured? Note that bash by
> default does not share history between sessions, so even if someone
> logged in as root (via other ssh session) and typed them, they would
> not appear in your ssh session.

I use bash.  More details in a reply to Greg in a moment.

>
> See https://mywiki.wooledge.org/BashFAQ/088, or wait for Greg to chime
> in with details and corrections.
>
>

--

Jesper Dybdal
https://www.dybdal.dk

[Jesper Dybdal]

On 2023-04-16 16:33, David Wright wrote:

> On Sun 16 Apr 2023 at 14:19:34 (+0200), Jesper Dybdal wrote:
>> The 4 lines were:
>>> md5users
>>> sp md5users
>>> sp /x/md5users
>>> ps /x/md5users
>>

> Just FTR and clarity's sake, are the "> " characters (which my MUA has
> unhelpfully doubled by quoting) part of what was typed in the putty
> session, or did you type them into the post to make them stand out?
They were not part of what was typed, and I did add them to make the
lines stand out.  Sorry for the unclear text.

Here is a correct and clear, I hope, version:

---------------- The 4 lines were:

md5users
sp md5users
sp /x/md5users
ps /x/md5users

---------------- End of the 4 lines

[Jesper Dybdal]

On 2023-04-16 15:08, Greg Wooledge wrote:
> On Sun, Apr 16, 2023 at 02:19:34PM +0200, Jesper Dybdal wrote:
>> And there in the bash history were 4 lines that I had not written :-(
> I would initially ask "who else lives with you"....

So would I - if I didn't know that the few people with physical access
to my apartment, including my wife, haven't the faintest idea that there
is a thing called "md5".

>> I am certain that nobody had been in my apartment while I was gone. And even
>> if they had, nobody with a key to my apartment would dream of writing things
>> like the 4 lines that I found in the history file.
>>
>> The 4 lines were:
>>> md5users
>>> sp md5users
>>> sp /x/md5users
>>> ps /x/md5users
>> There is no file named "md5users" or directory named "/x" or command named
>> "sp" on the Debian machine.
> This certainly sounds like someone walking up to your machine and typing
> the commands. Did you see the commands within the PuTTY session, or were
> they *only* in the shell history?

I saw them only in the shell history.  And I am pretty sure that I
cannot have overlooked them if they were visible on the screen.

>> * Is it probable that somebody can remote control one or both machines?  Do
>> those 4 lines ring a bell?  What are they all about?
> If someone did this with remote access, it would have to be access to
> the Windows machine. Nothing that could be done to the Debian machine
> would affect the in-memory shell history of a running instance of bash.
> Even writing to the /root/.bash_history file wouldn't cause the PuTTY
> session's bash to read those lines into its in-memory history. At least,
> not without a heavily altered bash history configuration.
>
> (Have you altered root's bash history configuration on that Debian system?
> If so, how?)

My .bashrc has:
> export HISTCONTROL=ignoreboth

and that's all.  And your description of the default behaviour matches
what I experience with bash.

>
> Those commands look like nonsense to me. However, the fact that they're
> evolving from line to line looks like someone was trying to "get it
> right", and failing to do so. Again, it looks like something that was
> typed by an (ignorant) human. However, I can't even guess what the
> intent was.
>
> I tried googling "ps /x/md5users" and even "md5users" and got no useful
> results. So, it doesn't look like a known malware worm. Also, one would
> think that a malware worm would simply issue the desired command the first
> time, and not have to fumble around trying to type it correctly.

Exactly.  I don't understand it.
Of the two machines, the Debian has the most important data: mailboxes
with mail archives for about 25 persons.  And just about all my data
files (docs, source code, photos, ...) live there.

If the WIndows machine has a virus that can sniff my typed key
decryption pass phrase and use it, then the Debian's root account may
have been compromised.  And I really need to be able to run ssh to root
to administer that machine (which normally has neither keyboard nor
monitor).

And the Windows machine has access to the data files on the Debian
machine.  So even if root is not compromised, the Windows machine can
make problems by destroying/modifying data files.  I now back up the
data files more frequently than I used to.

Thanks a lot to those who have answered.  And please keep the good ideas
coming...

[David Wright]

On Sun 16 Apr 2023 at 14:19:34 (+0200), Jesper Dybdal wrote:

> And there in the bash history were 4 lines that I had not written :-(
>
> I am certain that nobody had been in my apartment while I was gone.
> And even if they had, nobody with a key to my apartment would dream of
> writing things like the 4 lines that I found in the history file.
>
> The 4 lines were:
> > md5users
> > sp md5users
> > sp /x/md5users
> > ps /x/md5users
> There is no file named "md5users" or directory named "/x" or command
> named "sp" on the Debian machine.

Just FTR and clarity's sake, are the "> " characters (which my MUA has
unhelpfully doubled by quoting) part of what was typed in the putty
session, or did you type them into the post to make them stand out?

The reason I ask is that most people have their PS2 set to "> ",
suggesting that these might have been some sort of continuation—
of what, we don't know.

Cheers,
David.

[Jeffrey Walton]

On Sun, Apr 16, 2023 at 10:08 AM Jesper Dybdal <jd-debi...@dybdal.dk> wrote:
> ...

> In the long term, now that I'm retired, I hope to drop Windows
> completely - but not quite today :-).

++

My family went Windows-free about 2014. Grandparents, parents and me
are all using Linux. I cut them over to Linux because of the malware
and adware they kept installing.

You won't miss Windows.

Jeff

[Michel Verdier]

Le 16 avril 2023 Jesper Dybdal a écrit :

>> Perhaps a bot trying to execute some commands. As they do not apply to
>> debian you debian machine should not be compromised.
> Unless the malware on the windows machine is smart enough to use my secret key
> and decrypt it with a password retrieved from a key logger ...

But you go out leaving ssh session on ? And if no one could physically go
to your keyboard I can't imagine another possibility. Even if as Greg
says it looks like a human typing.

[Michel Verdier]

Le 16 avril 2023 Greg Wooledge a écrit :

> Do you mean that if you open two simultaneous bash sessions, and type
> a command into Session A, that it immediately appears in the history
> of Session B? (Or, immediately after hitting Enter in Session B, maybe.)

Ok I understand. I was meaning bash shares sessions but after closing
session in history file. Of course not in memory history.

[to...@tuxteam.de]

Sometimes, some tools rely on a shell at the "other side" to do
their job. Emacs's Tramp is known for leaving traces in the shell
history, quite possibly abominations like VSCode do their thing
in a similar way.

That said, the above commands look more like a human not quite
knowing what (s)he's doing. If that were an intruder, I'd not
worry too much ;-)

Cheers
--
t

[Greg Wooledge]

On Sun, Apr 16, 2023 at 04:30:51PM +0200, Jesper Dybdal wrote:
> On 2023-04-16 15:08, Greg Wooledge wrote:
> > (Have you altered root's bash history configuration on that Debian system?
> > If so, how?)
> My .bashrc has:
> > export HISTCONTROL=ignoreboth
>
> and that's all.  And your description of the default behaviour matches what
> I experience with bash.

There is simply no scenario where all of these things can be simultaneously
true.

Bash doesn't read the contents of the history file into the in-memory
history unless you run "history -r". If you had some kind of ksh-like
setup where you combined "history -w" and "history -r" commands in your
PROMPT_COMMAND or other variables, then we might be able to reconcile
the statements we've been given.

In the absence of that, there's just no way you could have commands in
your shell history that were not typed in that same shell session.

The most stupidly paranoid, off-the-wall, tin-foil-hat scenario that I
can come up with, which holds all these statements true, is that someone
hacked into the Debian system as root, attached gdb (or some similar
program) to a running bash, and used this opportunity to modify your
shell's history. Not to do anything. Just to fuck with you. To make
it LOOK like someone hacked you... and that the hacker was a halfwit.

The other scenario that comes to mind is that you actually typed the
commands yourself, and forgot doing it. Maybe you have dissociative
identity disorder or something, who knows.

The last scenario... is that one or more of the statements you've given
us are false.

[Jesper Dybdal]

On 2023-04-16 17:57, Greg Wooledge wrote:
> On Sun, Apr 16, 2023 at 04:30:51PM +0200, Jesper Dybdal wrote:
>> My .bashrc has:
>>> export HISTCONTROL=ignoreboth
>> and that's all.  And your description of the default behaviour matches what
>> I experience with bash.
> There is simply no scenario where all of these things can be simultaneously
> true.
>
> Bash doesn't read the contents of the history file into the in-memory
> history unless you run "history -r". If you had some kind of ksh-like
> setup where you combined "history -w" and "history -r" commands in your
> PROMPT_COMMAND or other variables, then we might be able to reconcile
> the statements we've been given.
>
> In the absence of that, there's just no way you could have commands in
> your shell history that were not typed in that same shell session.

Or somehow inserted into the PuTTY ssh client on the Windows machine to
become part of the ssh session.  You've just about convinced me that
that must be the situation.

I've just checked once more: the Windows machine does not have any
remote control server (VNC or Remote Desktop) enabled.  If it had, it
could have been an intrusion into the WiFi LAN.  So it's not that simple
- unless there is some hidden remote control server functionality.

> The most stupidly paranoid, off-the-wall, tin-foil-hat scenario that I
> can come up with, which holds all these statements true, is that someone
> hacked into the Debian system as root, attached gdb (or some similar
> program) to a running bash, and used this opportunity to modify your
> shell's history. Not to do anything. Just to fuck with you. To make
> it LOOK like someone hacked you... and that the hacker was a halfwit.

*Too* paranoid. And it would make sense only if I discovered it - if I
hadn't happened to use the history, I would never have seen anything
strange.

> The other scenario that comes to mind is that you actually typed the
> commands yourself, and forgot doing it.

Yes.  I certainly do not claim to always remember which commands I typed
an hour earlier, so if those lines had been something that could
remotely make sense to me, then I might well think I had done it
myself.  But those 4 lines?  If I had somehow typed such a mess, I would
remember it.

> Maybe you have dissociative
> identity disorder or something, who knows.

Who knows?  But if so, this is the first time it has shown symptoms.

> The last scenario... is that one or more of the statements you've given
> us are false.

Well - I happen to know that they're not.

Sometimes I almost think I must have dreamt it, but then I look at the 4
lines that I cut from the history file and saved.

Thanks for your help - you've made it clear that the problem originated
from the Windows machine.

Though that doesn't quite rule out that damage could have been done to
the Debian system at the same time, I think that in combination with the
apparent clumsiness of those commands, I can almost trust that the
Debian system is ok.

The question then remains: what to do with the Windows system before I
dare run a root ssh session from that machine again?  Perhaps restore a
backup, but from when?

Thanks,

[Michel Verdier]

Le 16 avril 2023 Jesper Dybdal a écrit :

> The question then remains: what to do with the Windows system before I dare
> run a root ssh session from that machine again?  Perhaps restore a backup, but
> from when?

As you don't know *how* you can't guess *when* and should reinstall from
scratch. If all your files are on debian it's not so hard.

[Thomas Schmitt]

Hi,

to make this mail on-topic:

Jesper Dybdal, do you see the riddling lines in file ~/.bash_history
of the superuser ?
If so: Do you see other strange lines there ? (Do they give more clue ?)


A bit less on-topic:

Greg Wooledge wrote:
> Bash doesn't read the contents of the history file into the in-memory
> history unless you run "history -r". If you had some kind of ksh-like
> setup where you combined "history -w" and "history -r" commands in your
> PROMPT_COMMAND or other variables, then we might be able to reconcile
> the statements we've been given.
>
> In the absence of that, there's just no way you could have commands in
> your shell history that were not typed in that same shell session.

My Debians always behaved that way. I remember that in Debian 8 i got
several different readline histories in the first shell terminals which
i started. With Debian 11 it's only one history per user. It seems to be
a collection of the last commands of shell sessions when the recent
shutdowns happened.
For example i create a new xterm and get to see (from new to old):

su -
firefox-esr &
top
ps -ef | grep pulseaudio
umgebung
view /u/x
su -
firefox-esr &
xset r rate 250 30

In man bash i read:

When the -o history option to the set builtin is enabled, the shell
provides access to the command history, [...]
On startup, the history is initialized from the file named by the vari‐
able HISTFILE (default ~/.bash_history).
[...]
set [--abefhkmnptuvxBCEHPT] [-o option-name] [arg ...]
[...]
-o option-name
The option-name can be one of the following:
[...]
history Enable command history, as described above under
HISTORY. This option is on by default in inter‐
active shells.

At the end of ~/.bash_history of my desktop user i see indeed above
history in old-to-new sequence:

xset r rate 250 30
firefox-esr &
su -
view /u/x
umgebung
ps -ef | grep pulseaudio
top
firefox-esr &
su -

I did not execute them in this sequence in the recent sessions when i tried
to find out what made pulseaudio permenantly busy. They surely stem from
different xterms. It is not clear whether one per shutdown was memorized
or whether more then one stem from a single shutdown.

I become superuser by "su -" and get to see

shutdown -h now
shutdown -r now
systemctl --global disable pulseaudio.service pulseaudio.socket
shutdown -r now
shutdown -h now
shutdown -r now
systemctl --global enable pulseaudio.service pulseaudio.socket
shutdown -h now

Since i only had one superuser xterm, i can quite surely state that the
systemctl commands were not the last ones which were performed in their
respective shells before shutdown.


Have a nice day :)

Thomas

[David Wright]

On Sun 16 Apr 2023 at 19:35:20 (+0200), Thomas Schmitt wrote:
>
> Jesper Dybdal, do you see the riddling lines in file ~/.bash_history
> of the superuser ?
> If so: Do you see other strange lines there ? (Do they give more clue ?)
>
>
> A bit less on-topic:
>
> Greg Wooledge wrote:
> > Bash doesn't read the contents of the history file into the in-memory
> > history unless you run "history -r". If you had some kind of ksh-like
> > setup where you combined "history -w" and "history -r" commands in your
> > PROMPT_COMMAND or other variables, then we might be able to reconcile
> > the statements we've been given.
> >
> > In the absence of that, there's just no way you could have commands in
> > your shell history that were not typed in that same shell session.
>
> My Debians always behaved that way. I remember that in Debian 8 i got
> several different readline histories in the first shell terminals which
> i started. With Debian 11 it's only one history per user. It seems to be
> a collection of the last commands of shell sessions when the recent
> shutdowns happened.

Me too: each shell starts with the contents of ~/.bash_history at that
instant, and adds its freshly typed lines to the end of the disk file
when it exits. And I spent a while searching unsuccessfully for some
HISTFILE=~/.bash_history or history -r command squirrelled away in
some startup file.

I set export HISTCONTROL=ignoreboth and don't know whether that
has side effects.

$ echo "$SHELLOPTS"
braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor:noclobber
$ echo "$BASHOPTS"
checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob:extquote:force_fignore:globasciiranges:interactive_comments:progcomp:promptvars:sourcepath
$

However, I can't confirm your Debian 8 behaviour.

Cheers,
David.

[Jesper Dybdal]

On 2023-04-16 19:35, Thomas Schmitt wrote:
> Hi,
>
> to make this mail on-topic:
>
> Jesper Dybdal, do you see the riddling lines in file ~/.bash_history
> of the superuser ?

Yes.

> If so: Do you see other strange lines there ? (Do they give more clue ?)

No.  I stupidly did not save the rest of .bash_history - only the 4
lines.  I did, however, take a quick look in the file, and saw nothing
conspicuous  other than those 4 lines.

[David Christensen]

On 4/16/23 05:19, Jesper Dybdal wrote:
> I have a Debian pc functioning as router, firewall, file server, name
> server, webserver, ...
> It has very recently been upgraded to Bullseye.
>
> On the internal network I have a Windows 10 pc.

> And there in the bash history were 4 lines that I had not written :-(

>> md5users
>> sp md5users
>> sp /x/md5users
>> ps /x/md5users

On 4/16/23 07:30, Jesper Dybdal wrote:

> ... I really need to be able to run ssh [on Windows to] administer
> [the Debian] machine (which normally has neither keyboard nor
> monitor).


What about installing a KVM switch and administering the Debian computer
from the console?


If the two computers are separated, you could use a KVM extender and put
the KVM switch near the Windows computer. Or, you could get networked
KVM equipment.


That said, using one computer as router, firewall, file server, name
server, web server, and more represents "all of your eggs in one
basket". I suggest using dedicated hardware for networking, network
segmentation (e.g. DMZ), and kernel or hypervisor compartmentalization
of services. Qubes looks very appealing:

https://www.qubes-os.org/


David

[Michel Verdier]

Le 17 avril 2023 David Christensen a écrit :

> That said, using one computer as router, firewall, file server, name server,
> web server, and more represents "all of your eggs in one basket". I suggest
> using dedicated hardware for networking, network segmentation (e.g. DMZ), and
> kernel or hypervisor compartmentalization of services. Qubes looks very
> appealing:

What are its advantages vs debian ? Debian can chroot most services. Some
are chrooted by default (postfix, etc). And you can use Xen and tor on
debian.

[Curt]

On 2023-04-16, Jesper Dybdal <jd-debi...@dybdal.dk> wrote:
>
> On 2023-04-16 15:08, Greg Wooledge wrote:
>> On Sun, Apr 16, 2023 at 02:19:34PM +0200, Jesper Dybdal wrote:
>>> And there in the bash history were 4 lines that I had not written :-(
>> I would initially ask "who else lives with you"....
>
> So would I - if I didn't know that the few people with physical access
> to my apartment, including my wife, haven't the faintest idea that there
> is a thing called "md5".

Maybe it's some trivial corollary of the infinite monkey theorem, and
it's really the dog.

--

[Stefan Monnier]

> That said, using one computer as router, firewall, file server, name server,
> web server, and more represents "all of your eggs in one basket". I suggest
> using dedicated hardware for networking, network segmentation (e.g. DMZ),
> and kernel or hypervisor compartmentalization of services.

Dedicated hardware has its upsides, indeed, but it also
has its downsides (e.g. in terms of impact to the planet).


Stefan

[Tim Woodall]

This is very true.

I switched to using one of these:
https://www.asrockrack.com/general/productdetail.asp?Model=J1900D2Y
in a xen configuration, from multiple hardware - but multiple services
on one hardware instance.

it's low power and fanless and has built in IPMI.

For me that ticks all the boxes I need. Having each of nameserver, dhcp
server, web server, firewall, "file server"[1] etc, running on separate
guests makes upgrading so much simpler too. I used to dread upgrades,
now they're relatively painless.

[1] I don't really run a fileserver but I do run iscsi targets.

[David Wright]

OK, you wrote that you "pressed up-arrow a few times. And there in the
bash history were 4 lines …". If those 4 lines were not the first
things to appear when you pressed up-arrow, then I would assume that
the commands you typed /just/ before you went out with the dog were
the first lines to appear, and then your 4 lines after more up-arrows.

If that's the case, then your 4 lines could have been typed
in a previous login as root, and that could have been some time
ago. They would have been languishing at the end of the file
/root/.bash_history before you logged in as root this time.

There is an option to timestamp entries in the history file. I've
never used it, nor heard of its being used. That might disambiguate
things if you ever suspect it might happen again.

Cheers,
David.

[David]

On Tue, 18 Apr 2023 at 04:42, David Wright <deb...@lionunicorn.co.uk> wrote:

> There is an option to timestamp entries in the history file. I've
> never used it, nor heard of its being used. That might disambiguate
> things if you ever suspect it might happen again.

Hi, on my machines I use Bash as interactive
shell, with:
HISTTIMEFORMAT=: %Y%m%d_%H%M%S ;

That provides a couple of benefits:

1) it writes a commented Unix timestamp with
each addition to the ~/.bash_history file, so that
the history file not only logs what commands were
run interactively, but also when.

2) when I run the 'history' command, the outpt
is formatted like this:
501 : 20230418_151124 ; help history
502 : 20230418_151406 ; env
503 : 20230418_151749 ; history
The colon and semicolon allow the timestamp
to function as a no-operation command.
That means that history expansion
can still function, for example entering !502
interactively will run line number 502, but
only the 'env' that comes after the semicolon
will have any effect.

[to...@tuxteam.de]

On Tue, Apr 18, 2023 at 05:29:43AM +0000, David wrote:
> On Tue, 18 Apr 2023 at 04:42, David Wright <deb...@lionunicorn.co.uk> wrote:
>
> > There is an option to timestamp entries in the history file. I've
> > never used it, nor heard of its being used. That might disambiguate
> > things if you ever suspect it might happen again.
>
> Hi, on my machines I use Bash as interactive
> shell, with:
> HISTTIMEFORMAT=: %Y%m%d_%H%M%S ;
>
> That provides a couple of benefits:
>
> 1) it writes a commented Unix timestamp with
> each addition to the ~/.bash_history file, so that
> the history file not only logs what commands were
> run interactively, but also when.
>
> 2) when I run the 'history' command, the outpt
> is formatted like this:
> 501 : 20230418_151124 ; help history
> 502 : 20230418_151406 ; env
> 503 : 20230418_151749 ; history
> The colon and semicolon allow the timestamp
> to function as a no-operation command.

At least in bash, this doesn't seem necessary, as you are
only seeing an external representation: internally, bash
keeps the timestamp separate (as happens to the seq number,
too).

In the external file, the timestamps are kept as #-comments
in separate lines (with the UNIX timestamps in them).

> That means that history expansion
> can still function, for example entering !502
> interactively will run line number 502, but
> only the 'env' that comes after the semicolon
> will have any effect.

I tried it out, and this also works with a "naked" timestamp,
without the : ... ; wrapping.

Caveat: I only tried with bash.

Cheers
--
t

[Richmond]

It's a long shot, but does either computer have wifi? Is it secured with
wpa2?

[debia...@howorth.org.uk]

bash seems to treat root and a normal user differently.

[to...@tuxteam.de]

On Tue, Apr 18, 2023 at 11:51:42AM +0100, debia...@howorth.org.uk wrote:
> <to...@tuxteam.de> wrote:

[...]

> > At least in bash, this doesn't seem necessary, as you are
> > only seeing an external representation: internally, bash
> > keeps the timestamp separate (as happens to the seq number,
> > too).
> >
> > In the external file, the timestamps are kept as #-comments
> > in separate lines (with the UNIX timestamps in them).
>
> bash seems to treat root and a normal user differently.

On my box, history, .bash_history and HISTTIMEFORMAT behave as
I described both for root and for a regular user.

Just to be sure:
# tomas@trotzki:~$ bash --version
# GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
# Copyright (C) 2020 Free Software Foundation, Inc.
# License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
#
# This is free software; you are free to change and redistribute it.
# There is NO WARRANTY, to the extent permitted by law.

Cheers
--
t

[David]

Hi, it could well be unnecessary, I haven't played with it
for a long time.

I'm sure that there would have been some reason at the
time why I chose to configure it that way, but it is so many years
ago that I can't recall the reason.

Guessing, it could just have been that I was lazy and doing something
odd. Perhaps I wanted to dump the history output into a file, preserve
the timestamps for some long-forgotten reason, and also put a shebang
at the top of the file and run it again with minimal editing. Or maybe
I wanted a known delimiter that I could strip automatically from
the history output. I dunno.

It also would have been several Bash versions ago, Bash 2 or 3, so
perhaps the behaviour changed since I configured it.

Anyway, if it isn't necessary now, there's no reason for me to advocate
doing that, so I appreciate that you have let everyone know about that.

[to...@tuxteam.de]

On Tue, Apr 18, 2023 at 11:59:58AM +0000, David wrote:
> On Tue, 18 Apr 2023 at 07:51, <to...@tuxteam.de> wrote:
> > On Tue, Apr 18, 2023 at 05:29:43AM +0000, David wrote:

[...]

> > > The colon and semicolon allow the timestamp
> > > to function as a no-operation command.
> >
> > At least in bash, this doesn't seem necessary, as you are
> > only seeing an external representation: internally, bash
> > keeps the timestamp separate (as happens to the seq number,
> > too).
>
> Hi, it could well be unnecessary, I haven't played with it
> for a long time.

[...]

> It also would have been several Bash versions ago, Bash 2 or 3, so
> perhaps the behaviour changed since I configured it.
>
> Anyway, if it isn't necessary now, there's no reason for me to advocate
> doing that, so I appreciate that you have let everyone know about that.

Definitely. I just pointed that out as a request for discussion.
Perhaps this isn't portable across shells or even different
versions of bash. So caveat emptor :)

Cheers and thanks -- I didn't know about HISTTIMEFORMAT before!

--
t

[Jesper Dybdal]

On 2023-04-16 14:19, I wrote:
> ...

> And there in the bash history were 4 lines that I had not written :-(

To summarize:

* Greg has convincingly argued that there is no way for the running
shell to get those lines into its history, other than by issuing them
over the ssh connection.

* We can therefore assume that the problem originated at the Windows
machine (the ssh client).

* It seems that an intruder has had control over the Windows machine,
including the ssh session, and thus, at least in principle, could have
done harm also to the Linux machine.

* There is, however, no sign of an infection of the Linux machine. And
the 4 lines do not suggest that whoever issued them knows what he's doing.

* So I am going to assume that the Linux machine is ok.

* The Windows machine could be infected with something that allows
remote control.

* So I should probably reinstall the Windows machine from scratch - or
perhaps restore a really old backup (I have one from July 2022, one from
2020, and one taken shortly after the original install in 2016).

Many thanks to everybody who answered!

[Jesper Dybdal]

On 2023-04-18 07:29, David wrote:
> On Tue, 18 Apr 2023 at 04:42, David Wright <deb...@lionunicorn.co.uk> wrote:
>
>> There is an option to timestamp entries in the history file. I've
>> never used it, nor heard of its being used. That might disambiguate
>> things if you ever suspect it might happen again.
> Hi, on my machines I use Bash as interactive
> shell, with:
> HISTTIMEFORMAT=: %Y%m%d_%H%M%S ;
>
> That provides a couple of benefits:

...

Thanks to David, David, Tomas, and debia...@howorth.org.uk for the
suggestion of using time stamps on the history lines.  I intend to do
that in the future.

[Jesper Dybdal]

On 2023-04-18 10:25, Richmond wrote:
> It's a long shot, but does either computer have wifi?

Those two computers do not, but the LAN they're connected to does have a
WiFi access point.  So yes, if anybody could access the LAN through the
WiFi and find a security hole in Windows to exploit, then that could be
the problem.  However:

> Is it secured with
> wpa2?

Yes.  The password is not easy to guess, and the neighbors do not know
it.  I think (but I may remember that incorrectly) that I checked the
log file in the access point and found nothing suspicious.

Thanks,
Jesper

[Jeremy Ardley]

On 18/4/23 21:36, Jesper Dybdal wrote:
>
>> Is it secured with
>> wpa2?
> Yes.  The password is not easy to guess, and the neighbors do not know
> it.  I think (but I may remember that incorrectly) that I checked the
> log file in the access point and found nothing suspicious.
>
>

Coincidentally I was checking on ways to break into a WiFi network today.

I should explain that for the first time yesterday I added a laptop to
my network by pressing a button on the wireless access point. It avoids
the need to type in my particularly long WPA2 password.

On reflection this sounded odd to me on and researching and I found
there are no security checks. The button is pressed and for a few
seconds any device in range can make a permanent connection without
password.

There is a second PIN mode where an attacker can brute force an 8 digit PIN.

The final mode that I'm used to is the Private Shared Key mode where the
user knows the WPA2 password.

The only way to be sure you are secure is to check the client list on
the router. If you have something you don't recognise then that may be
an intruder. However even that may not be sufficient so tools like nmap
can locate active hosts on your wifi network. Finally, wireshark can
monitor at the layer 2 level to identify unusual MAC addresses.


--
Jeremy
(Lists)

[Charles Curley]

On Tue, 18 Apr 2023 22:07:16 +0800
Jeremy Ardley <jer...@ardley.org> wrote:

> The only way to be sure you are secure is to check the client list on
> the router. If you have something you don't recognise then that may
> be an intruder.

You might also look into arpwatch and arpalert.

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[songbird]

<to...@tuxteam.de> wrote:
...

> Definitely. I just pointed that out as a request for discussion.
> Perhaps this isn't portable across shells or even different
> versions of bash. So caveat emptor :)
>
> Cheers and thanks -- I didn't know about HISTTIMEFORMAT before!

just as an aside for those who think about this sort of thing. :)

i like to start with a known state including the shell history
so upon starting up a terminal i determine what commands i want
in the history by detecting which directory i'm in (which tells
me which project i'm working on). it's very easy then for me
to start things by using the !<number> or !<abbrv>.

i also do not like history stuff hanging around so i may
attempt to clear things out upon logging out or shutting down
but since some crashes can happen i do not rely upon that
being 100%. which is why when i do start up i set things up
how i like and do clear the history at that point (before
putting the commands in i want).


songbird

[Michel Verdier]

Le 18 avril 2023 songbird a écrit :

> i like to start with a known state including the shell history
> so upon starting up a terminal i determine what commands i want
> in the history by detecting which directory i'm in (which tells
> me which project i'm working on). it's very easy then for me
> to start things by using the !<number> or !<abbrv>.

I recently learned the ctrl-r key which launch a regex search in
history. It's more powerful than !<abbrv> as it search the full
lines so not only commands but also parameters.

[Andy Smith]

Hello,

On Tue, Apr 18, 2023 at 06:22:16PM +0200, Michel Verdier wrote:
> I recently learned the ctrl-r key which launch a regex search in
> history. It's more powerful than !<abbrv> as it search the full
> lines so not only commands but also parameters.

Now step in to the late 2010s and look into "fzf" 😀

(It is packaged in Debian)

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[David Christensen]

I do not believe the analysis is complete -- I never saw an answer to
the following question (?); it is important:

On 4/17/23 21:42, David Wright wrote:
> OK, you wrote that you "pressed up-arrow a few times. And there in the
> bash history were 4 lines …". If those 4 lines were not the first
> things to appear when you pressed up-arrow, then I would assume that

> the commands you typed/just/ before you went out with the dog were

> the first lines to appear, and then your 4 lines after more up-arrows.
>
> If that's the case, then your 4 lines could have been typed
> in a previous login as root, and that could have been some time
> ago. They would have been languishing at the end of the file
> /root/.bash_history before you logged in as root this time.

Where the four commands in questions the very last commands entered into
Putty, or were they prior? If the latter, how far back? Same session?
Same day? A day ago? Week? Month?


David

[Jesper Dybdal]

I unfortunately did not note the exact sequence including context.

I think that I would have noticed it if those commands were visible on
the screen before I pressed up-arrow.

> > If that's the case, then your 4 lines could have been typed
> > in a previous login as root, and that could have been some time
> > ago. They would have been languishing at the end of the file
> > /root/.bash_history before you logged in as root this time.
> Where the four commands in questions the very last commands entered
> into Putty, or were they prior?  If the latter, how far back?  Same
> session? Same day?  A day ago?  Week? Month?

You're right.  I had overlooked that possibility: they might indeed be
from a previous session.  I don't think it is likely, though, since I
had used the active session for some time before walking the dog, and I
did not go far back in the history before I found those commands.

If something like that happens again, I'll take better care to gather
and keep the evidence :-(

Many thanks for your effort,