Howto log multiple sftpd instances with their chroot shared via NFS
Hildegard Meier
syslog-ng 3.3.4 + 3.13.2
Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
and each sftpd user's chroot home is /var/data/chroot/<username>/
and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
log { source(s_chroot_<username>); destination(d_sftp_<username>); };
Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
so every user can login on the one server or on the other. This is for high availability. This works so far.
What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
Thanks
Crossposting (because unanswered)
https://unix.stackexchange.com/questions/666641/howto-log-multiple-sftpd-servers-activity-which-users-chrooted-home-is-on-shar
_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Damien Miller
> OpenSSH 5.9p1 + 7.6p1
>
> syslog-ng 3.3.4 + 3.13.2
>
> Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
> and each sftpd user's chroot home is /var/data/chroot/<username>/
> and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>
> Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
> so every user can login on the one server or on the other. This is for high availability. This works so far.
>
> What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
> because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
>
> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
>
> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
some proposals for post-auth logging to be proxied via the priviledged
sshd monitor process but we haven't pursued them yet.
Maybe someone with more Linux/NFS wit could suggest an OS-side solution
for you?
-d
David Newall
On Tue, 21 Sep 2021, Hildegard Meier wrote:
> Now I have a second sftpd server in parallel, with the same user
> database and also mounts /var/data/chroot/ via NFS, and has the same
> syslog-ng config,
> so every user can login on the one server or on the other. This is for
> high availability. This works so far.
>
> What is not working now is the sftpd logging: The sftp user's log is
> only available on one sftp server exclusively, and that is the one
> where syslog-ng was started least, because as I understand it takes
> the exclusive unix socket file lock for each user's /dev/log.
>
> So, if a user logs in on the first server, where syslog-ng was started
> least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is
> not logged, neither on the second nor on the first server.
destination d_tcp {
network("log_host" port(1999));
};
Regards,
David
Darren Tucker
[...]
> Now I have a second sftpd server in parallel, with the same user database
> and also mounts /var/data/chroot/ via NFS
> How can I get the sftp user's activity be logged on each sftp server, when
> a user logs in to that server, while the user's home is shared on both
> servers via NFS?
>
--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
Jochen Bern
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>> So, if a user logs in on the first server, where syslog-ng was started
>> least, the user's sftp activity is logged on the first server.
>> But if the user logs in on the second server, it's sftp activity is
>> not logged, neither on the second nor on the first server.
>
> Forward the log entries on both machines to a log host.
that it'll have anything to forward to a log host, either.
The problem *presumably* is that the syslogd on server A has put some
sort of file lock on the device that propagates through the NFS server
and interferes with syslogd on server B using it.
One solution might be to reconfigure the syslogd's to use a method of
locking that does *not* propagate through NFS. I'm afraid I don't know
syslog-ng well enough to advise on that.
Then there's the possibility of reconfiguring *NFS* to stop the
forwarding, but "breaking" file locking on NFS is, of course, a can of
worms of possible side effects ...
(Bind) mounting a local .../dev over the NFS-shared chroot dirtree ...
ought to work, but complicates unmounting/remounting, which was already
enough of a hair-puller in failure scenarios when I last worked with NFS.
What do the chrooted users have for a homedir *within* the chroot? Would
it be possible to have /var/data/chroot be a local FS and mount only
/var/data/chroot/home from the NFS server? (If there are files that you
need to keep identical on both servers, e.g., under
/var/data/chroot/etc, you can still symlink those to some special subdir
like /var/data/chroot/home/ETC to put the actual data onto the NFS share.)
Regards,
--
Jochen Bern
Systemingenieur
Binect GmbH
Hildegard Meier
Stuart Henderson
> What do the chrooted users have for a homedir *within* the chroot? Would it
> be possible to have /var/data/chroot be a local FS and mount only
> /var/data/chroot/home from the NFS server? (If there are files that you need
> to keep identical on both servers, e.g., under /var/data/chroot/etc, you can
> still symlink those to some special subdir like /var/data/chroot/home/ETC to
> put the actual data onto the NFS share.)
user has their own separate chroot. So this type of approach would
require mounting a local fs of some sort over the top of each user's dir
which soon gets messy.
This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
kernel socket to receive those messages, and processes which want to
write a log entry just call the standard syslog functions which use
sendsyslog(2) rather than /dev/log, so it works even through FD
exhaustion, in chroot, and with syscall filters that prohibit
filesystem access.
Not entirely pleasant but I suppose it could alternatively be done by
using a LD_PRELOAD wrapper to override syslog functions (I think just
syslog_r is probably enough for openssh) and have them send over a
network socket instead.
Thorsten Glaser
> This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
> https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
> kernel socket to receive those messages, and processes which want to
> write a log entry just call the standard syslog functions which use
Oh, nice.
> The description was for /var/data/chroot/<username>/dev/log i.e. each
> user has their own separate chroot. So this type of approach would
> require mounting a local fs of some sort over the top of each user's dir
This made me curious, and I tried¹ this. It is possible to bind-mount
sockets on Linux iff the target exists as regular file.
sudo touch /var/data/chroot/<username>/dev/log # but beware of
# filesystem-based
# attacks here!
sudo mount --bind /dev/log /var/data/chroot/<username>/dev/log
① I went and began using this technology here:
https://github.com/mirabilos/shellsnippets/blob/master/posix/debchroot.sh
bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
╳ HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!
****************************************************
Douglas E Engert
>
> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88
This might work for a few sftp servers.
On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
syslog-ng.conf:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
(May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
Do the above for second server, hostname2, use /dev/<hostname2>/log
Then add these to the chroot directories in NFS:
/var/data/chroot/<username>/dev/<hostname1>/log
/var/data/chroot/<username>/dev/<hostname2>/log
This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
because the syslog-ng does not really look at the syslog-ng.conf in the chroot.
>
> Thanks
>
> Crossposting (because unanswered)
> https://unix.stackexchange.com/questions/666641/howto-log-multiple-sftpd-servers-activity-which-users-chrooted-home-is-on-shar
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-...@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
Douglas E. Engert <DEEn...@gmail.com>
Hildegard Meier
More ideas appreciated, since it would be relly important for us to have a solution for that.
>Date: Wed, 22 Sep 2021 13:06:43 +0200
>From: Jochen Bern <Joche...@binect.de>
>To: openssh-...@mindrot.org
>Subject: Re: Howto log multiple sftpd instances with their chroot
>shared via NFS
On 22.09.21 11:18, David Newall wrote:
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>> least, the user's sftp activity is logged on the first server.
>> But if the user logs in on the second server, it's sftp activity is
>> not logged, neither on the second nor on the first server.
>
>Considering that server B is not logging *at all* right now, I doubt
>that it'll have anything to forward to a log host, either.
>The problem *presumably* is that the syslogd on server A has put some
>sort of file lock on the device that propagates through the NFS server
>and interferes with syslogd on server B using it.
As I understand, syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
This seems also to be called IPC socket (inter-process communication socket) or AF_UNIX socket.
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."
"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )
>One solution might be to reconfigure the syslogd's to use a method of
>locking that does *not* propagate through NFS. I'm afraid I don't know
>syslog-ng well enough to advise on that.
I think the problem is that the chrooted sftp users only have write access (for logging) under their chroot home dir (and there /devlog) , and since that is on NFS, how could one escape that. A possibility would be a symlink to a local unix stream device, but chroot does not allow to access out of it, that's the whole point of being chrooted (jailed).
>Then there's the possibility of reconfiguring *NFS* to stop the
>forwarding, but "breaking" file locking on NFS is, of course, a can of
>worms of possible side effects ...
Do not know what you exactly mean, but I think we cannot fiddle with NFS file locking since we have several hundrets of sftp customer accounts on the NFS chroot, which transfer many many files.
>(Bind) mounting a local .../dev over the NFS-shared chroot dirtree ...
>ought to work, but complicates unmounting/remounting, which was already
>enough of a hair-puller in failure scenarios when I last worked with NFS.
man bind says: "mount --bind olddir newdir"
/dev/log is not a directory, but created by syslog-ng as a unix stream socket (see above). I guess if it would a directory, the sftpd process could not log to it?
# mkdir -p /my/local/dir/
# mount -v --bind /var/data/chroot/test/dev/log /my/local/dir
mount: /my/local/dir: mount(2) system call failed: Not a directory.
>What do the chrooted users have for a homedir *within* the chroot? Would
>it be possible to have /var/data/chroot be a local FS and mount only
>/var/data/chroot/home from the NFS server? (If there are files that you
>need to keep identical on both servers, e.g., under
>/var/data/chroot/etc, you can still symlink those to some special subdir
>like /var/data/chroot/home/ETC to put the actual data onto the NFS share.)
Strict technically, we really _need_ to share every sftp user's chrooted upload directory (/var/data/chroot/<username>/in/) and download directory (/var/data/chroot/<username>/out/), since we have a single backend process that reads all files that the customers have uploaded into their chroot /in/ directory (be it that they have logged in to the one sftp server or the other, both sftp servers are TCP load balanced, but the time order of the uploads needs to be honoured), and writes files to the customers chroot /out/ dir (and the customer can download the files from it's chrooted /out/ dir whether having sftp logged in to the one sftp server or the other).
But since we have 800 sftp customers, so 800 /var/data/chroot/<username>/in/ and 800 /var/data/chroot/<username>/out/ it is not feasable to have 1600 single NFS mounts, to achieve that /var/data/chroot/<username>/dev/log is local. That is the problem.
>Mount a tiny local ramfs or tmpfs over /var/data/chroot/dev?
For the same reason, since we have so many /var/data/chroot/<username>/dev/log , this would not be feasable (apart of the question if this would technically be a solution).
We need also a single log source for each every sftp user, because we need a single log file for every single sftp user, to be able to view each sftp user's sftp activity.
Best regards
Hildegard
Hildegard Meier
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
/var/data/chroot/<username>/dev/hostname1/log
/var/data/chroot/<username>/dev/hostname2/log
?
> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
/var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname1/log
?
> Do the above for second server, hostname2, use /dev/<hostname2>/log
/var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
since /var/data/chroot/<username>/dev/log is already a Symlink to /var/data/chroot/<username>/dev/hostname1/log
(see first step), and it is the same NFS filesystem.
> Gesendet: Freitag, 24. September 2021 um 16:08 Uhr
> Von: "Douglas E Engert" <deen...@gmail.com>
> An: openssh-...@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
> On 9/21/2021 4:49 AM, Hildegard Meier wrote:
>
> >
> > How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>
> based on:
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88
>
> This might work for a few sftp servers.
> On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
> syslog-ng.conf:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
>
> Do the above for second server, hostname2, use /dev/<hostname2>/log
>
> Then add these to the chroot directories in NFS:
> /var/data/chroot/<username>/dev/<hostname1>/log
> /var/data/chroot/<username>/dev/<hostname2>/log
>
> This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
> and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
> because the syslog-ng does not really look at the syslog-ng.conf in the chroot.
>
Thorsten Glaser
> It would not be possible to have a Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
>
> since /var/data/chroot/<username>/dev/log is already a Symlink to
> /var/data/chroot/<username>/dev/hostname1/log (see first step), and it
> is the same NFS filesystem.
Not sure I understand the entire situation, but can’t you make
/var/data/chroot/<username>/dev/log a regular file (on the NFS
mount) then bind-mount (see my earlier mail) /dev/log from the
host over it?
bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
╳ HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!
****************************************************
Hildegard Meier
I think, the sftpd process should just not write to the /dev/log unix socket (because this leads to the problem here), but to the local kernel directly, something like what you describe here.
But how could I do this concrete with Ubuntu Linux? What you write is rather abstract and I am not so expert that I understand what you mean with LD_PRELOAD wrapper.
Unfortunately, I could not change our sftp server to OpenBSD operating system since we would not have the capacity to maintain this one special operating system. We maintain our 350 Ubuntu Linux servers with already established processes.
Douglas E Engert
On 9/24/2021 10:03 AM, Hildegard Meier wrote:
> Thanks, this sounds interesting and maybe in this direction could be the solution? But I do not understand you precisely.
>
>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
> You mean there should be
> /var/data/chroot/<username>/dev/hostname1/log
> /var/data/chroot/<username>/dev/hostname2/log
>
var/data/chroot/<username>/dev/hostnameX/log
Thus each sever has it own lock for each user.
So any lock done while application is running in chroot will lock will lock on different file.
>
>> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
> You mean there should be the Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname1/log
>
Douglas E Engert
On 9/24/2021 10:11 AM, Thorsten Glaser wrote:
> On Fri, 24 Sep 2021, Hildegard Meier wrote:
>
>> It would not be possible to have a Symlink
>> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
Correct that is not needed.
It looks like the server has a line:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
It is not clear if this is created ahead of time or on the fly when the NFS volume is mounted
or for each possible user to create a unix-stream socket in NFS.
What I was proposing is each of the two servers has it own /var/data/chroot/<username>/dev/hostnameX.log
What is not clear is when running in chroot, when an application does
an openlog, syslog or vsyslog are these system calls, or library calls.
i.e. do the read any configuration from the chroot directory?
I think it expects to find /dev/log in the chroot directory and this name
matches what is in the real root directory. So I was proposing
that each sftpd server has a different /dev/hostname.log so a user
could be on both of the servers at the same time.
Hildegard said these are Ubuntu servers, and Ubuntu uses rsyslogd that looks
similar to syslog-ng.
Google'ing for: rsyslog chroot /dev/log
These look interesting.
https://www.the-art-of-web.com/system/sftp-logging-chroot/
https://serverfault.com/questions/878627/logging-chrooted-sftp-user-activity
>>
>> since /var/data/chroot/<username>/dev/log is already a Symlink to
>> /var/data/chroot/<username>/dev/hostname1/log (see first step), and it
>> is the same NFS filesystem.
>
> Not sure I understand the entire situation, but can’t you make
> /var/data/chroot/<username>/dev/log a regular file (on the NFS
> mount) then bind-mount (see my earlier mail) /dev/log from the
> host over it?
Is this what you are trying to do?
https://serverfault.com/questions/878627/logging-chrooted-sftp-user-activity
Instead, use input(type="imuxsock" Socket="/opt/bindmount/dev/log") to create an additional
socket (which you can then bind-mount such that all chroots see it at /dev/log).
>
> bye,
> //mirabilos
>
--
Douglas E. Engert <DEEn...@gmail.com>
_______________________________________________
Thorsten Glaser
> Instead, use input(type="imuxsock" Socket="/opt/bindmount/dev/log") to create
> an additional
> socket (which you can then bind-mount such that all chroots see it at
> /dev/log).
Wasn’t it said earlier in the thread that this doesn’t scale across
multiple chroots, especially when they may be added and removed dy‐
namically? Restarting the syslog daemon all the time isn’t going to
fly… and this can avoid that.
bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
╳ HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!
****************************************************
Peter Stuge
Hildegard Meier wrote:
> I think, the sftpd process should just not write to the /dev/log
> unix socket (because this leads to the problem here), but to the
> local kernel directly, something like what you describe here.
the system C library, without any interface to control the setting.
> But how could I do this concrete with Ubuntu Linux? What you write
> is rather abstract and I am not so expert that I understand what
> you mean with LD_PRELOAD wrapper.
starting a program loads that library into memory before the system C
library and the program itself. Symbols (ie. functions) loaded from
the preloaded library are not replaced by later (system) libraries,
making it possible to override and/or wrap libc functionality in this
fairly brutish way, hence a bit unpleasant.
Attached is source code for a simple such wrapper overriding openlog(),
syslog() and closelog(). It might work only on the older system, or on
both. You only need this hack on one server, the other can use /dev/log.
The wrapper directs logging to /dev/log_<hostname> instead of /dev/log.
Since libc has no interface to manipulate the socket path these functions
override the normal functions with a very limited implementation.
Adjust the template on line 49 if you want to change the destinateion.
Although functionality is limited it could work well for sftp-server.
One simplification is that this openlog() ignores all option flags
except LOG_NDELAY and that syslog() always behaves as if openlog() was
called with the LOG_PID option. Since this is exactly what OpenSSH log.c
does all log messages should look the same as with the libc implementation.
Note that log messages are silently lost on any error, e.g. syslog-ng
not running. Also note that if log messages, the openlog() ident
parameter or the username (when ident == NULL) are very long then
messages are silently truncated to 2048 chars. OpenSSH sends max 500
chars.
You didn't mention if your two sshd are configured to use the in-process
sftp-server (Subsystem sftp internal-sftp in sshd_config) - if so then
this is probably not really the best solution, since LD_PRELOAD would
have to be set for the entire sshd process, which I don't recommend
because this code is so simplistic compared to libc.
Anyway, try it if you like.
The comment near the top of the file shows how to compile. sshd can't
be configured to set LD_PRELOAD so to use the compiled .so with
sftp-server you have to create a simple script:
--8<-- /usr/local/libexec/sftp-server-hostlog
#!/bin/sh
export LD_PRELOAD=/usr/local/libexec/hostlog.so
exec /usr/lib/sftp-server "$*"
-->8--
and then configure sshd to call that instead of sftp-server:
--8<-- /etc/ssh/sshd_config
Subsystem sftp /usr/local/libexec/sftp-server-hostlog
-->8--
Adjust paths as needed, also in the compile command.
For anyone else interested please go ahead but note that I haven't
put huge effort into portability; it might work outside Linux but
I haven't tested anywhere else so YMMV.
Kind regards
//Peter
Jochen Bern
> More ideas appreciated, since it would be relly important for us
> to have a solution for that.
have no idea how portable it is to other distribs, so just give it a try:
> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> Subsystem sftp internal-sftp
> Match group mandanten
> ForceCommand internal-sftp -l INFO -u 0077
> # ls -al ~lvinq4/dev
> insgesamt 0
> drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
As you can see, the chroots have an *empty* /dev subdir, but logging
*still works*, apparently because the chrooted process just keeps using
the system-central /dev/log it opened *before* chroot()ing.
Kind regards,
Peter Stuge
> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
> have no idea how portable it is to other distribs, so just give it a try:
>
> > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > Subsystem sftp internal-sftp
> > Match group mandanten
> > ForceCommand internal-sftp -l INFO -u 0077
>
>
> > # ls -al ~lvinq4/dev
> > insgesamt 0
> > drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>
> As you can see, the chroots have an *empty* /dev subdir, but logging
> *still works*, apparently because the chrooted process just keeps using
> the system-central /dev/log it opened *before* chroot()ing.
when using internal-sftp as opposed to executing the sftp-server binary.
If syslog-ng can be made to reliably route internal-sftp messages to
user-specific log files then this approach would work well in Hildegard's
setup and would be a lot more pleasant than messing with LD_PRELOAD.
//Peter
Douglas E Engert
On 9/26/2021 3:20 PM, Peter Stuge wrote:
> Jochen Bern wrote:
>> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
>> have no idea how portable it is to other distribs, so just give it a try:
>>
>>> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
>>> Subsystem sftp internal-sftp
>>> Match group mandanten
>>> ForceCommand internal-sftp -l INFO -u 0077
>>
>>
>>> # ls -al ~lvinq4/dev
>>> insgesamt 0
>>> drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
>>> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>>
>> As you can see, the chroots have an *empty* /dev subdir, but logging
>> *still works*, apparently because the chrooted process just keeps using
>> the system-central /dev/log it opened *before* chroot()ing.
>
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
>
and if the sftp server can run in interactive mode. i.e. if a shell is started,
and if it can run any commands that may require logging.
OpenSSH sshd_config man page discusses ChrootDirectory says: "For an interactive session this requires at least a shell"
and also says: "sessions which use logging may require /dev/log inside the chroot directory on some operating systems."
So I would say the /dev/log or logs are required.
Based on the problem in original note:
> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
She also says: "for every user has a log device /var/data/chroot/<username>/dev/log"
(not sure what it changes, it may save the socket handle/index (or whatever it is) to be used in that server's kernel.)
Each server is overwriting the /dev/log in the chroot directories.
So if interactive is not required, then no chroot /dev/log would be needed and sftp-initeral could work.
But it sounds like there is a need to share these NFS volumes
So we are back to using bind-mounting as it is done in the kernel and NFS version of /dev/log is not touched.
OR
Each server has its own /dev/log_<hostname> and the syslog-ng files on each server do the same
and use LD_PRELOAD= as proposed by Peter Stuge. I have used the LD_PRELOAD before when running
valgrind and/or gdb to not unload modules so as the debug tables stay available,
> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD.
change to chroot.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-...@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
Hildegard Meier
Since the discussion has been shared between several mails now, I would like to try to summarize all up in one mail again here
and fill in the informations that where missing before.
We have 800 (eight hundred) sftp customers, each sftp customer has the same simple local Linux account on both sftp servers (simple entry in /etc/passwd and /etc/shadow etc.).
(Before we had only one sftp server, but for higher availability we want now to run two or more sftp servers parallel, accessed via TCP (sftp) load balancer)
Each customer account is in the group "sftp-cust", and only members of that group are allowed to login via sftp.
Each customer has it's chrooted home dir in /var/data/chroot/<username>/.
Here the (relevant part) of the sftpd config:
---------------------------------------------------------------
AllowGroups sftp-cust
Subsystem sftp internal-sftp -f LOCAL5 -l INFO
Match Group sftp-cust
ChrootDirectory %h
ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
AllowTcpForwarding no
---------------------------------------------------------------
Each sftp customer's sftp activity needs to be available in a dedicated log file for each sftp customer, for our support to be able to look into it. Here is an example of this log file:
Sep 28 15:54:25 myhostname internal-sftp[1618]: session opened for local user <username> from [1.2.3.4]
Sep 28 15:55:52 myhostname internal-sftp[27918]: remove name "/in/file.dat"
Sep 28 15:55:52 myhostname internal-sftp[27918]: sent status No such file
Sep 28 15:55:52 myhostname internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
Sep 28 15:55:52 myhostname internal-sftp[27918]: close "/in/file.dat" bytes read 0 written 2966
Sep 28 15:55:52 myhostname internal-sftp[27918]: set "/in/file.dat" modtime 20210928-15:55:46
If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
So a solution for the log problem would be, to not use the chroot logging, but to parse the global sftpd log (which is available on both sftpd servers locally for all local logins separated).
But I think this is not trivial to make this reliable and robust, since we need to parse the process id (sftpd session), check which username that session belongs to and write that log lines in the username specific log file.
Of course, process IDs are reused for different sessions, we have overlapping sessions etc. But if there is no other solution, than we need to try that. Maybe someone has already written such a session tracking parser.
Does somebody know of a log analyzer that can do this? A problem would be e.g. if the user never logs out, and the log file rotates daily, so after one day, the parser could not find any "session opened for local user xxx" log lines anymore.
If we want to try to keep using the available session chroot logging, man (8) sftp-server says:
"For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
syslogd(8) establish a logging socket inside the chroot directory."
As Peter has written here
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
as I understand it is hard coded in the system C library that the log devices name is /dev/log, so this cannot be changed (e.g. to log to chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2 on hostname2).
With the syslog-ng we use, we only need to create the directory /var/data/chroot/<username>/dev for each user once we create a new account.
We have the following syslog-ng config snippet file for each sftp user:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
But the problem is that the last started syslog-ng aquires the lock for the NFS shared /var/data/chroot/<username>/dev/log so the other server cannot read it anymore and so there is no sftp log for the sessions on the other server.
I guess this is because syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."
"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )
We need to stick with Ubuntu Linux as we have established management process only for this operating systems.
I hope I did not forget some information that was missing before. Thanks Hildegard
> Gesendet: Mittwoch, 22. September 2021 um 09:19 Uhr
> Von: "Damien Miller" <d...@mindrot.org>
> An: "Hildegard Meier" <daku...@gmx.de>
> Cc: openssh-...@mindrot.org
>
Peter Stuge
thanks for the summary.
Hildegard Meier wrote:
> ---------------------------------------------------------------
> AllowGroups sftp-cust
> Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> Match Group sftp-cust
> ChrootDirectory %h
> ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> AllowTcpForwarding no
> ---------------------------------------------------------------
LD_PRELOAD approach a bit less desirable. It will still work though.
> So a solution for the log problem would be, to not use the chroot
> logging, but to parse the global sftpd log
internal-sftp command line and run sshd with stderr connected to that
parser, to be able to keep track of events in a sensible way.
Log rotation would happen only after (or in) the parser.
> If we want to try to keep using the available session chroot
> logging, man (8) sftp-server says:
>
> "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> syslogd(8) establish a logging socket inside the chroot directory."
>
> As Peter has written here
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
>
> as I understand it is hard coded in the system C library that the log
> devices name is /dev/log, so this cannot be changed (e.g. to log to
> chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2
> on hostname2).
replacing the C library syslog functionality. My attachment does
indeed log to /dev/log_hostname1 on hostname1 instead of /dev/log.
However because of internal-sftp (which I still prefer over the alternative;
maintaining an sftp-server binary and possibly libraries in every homedir)
the entire sshd would have to be run with LD_PRELOAD.
Note that LD_PRELOAD is only neccessary on n-1 servers; the first
server can still use /dev/log and the full C library syslog() code.
If the second server where LD_PRELOAD is used does not need to
support anything besides SFTP, or if you can mandate that, then
it could be okay to run the entire sshd with LD_PRELOAD. Your IT
security colleagues may well object and they're not wrong; it's not
much code but still quite a mighty hammer.
> > Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> > for you?
It would be nice, but the writable (upper) directory can't be NFS
with overlayfs so it can not work on the SFTP servers.
But using overlayfs can work on the NFS server. There would then be
one export for each SFTP server, all of them reaching the same
homedir contents except for different (empty) <homedir>/dev/ subdirs.
This requires maintaining a second homedir structure on the NFS server,
but that structure only needs to contain a homedir with one empty dev
subdir under and nothing else, so it may be doable. Homedirs need
correct permissions and only one of the two directory inodes for the
homedir itself will be changed by the respective SFTP server, but on
the plus side it requires only a single overlay mount, nothing per-user
and no LD_PRELOAD stuff on the SFTP servers.
On the NFS server, assuming that /home is where real homedirs exist
(this gets mounted directly by one SFTP server), that /var/home2 is
the second homedir structure with only homedirs and an empty dev subdir
in each, that /var/home2_work is an empty directory on the same filesystem
as /var/home2 and that /nfs/home2 is the second export mounted by the
second SFTP server, then set it up like so:
mount -t overlay overlay -olowerdir=/home,upperdir=/var/home2,workdir=/var/home2_work /nfs/home2
Then export /home and /nfs/home2 via NFS.
The SFTP server mouting /home will access /home/<user>/dev/log
(both syslog-ng and internal-sftp) and the SFTP server mouting
/nfs/home2 will instead access /var/home2/<user>/dev/log but
everything else within the homedir is to and from /home/<user>.
If this change can be done on the NFS server I think it's what I'd prefer.
Kind regards
//Peter
Douglas E Engert
But here is another proposal.
You already have 800 NFs volumes and they are all mounted on each server.
(This is based on the syslog-ng configure which create a unix-stream in every volume when started.)
I assume that the only difference between these volumes is the user's home directory with
files they transfer to and from. And the user has only read access to whatever else is in the volume.
I assume they must have a /etc/passwd file with only one user.
Create 800 duplicate volumes on each server (not in NFS), but without the user's home directory.
for example /var/data/sftp/<username>/chroot/
modify the current 800 NFS volumes so they can be NFS mounted as the user's home in the above
new volumes.
Thus each sftp server uses the /var/data/sftp/<username>/chroot/dev/log which is on the same host
rather then a shared /dev/log. Its more complicated, but no local modifications.
Jochen Bern
> Each sftp customer's sftp activity needs to be available in a
> dedicated log file for each sftp customer, for our support to
> be able to look into it. Here is an example of this log file:
>
> Sep 28 15:54:25 myhostname internal-sftp[1618]:
> session opened for local user <username> from [1.2.3.4]
*earlier* than that, from before the client successfully authenticated
for the login as <username>. Having those messages go into a per-account
log(file) is nontrivial because sshd *does not yet know* what username
it will be, and if your support staff cannot see those lines, they will
not be able to debug *failed* connection attempts.)
At this point, I'd say that the best approach would be to have the
post-auth sftp-server process log to a *file* within the chroot, with
the filename set to the server's hostname (in the ForceCommand statement
of the server's /etc/ssh/sshd_config). Unfortunately, while sftp-server
has command line options for logging to syslog and for logging to
stderr, it does *not* have a command line option to log to a *file* (or,
for that matter, to set/change the "ident" parameter of openlog(3) to
include the username) ...
> you have a global sftpd log (I think in /var/log/messages on the
> server or something like that)
my experience, uses /var/log/syslog instead.)
> But I think this is not trivial to make this reliable and robust,
> since we need to parse the process id (sftpd session), check which
> username that session belongs to and write that log lines in the
> username specific log file.
when they make a difference between "logs" and "traces", but I suppose
that throwing one of those at your problem could easily prove to be
overkill ...
https://opentelemetry.io/docs/concepts/data-sources/
> A problem would be e.g. if the user never logs out, and the log file
> rotates daily, so after one day, the parser could not find any
> "session opened for local user xxx" log lines anymore.
ConSol's check_logfiles, etc.) can be configured to find and include the
rotated-away previous logfile instances ...
> as I understand it is hard coded in the system C library that the
> log devices name is /dev/log, so this cannot be changed
[still fondly remembers AIX "softlinks" where you could "ln -s"
something to "/foo/$BAR/baz" and the value of the env var BAR would be
taken into account at access/expansion time]
> But the problem is that the last started syslog-ng aquires the lock
> for the NFS shared /var/data/chroot/<username>/dev/log so the other
> server cannot read it anymore and so there is no sftp log for the
> sessions on the other server.
a file locking method that just does *not* work across NFS shares -
there's about half a dozen different methods available, see, e.g.,
https://dovecot.org/pipermail/dovecot/2011-July/060149.html
-, you could circumvent that effect from the get-go ...)
Peter Stuge
> But the problem is that the last started syslog-ng aquires the lock
> for the NFS shared /var/data/chroot/<username>/dev/log so the other
> server cannot read it anymore
the NFS server?
Douglas E Engert wrote:
> You already have 800 NFs volumes and they are all mounted on each server.
and avoiding per-user runtime setup such as mounts is a requirement.
Jochen Bern wrote:
> I *still* suspect that if only you could configure the syslogd's to use
> a file locking method that just does *not* work across NFS shares -
> there's about half a dozen different methods available, see, e.g.,
>
> https://dovecot.org/pipermail/dovecot/2011-July/060149.html
>
> -, you could circumvent that effect from the get-go ...
I'm curious what kind of locking it is. Maybe the contention is
all within the NFS layer and could be overcome by setting a nolock
or local_lock mount option on the SFTP servers, if either option is
acceptable for the use case.
Kind regards
//Peter
Douglas E Engert
On 9/28/2021 6:29 PM, Peter Stuge wrote:
> Hildegard Meier wrote:
>> But the problem is that the last started syslog-ng aquires the lock
>> for the NFS shared /var/data/chroot/<username>/dev/log so the other
>> server cannot read it anymore
>
> Is it known what kind of lock this is? Was it investigated? Maybe on
> the NFS server?
some index into the server's kernal unix-stream like process and fd used in the bind is stored in the chrooted /dev/log when
the syslog-ng creates it. The would match the statement:
> So, if a user logs in on the first server, where syslog-ng was started least(last?), the user's sftp activity is logged on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
Says: "The AF_UNIX (also known as AF_LOCAL) socket family is used to communicate between
processes on the same machine efficiently.
The "same machine" is the key.
netstat should show what sockets are in use.
>
>
> Douglas E Engert wrote:
>> You already have 800 NFs volumes and they are all mounted on each server.
>
> AIUI there's only one NFS export with all homedirs mounted on each server,
> and avoiding per-user runtime setup such as mounts is a requirement.
>
>
> Jochen Bern wrote:
>> I *still* suspect that if only you could configure the syslogd's to use
>> a file locking method that just does *not* work across NFS shares -
>> there's about half a dozen different methods available, see, e.g.,
>>
>> https://dovecot.org/pipermail/dovecot/2011-July/060149.html
>>
>> -, you could circumvent that effect from the get-go ...
>
> Looking through the afsocket module in syslog-ng it does no file locking.
>
>
> I'm curious what kind of locking it is. Maybe the contention is
> all within the NFS layer and could be overcome by setting a nolock
> or local_lock mount option on the SFTP servers, if either option is
> acceptable for the use case.
>
>
> Kind regards
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-...@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
Jochen Bern
> On 9/28/2021 6:29 PM, Peter Stuge wrote:
>> Hildegard Meier wrote:
>>> But the problem is that the last started syslog-ng aquires the lock
>>> for the NFS shared /var/data/chroot/<username>/dev/log so the other
>>> server cannot read it anymore
>>
>> Is it known what kind of lock this is? Was it investigated? Maybe on
>> the NFS server?
>
> I suspect it is not a lock, but the "struct sockaddr_un" used with bind
> and connect or
> some index into the server's kernal unix-stream like process and fd
> used in the bind is stored in the chrooted /dev/log when
> the syslog-ng creates it. The would match the statement:
> > So, if a user logs in on the first server, where syslog-ng was
> started least(last?), the user's sftp activity is logged on the first
> server.
> > But if the user logs in on the second server, it's sftp activity
> is not logged, neither on the second nor on the first server.
> >
> > If the syslog-ng is then restarted on the second server, the sftp
> user's activity is exclusively logged only on the second server and only
> for logins on the second server.
indeed REMOVE AND RECREATE the /dev/log sockets, then any user logging
into server B afterwards would find and open a socket that is not the
same (inode number) as the one server B's syslogd (created and) opened
when *it* started. Presto, implicit disconnect.
(The difference being that the socket itself would not have to store any
additional state about the open FDs pointing to it, which I doubt it has
any capacity for.)
If that theory holds, then all we'd need to do is to force both
syslogd's to use *pre-existing* sockets instead ... (And add a little
housekeeping background job that detects new users / chroots, creates
the matching socket, and HUP's both syslogd's so as to reopen them all.)
Regards,
Hildegard Meier
Add the following line to /etc/fstab:
/var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2
Then:
mkdir /usr/local/sftp_chroot_bind_mount
mount /usr/local/sftp_chroot_bind_mount
to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
Then replace in the syslog-ng config:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
Restart syslog-ng
That would have been such a simple workaround.
But unfortunately, the problem is not solved with this.
The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
Hildegard Meier
internal-sftp[27918]: session opened for local user <username> from [1.2.3.4]
sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504 ssh2
? With Ubuntu 18.04.1 LTS and it's shipped OpenSSH 7.6p1-4ubuntu0.5 I can not get any sftp session logging when having only an empty /dev directory under the user's chroot directory, without it's "log" file.
I wonder if it would be a bug or a feature if you can manage to get sftp session logging without /dev/log in the sftp user's chroot dir?
What CentOS and OpenSSH version do you have exactly?
What could be the difference of the CentOS OpenSSH? Do you have special starting options?
Peter,
> when using internal-sftp as opposed to executing the sftp-server binary.
First login:
ps auxww |grep sftp
root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
Then logout and login again, second login:
ps auxww |grep sftp
root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
root 4630 0.5 0.1 74736 6596 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
sftp_na+ 4632 0.0 0.0 74736 3552 ? S 11:05 0:00 sftpd: sftp_nagios@notty
sftp_na+ 4633 0.0 0.0 74736 3164 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
> Gesendet: Sonntag, 26. September 2021 um 22:20 Uhr
> Von: "Peter Stuge" <pe...@stuge.se>
> An: openssh-...@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
Hildegard Meier
but just the whole sftp session logs from all users wnet to facility LOCAL5 and I fetched facility LOCAL5 into one log file. But /dev/log in each sftp user's chroot dir was required for that, though.
Hildegard Meier
I think both the LD_PRELOAD and overlay fs approach are too complicated and hacky for me. Additionally, the user's chroot NFS share is exported by a proprietary NFS storage appliance (Tegile), and I would like to keep that as a simple nfs server, as we experienced already some stramnge behaviour with that appliance in the past, and I am happy it is working now stable finally.
> Gesendet: Dienstag, 28. September 2021 um 20:27 Uhr
> Cc: "Hildegard Meier" <daku...@gmx.de>
>
David Newall
On 29/9/21 6:54 pm, Hildegard Meier wrote:
>> Von: "Hildegard Meier"<daku...@gmx.de>
>> If one does not use the/dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
> Sorry I think this is not true, I think I confused this with the first situation when I did not fetch each sftp chrooted user's /dev/log with a specific syslog-ng source config,
> but just the whole sftp session logs from all users wnet to facility LOCAL5 and I fetched facility LOCAL5 into one log file. But /dev/log in each sftp user's chroot dir was required for that, though.
I don't think it should be this hard to fix. I'd like to help.
I'm sorry if the questions have already been asked; if so, I missed it.
What is /dev/log on each server (not the one in chroot). I.e. output of
ls -l /dev/log
As Peter Stuge said, if you're using internal-sftp, the log device would
be opened at start of sshd. For external sftp, what is /dev/log in each
NFS chroot? I.e. output of ls -l /var/data/chroot/dev/log
What OS are used on your sftp server (i.e. not the NFS server, and not
the sftp client machine.)
Regards,
David
Hildegard Meier
> (This is based on the syslog-ng configure which create a unix-stream in every volume when started.)
All the sftp user's chrrot directories are under that NFS mount.
There are only 800 sftp config objects, each reads the /dev/log of each sftp user (/var/data/chroot/<username>/dev/log)
The directory structure is like this (for the user sftp_nagios):
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 .
drwxr-xr-x+ 1143 root root 1145 Sep 23 15:07 ..
drwxr-x---+ 2 root sftp_nagios 3 Sep 29 11:26 dev
drwxr-xr-x+ 2 root root 3 Oct 31 2014 etc
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:19 in
dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .ssh
ls -al /var/data/chroot/sftp_nagios/dev
total 4
drwxr-x---+ 2 root sftp_nagios 3 Sep 29 11:26 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 log
ls -al /var/data/chroot/sftp_nagios/etc/
total 6
drwxr-xr-x+ 2 root root 3 Oct 31 2014 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
-rw-r--r--+ 1 root root 2309 Oct 31 2014 localtime
ls -al /var/data/chroot/sftp_nagios/in/test/
total 5
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:21 .
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:19 ..
drwxrwx--- 2 sftp_nagios sftp_nagios 5 Sep 21 16:47 nagios
(nagios does test uploads to it's chrroted dir /in/test/nagios/)
ls -al /var/data/chroot/sftp_nagios/.ssh/
total 4
dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
-r--r-----+ 1 root sftp_nagios 401 Sep 10 09:30 authorized_keys
(this is for public key auth, in the future this shall be moved out of the user's chroot dir structure as it is unwanted that the users can change/view that file)
Hildegard Meier
> indeed REMOVE AND RECREATE the /dev/log sockets,
It is never removed afterwards.
Hildegard Meier
Thanks David, your help is much appreciated.
>What is /dev/log on each server (not the one in chroot). I.e. output of ls -l /dev/log
On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
ls -l /dev/log
lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log
On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
ls -l /dev/log
srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log
>For external sftp, what is /dev/log in each NFS chroot? I.e. output of ls -l /var/data/chroot/dev/log
Both above sftp servers (old and new generation) mount /var/data/chroot/ from a third NFS server appliance.
Every sftp user has it's (chroot) home dir /var/data/chroot/<username>/ under that single NFS mount.
E.g. for the user "sftp_nagios" this is
ls -l /var/data/chroot/sftp_nagios/dev/log
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log
Best regards
Jochen Bern
> Jochen, are you sure that you see the real sftp user session detailed activity log, e.g.
>
> internal-sftp[27918]: session opened for local user <username> from [1.2.3.4]
> internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
>
> etc. and not just the sshd auth log, e.g.
>
> sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504 ssh2
connectivity *and* concurrent-file-operations problems, I'm *quite* sure
of that. :-)
> sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
> sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
> sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
> sshd[27049]: sent status No such file [postauth]
> sshd[27049]: sent status No such file [postauth]
> sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
> sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
> sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
> sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]
- all from today's /var/log/messages .
> I wonder if it would be a bug or a feature if you can manage to get sftp
> session logging without /dev/log in the sftp user's chroot dir?
exactly), hence my reluctance to openly recommend my setup to others ...
> What CentOS and OpenSSH version do you have exactly?
(openssh-7.4p1-21.el7.x86_64).
> Do you have special starting options?
*logging* in *any* way would be:
> SyslogFacility AUTHPRIV
> UsePAM yes # That's why there's messages from PAM in the log above
> UsePrivilegeSeparation sandbox
> Match group mandanten
>> indeed REMOVE AND RECREATE the /dev/log sockets,
>
> If /dev dir under sftp user's chroot dir exists but there is no "log" file in it, it gets created by syslog-ng.
> It is never removed afterwards.
share*, and never removed, that means that all the .../dev/log's there
are were created *ONCE* by whichever syslogd got restarted *first* after
the user was created, correct? But still only the syslogd restarted
*last*, no matter whether on the same server or the other, gets that
user's log messages? I'm getting a murder mystery vibe here ...
Regards,
Jochen Bern
> ls -al /var/data/chroot/sftp_nagios/etc/
> total 6
> drwxr-xr-x+ 2 root root 3 Oct 31 2014 .
> drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
> -rw-r--r--+ 1 root root 2309 Oct 31 2014 localtime
> # ls -al ~binect/etc
> insgesamt 8
> drwx--x---. 2 root mandanten 31 26. Jan 2018 .
> drwxr-x---. 5 root mandanten 62 4. Nov 2019 ..
> -rw-r-----. 1 root mandanten 24 26. Jan 2018 group
> -rw-r-----. 1 root mandanten 90 26. Jan 2018 passwd
> # grep . ~binect/etc/*
> /home/chroot/binect/etc/group:root:x:0:
> /home/chroot/binect/etc/group:users:x:[GID of group "mandanten"]:
> /home/chroot/binect/etc/passwd:root:x:0:0:root:/:/usr/sbin/nologin
> /home/chroot/binect/etc/passwd:binect:x:[UID of "binect"]:[GID of "mandanten"]:Mandant binect:/:/usr/sbin/nologin
- just so that the user's "ls -l" output is more readable than listing
raw UIDs and GIDs.)
> ls -al /var/data/chroot/sftp_nagios/.ssh/
> total 4
> dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .
> drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
> -r--r-----+ 1 root sftp_nagios 401 Sep 10 09:30 authorized_keys
>
> (this is for public key auth, in the future this shall be moved out of the user's chroot dir structure as it is unwanted that the users can change/view that file)
> Match group mandanten
> AuthorizedKeysCommand /usr/local/sbin/MKLookup
> AuthorizedKeysCommandUser akcu
> # cat /usr/local/sbin/MKLookup
> #!/bin/sh
>
> MAIN_FILE="/etc/mand/pubkeys"
> MASTER_FILE="/etc/mand/masterkeys"
>
> MANDANT="$1"
> if [ "`echo $MANDANT | tr 'A-Za-z0-9-' _ | sed -e 's/^_*$/_/'`" != "_" ]; then
> # Unsupported characters in username. Refuse to work.
> exit 0
> fi
> if [ -r "$MAIN_FILE" ]; then
> grep '^ *#'"$MANDANT"'# *ssh-' "$MAIN_FILE" | sed -e 's/^ *#'"$MANDANT"'# *//'
> fi
> if [ -r "$MASTER_FILE" ]; then
> cat "$MASTER_FILE"
> fi
> exit 0
> # grep '^#binect#ssh-r.*Bern' /etc/mand/pubkeys | sed -e 's/ .* / ... /'
> #binect#ssh-rsa ... Joche...@Binect.de
(Making the entries' format so that they'd be *nonfunctional comments*
if they'd ever be read as normal authorized_keys lines is an extra
security precaution by paranoid /me ;-)
Regards,
--
Jochen Bern
Systemingenieur
F +49 6151 9067-290
E joche...@binect.de
W www.binect.de
Binect GmbH
Robert-Koch-Str. 9
64331 Weiterstadt
Geschäftspost.Einfach.Digital.
Wir sind nach ISO/IEC 27001:2013 und 9001:2015 zertifiziert.
BMWi fördert digitale Lösungen für KMU.
Geschäftsführung: Dr. Frank Wermeyer, Michael Imiolczyk
Unternehmenssitz: Weiterstadt
Register: Amtsgericht Darmstadt, HRB 94685
Umsatzsteuer-ID: DE 221 302 264
Hildegard Meier
> > sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
> > sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
> > sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
> > sshd[27049]: sent status No such file [postauth]
> > sshd[27049]: sent status No such file [postauth]
> > sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
> > sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
> > sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
> > sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]
> > SyslogFacility AUTHPRIV
> > UsePAM yes # That's why there's messages from PAM in the log above
> > UsePrivilegeSeparation sandbox
> > Subsystem sftp internal-sftp
> > Match group mandanten
> > PermitTTY no
> > ForceCommand internal-sftp -l INFO -u 0077
>
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039673.html
where you post the output of "egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config"
You have "ChrootDirectory" set in sshd_config, right?
E.g. I have set
ChrootDirectory %h
> If a .../dev/log is created within the .../dev/ directory *on the NFS
> share*, and never removed, that means that all the .../dev/log's there
> are were created *ONCE* by whichever syslogd got restarted *first* after
> the user was created, correct? But still only the syslogd restarted
> *last*, no matter whether on the same server or the other, gets that
> user's log messages?
Jochen Bern
>>> sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
>>> sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
>>> sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
>>> sshd[27049]: sent status No such file [postauth]
>>> sshd[27049]: sent status No such file [postauth]
>>> sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
>>> sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
>>> sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
>>> sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]
>
> Have all sftp log messages from today the prefix sshd[27049]?
-D"'s PID does not show up at all).
(Did I say in previous posts that I took this from /var/log/messages ?
Of course *not*, thanks to:
>>> SyslogFacility AUTHPRIV
and CentOS' default syslogd configs, it's all in /var/log/secure .)
> Sorry for this question, but [...]
>
> E.g. I have set
> ChrootDirectory %h
yes. I also see the cwd as "/" and the group ownerships as "users" in a
test login (while the GID is named "mandanten" in the *real* /etc/group
), so the chroot() definitely works. I'd be in HUGE trouble if it
didn't. :-3
Douglas E Engert
Another take on the direct bind mount:
If server1 did:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
Then for each user server1 would also do (or have in fstab):
mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1
and server2 did:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
and also do this:
mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2
This would allow each server to have its own /dev/log_host so they would not overwrite each others,
but when the libc went to write to the chrooted /dev/log, the kernel would write to /dev/log_serverX because
the direct bind is in the host's kernel.
I have not tried this.
Douglas E Engert
On 9/29/2021 8:34 AM, Douglas E Engert wrote:
> Thanks for the info on the single NFS volume mount.
>
> Another take on the direct bind mount:
>
>
> If server1 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
>
> Then for each user server1 would also do (or have in fstab):
> mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1
>
> and server2 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
Should read:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server2" optional(yes) ); };
David Newall
This, I believe, explains your problem and how to achieve your goal:
On 29/9/21 7:55 pm, Hildegard Meier wrote:
>> What is /dev/log on each server (not the one in chroot). I.e. output of ls -l /dev/log
> On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
> ls -l /dev/log
> lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log
>
> On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
> ls -l /dev/log
> srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log
On Linux, a process opens /dev/log when it first uses syslog(3). It
writes log messages there. On your new server, /dev/log is a link to
/run/systemd/journal/dev-log, which is a UNIX domain socket. On the old
server, /dev/log is /dev/log, which is also a UNIX domain socket.
The syslog daemon conventionally also opens /dev/log to read incoming
log messages. That's how log messages get from arbitrary process to
syslog daemon: they communicate via the same UNIX domain socket.
>> For external sftp, what is /dev/log in each NFS chroot? I.e. output of ls -l /var/data/chroot/dev/log
> Both above sftp servers (old and new generation) mount/var/data/chroot/ from a third NFS server appliance.
> Every sftp user has it's (chroot) home dir/var/data/chroot/<username>/ under that single NFS mount.
>
> E.g. for the user "sftp_nagios" this is
>
> ls -l /var/data/chroot/sftp_nagios/dev/log
> srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log
When the syslog daemon starts, it needs to be able to open (in fact,
create) the same UNIX domain socket that the external sftp server will
write to. When it starts! That means, the socket that the external
sftp server will write to must be visible in the chroot environment and
also must be visible when syslog starts.
I'm imagining you have an NFS share to mount over /var/data/chroot on
both of your SFTP servers so that you can pre-populate an empty
directory for each sftp user. I'm imagining that you NFS mount each
user's home directory inside that when the user logs in (i.e. connects
using sftp, in your case).
Finally, I infer that when a user connects with sftp, they get an
external sftp process chrooted to /var/data/chroot/<username> with a
/dev/log inside that. This can't work. The syslog daemon on neither old
nor new server can open /var/data/chroot/<username>/dev/log at startup
because there is no such file at startup.
What you should do is create /var/data/chroot/{dev,home} on both
servers. These are local directories, and not NFS mounts. This allows
you to configure syslog daemon to read log messages from
/var/data/chroot/dev/log (wich syslog will create when it starts.) You
can't NFS mount over /var/data/chroot because you've got two different
sftp servers, each running syslog daemon. The server that starts second
will remove the /var/data/chroot/dev/log socket that the first one
created, and create it's own socket in that place. The
/var/data/chroot/dev directory must be truly local on each machine.
You'll also have /var/data/chroot/lib/sftp-server (or
/var/data/chroot/lib/openssh/sftp-server) on each machine, so that sshd
can exec that inside the chroot when a user connects. The user will be
in chroot at /var/data/chroot.
You mount the NFS share containing the empty directories for each user.
You mount it at /var/data/chroot/home.
You dynamically mount the user's home directory at
/var/data/chroot/home/<username>.
When a user logs in using sftp, they're in a chroot at /var/data/chroot
with their home directory at /home/<username> (as they see it).
What you might not like about this is that while two different users are
logged on to the same server at the same time, their home directories
are also mounted and accessible at the same time. Mitigate this by
chmod a=x on the NFS share that you'll mount over
/var/data/chroot/home. This is security through obscurity: Users can't
see what's in home. Also, something like chmod go= on each users
NFS-shared home directory. This prevents anybody from writing in any
home directory other than their own.
Regards,
David
Hildegard Meier
did not know abount "mount move" before, interesting.
But unfortunateley, it would be an impractical complicated mess to have this hundrets of extra mounts, which additionally are server specific, so at least for me this would not be a possible approach.
Additionally, reading the mount man page I doubt this would be possible, anyway. I guess /dev/log would not be available in the chroot anymore.
And it says "moving a mount residing under a shared mount is invalid and unsupported".
But as said, this is approach is impractical for me anyways.
"The move operation
Move a mounted tree to another place (atomically). The call is:
mount --move olddir newdir
This will cause the contents which previously appeared under olddir to now be accessible under newdir. The physical location of the files
is not changed. Note that olddir has to be a mountpoint.
Note also that moving a mount residing under a shared mount is invalid and unsupported. Use findmnt -o TARGET,PROPAGATION to see the cur‐
rent propagation flags."
> Gesendet: Mittwoch, 29. September 2021 um 15:34 Uhr
> Von: "Douglas E Engert" <deen...@gmail.com>
> An: openssh-...@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
Hildegard Meier
I think you describe the problem correctly, and your approach could work, but unfortunately it would be a way too complicated mess.
As I had written in my summary
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039677.html
"Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share."
Best regards
Gesendet: Mittwoch, 29. September 2021 um 17:00 Uhr
Von: "David Newall" <ope...@davidnewall.com>
An: "Hildegard Meier" <daku...@gmx.de>
Cc: "Damien Miller" <d...@mindrot.org>, "OpenSSH List" <openssh-...@mindrot.org>
Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
Hi Hildegard,
This, I believe, explains your problem and how to achieve your goal:
On 29/9/21 7:55 pm, Hildegard Meier wrote:
What is /dev/log on each server (not the one in chroot). I.e. output of ls -l /dev/log
On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
ls -l /dev/log
lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log
On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
ls -l /dev/log
srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log
On Linux, a process opens /dev/log when it first uses syslog(3). It writes log messages there. On your new server, /dev/log is a link to /run/systemd/journal/dev-log, which is a UNIX domain socket. On the old server, /dev/log is /dev/log, which is also a UNIX domain socket.
The syslog daemon conventionally also opens /dev/log to read incoming log messages. That's how log messages get from arbitrary process to syslog daemon: they communicate via the same UNIX domain socket.
For external sftp, what is /dev/log in each NFS chroot? I.e. output of ls -l /var/data/chroot/dev/log
Both above sftp servers (old and new generation) mount /var/data/chroot/ from a third NFS server appliance.
Every sftp user has it's (chroot) home dir /var/data/chroot/<username>/ under that single NFS mount.
David Newall
On 30/9/21 1:01 am, Hildegard Meier wrote:
> Thanks David,
>
> I think you describe the problem correctly, and your approach could work, but unfortunately it would be a way too complicated mess.
>
> As I had written in my summary
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039677.html
>
> "Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share."
I may have explained myself poorly. No bind mount is required. You need
one NFS mount over /var/data/chroot/home (yielding /home/<user> for all
possible users), and one dynamic mount of user's home directory when
they log in. (Dynamically mounting users home directory is quite common
and is handled by automount.)
I might also have misunderstood your desire to automount home
directories. Perhaps you don't want to do that, and have all user home
directories on the one NFS share, mounted over /var/data/chroot/home.
If so, great, that's all you need: create /var/data/chroot (a local
directory containg dev, home and lib directories, as well as ssh-server
executable) and NFS mount the share containing all of your user
directories over /var/data/chroot/home.
If I've still not got it right, you might explain to me what is
complicated so that I can better understand.
Peter Stuge
Hildegard Meier wrote:
> > I think that works specifically because *no* new process is created
> > when using internal-sftp as opposed to executing the sftp-server binary.
>
> For every sftp subsystem login (here with user "sftp_nagios"), I
> see a new sftpd process created:
> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
have written different, refering to sshd vs. sftp-server:
sshd creates one child process for each incoming TCP connection (4590)
and after authentication it creates another child process (4592) which
runs as the authenticated user to handle the session, and then the
sftp subsystem channel is handled by yet another child process (4593).
These processes all run (different parts of) sshd, they do not
execute any other program, when using internal-sftp not even 4593.
This is relevant because as mentioned there is then no point where
sshd executes the sftp-server binary, so also no opportunity to use
LD_PRELOAD only for SFTP. All of sftp-server is built into sshd,
specifically to enable internal-sftp, with the great benefit that no
sftp-server binary is needed inside the chroot.
My thought was still wrong though. It could have been relevant for
the syslog() socket because that is closed automatically on exec()
and not on fork(), but sshd re-opens the syslog() socket right after
fork() so the internal-sftp vs. sftp-server makes no difference for
logging, only for LD_PRELOAD convenience.
Hildegard Meier wrote:
> > If a newly-started syslogd on server A does
> > indeed REMOVE AND RECREATE the /dev/log sockets,
>
> file in it, it gets created by syslog-ng. It is never removed afterwards.
if the socket inode does not already exist in the filesystem.
So syslog-ng does what all AF_UNIX socket servers must: it unlinks the
socket from the filesystem first, so that a new socket inode can be
created by bind().
Jochen identified the problem.
Jochen Bern wrote:
> (The difference being that the socket itself would not have to store any
> additional state about the open FDs pointing to it, which I doubt it has
> any capacity for.)
filesystem inode to its socket object. Since the inode created by
daemon B means nothing within the kernel on system A, connecting to
that inode now also visible on system A doesn't reach daemon A.
Jochen Bern wrote:
> If that theory holds, then all we'd need to do is to force both
> syslogd's to use *pre-existing* sockets instead ...
Hildegard Meier wrote:
> I think both the LD_PRELOAD and overlay fs approach are too
> complicated and hacky for me.
There aren't very many other solutions:
* A patch to change SFTP logging
The attached patch adds an -E option to sftp-server (as exists for sshd)
to log to a specific file. With -E /sftp.log on the internal-sftp command
line in sshd_config on both servers they would then both log to a file in
the user's home directory. Beware of write contention on that log file
when the same user is connected to both servers.
Ensure that the file exists before using this configuration. Make file
ownership root:user and mode 620 if the user shouldn't be able to read it.
The patch is small, but until an equivalent is accepted in OpenSSH,
if ever, it requires maintaining a local package.
Note that as proposed this patch only works with a regular file, not
a syslog socket. Supporting a socket is of course possible but requires
a larger patch re-implementing much of syslog(). I can write that if
it would be accepted into OpenSSH.
* Add -e to the internal-sftp command line and create a custom parser
Because internal-sftp is still the sshd process this log message
parser must be connected already to the initial sshd stderr and must
handle anything that can be output on stderr by any sshd child process,
not just SFTP messages, making this alternative less attractive.
Kind regards
//Peter
Peter Stuge
Hildegard Meier wrote:
> ps auxww |grep sftp
> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
I guess that this is a TLS FTP server.
Try ps fax to see process relationships.
The "sftpd:" prefix for the sshd processes may be related to SELinux;
nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
Jim Knoble
If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).
Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....
If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.
____________________
[daemontools]: https://cr.yp.to/daemontools.html
--
jmk
> On Sep 29, 2021, at 19:44, Peter Stuge <pe...@stuge.se> wrote:
>
> Small clarification:
Hildegard Meier
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
To have a clean seperation of the standard sshd service on port 22 and our sftp server that listens on another port,
and both have and shall have really nothing to do which each other,
I created a new sshd instance named "sftpd":
/usr/sbin/sftpd is a symlink to /usr/sbin/sshd
Copied /etc/ssh/ and /etc/default/ssh to /etc/sftpd/ and /etc/default/sftpd
Copied /lib/systemd/system/ssh.service to /etc/systemd/system/sftpd.service
end replaced in the copy all occurences of regex "sshd?" with sftpd
E.g.
[Service]
EnvironmentFile=-/etc/default/sftpd
ExecStartPre=/usr/sbin/sftpd -t
ExecStart=/usr/sbin/sftpd -D $SFTPD_OPTS -f /etc/sftpd/sftpd_config
Important for correct autostart of sftpd on boot is:
[Unit]
After=network.target auditd.service sshd.service
Because sftpd needs to be started after sshd because only sshd creates /run/sshd/
and that is needed by sftpd (hard compiled in sshd).
Pitfall:
When stopping sshd /run/sshd/ gets deleted and you get the following sftpd error:
fatal: Missing privilege separation directory: /run/sshd
and sftpd does not accept new connections anymore etc.
Here ps fax output :)
12981 ? Ss 0:02 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
18846 ? Ss 0:00 \_ sftpd: sftp_nagios [priv]
18848 ? S 0:00 \_ sftpd: sftp_nagios@notty
18849 ? Ss 0:00 \_ sftpd: sftp_nagios@internal-sftp
Peter Stuge
Hildegard Meier wrote:
> > 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
>
> and our sftp server that listens on another port, and both have and
> shall have really nothing to do which each other, I created a new
> sshd instance named "sftpd":
build for the standalone sftp server. Does the patch idea seem viable?
> [Service]
> EnvironmentFile=-/etc/default/sftpd
> ExecStartPre=/usr/sbin/sftpd -t
> ExecStart=/usr/sbin/sftpd -D $SFTPD_OPTS -f /etc/sftpd/sftpd_config
>
> Important for correct autostart of sftpd on boot is:
> [Unit]
> After=network.target auditd.service sshd.service
>
> Because sftpd needs to be started after sshd because only sshd
> creates /run/sshd/ and that is needed by sftpd (hard compiled in sshd).
the sftpd service needs to use the same path.
A local sshd build allows cleanly solving that as well.
These commands build an OpenSSH-8.8p1 sshd with my sftp-server patch
and the alternate privsep directory:
git clone https://anongit.mindrot.org/openssh.git &&
cd openssh &&
git checkout -b v88_sftplog V_8_8_P1 &&
wget -O 0001-sftplog.patch \
https://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20210930/9b40e2d4/attachment.bin &&
git am 0001-sftplog.patch &&
autoreconf -fi -I m4 &&
./configure --prefix=/usr/local/sftpd --with-privsep-path=/run/sftpd &&
make sshd
Thanks to --prefix this sshd will not interfere with anything installed
via package management. Place configuration and keys (or symlinks) in
/usr/local/sftpd/etc/ and remember PidFile /run/sftpd.pid in sshd_config.
In any case you can express the current dependency to systemd to avoid
errors:
> When stopping sshd /run/sshd/ gets deleted and you get the
> following sftpd error:
> fatal: Missing privilege separation directory: /run/sshd
> and sftpd does not accept new connections anymore etc.
tell systemd that sftpd requires sshd, and should be stopped first
if sshd is being stopped.
Also create a Wants:
mkdir /etc/systemd/system/sshd.service.wants
ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/
..to tell systemd that it should try to start sftpd when sshd starts.
Keep After=sshd.service in sftpd.service.
With those changes you can stop and start sftpd while sshd continues
running like before, if you stop sshd then sftpd is now stopped first,
and if you start either sshd or sftpd when neither is running then
both are started in the right order.
> Here ps fax output :)
Kind regards
//Peter
Nico Kadel-Garcia
figure out "why are you hurting yourself doing this"? In the amount of
time burned trying to integrate hundreds of NFS mounts and chroot cage
configurations, you could have "vsftpd" to support FTPS rather than
SFTP, gotten clean isolated upload and download cages without having
to embed /dev/log in the chroot cages, enabled much simpler and stable
automounting, and spent the rest of the time on the lobbying effort
necessary to convince your user community to switch to FTPS enabled
clients.
Unless you have some compelling need for SSH key based access, why are
you spending these cycles on this work?
Jim Knoble
I was able to get logging from sshd to stderr using 'sshd -e', but it appears as if internal-sftp does not send messages to stderr (or, at least, to the same file handle). With 'LogLevel VERBOSE', sshd emits expected messages about connection and accepting keys, then:
User child is on pid 1234
Starting session: subsystem'sftp' for user1 from 1.2.3.4 port 5678 id 0
It's not until closing the session that we get more:
Close session: user user1 from 1.2.3.4 port 5678 id 0
Received disconnect from 1.2.3.4 port 5678:12: disconnected by user
[...]
As expected, command-line flags in sshd_config for the internal-sftp command have no effect ('-e -l DEBUG', for example).
I haven't looked at the code to see whether there's an easy path here or not. Too bad, it seemed like a reasonable approach -- and could still be, if someone were to implement a reasonable "log-to-stderr" for internal sftp.
The only other approach that's seems at all reasonable to me is to add code and a config twiddle to sshd to send log messages directly to, say, an RFC5425 syslog-over-TLS server at a configurable address and port. That seems like a lot of code for not so much benefit.
--
jmk
> On Sep 29, 2021, at 20:44, Jim Knoble <jmkn...@pobox.com> wrote:
>
>
Hildegard Meier
I think I have found now a workaround for the problem.
> Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.
My workaround is the following:
sudo mkdir /var/data/dev # Create a directroy under which user subdirectories are created
For very username <username> and the user's primary group <groupname> do the following:
sudo mkdir /var/data/dev/<username>
sudo chmod 550 /var/data/dev/<username> # This restrictive permission is a requirement I think
sudo chgrp <groupname> /var/data/dev/<username> # so the user can read the directory
So the new directory is exactly the same as the existing /var/data/chroot/<username>/dev directory (which is on the nfs mount /var/data/chroot/).
Then do
mount --bind /var/data/dev/<username> /var/data/chroot/<username>/dev
so /var/data/chroot/<username>/dev is now effectively local on the sftp server, not anymore on nfs mount.
Then change the syslog-ng config
from
source s_chroot_<username> { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };
(this is not strictly needed, but I think it's nice having syslog-ng definitely now only reading from local file, guaranteed not from nfs mount anymore)
I have tested this successfully with one user so far - whether the user logs in on the one sftp server or the other, syslog-ng can now log the sftp session on the affected sftp server.
While it is still a mess to have 800 (or in the future maybe 2000 or 3000) bind mounts which I very much would like to avoid, I think this workaround is acceptable because the impact is limited to the logging functionality and not the sftp service itself. Speaking, if there are problems with the bind mounts, the impact is only that there is no sftp session logging.
Also, this is very simple, uniform and clear.
And one has 100% reliable sftp session logging.
No need to change anything else of the well established production system.
I think I will create systemd unit files for every bind mount (I experienced systemd dependency cycles with bind mounts in /etc/fstab on Ubuntu18 boot before anyway, so systemd units are needed anyway).
Maybe one can define a systemd dependency so that when you unmount the nfs mount /var/data/chroot/, all the bind mounts under it are automatically unmounted before (sftpd needs to be stopped first of course, otherwise mount would be busy), and also the bind-mounts automatically be mounted after /var/dat/chroot is mounted again.
So there should not be much hassle with all those simple uniform bind mounts.
I think I will implement a nagios monitoring check that checks if for every username /var/data/chroot/<username>/dev is a mountpoint, so logging is assured.
According to https://serverfault.com/questions/102588/maximum-numbers-for-file-system-mounts-in-linux/927860#927860 the Linux Kernel mount maximum is 100 000 so hopefully 2000 or 3000 will not be a problem.
--
Possible alternative
I guess (not tested) an alternative could be to hardlink (so needs to be on the same (nfs) file system) every /var/data/chroot/<username>/dev/log to e.g. /var/data/chroot/general_dev_log (which would need access permission for all users of course)
and do
mount --bind /var/data/general_dev_log /var/data/chroot/general_dev_log
so every sftp session would log to the one and only log device (unix socket), again using the local bind mount.
So you could write all the sftp session logs of all users into one log file with syslog-ng.
Advantage:
- only one bind mount needed
- only one syslog-ng unix-stream configuration needed
Disadvantage:
You need to parse and filter the big one session log to create user-specific log files which only conatin the sessions for that user.
I have no experience with such tools and I think this filtering can not be 100% reliable, there will always be some cases where the filter will fail and log entries will be missing.
Best regards
Hildegard Meier
was to be able to have maintenance of the single sftp servers without service outage, to be able to apply operating system security patches delivered by the distribution (Ubuntu in this case).
I have no capacity to follow the OpenSSH security issues myself an then if needed re-compile newer patched versions (and not even then apply your patch additionally every time to it :)
We do here industry production service and need to stick with vanilla distribution OpenSSH and leave the delivery security patches to Ubuntu.
> Set BindsTo=sshd.service in the sftpd.service [Unit] section to
> tell systemd that sftpd requires sshd, and should be stopped first
> if sshd is being stopped.
>
> Also create a Wants:
>
> mkdir /etc/systemd/system/sshd.service.wants
> ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/
>
> ..to tell systemd that it should try to start sftpd when sshd starts.
>
> Keep After=sshd.service in sftpd.service.
>
Hildegard Meier
> you could have "vsftpd" to support FTPS rather than SFTP
(if customers insist to use FTPS they can use it on the old server, though)
> convince your user community to switch to FTPS enabled clients
The decision for SFTP was made ten years ago, and we need to support SFTP for decades now on :)
It took years to achieve the sftp service to become reasonably robust stable, there popped up many problems through the years which was then fixed, and with so many customers you experience ever new unexpected fancy client behaviour.
And then we have the forwarding from the sftp service to the backend processing and back, which is also not trivial.
Just to mention, 18 million files transferred via SFTP last month:
inbound 8 252 591
outbound 10 074 881
David Newall
On 1/10/21 2:44 pm, Hildegard Meier wrote:
> mount --bind/var/data/dev/<username>/var/data/chroot/<username>/dev
>
> so/var/data/chroot/<username>/dev is now effectively local on the sftp server, not anymore on nfs mount.
That's not right. This doesn't magically turn an NFS mount into a local
mount. If /var/data/chroot/<username>/dev is NFS mounted,
/var/data/dev/<username> is also NFS mounted.
I believe I explained what you need to do but for some reason you didn't
like it, and when I asked why, you never replied.
To repeat myself:
1. Create /var/data/chroot/dev, /var/data/chroot/home and
/var/data/chroot/lib on both machines.
2. Configure your syslog daemon to read from /var/data/chroot/dev/log.
3. Configure SSH to chroot for the SFTP users to /var/data/chroot.
4. Mount your current NFS share which contains all 800 user directories
over /var/data/chroot/home.
For extra points, instead of step 4, although I think is the bit you
didn't like, while at the same time I think is the bit that you said you
really want, so there's cognitive dissonance for you:
4. Mount a NFS share which contains empty directories for all 800 users
over /var/data/chroot/home.
5. When a user logs in use automount to NSF mount their home directory
over /var/data/chroot/home/<username>.
This is not hard.
Regards,
David
Peter Stuge
Hildegard Meier wrote:
> then if needed re-compile newer patched versions (and not even then
> apply your patch additionally every time to it :)
OpenSSH has a few releases per year (including security updates), there's
a release announcement mailing list with no other traffic, and the build
commands I sent need only minimal adjustment for new version numbers in
the git checkout command - or remove the git checkout command to always
build the very latest code, then the commands stay identical, including
applying the patch. The patch is small so should apply cleanly for some
time.
> We do here industry production service and need to stick with
> vanilla distribution OpenSSH and leave the delivery security
> patches to Ubuntu.
distributions usually ship OpenSSH with their own patches, which
every now and then introduce some problems.
And Ubuntu like all other distributions usually take maybe a day or two
even for critical updates, full new releases can take months or years,
so a local build on release day is much faster, but you are right:
A local build does require tracking releases and running commands other
than the standard update process, and while few packages are as easy to
build as OpenSSH I understand that it can still be out of scope.
> > BindsTo=sshd.service
> > ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/
>
> Thanks for the hint, I will look into it, I am not so experienced
> with systemd yet.
for neat yet precise configuration. I like it a lot. Can recommend.
Hildegard Meier wrote:
> My workaround is the following:
>
> so /var/data/chroot/<username>/dev is now effectively local on the
> sftp server, not anymore on nfs mount.
> Then change the syslog-ng config
>
> from
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>
> to
> source s_chroot_<username> { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };
>
> (this is not strictly needed, but I think it's nice having syslog-ng
> definitely now only reading from local file, guaranteed not from nfs
> mount anymore)
> I think I will create systemd unit files for every bind mount
> (I experienced systemd dependency cycles with bind mounts in /etc/fstab
> on Ubuntu18 boot before anyway, so systemd units are needed anyway).
dynamically (on boot and on systemctl daemon-reload), otherwise units
must be created and deleted explicitly by user account management.
I've attached a suggested generator script that reads /etc/passwd and
creates units for all members of the group set at the top of the script.
The sshd_config you sent matches on group sftp-cust so I used that.
The generator can be tested in a temporary directory:
dir=$(mktemp -d); ./sftplog-generator.sh "$dir" && ls -lR "$dir"
When all is good copy the script to /etc/systemd/system-generators/
and after running systemctl daemon-reload the generated units will
be in /run/systemd/system-generators/ (never stored on disk) and have
all been loaded.
systemctl restart sftpd.service then mounts everything before
starting the new sftpd process.
> Maybe one can define a systemd dependency so that when you unmount
> the nfs mount /var/data/chroot/, all the bind mounts under it are
> automatically unmounted before (sftpd needs to be stopped first of
> course, otherwise mount would be busy),
to unmount. systemd doesn't know about unsuccessful attempts to
directly umount a busy /var/data/chroot, but when stopping the
mount unit it knows to first stop the dependency tree. sftpd can
be included there.
> and also the bind-mounts automatically be mounted after /var/dat/chroot
> is mounted again.
systemd implicitly[2] adds a Requires= dependency with After= ordering
for directories above the target of mount units, so the generated units
get this for /var/data/chroot automatically, ensuring that /var/data/chroot
is mounted first.
The generator creates units with Before=sftpd.service ordering, it creates
Wants= dependencies for var-data-chroot.mount on each bind mount using a
symlink in var-data-chroot.mount.wants/, and finally it creates
BindsTo= dependencies for sftpd.service on each bind mount using so-called
drop-in[3] files in sftpd.service.d/.
The Wants= symlinks automatically mount the bind mounts when /var/data/chroot
is mounted, and the BindsTo= drop-in files ensure that when sftpd.service
is started all bind mounts are mounted first, and also that if any bind
mount is unmounted then sftpd.service is stopped.
A good example of the dependency model capabilities!
> I think I will implement a nagios monitoring check that checks if
> for every username /var/data/chroot/<username>/dev is a mountpoint,
> so logging is assured.
without all bind mounts and will stop sftpd if any bind mount is unmounted.
If stopping sftpd.service is too violent then change BindsTo=${unit}
to Requires=${unit} near the bottom of the generator, but add
BindsTo=var-data-chroot.mount to sftpd.service, otherwise sftpd will not
be stopped when stopping var-data-chroot.mount, causing that to fail.
I like how BindsTo= ensures that sftpd either works fully or not at all,
but that may not suit everyone.
> Possible alternative
> --
> to be on the same (nfs) file system) every
> /var/data/chroot/<username>/dev/log
syslog-ng creates the general socket, but syslog-ng A and syslog-ng B
will create different general socket inodes, the later will overwrite
the earlier, so it's back to square one.
> Disadvantage:
> You need to parse and filter the big one session log to create
> user-specific log files which only conatin the sessions for that user.
> I have no experience with such tools and I think this filtering can
> not be 100% reliable
messages so it's difficult to reliably associate messages with a user.
Kind regards
//Peter
[1] https://www.freedesktop.org/software/systemd/man/systemd.generator.html
[2] https://www.freedesktop.org/software/systemd/man/systemd.mount.html
[3] https://www.freedesktop.org/software/systemd/man/systemd.unit.html
Peter Stuge
David Newall wrote:
> I believe I explained what you need to do but for some reason you didn't
> like it, and when I asked why, you never replied.
> This is not hard.
I find your tone condescending, even if unintended, and I probably
wouldn't reply to it either. If you'd like to discuss that further
please email me off-list.
Have a nice weekend everyone
//Peter
David Newall
On 2/10/21 1:52 am, Peter Stuge wrote:
> David,
>
> David Newall wrote:
>> I believe I explained what you need to do but for some reason you didn't
>> like it, and when I asked why, you never replied.
> ..
>> This is not hard.
> I find your tone condescending, even if unintended
Apologies. I didn't intend to be condescending. I was trying to help
with what seems to me to be an already solved problem.
David
Hildegard Meier
> Gesendet: Freitag, 01. Oktober 2021 um 06:14 Uhr
> Von: "Hildegard Meier" <daku...@gmx.de>
> Cc: openssh-...@mindrot.org
>