Need help understanding systemd.socket concept

[Robert Latest]

Hello,

I understand how a .socket unit file with Accept=yes works: Upon connection to
the socket, systemd starts up the associate service, it communicates with
stdout and stdin connected to the socket, and shuts down.

That is not what I want. I need to have one service permanently active which
listens to stuff being passed in over the socket. I understand that this is
done by setting the 'Accept' option to false.

The manpage says: "If false, all listening sockets themselves are passed to the
started service unit, and only one service unit is spawned for all connections
(also see above)."

Frankly, I have no idea what "all listening sockets themselves are passed to
the service" means. Passed how? As file descriptors on the command line?

I looked at some .socket and the associated .service files in the /lib/systemd
hierarchy but got none the wiser.

The service is actually a Python program that collects pickled Python objects
as input from the socket, but that doesn't really matter. I wonder if systemd
can help me with this scenario or if I need to write the whole server in
Python.

The incoming data originates to the same system so I can use what I understand
to be "Unix domain sockets." The unit.socket manpage makes no reference to that
at all. And since I want to receive one whole object each time, it will have to
be a "datagram" socket, right?

As you can see I'm in way over my head here. I am hoping that systemd can make
short work of the nitty-gritty socket programming details.

robert

[Henning Hucke]

On 2021-03-20, Robert Latest <bobl...@yahoo.com> wrote:

> Hello,

Hi Robert,

> I understand how a .socket unit file with Accept=yes works: Upon connection to
> the socket, systemd starts up the associate service, it communicates with
> stdout and stdin connected to the socket, and shuts down.
>

> That is not what I want. [...]
> [...]

please start googling for more informations on the concept of sockets
under Unix flavoured operating systems.

Aside of this be told that one or the other way runing services usually
hand over incoming connections to spawned worker processes. This is true
for everything which needs more than just some milliseconds to process
the request and doesn't work with connection identifiers within the
application protocol and uses connection oriented protocols - like tcp.

SystemD socket units accept connections and spawn the specified program
to handle the specific connection. Nonetheless SystemD is permanently
listening to the specified socket. In this sense the service is
permanently available. As far as I know its not possible to write a
stdin/stdout oriented process which works with "Accept=false". Programs
which work with "Accept=false" need to be aware of the socket and need
to handle the socket by themselfs.

Concerning the concept of sockets under Unix flavoured operating
systems: As far as it can be driven it doesn't make a difference to the
program whether it listens to a network socket or to a unix domain socket.

Best regards
Henning
--
If you think technology can solve your problems you don't understand
technology and you don't understand your problems. (Bruce Schneier)

[Richard Kettlewell]

Robert Latest <bobl...@yahoo.com> writes:

> Hello,
>
> I understand how a .socket unit file with Accept=yes works: Upon
> connection to the socket, systemd starts up the associate service, it
> communicates with stdout and stdin connected to the socket, and shuts
> down.
>
> That is not what I want. I need to have one service permanently active
> which listens to stuff being passed in over the socket. I understand
> that this is done by setting the 'Accept' option to false.
>
> The manpage says: "If false, all listening sockets themselves are
> passed to the started service unit, and only one service unit is
> spawned for all connections (also see above)."
>
> Frankly, I have no idea what "all listening sockets themselves are
> passed to the service" means. Passed how? As file descriptors on the
> command line?

From systemd.socket(5):

Note that the daemon software configured for socket activation
with socket units needs to be able to accept sockets from
systemd, either via systemd's native socket passing interface
(see sd_listen_fds(3) for details) or via the traditional
inetd(8)-style socket passing (i.e. sockets passed in via
standard input and output, using StandardInput=socket in the
service file).

--
https://www.greenend.org.uk/rjk/

[Robert Latest]

Richard Kettlewell wrote:
> From systemd.socket(5):
>
> Note that the daemon software configured for socket activation
> with socket units needs to be able to accept sockets from
> systemd, either via systemd's native socket passing interface
> (see sd_listen_fds(3) for details) or via the traditional
> inetd(8)-style socket passing (i.e. sockets passed in via
> standard input and output, using StandardInput=socket in the
> service file).

Thanks, I missed that because I already got caught up in the "Accept" section
of the manpage.

So basically if I use Accept=no and StandardInput=socket, systemd will start
one service process, and that process will get connected via stdin/out to
whatever is connecting to that socket? Which means that it can't handle
concurrent connections of course.

robert

[Richard Kettlewell]

Robert Latest <bobl...@yahoo.com> writes:
> So basically if I use Accept=no and StandardInput=socket, systemd will
> start one service process, and that process will get connected via
> stdin/out to whatever is connecting to that socket? Which means that
> it can't handle concurrent connections of course.

It won’t be connected to anything; it will get the listening socket. How
you deal with concurrency (with this configuration) is up to your
server.

Accept=
Takes a boolean argument. If true, a service instance is
spawned for each incoming connection and only the connection
socket is passed to it. If false, all listening sockets

themselves are passed to the started service unit, and only
one service unit is spawned for all connections (also see
above).

--
https://www.greenend.org.uk/rjk/

[The Natural Philosopher]

Oh I think it can. It will simply invoke another copy of your program
That's how inetd works - it forks and connects to an IO based daemon

remember each inbound connection will have a unique source IP and port
and that is how e.g. inetd sorts out which packet goes to which daemon
instance


--
“I know that most men, including those at ease with problems of the
greatest complexity, can seldom accept even the simplest and most
obvious truth if it be such as would oblige them to admit the falsity of
conclusions which they have delighted in explaining to colleagues, which
they have proudly taught to others, and which they have woven, thread by
thread, into the fabric of their lives.”

― Leo Tolstoy

[Robert Latest]

Richard Kettlewell wrote:
> It won’t be connected to anything; it will get the listening socket. How
> you deal with concurrency (with this configuration) is up to your
> server.

Yeah I get that now. Also to obtain the listening socket I'd have to use a
systemd-specific API. Since my client will be written in Python I'll just use
the socketserver library to implement it and make it a regular service. It's
pretty easy anyway but I thought that systemd would make it even easier.

robert

[Teemu Likonen]

* 2021-03-21 15:20:13+0000, Robert Latest wrote:

> Yeah I get that now. Also to obtain the listening socket I'd have to
> use a systemd-specific API. Since my client will be written in Python
> I'll just use the socketserver library to implement it and make it a
> regular service. It's pretty easy anyway but I thought that systemd
> would make it even easier.

You can make systemd spawn a service instance for each connection. You
create a service unit template file named "myservice@.service". See the
"@" character.


[Unit]
Description=My Service

[Service]
ExecStart=/bin/sh -c 'while read line; do echo "$line" | sed -e s/./X/g; done'
StandardInput=socket
StandardOutput=socket


That is activated by "myservice.socket" unit file like this:


[Unit]
Description=My service Listener

[Socket]
ListenStream=2000
Accept=true


Each service instance unit will be named "myse...@SOMETHING.service".
You can try the above service example with "telnet localhost 2000".

--
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[The Natural Philosopher]

On 21/03/2021 15:20, Robert Latest wrote:
> but I thought that systemd would make it even easier.
>

Bless!



--
Canada is all right really, though not for the whole weekend.

"Saki"

[Robert Latest]

Teemu Likonen wrote:
>
> You can make systemd spawn a service instance for each connection.

I know but I don't want to (as I explained in the first post). My service needs
to make decisions based on all subsequent connections it gets.

robert

[The Natural Philosopher]

> ^^^^^^^^^^
Time travel? Or prescience?

I'd like to see your code...

> robert
>


--
"The great thing about Glasgow is that if there's a nuclear attack it'll
look exactly the same afterwards."

Billy Connolly

[Richard Kettlewell]

Robert Latest <bobl...@yahoo.com> writes:
> Richard Kettlewell wrote:
>> It won’t be connected to anything; it will get the listening
>> socket. How you deal with concurrency (with this configuration) is up
>> to your server.
>
> Yeah I get that now. Also to obtain the listening socket I'd have to use a
> systemd-specific API.

No, you can get the listening socket on stdin and call accept() on it
yourself. See the systemd.socket(5) fragment quoted earlier.

--
https://www.greenend.org.uk/rjk/

[Arnold Ziffel]

Henning Hucke <h_hucke+...@newsmail.aeon.icebear.org> wrote:

> please start googling for more informations on the concept of sockets
> under Unix flavoured operating systems.

Here's a good link:

https://beej.us/guide/bgnet/

Don't be discouraged by the "buy" link above -- the guide is browseable
and downloadable below it.

--
The ship of state is the only ship that leaks at the top.

[Robert Latest]

The Natural Philosopher wrote:
> On 22/03/2021 07:12, Robert Latest wrote:
>> Teemu Likonen wrote:
>>>
>>> You can make systemd spawn a service instance for each connection.
>>
>> I know but I don't want to (as I explained in the first post). My service
>> needs to make decisions based on all subsequent connections it gets.
>> ^^^^^^^^^^
> Time travel? Or prescience?

No. I should have written "previous." It's a simple service that receives and
collects logging messages from other processes in a single queue to mail out to
a maintainer given certain conditions. For example, it only sends out the
queued messages if the last mail was sent at leat 24h ago, or if the current
message is of a higher priority than the last one from that sender, stuff like
that. Obviously this can't be done if each incoming message spawns a new
instance of the receiving process. Since I had no idea how to implement this I
thoght maybe systemd could help me.

I ended up implementing using Python's socketserver library, it's simple, about
80 lines of code.

--
robert

[Phase3]

On 20 Mar 2021 17:57:10 GMT, Robert Latest <bobl...@yahoo.com>
wrote:

Ummmmmmmmmm ... I'm not entirely sure what you WANT
to accomplish here. A couple months back I wrote a service
that bash and ksh can run using their fake /dev/tcp device.
You wake it up, send it a param line and it spits back what
you need over stdout. Perfect for shell scripts - a 3 line
discussion.

This was a 'C' service ... Python works, but has a few more
complications, mostly involving that annoying "byte-like"
data format. I know all this because last year I was writing
a 'C' service with a Python client in mind ... until I realized
I was basically re-inventing FTP :-) It was beautiful, now
it's junk .......

You may not need a datagram socket at all, depending on
what you want to consume the data.