Logging library with network support

[Victor]

Hello.

I'm writing a set of applications running on different hosts and
communicating via network.

All applications produce some logs. I need to receive and filter log
messages from all that applications and visualize them at runtime.

So the question is: is there already a library for C or C++ that implements
both sending and receiving log messages, hopefully with filtering and
multiple receivers allowed?

The solution I've found for now is to send messages using syslog() and
configure syslog-ng to transmit, receive and pass received messages to local
application.

Now I'm looking for alternatives to compare.

Thanks.

-- Victor

[Victor]

Sorry, forgot to mention: I would like to be able to connect to server
dynamically from any host and start receiving messages.

With syslog-ng, I guess, this will require to add new rule to server syslog-
ng config and (maybe) restart syslog-ng, that's why I'm looking for other
solutions.

-- Victor

[Jorgen Grahn]

On Sun, 2012-12-16, Victor wrote:
> Victor wrote:
>
>>
>> Hello.
>>
>> I'm writing a set of applications running on different hosts and
>> communicating via network.
>>
>> All applications produce some logs. I need to receive and filter log
>> messages from all that applications and visualize them at runtime.
>>
>> So the question is: is there already a library for C or C++ that
>> implements both sending and receiving log messages, hopefully with
>> filtering and multiple receivers allowed?

At this point the answer is yes: the standard C library's syslog(3).

>> The solution I've found for now is to send messages using syslog() and
>> configure syslog-ng to transmit, receive and pass received messages to
>> local application.

Another option is to let all (the distributed) applications log, via
syslog, to a central log server. There you can let other tools read
the log files -- while still enjoying the benefits from standard
syslog logging: a familiar interface for controlling it, and things
like log compression and rotation.

>> Now I'm looking for alternatives to compare.
>>
>> Thanks.
>>
>> -- Victor
>
> Sorry, forgot to mention: I would like to be able to connect to server
> dynamically from any host and start receiving messages.

A bit unclear, but I assume this means you want to access the logs
from anywhere; you don't want to personally sit down at a specific
server to do it.

> With syslog-ng, I guess, this will require to add new rule to server syslog-
> ng config and (maybe) restart syslog-ng, that's why I'm looking for other
> solutions.

Why? With the central syslog server approach above, I guess your
requirement would translate roughly to:

ssh logserver tail -F /var/log/my_set_of_applications

or

ssh logserver tail -F /var/log/my_set_of_applications |\
my_visualization_stuff

Would that be enough?

I'm not saying syslog-to-file is enough for you (because I don't know
exactly what you need) but don't jump on exotic solutions before
you've tried the normal ones. You can do rather impressive things
without changing how the application logs.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Victor]

I like this, because it's simple and there is no need to write new code at
all.

> I'm not saying syslog-to-file is enough for you (because I don't know
> exactly what you need) but don't jump on exotic solutions before
> you've tried the normal ones. You can do rather impressive things
> without changing how the application logs.
>
> /Jorgen
>

However, there are some problems in my case:

1) There are multiple message sources, some on remote hosts and some on
local machine.

So one central log server is not enough because there are always components
individual for every user.

2) Some filtering should be used: it should be possible to receive only
messages from given components on remote machine and only with given log
level.

2) I don't want to force user to have ssh access to all hosts he (or she)
wants to monitor.

So, it will be ideal to have a software that allows:
- to create a "message bus" that unifies messages from remote and local
components
- to connect components to that bus dynamically
- to fetch messages from that bus
- to use individual buses in multiple applications

With "ssh tail -f" approach, I'm still required to write this manually. In
this case syslog work is virtually limited to saving syslog() arguments to
file, that is not too much.

So, is it possible to implement the whole system using syslog or may be
other existing software?

-- Victor

[Rainer Weikusat]

Victor <vic...@enise.org> writes:
> Jorgen Grahn wrote:

[...]

>> I'm not saying syslog-to-file is enough for you (because I don't know
>> exactly what you need) but don't jump on exotic solutions before
>> you've tried the normal ones. You can do rather impressive things
>> without changing how the application logs.

[...]

> However, there are some problems in my case:
>
> 1) There are multiple message sources, some on remote hosts and some on
> local machine.
>
> So one central log server is not enough because there are always components
> individual for every user.

Why do you think the second statement followed from the first?

> 2) Some filtering should be used: it should be possible to receive only
> messages from given components on remote machine and only with given log
> level.

The two usual ways to do this would be

- preprocess the complete log file to filter out unwanted
messages before doing 'a processing step' (visualization)

- inhibit generation of unwanted messages based on some (user
configurable) per-component log level setting

A suitably capapble syslog daemon (eg rsyslogd) can probably also be
configured to perform filtering-on-forward but letting one piece of
software generate a message and use a second one to throw it away is
sort-of stupid.

> 2) I don't want to force user to have ssh access to all hosts he (or she)
> wants to monitor.

In case of a 'log server', only access to the log files stored on this
server is necessary. Again, a 'suitably capable syslog daemon' should
also be capable to log to a variety of other 'log sinks' instead of
only to file (eg, 'some RDBMS').

> So, it will be ideal to have a software that allows:
> - to create a "message bus" that unifies messages from remote and local
> components
> - to connect components to that bus dynamically
> - to fetch messages from that bus
> - to use individual buses in multiple applications

You should try to meet a guy called 'Lennart Poettering' (working for
RedHat). Both of you could probably spend many-an-evening with
designing hypercomplicated and seriously overgeneralized solutions for
simple problems (which were already solved to begin with) ...

[Victor]

Rainer Weikusat wrote:

> Victor <vic...@enise.org> writes:
>> Jorgen Grahn wrote:
>
> [...]
>
>>> I'm not saying syslog-to-file is enough for you (because I don't know
>>> exactly what you need) but don't jump on exotic solutions before
>>> you've tried the normal ones. You can do rather impressive things
>>> without changing how the application logs.
>
> [...]
>
>> However, there are some problems in my case:
>>
>> 1) There are multiple message sources, some on remote hosts and some on
>> local machine.
>>
>> So one central log server is not enough because there are always
>> components individual for every user.
>
> Why do you think the second statement followed from the first?

Because central log server doesn't know about every user's local application
messages that should be merged together with remote ones.

Am I missing something?

>> 2) Some filtering should be used: it should be possible to receive only
>> messages from given components on remote machine and only with given log
>> level.
>
> The two usual ways to do this would be
>
> - preprocess the complete log file to filter out unwanted
> messages before doing 'a processing step' (visualization)
>
> - inhibit generation of unwanted messages based on some (user
> configurable) per-component log level setting
> A suitably capapble syslog daemon (eg rsyslogd) can probably also be
> configured to perform filtering-on-forward but letting one piece of
> software generate a message and use a second one to throw it away is
> sort-of stupid.

I see. The question in previous post was: "should I write this client-side
code myself _in case_ there is no central log server, or is there already
existing solution".

>> 2) I don't want to force user to have ssh access to all hosts he (or she)
>> wants to monitor.
>
> In case of a 'log server', only access to the log files stored on this
> server is necessary. Again, a 'suitably capable syslog daemon' should
> also be capable to log to a variety of other 'log sinks' instead of
> only to file (eg, 'some RDBMS').
>
>> So, it will be ideal to have a software that allows:
>> - to create a "message bus" that unifies messages from remote and local
>> components
>> - to connect components to that bus dynamically
>> - to fetch messages from that bus
>> - to use individual buses in multiple applications
>
> You should try to meet a guy called 'Lennart Poettering' (working for
> RedHat). Both of you could probably spend many-an-evening with
> designing hypercomplicated and seriously overgeneralized solutions for
> simple problems (which were already solved to begin with) ...

Yees, you're absolutely right.

After posting all this stuff and summarizing what the complete solution with
such requirements should be I came to reviewing my optional requirements and
choosing something much more simpler.

---> You may skip this

I think I'll keep components and their logs isolated without merging logs
at application level at all. User can configure syslog himself, if he
needs it.

Finally, all components anyway should have simple (text) notification
protocol and it is possible to reuse it adding another connection for log
messages.

Client application can use such connection to display separate window for
every component with its log which is useful for debugging.

This will allow to troubleshoot without requiring syslog, ssh, etc. All
enhanced log management still can be done using syslog.

What do you think about it?

<----

So, thanks for help and right questions.

-- Victor

[Jorgen Grahn]

On Sun, 2012-12-16, Rainer Weikusat wrote:
> Victor <vic...@enise.org> writes:

...

>> So, it will be ideal to have a software that allows:
>> - to create a "message bus" that unifies messages from remote and local
>> components
>> - to connect components to that bus dynamically
>> - to fetch messages from that bus
>> - to use individual buses in multiple applications
>
> You should try to meet a guy called 'Lennart Poettering' (working for
> RedHat). Both of you could probably spend many-an-evening with
> designing hypercomplicated and seriously overgeneralized solutions for
> simple problems (which were already solved to begin with) ...

You mean 'systemd' -- the init which restarts your daemons when they
crash ... Haven't those guys already designed a syslog replacement
fitting your description? I seem to recall a Slashdot article and a
flamewar a year ago or so ...

[James K. Lowden]

On Sun, 16 Dec 2012 21:06:05 +0400
Victor <vic...@enise.org> wrote:

> 1) There are multiple message sources, some on remote hosts and some
> on local machine.

...

> 2) Some filtering should be used: it should be possible to receive
> only messages from given components on remote machine and only with
> given log level.

I see nothing in your problem statement outside what syslog(3) can do
in terms of data capture. Your mistake, ISTM, is that you're trying to
use the logging system simultaneously to capture the data and to
collect it in discrete files (whose access permissions can be controlled
independently). The basic problem with that approach is that what
someone wants to see is often not known at the time the data are
captured. In fact, one answer often leads to another question; that is
how data analysis proceeds.

By capturing all logs on a single log server with syslog, you use
a bog-standard service. You ensure all log data can be made available
to all users. Nothing is lost, and access to the log is governed by one
set of permissions. The syslog priority can be used to control log
granularity i.e., to filter out events belonging to a predetermined
group on a per-process basis.

If you don't want all users to have access to the whole log, or if
you want to offer some kind of filtered access as a convenience, you
have some work ahead of you. A simple solution would periodically scan
the log (perhaps with awk) and produce reports, permissions for which
could be set individually. A sophisticated solution would parse the
log into a DBMS, where data access could be controlled by row and
column, and where the data can be conveniently grouped and summarized.

--jkl

[Jorgen Grahn]

I don't understand that part. The term "user" ... perhaps it's not
logging in the usual sense you're talking about, but some kind of
personal notification service?

For example, a mail server does logging, but the audiences for those
logs are pretty limited:
- a local user seeing critical errors on the mail server's console
- an admin having read access to the log files, on whatever server
they end up on
- perhaps, an admin with fewer privileges who can't normally see who
sent a mail to whom, but can check the error logs for severe
problems.

There's nothing like "I have a mailbox on this server and would like
to see the logs for my own mailbox". If you want that, you should
probably not use syslog, or any other logging package.

> 2) Some filtering should be used: it should be possible to receive only
> messages from given components on remote machine and only with given log
> level.
>
> 2) I don't want to force user to have ssh access to all hosts he (or she)
> wants to monitor.

(In my mind, another sign you're not talking about logging in the
usual sense.)

Ok, but if so, *which* access can you force the user to have? You
probably have to implement end-to-end authorization, authentication
and encryption in /some/ way.

[Victor]

James K. Lowden wrote:

> On Sun, 16 Dec 2012 21:06:05 +0400
> Victor <vic...@enise.org> wrote:
>
>> 1) There are multiple message sources, some on remote hosts and some
>> on local machine.
> ...
>> 2) Some filtering should be used: it should be possible to receive
>> only messages from given components on remote machine and only with
>> given log level.
>
> I see nothing in your problem statement outside what syslog(3) can do
> in terms of data capture. Your mistake, ISTM, is that you're trying to
> use the logging system simultaneously to capture the data and to
> collect it in discrete files (whose access permissions can be controlled
> independently). The basic problem with that approach is that what
> someone wants to see is often not known at the time the data are
> captured. In fact, one answer often leads to another question; that is
> how data analysis proceeds.

I see now.

> By capturing all logs on a single log server with syslog, you use
> a bog-standard service. You ensure all log data can be made available
> to all users. Nothing is lost, and access to the log is governed by one
> set of permissions. The syslog priority can be used to control log
> granularity i.e., to filter out events belonging to a predetermined
> group on a per-process basis.

Thanks. But consider the following situation:

server
+---------------+ +---------------+------+
| |---------| local-daemon1 | gui1 | client1
| | +---------------+------+
| remote-daemon |
| | +---------------+------+
| |---------| local-daemon2 | gui2 | client2
+---------------+ +---------------+------+

Gui1 should show logs from remote-daemon and local-daemon1.
Gui2 should show logs from remote-daemon and local-daemon2.

Speaking about central log server, do you mean that local-daemon of every
client should send its log to server and local GUI should fetch logs back
from the server?

It seems odd to me.

And otherwise every client has multiple log sources and possibly it should
merge them.

So initially I thought about a library that will automate this, but now I'm
going to leave it to user to configure syslog (also please see my previous
post).

> If you don't want all users to have access to the whole log, or if
> you want to offer some kind of filtered access as a convenience, you
> have some work ahead of you. A simple solution would periodically scan
> the log (perhaps with awk) and produce reports, permissions for which
> could be set individually. A sophisticated solution would parse the
> log into a DBMS, where data access could be controlled by row and
> column, and where the data can be conveniently grouped and summarized.

Well, I'm not configuring a server, I'm writing an application, so I guess I
should provide a mechanism, not policy and it's not my application task to
manage log access rights.

And yes, its now clear that when one doesn't use syslog, he should either
reinvent it or lack features you described above :)

>
> --jkl

-- Victor

[Victor]

Jorgen Grahn wrote:

> On Sun, 2012-12-16, Victor wrote:

[...]

>> However, there are some problems in my case:
>>
>> 1) There are multiple message sources, some on remote hosts and some on
>> local machine.
>>
>> So one central log server is not enough because there are always
>> components individual for every user.
>
> I don't understand that part. The term "user" ... perhaps it's not
> logging in the usual sense you're talking about, but some kind of
> personal notification service?
>
> For example, a mail server does logging, but the audiences for those
> logs are pretty limited:
> - a local user seeing critical errors on the mail server's console
> - an admin having read access to the log files, on whatever server
> they end up on
> - perhaps, an admin with fewer privileges who can't normally see who
> sent a mail to whom, but can check the error logs for severe
> problems.
>
> There's nothing like "I have a mailbox on this server and would like
> to see the logs for my own mailbox". If you want that, you should
> probably not use syslog, or any other logging package.

Well, yes, it is unclear -- my mistake.

There is a server application running simulation/game engine.

There is also multiple clients that can:

- connect to that server

- control server interactively, i.e. execute some operations
(well, authorized clients can)

- monitor server log, because it is very useful to see what is going
on

And in fact, server and client are not monoliths, but includes several
processes, all producing some logs.

Some messages are important and should be saved to file (for example, user
authorization). Others are not, but they are useful when controlling server
remotely.

>> 2) Some filtering should be used: it should be possible to receive only
>> messages from given components on remote machine and only with given log
>> level.
>>
>> 2) I don't want to force user to have ssh access to all hosts he (or she)
>> wants to monitor.
>
> (In my mind, another sign you're not talking about logging in the
> usual sense.)
>
> Ok, but if so, *which* access can you force the user to have? You
> probably have to implement end-to-end authorization, authentication
> and encryption in /some/ way.

Yes, I have to.

PS. Actually, all this is not ready yet and is in early development stage.

>
> /Jorgen
>
-- Victor

[Victor]

Ha, they are:
- systemd-journald [1] (syslog implementation, AFAIK)
- systemd-journal-gatewayd [2] (allows to fetch server log messages
via http and json).

So one can run journald and gatewayd at all hosts, fetch logs via http and
collect them in one place.

Personally for me, it sounds good, speaking about idea, not implementation.

However, I believe it's a bad idea to design your application on top of
concrete syslog and even more so on concrete init (!), because it limits
user to that concrete implementation and leaves him no choice.

[1] http://www.dsm.fordham.edu/cgi-bin/man-cgi.pl?topic=systemd-
journald.service&ampsect=8

[2] http://www.freedesktop.org/software/systemd/man/systemd-journal-
gatewayd.service.html

-- Victor

[Rainer Weikusat]

Victor <vic...@enise.org> writes:

[...]

> Ha, they are:
> - systemd-journald [1] (syslog implementation, AFAIK)
> - systemd-journal-gatewayd [2] (allows to fetch server log messages
> via http and json).
>
> So one can run journald and gatewayd at all hosts, fetch logs via http and
> collect them in one place.

Or any number of other logging daemons which provide similar
facilities but without creating files using some purposely undocumented
'binary record format' and without requiring use of a specific init
(the other difference would be that this is supposed to be a
'trustworthy' system audit log which syslog isn't).

> Personally for me, it sounds good, speaking about idea, not
> implementation.

Chances are that what you consider to be 'the idea' is exacly what
other syslog daemons also provide ...

> However, I believe it's a bad idea to design your application on top of
> concrete syslog and even more so on concrete init (!), because it limits
> user to that concrete implementation and leaves him no choice.

It is a bad idea to implement 'roll your own' schemes for collecting
diagnostic output from different applications. In the end, this will
boil down to some file which sits in some obscure directory nobody
ever looks at until the machines runs out of space. At this point,
some sysadmin (more likely, some hundreds or thousands of sysadmins)
have to reconfigure their logging system to deal with YAOTC (yet
another overflowing trashcan) and cleanup the resulting mess.

The much better idea is to use an existing mechanism. For
general-purpose diagnostic output, syslog ought to be good enough,
especially considering that a user can use all kinds of 'fancy
implementations' with extended facilities in almost arbitrarily
complicated 'network topologies' if he so desires. Isn't the purpose
of your application something other than generating diagnostics, IOW,
printing 'Hello, world!'? If so, why not leave 'handling of
diagnostics' to already existing software completely?

[Rainer Weikusat]

Jorgen Grahn <grahn...@snipabacken.se> writes:

[...]

> You mean 'systemd' -- the init which restarts your daemons when they
> crash ...

init would do that as well, except that (almost) nobody uses this
facility (it is only commonly used for gettys). But (I've been doing
'special-purpose system design and implementation' for varying usage
scenarios since 2004) you've accidentally hit a pet peeve of mine so
I'm going to expand a little on that ---

The traditional way to orchestrate a complex system startup
procedured, the SysV run-level system, provides essentially no process
control/ management facilities: init executes a particular script
(/etc/init.d/rc) in case of run-level changes which - in turn -
executes some set of other scripts symlinked (on Linux. At least
Solaris uses or used to use hardlinks instead) into the /etc/rc?.d
directories with specific arguments in order to start or stop
services. Anything beyond that has to be implemented within the
scripts themselves or in application code. This implies that there's a
huge amount of code duplication here and some features, such as
monitoring processes and restarting them in case of unexpected
termination, are usually not provided at all.

'systemd' is one approach for improving this situation. AFAIK, the
idea is roughly to implement a 'huge', monolithic, ad-hoc scriptable
'master control program' which supports every conceivable
process-control feature Linux supports, no matter if it is actually
needed for something, which has to be loaded into memory (at least
partially) all the time despite most of its code is unused most of the
time, which has been written in C 'for performance' (amusingly, the
/etc/init.d/sendmail script on my 'office workstation' can start
sendmail about an order of magnitude faster than the average service
start time systemd is reportedly capable of achieving) and which
provides some other grotty, old performance hacks everyone stopped
using for good years ago (the idea of a super server invoking
special-purpose servers on demand).

As I wrote in he first paragraph, I have had to deal with 'system
startup code' for some time and so far, I've been favoring a very much
different approach:

- I don't want to invent a scripting language if there are
numerous implementation of a standardized one I can just
use (the Bourne-shell language)

- I don't want to implement or even just have support for
features I don't need

- I don't want a 'system super daemon' consuming resources
while hanging around doing nothing

- I'm not interested in solving hypothetical performance
problems in rarely executed code

- I'm not smarter than everyone else and absolutely not more
knowledgable/ experienced. If people have been successfully
'gettting stuff done' in certain ways for several decades,
this looks like 'an approach which works' as opposed to 'the
legacy we need to overcome ASAP' to me. Especially, this
means the concept of solving complex problems with the help
of interacting simple tools.

In practice, this means whenever I need a new process management
feature, I write a small C program which does that and only that which
is supposed to be executed by the shell in 'a suitable way', relying
on the UNIX(*) feature that a program can modify (parts of) the kernel
state of a process and then execute another program in the modified
environment. A practical example of that looks like this

daemon chdir $DIR monitor -n jboss -g $RUNAS chids -u $RUNAS $JBOSS_HOME/bin/run.sh -b 127.0.0.1

Each of these primtives performs a particular action and passes the
remaining parts of its command line to execv in order to execute the
next. So far, I've accumulated the following tools

chdir

Changed the working directory of the process and execute
another program.

check-locks

Walks up through the chain of parent processes and check if a
certain set of fcntl record locks are in effect ATM.
chids

Change the uid and/or gid of the process and execute another
program.

daemon

'Daemonize' and execue another program in the backgrounded
process.

lock

Acquire a set of fcntl record locks in a particular order and
execue another program.

mail-output

Execute a program and mail its stdout (and possibly, stderr)
output to some address.

monitor

Monitor another processes' health, restart it in case of
unexpected termination and provide some process control
features (start/ stop/ restart).

rand-periodic

Run some other program such that it executes 'every n seconds'
on average, using a randomized interval.

run-at

Execute a program at a specific time (passed as UNIX(*)
timestamp).

send_args/ recv_args

Send a set of arguments over an 'argument munging transport'
(aka ssh)/ execute a command with the arguments received from
a send_args invocation.

This grew out of earlier work done for a 'Embedded Linux' OS for a UTM
appliance except that I religiously stuck to the 'one feature per
program' idea for the 2nd iteration. So far, this has worked out very
nicely.

[James K. Lowden]

On Mon, 17 Dec 2012 03:04:50 +0400
Victor <vic...@enise.org> wrote:

> > By capturing all logs on a single log server with syslog, you use
> > a bog-standard service. You ensure all log data can be made
> > available to all users. Nothing is lost, and access to the log is
> > governed by one set of permissions. The syslog priority can be used
> > to control log granularity i.e., to filter out events belonging to
> > a predetermined group on a per-process basis.
>
> Thanks. But consider the following situation:
>
> server
> +---------------+ +---------------+------+
> | |---------| local-daemon1 | gui1 | client1
> | | +---------------+------+
> | remote-daemon |
> | | +---------------+------+
> | |---------| local-daemon2 | gui2 | client2
> +---------------+ +---------------+------+
>
> Gui1 should show logs from remote-daemon and local-daemon1.
> Gui2 should show logs from remote-daemon and local-daemon2.
>
> Speaking about central log server, do you mean that local-daemon of
> every client should send its log to server and local GUI should fetch
> logs back from the server?

Yes. Keep in mind "the server" is simply the machine running syslogd.
It need not be any of the machines depicted above.

> It seems odd to me.

Perhaps, but it's simpler. Put your faith in the math. If each process
is responsible for its data and for collecting data from others (or
writing to the others' logs), there are potentially N!/2 connections.
If all processes write to a single master log, there are only N
connections. For 3 clients and a server, that's the difference between
(4*3*2)/2 = 12 connections and only 4.

> And otherwise every client has multiple log sources and possibly it
> should merge them.

Exactly. Restricting it to your description above, each GUI has to
deal with two information sources: its own and the server's. If all
processes write to a single log, each one need read from only one
log, too.

> Well, I'm not configuring a server, I'm writing an application, so I
> guess I should provide a mechanism, not policy and it's not my
> application task to manage log access rights.

That's good. If you do run into administrative concerns about
commingling your application's messages with other system messages,
remember syslogd can be configured to segregate different messages for
different facilities into separate files. Based on what you've said,
selecting the appropriate lines from the log could be based on the
ident parameter to openlog(3).

HTH.

--jkl

[Victor]

Looks nice. I'll take a note.

Personally I would prefer this "one feature per program" approach, but I
think I understand why one may want a thing like one big master control
program.

If your goal is a very stable and user-friendly GUI-only OS, you maybe want
to have more control of all its components and want to provide only
restricted set of configuration options lacking flexibility - if you want
simple GUI for _everything_, you should keep your back-end simple too,
right?

So, I'm considering this as just another effort that maybe has nothing to do
with Unix - I don't need it, however other people may need.

Probably, if they complete that goal, the only significant component from
Unix world they will use will be Linux kernel, like in Android.

PS. AFAIK, systemd is not absolutely monolithic - it includes several
daemons; but they are probably useless as separate programs.

-- Victor

[Victor]

Sounds convincing.

There is probably one problem -- if there are dozens of clients and one
server, server will be either overloaded with client messages or it will be
required to manage multiple log servers otherwise.

But for this moment it's not clear if this is a real problem in my case, I
need to play with various variants at first.

Thanks!

-- Victor

[James K. Lowden]

On Mon, 17 Dec 2012 23:53:15 +0400
Victor <vic...@enise.org> wrote:

> There is probably one problem -- if there are dozens of clients and
> one server, server will be either overloaded with client messages or
> it will be required to manage multiple log servers otherwise.

How much information do you intend to log? If each client produces one
100-byte message every second, any server built since the Clinton
administration could easily handle a thousand clients.

If server and client share an outmoded 10 Mb/s Ethernet, they can
transmit ~1 MB/s. That's the slowest link; it's about 2% of the speed
of the hard disk.

If your client generates 0.01 MB/s -- 10 KB/s, 1% of ye olde Ethernet,
0.02% of the disk bandwidth -- that would be 100 messages of 100 bytes
each, every second. That's a tiny fraction of the capacity of a $500
computer, and it would create a log file of 864 MB = 10K * 3600 * 24,
i.e. 8.6 million lines per client, every day.

At that rate, yes, a few dozen clients would start to tax a garden
variety Best Buy special. The problem disappears with a server using
slightly better hardware, costing perhaps $5000: server-grade SATA
RAID-10 and dual 100 Mb/s Ethernet. It also disappears if your log
messages are on the order of 1/second, not 100/second. :-)

Pretending you were to collect roughly 10 million rows per client per
day, it's interesting to note that, while syslogd could handle the job
easily (as shown above), reporting on that information could quickly
become a challenge, especially historical summaries. Fifty clients at
that rate produce 1.5 billion rows = 30 * 50 * 10M each month, almost
20 billion/year. At that point, awk(1) is no longer your friend.

Figuring out what it all means is much harder than collecting it in the
first place. If you don't believe me, ask Donald Trump!

--jkl

[William Ahern]

Victor <vic...@enise.org> wrote:

> Hello.

> I'm writing a set of applications running on different hosts and
> communicating via network.

> All applications produce some logs. I need to receive and filter log
> messages from all that applications and visualize them at runtime.

> So the question is: is there already a library for C or C++ that
> implements both sending and receiving log messages, hopefully with
> filtering and multiple receivers allowed?

Just log to stderr, then you can use any library you want, in any language,
or even any external program or ad hoc script. Writing to stderr allows you
to change the way you process logs as much as you'd like.

> The solution I've found for now is to send messages using syslog() and
> configure syslog-ng to transmit, receive and pass received messages to
> local application.

> Now I'm looking for alternatives to compare.

Logging to syslog would be the second best option, as it provides almost the
same capabilities, albeit with more-or-less baggage, depending on how you
end up processing the output. The biggest problem with syslog is that it
gets complicated if you have more than one instance of your daemon, e.g.
during development or on multi-homed systems. And in any event, there are
utilities to forward stderr to syslog.

[Rainer Weikusat]

William Ahern <wil...@wilbur.25thandClement.com> writes:

[...]

> Just log to stderr, then you can use any library you want, in any language,
> or even any external program or ad hoc script. Writing to stderr allows you
> to change the way you process logs as much as you'd like.

[...]

> And in any event, there are utilities to forward stderr to syslog.

In plain English, this means little but "I'm going to write it to a
file, dammit!, and if someone really wants to use this weird
'suesslock' facility, let it be his problem!" (I vaguely remember that
there used to be a Daniel "everything somebody else did was not done
as I had done it !!1[*]" J. Bernstein rant on this topic as well but I
couldn't locate it quickly if it actually exists or existed).

[*] And every Kerberos replay cache I ever replaced had
serious technical shortcomings which manifested itself in a
particular problem situation I had to deal with some years ago
and every guy who wrote the code demonstrably didn't have the
slightest idea regarding what the dynamic behaviour of its
creation would be ...

The problem with this is that a system running a sizeable number of
unattended and possibly even interacting/ interdependent tasks needs
some factiliy for collecting and correlating diagnostic output from
all of them for troubleshooting of both ongoing and past problems and
for manageing the data collected in this way.

There's a standardized UNIX(*) interface whose description starts with

The syslog() function shall send a message to an
implementation-defined logging facility, which may log it in
an implementation-defined system log, write it to the system
console, forward it to a list of users, or forward it to the
logging facility on another host over the network. The logged
message shall include a message header and a message body. The
message header contains at least a timestamp and a tag string.

And using this 'implentation-defined logging facilitity' is The Right
Thing To Do (see last paragraph) for every program which either isn't
supposed to be interactive or is producing diagnostic output not
supposed to be 'usually useful' for someone using it interactively (I
actually started to use syslog [with the non-standard LOG_PERROR] even
for programs which might be useful an an interactive shell, simply
because this means that any diagnostics get also logged, no matter
what the execution situation was, and I can find them later
when I need them).

[Jorgen Grahn]

On Mon, 2012-12-17, James K. Lowden wrote:
> On Mon, 17 Dec 2012 23:53:15 +0400
> Victor <vic...@enise.org> wrote:
>
>> There is probably one problem -- if there are dozens of clients and
>> one server, server will be either overloaded with client messages or
>> it will be required to manage multiple log servers otherwise.

[snip good reality check]

> Pretending you were to collect roughly 10 million rows per client per
> day, it's interesting to note that, while syslogd could handle the job
> easily (as shown above), reporting on that information could quickly
> become a challenge, especially historical summaries. Fifty clients at
> that rate produce 1.5 billion rows = 30 * 50 * 10M each month, almost
> 20 billion/year. At that point, awk(1) is no longer your friend.

Maybe not at *that* point, but standard tools take you a long way.
Longer, I think, than many believe.

I've successfully[1] grepped[2] hundreds of megabytes of logs, via
NFS. I don't remember if they were gzipped or not; as I recall it
was slightly faster if they were, trading local CPU for I/O.

/Jorgen

[1] "Successfully" meaning I could do it regularly, without a lot of
up-front planning. I could refine my searches when needed. And I
didn't have time to fetch coffee between runs; they took something
like a minute.

[2] "Grepping" meaning find, grep, sort, uniq, perl, tee and so on,
in pipelines.

[William Ahern]

Rainer Weikusat <rwei...@mssgmbh.com> wrote:
<snip>

> There's a standardized UNIX(*) interface whose description starts with

> The syslog() function shall send a message to an
> implementation-defined logging facility, which may log it in
> an implementation-defined system log, write it to the system
> console, forward it to a list of users, or forward it to the
> logging facility on another host over the network. The logged
> message shall include a message header and a message body. The
> message header contains at least a timestamp and a tag string.

> And using this 'implentation-defined logging facilitity' is The Right
> Thing To Do (see last paragraph) for every program which either isn't
> supposed to be interactive or is producing diagnostic output not
> supposed to be 'usually useful' for someone using it interactively (I
> actually started to use syslog [with the non-standard LOG_PERROR] even
> for programs which might be useful an an interactive shell, simply
> because this means that any diagnostics get also logged, no matter
> what the execution situation was, and I can find them later
> when I need them).

Pray tell, where might I find that log output?

The syslog library interface and wire format may be standardized, but that's
really not the biggest pain point. Various syslog daemons do have lots of
fancy features for redirecting output. But why wouldn't I just redirect the
log properly in the first instance, without having to muck around with a
centralized, root-permission-only control file?

If I'm redirecting to stderr, I know exactly where that output will be on
any system.

For the sake of argument, let's not even get into the whole systemd journal
fiasco.

[Markus Raab]

Hello!

Rainer Weikusat wrote:
> monitor
>
> Monitor another processes' health, restart it in case of
> unexpected termination and provide some process control
> features (start/ stop/ restart).

Can you elaborate?

How do you start/ stop/ restart a monitored process?
Do you have some nasty while loop in there?
Why not use the init feature?

best regards
Markus

--
http://www.markus-raab.org | Ich kenne keine Software die sich einfacher
-o) | bauen und installieren lￜￜt als cdrecord.
Kernel 2.6.32-5-a /\ | -- Joerg Schilling
on a x86_64 _\_v |

[Rainer Weikusat]

Markus Raab <use...@markus-raab.org> writes:
> Rainer Weikusat wrote:
>> monitor
>>
>> Monitor another processes' health, restart it in case of
>> unexpected termination and provide some process control
>> features (start/ stop/ restart).
>
> Can you elaborate?
>
> How do you start/ stop/ restart a monitored process?
> Do you have some nasty while loop in there?
> Why not use the init feature?

As it stands, this question is somewhat incomprehensible/ doesn't
make any sense: Eg, what is 'the init feature' and did you really
intend to ask how parent processes get notified of status changes in
child processes and how processes can control other processes with
signals in UNIX(*)?

[Victor]

Don't know now concrete numbers - the work is in progress, unfortunately I
have no complete system yet.

Don't think more than about 100 messages per second per client. But the
client and server are not in LAN, they're connected to Internet instead. So
the client-server line seems to me to be usually <= 1-5 MBps.

Well, thanks, I'll keep in mind to measure real numbers and try syslog and
standard tools first.

-- Victor

[Victor]

Why not just to provide an option that allows user to switch between stderr
and syslog?

-- Victor

[James K. Lowden]

On Tue, 18 Dec 2012 12:10:04 -0800
William Ahern <wil...@wilbur.25thandClement.com> wrote:

> But why wouldn't I just redirect the
> log properly in the first instance, without having to muck around
> with a centralized, root-permission-only control file?

Er, define "properly"? Would that be "using shell redirection
syntax"?

A daemon is a usually a system-wide service under administrative
control. If you're setting one up, a root-owned file in /etc shouldn't
present a problem.

> If I'm redirecting to stderr, I know exactly where that output will
> be on any system.

The log, if there is to be one, has to be somewhere, has to be owned by
someone. Its location has to be determined by something.

If the location is determined by redirection in a script, changing it
becomes a *logic* problem: one modifes the script, a program. In
the process risking breakage, maybe by entering a stray comma or
something by mistake. And perforce you're permitting the script to be
reprogrammed -- to do *anything* else, as the superuser -- when what's
needed is merely to be able to change the log location.

syslog.conf reduces the log location to *data*, independent of the
logic. It allows different categories and classes of messages to be
handled in different ways. Like hier(7), it simplifies the job of
programmer and admin by creating a canonical structure for each to
use.

That's why. :-)

--jkl

[William Ahern]

James K. Lowden <jklo...@speakeasy.net> wrote:
> On Tue, 18 Dec 2012 12:10:04 -0800
> William Ahern <wil...@wilbur.25thandClement.com> wrote:

> > But why wouldn't I just redirect the
> > log properly in the first instance, without having to muck around
> > with a centralized, root-permission-only control file?

> Er, define "properly"? Would that be "using shell redirection
> syntax"?

At a minimum, yes. Although I keep a small library, redir.c, in my toolbox
which I use to redirect stderr (and stdout and stdin) according to
command-line arguments. That way early errors are able to reach any attached
console. It also executes any shell code in a different session, detached
from any console.

> A daemon is a usually a system-wide service under administrative control.
> If you're setting one up, a root-owned file in /etc shouldn't present a
> problem.

That statement was admittedly pretty clear when syslog was invented, even if
disputable--I used to run lots of daemons from my personal account on big
university machines, including web servers and SLIP gateways. But now that
Unix dominates many embedded spaces, among others, "system-wide" and
"administrative" are fuzzier concepts--there are many presumptions built
into those phrases as historically used. In my experience, unless there's an
existing need to interoperate with syslog compliant services on the network,
the syslog infrastructure is more often avoided--both reasonably and
unreasonably, depending on the context.

> > If I'm redirecting to stderr, I know exactly where that output will be
> > on any system.

> The log, if there is to be one, has to be somewhere, has to be owned by
> someone. Its location has to be determined by something.

If you're documenting a service or utility, do you document that "this
program logs to where ever your syslog daemon dumps its output, possibly to
multiple different files depending on log level". Or do you document, "this
program logs to /var/log/foo/foo-%Y%m%d.log"?

> If the location is determined by redirection in a script, changing it
> becomes a *logic* problem: one modifes the script, a program. In
> the process risking breakage, maybe by entering a stray comma or
> something by mistake. And perforce you're permitting the script to be
> reprogrammed -- to do *anything* else, as the superuser -- when what's
> needed is merely to be able to change the log location.

I'd argue that, in practice, it's a distinction without a difference. Don't
forget that you can always parameterize the redirection code. But having the
capability to redirect log output with ad hoc code can be at times
invaluable. And you don't have to leave the users entirely up to their own
devices to handle the redirection. As with anything, the default behavior
should be the obvious and most convenient.

> syslog.conf reduces the log location to *data*, independent of the logic.
> It allows different categories and classes of messages to be handled in
> different ways. Like hier(7), it simplifies the job of programmer and
> admin by creating a canonical structure for each to use.

Except unlike hier(7), there's far more variance. I debug most of my work on
Linux, OpenBSD, and OS X. And I maintain more--FreeBSD, NetBSD, and
Solaris--for some of it. And, as I said before, there's nothing preventing
someone from redirecting stderr to syslog, or from maintaining their logs in
substantially the same format and location as syslog managed logs.

FWIW, though I obviously have a preference for logging to stderr, I'm not
going to take someone to task after the face for choosing syslog. It's a
decent runner-up in my book, for all the reasons you've mentioned and more.

[Jorgen Grahn]

On Thu, 2012-12-20, William Ahern wrote:
> James K. Lowden <jklo...@speakeasy.net> wrote:
>> On Tue, 18 Dec 2012 12:10:04 -0800
>> William Ahern <wil...@wilbur.25thandClement.com> wrote:
>
>> > But why wouldn't I just redirect the
>> > log properly in the first instance, without having to muck around
>> > with a centralized, root-permission-only control file?
>
>> Er, define "properly"? Would that be "using shell redirection
>> syntax"?
>
> At a minimum, yes. Although I keep a small library, redir.c, in my toolbox
> which I use to redirect stderr (and stdout and stdin) according to
> command-line arguments. That way early errors are able to reach any attached
> console. It also executes any shell code in a different session, detached
> from any console.
>
>> A daemon is a usually a system-wide service under administrative control.
>> If you're setting one up, a root-owned file in /etc shouldn't present a
>> problem.
>
> That statement was admittedly pretty clear when syslog was invented, even if
> disputable--I used to run lots of daemons from my personal account on big
> university machines, including web servers and SLIP gateways.

That is a valid argument, but ...

> But now that
> Unix dominates many embedded spaces, among others, "system-wide" and
> "administrative" are fuzzier concepts--there are many presumptions built
> into those phrases as historically used.

... I don't understand what's more fuzzy. You basically have three
kinds of users on Unix: root, per-daemon accounts, and untrusted real
users. If "embedded" means no real users ever log in to do normal
computing, that still leaves the other categories as useful as ever.

> In my experience, unless there's an
> existing need to interoperate with syslog compliant services on the network,
> the syslog infrastructure is more often avoided--both reasonably and
> unreasonably, depending on the context.

One problem (if you like Unix) may be that a lot of embedded people
who don't understand or like Unix are forced to migrate to Linux.

I sometimes wish there was an easy way to indoctrinate newcomers. Eric
Raymond's "The art of Unix programming" comes close, but it can be
off-putting and lots of people don't read books anyway.

/Jorgen

[Rainer Weikusat]

Victor <vic...@enise.org> writes:

[...]

> Why not just to provide an option that allows user to switch between stderr
> and syslog?

NB: The text below reflects my present opinion on this but it is
something like 'an opinion in progress' ...

The problem with this is that 'file descriptor 2' is not anyhow
special and may refer to anything when a program is executed in an
'uncommon' environment[*], ie, not by an interactive shell or not
by a shell at all. Because of this, a program which isn't intended for
being used interactively from a shell or as tool in a shell script
shouldn't just dump diagnostic output on file descriptor 2 except if
there's a reason to assume that it is actually supposed to be used for
that. What I presently do in such a case is that I let the code log to
file descriptor 2 if it is an interactive device (isatty(3)) or if
this was explicitly requested via command-line option and I always log
to syslog as well --- I have lost too much output I had needed at some
point in time in the past already.

[*] This badly bit me about ten years ago with some code ran by the
then-current cfd version (the cfengine job execution daemon). The cfd
executed fclose(stdin); fclose(stdout); fclose(stderr) as part of its
daemonization code, consequently, the next three file descriptors
opened by my code got the descriptors 0, 1 and 2 and the diagnostic
output ended up being sent over a TCP connection supposed to be used
for 'something completely different'. That's easy to avoid if one is
aware of the problem but hard to debug when this is not the case.

[James K. Lowden]

On Wed, 19 Dec 2012 17:37:18 -0800
William Ahern <wil...@wilbur.25thandClement.com> wrote:

> > The log, if there is to be one, has to be somewhere, has to be
> > owned by someone. Its location has to be determined by something.
>
> If you're documenting a service or utility, do you document that "this
> program logs to where ever your syslog daemon dumps its output,
> possibly to multiple different files depending on log level". Or do
> you document, "this program logs to /var/log/foo/foo-%Y%m%d.log"?

Judging from a handful of daemons on my system, we're both right.
Most daemons (as you know) have some kind of debug mode that allows
the log to be written to an arbitrary file or standard error.

But syslog is not so arcane as you suggest. The manuals usually
just say "this system uses the system logger". For example:

ftpd: -l Each successful and failed FTP session is logged
using syslog with a facility of LOG_FTP.

sshd: -e When this option is specified, sshd will
send the output to the standard error instead of the system log.

ntpd: -l string, --logfile=string
path to the log file.
Specify the name and path of the log file. The default
is the system log file.

Many barely mention it. Postfix, for example, has parameters in
main.cf to set the syslog facility and name, and never says explicitly
that logging is handled by syslog.

Like a lot of things UNIX, if you just say what's true, you'll
communicate clearly to those with the requisite knowledge. To someone
who knows syslog, it's enough to say you're using the system
logger. To someone who doesn't, the system logger is another
complicated system for which he'll have to RTFM.

> > Like hier(7), it simplifies the job of programmer and
> > admin by creating a canonical structure for each to use.
>
> Except unlike hier(7), there's far more variance. I debug most of my
> work on Linux, OpenBSD, and OS X.

Is there so much variation? OS X syslog looks a lot like NetBSD to
me. I used Ubuntu syslog this summer without noticing any difference.
The Solaris man pages sure look familiar. If you mean different
sysadmins configure their syslog.conf differently, sure, but I'd say
that's a good thing. If you mean many systems lie outside the syslog
purview, sure, and I'd say that's a bad thing. :-)

--jkl

[Rainer Weikusat]

William Ahern <wil...@wilbur.25thandClement.com> writes:
> James K. Lowden <jklo...@speakeasy.net> wrote:

>> William Ahern <wil...@wilbur.25thandClement.com> wrote:

[...]

>> A daemon is a usually a system-wide service under administrative control.
>> If you're setting one up, a root-owned file in /etc shouldn't present a
>> problem.
>
> That statement was admittedly pretty clear when syslog was invented, even if
> disputable--I used to run lots of daemons from my personal account on big
> university machines, including web servers and SLIP gateways.
> But now that Unix dominates many embedded spaces, among others, "system-wide" and
> "administrative" are fuzzier concepts--there are many presumptions built
> into those phrases as historically used. In my experience, unless there's an
> existing need to interoperate with syslog compliant services on the network,
> the syslog infrastructure is more often avoided--both reasonably and
> unreasonably, depending on the context.

This is a pair of anecdotes:

- in 1847 or so, students used to run 'network servers' on
'big [time-shared] university machines', maybe even
occasionally in line with the applicable usage policy

- more-or-less POSIX/ UNIX(*) compliant kernels and 'basic
application programming environments' are nowadays used for
many embedded systems and (not entirely unlike the students
from 1847), the people who write code for such systems care
little about stuff which works or doesn't when other people
have to manage 24x7 'network appliances' all around the world and
are actually expected to fix software bugs, as opposed to
telling the user to power-cylce the device until he tires of
it or the problem has been perturbed out of existence (if
there is any kind of user support for the 'embedded
computer' at all)

If anything, this communicates that people tend to care little for
problems which don't affect them.

>>> If I'm redirecting to stderr, I know exactly where that output will be
>>> on any system.
>
>> The log, if there is to be one, has to be somewhere, has to be owned by
>> someone. Its location has to be determined by something.
>
> If you're documenting a service or utility, do you document that "this
> program logs to where ever your syslog daemon dumps its output, possibly to
> multiple different files depending on log level". Or do you document, "this
> program logs to /var/log/foo/foo-%Y%m%d.log"?

For obvious reasons, you (or anyone) can't 'document' details about
the configuration of 'some logging facility' whose configuration is
unknown to you. But I'd prefer 'This programs emits diagnostics via
syslog using a default facility of ... This can be change by ..." over
"unless forced to do otherwise, this program will keep writing
diagnostic output to

/some/random/location/I/happened/to/like/jeremias-<current monarch of thailand>.nasenbaer [*]

until your computer ran out of space" (or the much more common "don't
document anything, let the local admin discover the log file").

And there's still the issue that computers can run lots of
software-provided services and complex problems can and likely will
require tracing the interfactions of some set of program running
unattendedly. And this is much simpler if their diagnostic output ends
up in one file than in n different files sitting in random locations
on the file system, preferable (eg, the squid 'access log') even using
their own, custom timestamp/ header format.

[*] http://www.wilhelma.de/uploads/tx_templavoila/nasenbaer_dia_04.jpg

[Markus Raab]

Hi!

"init feature":
use /etc/inittab with "respawn" to restart the application if it crashed

No, it is no problem how to use wait(2). The question was how to communicate
with the "monitor" process. Typically I would also expect a parameter how
to handle continously crashing applications?

start-stop-daemon(8) from debian says:
-b, --background
Typically used with programs that don't detach on their own.
This option will force start-stop-daemon to fork before starting the
process, and force it into the background. WARNING: start-stop-daemon
cannot check
the exit status if the process fails to execute for any
reason. This is a last resort, and is only meant for programs that either
make no sense forking on their own, or where it's not feasible to add the
code for
them to do this themselves.


Which basically tells that --background is a hack and should not be used.
Does your "daemon" implementation have this problem, too?

best regards
Markus

[Rainer Weikusat]

Markus Raab <use...@markus-raab.org> writes:
> Rainer Weikusat wrote:
>> Markus Raab <use...@markus-raab.org> writes:
>>> Rainer Weikusat wrote:
>>>> monitor
>>>>
>>>> Monitor another processes' health, restart it in case of
>>>> unexpected termination and provide some process control
>>>> features (start/ stop/ restart).
>>>
>>> Can you elaborate?
>>>
>>> How do you start/ stop/ restart a monitored process?
>>> Do you have some nasty while loop in there?
>>> Why not use the init feature?
>>
>> As it stands, this question is somewhat incomprehensible/ doesn't
>> make any sense: Eg, what is 'the init feature' and did you really
>> intend to ask how parent processes get notified of status changes in
>> child processes and how processes can control other processes with
>> signals in UNIX(*)?
>
> "init feature":
> use /etc/inittab with "respawn" to restart the application if it
> crashed

I did this in the past, although with an init I wrote myself:
Controlling processes by modifying text files is doable but too
cumbersome. Also, the inittab file of the stock sysvinit wasn't designed to
be easily processed by other programs and sometimes, I'm forced to use
'weird distributions' (like Ubuntu) with equally weird replacement
inits (like upstart) whose 'interface' typically isn't very stable,
at least not until the code was finally abandoned because the 'original
offender' (author) found some other, relatively simple existing
arrangment he wanted to replace with something very much more
complicated of his own design. sysvinit lacks too many features I need
and the one bazillion replacements who have them are all incompabile
with each other and too much of an example of feeping creaturism. IMO,
the ideal init would deal with the system console, reap orphaned
processes and leave everything else to other programs. Lastly, I
wanted something which can also be used interactively/
'experimentally'. Eg, presently, I'm running an explicit vacuumdb on a
certain computer because the postgres version installed there is too
old for the autovacuum daemon to actually work in the situation it
needed to work in. For now, I've just started that with

daemon chdir / monitor -n vacuumdb chids -u postgres rand-periodic 10800 mail-output -s vacuumdb -t root -e vacuumdb -U postgres -d mad_database -v -z

Maybe, it will be turned into a proper system service later on or
postgres updated or the machine decommissioned altogether. Also, maybe
I (or someone else) comes up with a better way to do that.

> No, it is no problem how to use wait(2). The question was how to communicate
> with the "monitor" process.

It creates an AF_UNIX stream socket in a certain directory using
either the first 'word' of the command or the value given to the -n
option as name. An extremely simple-minded binary protocol is used to
request one of the two actions it is capable of (stop the monitored
process or restart it).

> Typically I would also expect a parameter how to handle continously
> crashing applications?

There's really only one sensible way to deal with this (init does this
as well): After the new process was started, start a timer. Increment
a counter whenever it exists before this timer has expired. If the
counter has reached a certain threshold value before the timer
expired, assume there's a problem, log a 'respawning too fast' message
and keep quiet for some time before starting over (I'm using 5s/ 3
exists/ 5s for no particular reason).

> start-stop-daemon(8) from debian says:
> -b, --background
> Typically used with programs that don't detach on their own.
> This option will force start-stop-daemon to fork before starting the
> process, and force it into the background. WARNING: start-stop-daemon
> cannot check
> the exit status if the process fails to execute for any
> reason. This is a last resort, and is only meant for programs that either
> make no sense forking on their own, or where it's not feasible to add the
> code for
> them to do this themselves.
>
> Which basically tells that --background is a hack and should not be
> used.

This 'basically' communicates that the author is confused[*] and doesn't
run the risk to get emergency phonecalls at 3 in the morning because a
crashed process on a computer in Japan needs to be
restarted.

[*] If the program forks itself into the background
'start-stop-daemon' (another of these 'one ring to rule them all'
monsters) also can't tell if it then immediately terminated itself
'for any reason' and the exit status is lost in this case: init will
reap the process and remain silent about that. The sensible way to
deal with this is to prevent the server from backgrounding itself [**],
even if this means that its code has to be modified. Should the exec
fail, the program which called it will continue to execute and can
thus log an error message.

[**] Whether a certain program should run 'in the foreground', 'in the
background' or 'as a daemon process' should be a descision made by the
invoker/ the invoking code, not by the program itself.

[Mark Wooding]

Rainer Weikusat <rwei...@mssgmbh.com> writes:

> Markus Raab <use...@markus-raab.org> writes:
> > start-stop-daemon(8) from debian says:
> > -b, --background
> > Typically used with programs that don't detach on
> > their own. This option will force start-stop-daemon to
> > fork before starting the process, and force it into
> > the background. WARNING: start-stop-daemon cannot
> > check the exit status if the process fails to execute
> > for any reason. This is a last resort, and is only
> > meant for programs that either make no sense forking
> > on their own, or where it's not feasible to add the
> > code for them to do this themselves.
> >
> > Which basically tells that --background is a hack and
> > should not be used.
>
> This 'basically' communicates that the author is confused[*] and doesn't
> run the risk to get emergency phonecalls at 3 in the morning because a
> crashed process on a computer in Japan needs to be
> restarted.
>
> [*] If the program forks itself into the background
> 'start-stop-daemon' (another of these 'one ring to rule them all'
> monsters) also can't tell if it then immediately terminated itself
> 'for any reason' and the exit status is lost in this case: init will
> reap the process and remain silent about that.

No, you've misunderstood. The `--background' option tells `start-stop-
daemon' that the (not-)daemon /doesn't/itself fork into the background:
it just runs forever. In this case, `start-stop-daemon' does the fork
itself. But this causes trouble. A typical daemon program will perform
some initialization, and then fork, leaving the child executing the main
loop and the parent can exit successfully. If the fork isn't there then
there's no generic way of telling when the new process has completed its
initialization. So `start-stop-daemon' just returns immediately and
leaves the outer script to deal with this mess.

> The sensible way to deal with this is to prevent the server from
> backgrounding itself [**], even if this means that its code has to be
> modified. Should the exec fail, the program which called it will
> continue to execute and can thus log an error message.
>
> [**] Whether a certain program should run 'in the foreground', 'in the
> background' or 'as a daemon process' should be a descision made by the
> invoker/ the invoking code, not by the program itself.

This leaves a remaining problem: often, daemon Y depends for its
functionality on some existing daemon X. How do you determine when is
the right time to start Y, in the absence of X clearly marking its
successful initialization by calling fork?

-- [mdw]

[Rainer Weikusat]

I don't quite understand how you managed to come up with this
proposition since this is literally stated as such in the first quoted
sentence of the documentation excerpt.

> In this case, `start-stop-daemon' does the fork itself. But this
> causes trouble. A typical daemon program will perform some
> initialization, and then fork, leaving the child executing the main
> loop and the parent can exit successfully. If the fork isn't there
> then there's no generic way of telling when the new process has
> completed its initialization.

Or so you assume. The uncooperative server process may as well fork
immediately and do anyhing else later. And it can of course 'crash'
immediately after it detached itself from its original parent or
terminate because of any kind of 'other runtime error' which will then
be lost and the service unavailable until someone happens to notice
this and happens to communicate this to someone else who can fix the
problem. But nobody's going to call the person who wrote
'start-stop-daemon' in the middle of the night because of that and
nobody is expecting said person 'to deal with the mess' (in the sense
of 'being responsible for it').

[...]

>> The sensible way to deal with this is to prevent the server from
>> backgrounding itself [**], even if this means that its code has to be
>> modified. Should the exec fail, the program which called it will
>> continue to execute and can thus log an error message.
>>
>> [**] Whether a certain program should run 'in the foreground', 'in the
>> background' or 'as a daemon process' should be a descision made by the
>> invoker/ the invoking code, not by the program itself.
>
> This leaves a remaining problem: often, daemon Y depends for its
> functionality on some existing daemon X. How do you determine when is
> the right time to start Y, in the absence of X clearly marking its
> successful initialization by calling fork?

This question makes no sense: Whether a 'server X' which 'forked' will
be available and capable of fullfilling a request by a 'server Y' (or
the 500th request, after successfully dealing with the 499 which came
before it) by the time Y actually makes this request can't be
determined from the outside at some earlier time.