syslog(3) issues

[Attilio Rao]

Hi,
I was trying to use syslog(3) in a port application that uses
threading , having all of them at the LOG_CRIT level. What I see is
that when the logging gets massive (1000 entries) I cannot find some
items within the /var/log/messages (I know because I started stamping
also some sort of message ID in order to see what is going on). The
missing items are in the order of 25% of what really be there.

Someone has a good idea on where I can start verifying for my syslogd
system? I have really 0 experience with syslogd and maybe I could be
missing something obvious.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Garrett Cooper]

On Sun, Sep 2, 2012 at 4:35 PM, Attilio Rao <att...@freebsd.org> wrote:
> Hi,
> I was trying to use syslog(3) in a port application that uses
> threading , having all of them at the LOG_CRIT level. What I see is
> that when the logging gets massive (1000 entries) I cannot find some
> items within the /var/log/messages (I know because I started stamping
> also some sort of message ID in order to see what is going on). The
> missing items are in the order of 25% of what really be there.
>
> Someone has a good idea on where I can start verifying for my syslogd
> system? I have really 0 experience with syslogd and maybe I could be
> missing something obvious.

I'd maybe use something like rsyslog and force TCP to verify that
the messages made it to their endpoints, and if all the messages make
it to the rsyslogd daemon use tcpdump/wireshark to figure out if the
UDP datagrams (default transport layer for syslog) aren't getting
dropped on the floor.
Cheers!
-Garrett

[Attilio Rao]

On Mon, Sep 3, 2012 at 1:20 AM, Garrett Cooper <yane...@gmail.com> wrote:
> On Sun, Sep 2, 2012 at 4:35 PM, Attilio Rao <att...@freebsd.org> wrote:
>> Hi,
>> I was trying to use syslog(3) in a port application that uses
>> threading , having all of them at the LOG_CRIT level. What I see is
>> that when the logging gets massive (1000 entries) I cannot find some
>> items within the /var/log/messages (I know because I started stamping
>> also some sort of message ID in order to see what is going on). The
>> missing items are in the order of 25% of what really be there.
>>
>> Someone has a good idea on where I can start verifying for my syslogd
>> system? I have really 0 experience with syslogd and maybe I could be
>> missing something obvious.
>
> I'd maybe use something like rsyslog and force TCP to verify that
> the messages made it to their endpoints, and if all the messages make
> it to the rsyslogd daemon use tcpdump/wireshark to figure out if the
> UDP datagrams (default transport layer for syslog) aren't getting
> dropped on the floor.

Forgot to mention: the logging is done completely locally so I don't
think network should play a role here. Also, I would like to
understand if I'm missing something subdle or if we actually may have
a bug in syslogd.

Attilio


--
Peace can only be achieved by understanding - A. Einstein

[Ian Lepore]

On Mon, 2012-09-03 at 00:35 +0100, Attilio Rao wrote:
> Hi,
> I was trying to use syslog(3) in a port application that uses
> threading , having all of them at the LOG_CRIT level. What I see is
> that when the logging gets massive (1000 entries) I cannot find some
> items within the /var/log/messages (I know because I started stamping
> also some sort of message ID in order to see what is going on). The
> missing items are in the order of 25% of what really be there.
>
> Someone has a good idea on where I can start verifying for my syslogd
> system? I have really 0 experience with syslogd and maybe I could be
> missing something obvious.

There's a chance this PR about syslogd incorrectly calculating socket
receive buffer sizes is related and the patch attached to it could fix
it...

http://www.freebsd.org/cgi/query-pr.cgi?pr=1604331

I filed the PR long ago, if the patches have drifted out of date I'll be
happy to re-work them.

-- Ian

[Ian Lepore]

On Sun, 2012-09-02 at 19:50 -0600, Ian Lepore wrote:
> On Mon, 2012-09-03 at 00:35 +0100, Attilio Rao wrote:
> > Hi,
> > I was trying to use syslog(3) in a port application that uses
> > threading , having all of them at the LOG_CRIT level. What I see is
> > that when the logging gets massive (1000 entries) I cannot find some
> > items within the /var/log/messages (I know because I started stamping
> > also some sort of message ID in order to see what is going on). The
> > missing items are in the order of 25% of what really be there.
> >
> > Someone has a good idea on where I can start verifying for my syslogd
> > system? I have really 0 experience with syslogd and maybe I could be
> > missing something obvious.
>
> There's a chance this PR about syslogd incorrectly calculating socket
> receive buffer sizes is related and the patch attached to it could fix
> it...
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=1604331
>
> I filed the PR long ago, if the patches have drifted out of date I'll be
> happy to re-work them.
>
> -- Ian
>

Oops, I glitched the PR number when I pasted it, this one should be
correct:

http://www.freebsd.org/cgi/query-pr.cgi?pr=160433

[Hooman Fazaeli]

On 9/3/2012 4:05 AM, Attilio Rao wrote:
> Hi,
> I was trying to use syslog(3) in a port application that uses
> threading , having all of them at the LOG_CRIT level. What I see is
> that when the logging gets massive (1000 entries) I cannot find some
> items within the /var/log/messages (I know because I started stamping
> also some sort of message ID in order to see what is going on). The
> missing items are in the order of 25% of what really be there.
>
>
>

Is not it caused by syslogd compression? As you may know syslogd compresses
consecutive identical messages into one message like "last message repeated N times".
The -cc option disables compression.

[Robert Watson]

On Mon, 3 Sep 2012, Attilio Rao wrote:

> I was trying to use syslog(3) in a port application that uses threading ,
> having all of them at the LOG_CRIT level. What I see is that when the
> logging gets massive (1000 entries) I cannot find some items within the
> /var/log/messages (I know because I started stamping also some sort of
> message ID in order to see what is going on). The missing items are in the
> order of 25% of what really be there.
>
> Someone has a good idea on where I can start verifying for my syslogd
> system? I have really 0 experience with syslogd and maybe I could be missing
> something obvious.

syslog(3)/syslogd(8) use datagram sockets for both local and networked
logging, and it is possible for those datagram sockets to fill and drop
messages. I'm not sure if we have per-socket counters that can easily be
queried by syslogd, but if we do, it might be beneficial to have syslogd wake
up once a second and check to see if the counters have changed -- if they
have, inject a log message indicating how many messages were dropped in the
last $epsilon. If we don't have counters along those lines, it might make
sense to add them. We might also find that it is appropriate to tune up the
limits if they no longer seem sensible in the current world order -- they may
have late 1980s/early 1990s values (or they may not).

Robert

[Eygene Ryabinkin]

Attilio, good day.

Mon, Sep 03, 2012 at 12:35:24AM +0100, Attilio Rao wrote:
> I was trying to use syslog(3) in a port application that uses
> threading , having all of them at the LOG_CRIT level. What I see is
> that when the logging gets massive (1000 entries) I cannot find some
> items within the /var/log/messages (I know because I started stamping
> also some sort of message ID in order to see what is going on). The
> missing items are in the order of 25% of what really be there.
>
> Someone has a good idea on where I can start verifying for my syslogd
> system?

This is almost obvious, but had you tried running syslogd with flag '-d'
that gives verbose information about incoming stuff and whether it will
be logged to some file(s).
--
Eygene Ryabinkin ,,,^..^,,,
[ Life's unfair - but root password helps! | codelabs.ru ]
[ 82FE 06BC D497 C0DE 49EC 4FF0 16AF 9EAE 8152 ECFB | freebsd.org ]

[Attilio Rao]

On Mon, Sep 3, 2012 at 3:23 AM, Ian Lepore
<fre...@damnhippie.dyndns.org> wrote:
> On Sun, 2012-09-02 at 19:50 -0600, Ian Lepore wrote:
>> On Mon, 2012-09-03 at 00:35 +0100, Attilio Rao wrote:
>> > Hi,
>> > I was trying to use syslog(3) in a port application that uses
>> > threading , having all of them at the LOG_CRIT level. What I see is
>> > that when the logging gets massive (1000 entries) I cannot find some
>> > items within the /var/log/messages (I know because I started stamping
>> > also some sort of message ID in order to see what is going on). The
>> > missing items are in the order of 25% of what really be there.
>> >
>> > Someone has a good idea on where I can start verifying for my syslogd
>> > system? I have really 0 experience with syslogd and maybe I could be
>> > missing something obvious.
>>
>> There's a chance this PR about syslogd incorrectly calculating socket
>> receive buffer sizes is related and the patch attached to it could fix
>> it...
>>
>> http://www.freebsd.org/cgi/query-pr.cgi?pr=1604331
>>
>> I filed the PR long ago, if the patches have drifted out of date I'll be
>> happy to re-work them.
>>
>> -- Ian
>>
>
> Oops, I glitched the PR number when I pasted it, this one should be
> correct:
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=160433

This patch fixes the problem for me, thanks a lot.

Alexander, do you have any reservation against it? When do you think
the patch can be merged to -CURRENTR?

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein