Sagan fifo buffer capacity

[Aleksey Chudov]

Hi,

I use rsyslog for sending data to Sagan (approx 15k messages per second). There are a lot of fifo errors according to rsyslog impstats module counters. After a quick research I thought about fifo buffer capacity. 

According to http://man7.org/linux/man-pages/man7/pipe.7.html

Since Linux 2.6.35, the default pipe capacity is 65536 bytes, but the capacity can be queried and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations (up to a maximum capacity which defaults to 1048576 bytes, but can be changed via /proc/sys/fs/pipe-max-size).

Below is a sample Python code

# python
>>> import fcntl
>>> fifo = open('/var/log/sagan/sagan.fifo')
>>> fcntl.fcntl(fifo, 1032)
65536
>>> fcntl.fcntl(fifo, 1031, 4096)
4096
>>> fcntl.fcntl(fifo, 1031, 1048576)
1048576
>>> exit()

The results confirmed the theory

Can Sagan set the pipe capacity after opening fifo file?

Regards,
Aleksey

[Champ Clark III]

I've thought about this in the past and thought that might be the responsibility for the writer to control.   I've seen some references to increasing the FIFO size in syslog-ng,  but I don't see a way to do it in rsyslog.    The results you've shared are pretty dramatic.  Let me do a bit more research and thinking about it.   I think this could become a Sagan option pretty easily. 

I'll get back to you shortly.

---

From: "Aleksey Chudov" <aleksey...@gmail.com>
To: "sagan-users" <sagan...@googlegroups.com>
Sent: Wednesday, April 29, 2015 3:11:08 PM
Subject: [sagan-users] Sagan fifo buffer capacity

--
You received this message because you are subscribed to the Google Groups "sagan-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sagan-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Champ Clark III]

I just committed some test code that allows Sagan to alter the capacity of the pipe.  If you look at:

https://github.com/beave/sagan/blob/master/etc/sagan.conf#L114

You'll see the new option "sagan_fifo_size".  You can set that to whatever works best for you.   If the option is _not_ set,  it defaults to a size of 1048576 bytes.   That is the current max default of the Linux kernel via/proc/sys/fs/pipe-max-size.  I can't think of any situation where setting the pipe to it's "max size" would be a bad thing.  

Please test and let me know your thoughts.   I've added some checks in the configure.ac to test F_GETPIPE_SZ and F_SETPIPE_SZ are available in the fcntl.h .   My main concern about doing this is portability but I think this will take care of any situation where the pipe capacity cannot be changed.   This works under Linux and my brief reading suggests that it'll work under FreeBSD.  I'll do compile/system checks later in the week. 

Thank you for bring this issue up!  It was pretty straight forward to implement.  Let me know how it works out for you.

---

From: "Champ Clark" <ccl...@quadrantsec.com>
To: "sagan-users" <sagan...@googlegroups.com>
Sent: Wednesday, April 29, 2015 5:37:04 PM
Subject: Re: [sagan-users] Sagan fifo buffer capacity

[Aleksey Chudov]

It works like a charm. So far, I haven't found any issues with it. Thank you for quick response!

According to Sagan documentation it is responsibility of a user to create fifo file before starting Sagan. Manually executing mkfifo is definitely not an option. I've solved the problem using external script that you have recently merged https://github.com/beave/sagan/blob/master/extra/RPM/sagan.service#L8 https://github.com/beave/sagan/blob/master/extra/RPM/sagan-setup.libexec. For me It is a bit of workaround but it works.

Since Sagan is managing fifo file some way can it create one if it does not exist?

[Champ Clark III]

I would disagree with the statement that "Sagan is managing the fifo".   Sagan is actually just a "reader" of the FIFO.  

In a way,  I'd say the "writer" should be doing more of the "managing" of the FIFO.  

One of the first steps in setting up Sagan is the configuration and verification that the writer (rsyslog/syslog-ng/etc) the FIFO.   If the "writer" isn't writing,  Sagan can't accomplish its mission.   If Sagan create's the FIFO for the user,  we're now getting out of step with the process.  My fear is that this will lead to confusion.   For example,  Sagan is creating & reading /var/sagan/sagan.fifo,  but the writer has never been configured.   "Why isn't Sagan working?!?!?" questions ensue.  :)  

Once the fifo is created,  it should be persistent.   That is,  if should survive reboots. 

Let me know your thoughts and thanks for the input.

---

From: "Aleksey Chudov" <aleksey...@gmail.com>
To: "sagan-users" <sagan...@googlegroups.com>

Sent: Thursday, April 30, 2015 6:00:28 AM

Subject: Re: [sagan-users] Sagan fifo buffer capacity

[Aleksey Chudov]

Thanks for your opinion. But let me disagree :)

In my point of view the main idea of FIFO interprocess communication is closely match the idea of socket communication. Usually it is assumed that the receiving application opens socket and sending application connects to this opened socket to send messages.

Lets consider for example Rsyslog application.

1. Rsyslog can receive messages from remote processes using udp, tcp, unix socket, etc. In this case Rsyslog is acting as receiver and it is responsibility of receiver to create socket. So, Rsyslog creates all kind of listening sockets.

2. Rsyslog can forward messages to remote processes using udp, tcp, unix sockets, fifo, pipe, etc. In this case Rsyslog is acting as sender but it is still responsibility of receiver to create socket. So, Rsyslog never creates any sockets to forward to.

I out case Rsyslog acting as sending application and Sagan acting as receiving application. It would be logical that Sagan opens FIFO to receive messages from on Rsyslog this FIFO.

[Aleksey Chudov]

Currently, we are in a situation where Rsyslog does not create the FIFO as described earlier, and Sagan does not create FIFO because it believe that the FIFO must be created by Rsyslog. As a result FIFO eventually created by user because no one else left in the scheme :)

[Aleksey Chudov]

After extensive tests I finally finished my configuration with the following settings:

# sysctl -w fs.pipe-max-size=16777216
sagan_fifo_size 16777216

Using large fifo size I can restart or even stop Sagan for a dozen of seconds (for example while updating to a newer version) and there were no fifo errors according to rsyslog counters. At the same time I haven't found any negative effect of increasing fifo size.

On Thursday, April 30, 2015 at 5:35:19 AM UTC+3, Da Beave wrote:

[Champ Clark III]

This is great information.   I'm going to add this to the Sagan wiki ASAP.   With proper credits of course.  

About how many log lines per/sec are you processing ?

Thank you.

--

Champ Clark III

ccl...@quadrantsec.com

----- Reply message -----
From: "Aleksey Chudov" <aleksey...@gmail.com>
To: <sagan...@googlegroups.com>
Subject: [sagan-users] Sagan fifo buffer capacity

[Aleksey Chudov]

Our setup is pretty simple. We have one Rsyslog server (collector) per data center. All data center machines send system logs to this central collector using Rsyslog RELP protocol. Logs are stored in local files and a copy is forwarded to the Sagan running on the same server for analysis. The busiest Rsyslog collector processes about 15 - 20 thousands messages per second with rare spikes up to 30 - 40 thousands.

Below is a graph. Action 1 is actually
*.* |/var/log/sagan/sagan.fifo;FormatSagan

[Da Beave]

Hello Aleksey, 

I haven't forgotten about this.   It is something that I'm still thinking about.  

You do bring up some valid points. 

I need to do some testing to see how confusing this might be.  For  example,  if I tell syslog-ng/rsyslog to "use" a non-existent FIFO,  when Sagan "creates" it,  what affect will that have?   It's easy enough to implement,  but I do want to think about this and maybe discuss with some of the guys at Quadrant to get other opinions. 

Thank you!

[Aleksey Chudov]

For rsyslog a non-existent FIFO is not a problem. It just reports the error and continues to operate as usual

For example
May  5 22:35:29 localhost rsyslogd-2039: Could no open output pipe '/var/run/sagan.fifo': No such file or directory [try http://www.rsyslog.com/e/2039 ]

Anyway writing to existing FIFO without reader on the other end eventually fails. This is not seen in Rsyslog logs (at least using info logging level) but it can be seen in Rsyslog impstats counters.

To prevent such problems Sagan can be started before syslog-ng/rsyslog https://github.com/beave/sagan/blob/master/extra/RPM/sagan.service#L4