Rotatelogs - multiple processes
Lewis G Rosenthal
me out of house and home... :-)
Actually, it's been one of those things which I've known I've had to
address for some time, and I'm rapidly reaching that point. I'm not even
sure I like the way I have it set on my test system right now, and may
look to use something else (I'd much prefer to pipe all of this to
syslog on another box, in fact). However, as a matter of curiosity, I
point up the following:
For testing purposes, I have a mirror setup on my ThinkPad of what I
have on my live server. In vhosts conf, I have all logs commented out,
and in httpd.conf, I have:
LoadModule log_config_module modules/log_conf.dll
LoadModule logio_module modules/logio.dll
and
ErrorLog "||bin/rotatelogs logs/error_log.%Y-%m-%d-%H_%M_%S 5M"
<IfModule log_config_module>
LogFormat "%v %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%v %l %u %t \"%r\" %>s %b" comonvhost
<IfModule logio_module>
LogFormat "%v %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\" %I %O" combinedio
</IfModule>
CustomLog "||bin/rotatelogs logs/access_log.%Y-%m-%d-%H_%M_%S 5M"
combinedio
</IfModule>
So, with only (3) httpd processes spawned, why do I have (9) instances
of rotatelogs running?
Something in my MPM, perhaps, as hinted by
https://issues.apache.org/bugzilla/show_bug.cgi?id=24805 (though closed,
and quite old)?
Running separate error & access logs for each vhost, I couldn't even get
Apache started (with all of these rotatelogs processes), so I
reconfigured for a combined log file, and will split later on.
Anyway, just curious about the number of processes, here.
TIA
--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS
Rosenthal& Rosenthal, LLC www.2rosenthals.com
Need a managed Wi-Fi hotspot? www.hautspot.com
visit my IT blog www.2rosenthals.net/wordpress
please do not add my address to any non-bcc mass mailings
-------------------------------------------------------------
Dave Saville
>
>I'm experimenting with rotatelogs, as these blasted log files are eating
>me out of house and home... :-)
<snip>
>
>So, with only (3) httpd processes spawned, why do I have (9) instances of
>rotatelogs running?
I think I saw this some years ago. The other problem I saw was if one of
the HTTPD process goes south then the rotate fails as it can't shut the log
file. I gave up and use consolidated logs for access and error and do the
rotate with an external script that bounces Apache whilst it does it.
--
Kind regards
Dave Saville
Zdenek Wagner
> You received this message because you are subscribed to the Google Groups
> "Apache for OS/2" group.
> To post to this group, send email to apa...@googlegroups.com.
> To unsubscribe from this group, send email to
> apache2+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/apache2?hl=en.
>
Have you tried my zwrotlogs? (http://hroch486.icpf.cas.cz/zwrotlogs/)
Many many years ago if Apache crashed, rotatelogs crashed too and
since the log files were not closed properly, the whole contents were
lost. In my program I can ask log rotation eg every 5 minutes but the
file name is day-based. Thus I have one file per day but it is closed
end reopened every 5 minutes. In case of crash I lose at most 5
minutes.
--
Zdeněk Wagner
http://hroch486.icpf.cas.cz/wagner/
http://icebearsoft.euweb.cz
Lewis G Rosenthal
On 11/15/11 05:25 am, Zdenek Wagner thus wrote :
> 2011/11/15 Dave Saville<da...@deezee.org>:
>
>> On Tue, 15 Nov 2011 02:15:59 -0500 Lewis G Rosenthal wrote:
>>
>>> I'm experimenting with rotatelogs, as these blasted log files are eating
>>> me out of house and home... :-)
>>>
>> <snip>
>>
>>
>>> So, with only (3) httpd processes spawned, why do I have (9) instances of
>>> rotatelogs running?
>>>
>> I think I saw this some years ago. The other problem I saw was if one of the
>> HTTPD process goes south then the rotate fails as it can't shut the log
>> file. I gave up and use consolidated logs for access and error and do the
>> rotate with an external script that bounces Apache whilst it does it.
>>
>>
I did indeed recall that you found rotatelogs unsuitable, Dave, but was
unclear as to the actual reason, at this point. Thanks for refreshing my
memory.
<snip>
> Have you tried my zwrotlogs? (http://hroch486.icpf.cas.cz/zwrotlogs/)
> Many many years ago if Apache crashed, rotatelogs crashed too and
> since the log files were not closed properly, the whole contents were
> lost. In my program I can ask log rotation eg every 5 minutes but the
> file name is day-based. Thus I have one file per day but it is closed
> end reopened every 5 minutes. In case of crash I lose at most 5
> minutes.
>
I think I inquired on-list some time ago about zwrotlogs, Zdenek, and
you pointed me back to rotatelogs at the time, though again, I am
unclear as to your reasoning. I'll surely have another look, though, as
the whole idea of losing the log simply because of a daemon crash. Also,
considering the number of times the child processes of my Apache
installation seem to fall over (in production; on this test system, I
haven't yet been able to duplicate that, but I have no real load on it),
I can only imagine the mess I'd end up with between runaway processes
and broken logs...
Thanks, both of you!
Steven Levine
on 11/15/11
at 11:25 AM, Zdenek Wagner <zdenek...@gmail.com> said:
Hi guys,
>Have you tried my zwrotlogs? (http://hroch486.icpf.cas.cz/zwrotlogs/)
>Many many years ago if Apache crashed, rotatelogs crashed too and since
>the log files were not closed properly, the whole contents were lost. In
>my program I can ask log rotation eg every 5 minutes but the file name is
>day-based. Thus I have one file per day but it is closed end reopened
>every 5 minutes. In case of crash I lose at most 5 minutes.
There's also the attached. It contains a few minor tweaks to your
zwrotlogs and builds with OpenWatcom.
Steven
--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> eCS/Warp/DIY etc.
www.scoug.com www.ecomstation.com
----------------------------------------------------------------------
Steven Levine
at 08:36 AM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
Hi guys,
>I think I inquired on-list some time ago about zwrotlogs, Zdenek, and
>you pointed me back to rotatelogs at the time, though again, I am
>unclear as to your reasoning. I'll surely have another look, though, as
>the whole idea of losing the log simply because of a daemon crash.
You will not lose the log if httpd crashes, only if the system crashes.
FWIW, I said this to massimo during on of our previous threads discussing
log rotation.
<snip>
There are a couple of nits. I find that both the executable and log file
names must be fully specified. Neither rotatelogs and zwrotlogs seems to
know to shut down when apache is shutdown.
There may be an issue with name based virtual hosts and the logging method
selection. In my tests, I ended up with multiple instances of
rotatelogs/zwrotlogs even though only one of the virtual hosts specified a
piped log CustomLog.
</snip>
Regarding my comment that zwrotlogs does not seem to know to shut down
when httdp dies, I'm not sure I still believe this. The zrotlogs code
looks like it should detect the broken pipe and exit cleanly.
Regards,
Zdenek Wagner
> In <4EC26AEE...@2rosenthals.com>, on 11/15/11
> at 08:36 AM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
>
> Hi guys,
>
>>I think I inquired on-list some time ago about zwrotlogs, Zdenek, and
>>you pointed me back to rotatelogs at the time, though again, I am
>>unclear as to your reasoning. I'll surely have another look, though, as
>>the whole idea of losing the log simply because of a daemon crash.
>
> You will not lose the log if httpd crashes, only if the system crashes.
>
> FWIW, I said this to massimo during on of our previous threads discussing
> log rotation.
>
> <snip>
> There are a couple of nits. I find that both the executable and log file
> names must be fully specified. Neither rotatelogs and zwrotlogs seems to
> know to shut down when apache is shutdown.
>
> There may be an issue with name based virtual hosts and the logging method
> selection. In my tests, I ended up with multiple instances of
> rotatelogs/zwrotlogs even though only one of the virtual hosts specified a
> piped log CustomLog.
> </snip>
>
> Regarding my comment that zwrotlogs does not seem to know to shut down
> when httdp dies, I'm not sure I still believe this. The zrotlogs code
> looks like it should detect the broken pipe and exit cleanly.
>
cannot test it. As I remember, I had no problems when httpd was
restarted.
> Regards,
>
> Steven
>
> --
> ----------------------------------------------------------------------
> "Steven Levine" <ste...@earthlink.net> eCS/Warp/DIY etc.
> www.scoug.com www.ecomstation.com
> ----------------------------------------------------------------------
>
> --
> You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
> To post to this group, send email to apa...@googlegroups.com.
> To unsubscribe from this group, send email to apache2+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/apache2?hl=en.
>
>
--
Lewis G Rosenthal
> In<CAC1phyY=RRXyyaN-9D4Ba4vHkDOGM...@mail.gmail.com>,
> on 11/15/11
> at 11:25 AM, Zdenek Wagner<zdenek...@gmail.com> said:
>
> Hi guys,
>
>
>> Have you tried my zwrotlogs? (http://hroch486.icpf.cas.cz/zwrotlogs/)
>> Many many years ago if Apache crashed, rotatelogs crashed too and since
>> the log files were not closed properly, the whole contents were lost. In
>> my program I can ask log rotation eg every 5 minutes but the file name is
>> day-based. Thus I have one file per day but it is closed end reopened
>> every 5 minutes. In case of crash I lose at most 5 minutes.
>>
> There's also the attached. It contains a few minor tweaks to your
> zwrotlogs and builds with OpenWatcom.
>
Steve, insofar as not losing the log if the daemon crashes, if
rotatelogs has the file open, how would a second instance (spawned when
the daemon restarted) know? How would it access the same file held open
by the (orphaned) rotatelogs process?
Again, these re points of curiosity for now, as I haven't settled on
anything just yet. Still all of the logs for all of those virtuals are
really getting to me, along with FTP logs which seem to grow at an
alarming rate (for what is, by my estimation, a relatively quiet
server)... I do need to get a handle on this stuff. A single, unifying
solution (Apache, PHP, FTP, at least) would be great, which is why I'm
just thinking along the lines of a syslog solution on another box.
Steven Levine
at 12:42 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
Hi Lewis,
>Steve, insofar as not losing the log if the daemon crashes, if
>rotatelogs has the file open, how would a second instance (spawned when
>the daemon restarted) know? How would it access the same file held open
>by the (orphaned) rotatelogs process?
I can't speak for rotatelogs, but zrotlogs will detect the broken pipe and
shutdown which will close the file. Zrotlogs opens logs for append, so
there's not problem with overwriting existing data.
Lewis G Rosenthal
> In<4EC2A49...@2rosenthals.com>, on 11/15/11
> at 12:42 PM, Lewis G Rosenthal<lgros...@2rosenthals.com> said:
>
> Hi Lewis,
>
>> Steve, insofar as not losing the log if the daemon crashes, if
>> rotatelogs has the file open, how would a second instance (spawned when
>> the daemon restarted) know? How would it access the same file held open
>> by the (orphaned) rotatelogs process?
> I can't speak for rotatelogs, but zrotlogs will detect the broken pipe and
> shutdown which will close the file. Zrotlogs opens logs for append, so
> there's not problem with overwriting existing data.
>
Still, I'm wondering whether some mechanism to log to a MySQL db might
be more efficient (something akin to what is described here:
http://www.rsyslog.com/doc/rsyslog_mysql.html ).
Joachim Benjamins
> On 11/15/11 01:57 pm, Steven Levine thus wrote :
>> In<4EC2A49...@2rosenthals.com>, on 11/15/11
>> at 12:42 PM, Lewis G Rosenthal<lgros...@2rosenthals.com> said:
>>
>> Hi Lewis,
>>
>>> Steve, insofar as not losing the log if the daemon crashes, if
>>> rotatelogs has the file open, how would a second instance (spawned when
>>> the daemon restarted) know? How would it access the same file held open
>>> by the (orphaned) rotatelogs process?
>> I can't speak for rotatelogs, but zrotlogs will detect the broken pipe
>> and
>> shutdown which will close the file. Zrotlogs opens logs for append, so
>> there's not problem with overwriting existing data.
>>
> Thanks. I'll give this a shot next time I look at this logging issue.
> Still, I'm wondering whether some mechanism to log to a MySQL db might
> be more efficient (something akin to what is described here:
> http://www.rsyslog.com/doc/rsyslog_mysql.html ).
>
This will most definitely not be more efficient. I tried
(http://www.logaholic.com) but MySQL really can't handle lot's of data
all that well (although this seems to have improved in recent versions).
Of course, the realtime analytics *are* very cool ;)
--
Kind regards / met vriendelijke groet,
Joachim Benjamins
+------------------------------------------------------------+
| Mensys BV - The Software Catalogue for Professionals |
+------------------------------------------------------------+
| Joachim Benjamins | Email: joa...@mensys.nl |
| Crayenestersingel 65 | Phone: +31-23-5482020 |
| 2012 PG Haarlem | Fax : +31-23-5482030 |
| The Netherlands | www : http://www.mensys.nl/ |
+------------------------------------------------------------+
Zdenek Wagner
> Lewis G Rosenthal wrote:
>>
>> On 11/15/11 01:57 pm, Steven Levine thus wrote :
>>>
>>> In<4EC2A49...@2rosenthals.com>, on 11/15/11
>>> at 12:42 PM, Lewis G Rosenthal<lgros...@2rosenthals.com> said:
>>>
>>> Hi Lewis,
>>>
>>>> Steve, insofar as not losing the log if the daemon crashes, if
>>>> rotatelogs has the file open, how would a second instance (spawned when
>>>> the daemon restarted) know? How would it access the same file held open
>>>> by the (orphaned) rotatelogs process?
>>>
>>> I can't speak for rotatelogs, but zrotlogs will detect the broken pipe
>>> and
>>> shutdown which will close the file. Zrotlogs opens logs for append, so
>>> there's not problem with overwriting existing data.
>>>
>> Thanks. I'll give this a shot next time I look at this logging issue.
>> Still, I'm wondering whether some mechanism to log to a MySQL db might be
>> more efficient (something akin to what is described here:
>> http://www.rsyslog.com/doc/rsyslog_mysql.html ).
>>
>
> This will most definitely not be more efficient. I tried
> (http://www.logaholic.com) but MySQL really can't handle lot's of data all
> that well (although this seems to have improved in recent versions).
>
insertions. It is suggested in the documentation to switch indexing
off before large insertions and to rebuild the index later but this is
not usable in logging.
> Of course, the realtime analytics *are* very cool ;)
>
> --
> Kind regards / met vriendelijke groet,
>
> Joachim Benjamins
>
> +------------------------------------------------------------+
> | Mensys BV - The Software Catalogue for Professionals |
> +------------------------------------------------------------+
> | Joachim Benjamins | Email: joa...@mensys.nl |
> | Crayenestersingel 65 | Phone: +31-23-5482020 |
> | 2012 PG Haarlem | Fax : +31-23-5482030 |
> | The Netherlands | www : http://www.mensys.nl/ |
> +------------------------------------------------------------+
>
Lewis G Rosenthal
> Lewis G Rosenthal wrote:
>> On 11/15/11 01:57 pm, Steven Levine thus wrote :
>>> In<4EC2A49...@2rosenthals.com>, on 11/15/11
>>> at 12:42 PM, Lewis G Rosenthal<lgros...@2rosenthals.com> said:
>>>
>>> Hi Lewis,
>>>
>>>> Steve, insofar as not losing the log if the daemon crashes, if
>>>> rotatelogs has the file open, how would a second instance (spawned
>>>> when
>>>> the daemon restarted) know? How would it access the same file held
>>>> open
>>>> by the (orphaned) rotatelogs process?
>>> I can't speak for rotatelogs, but zrotlogs will detect the broken
>>> pipe and
>>> shutdown which will close the file. Zrotlogs opens logs for append, so
>>> there's not problem with overwriting existing data.
>>>
>> Thanks. I'll give this a shot next time I look at this logging issue.
>> Still, I'm wondering whether some mechanism to log to a MySQL db
>> might be more efficient (something akin to what is described here:
>> http://www.rsyslog.com/doc/rsyslog_mysql.html ).
>>
>
> This will most definitely not be more efficient. I tried
> (http://www.logaholic.com) but MySQL really can't handle lot's of data
> all that well (although this seems to have improved in recent versions).
>
> Of course, the realtime analytics *are* very cool ;)
>
There used to be an Apache module for this (mod_log_sql), but it doesn't
seem to be around any longer.
Lewis G Rosenthal
<snip>