Zobacz stronę "file_appender crash during logging "

2 views
Skip to first unread message

odi

unread,
Apr 7, 2009, 6:39:36 AM4/7/09
to Log4erl


Kliknij http://groups.google.com/group/log4erl/web/file-appender-crash-during-logging
- a jeżeli to nie zadziała, skopiuj adres i wklej go w polu adresu
przeglądarki internetowej.

Ahmed Ali

unread,
Apr 7, 2009, 2:26:36 PM4/7/09
to log...@googlegroups.com
Hi Kuba,

Obviously, the crash is because many requests comming to file appender
and requests accumulate waiting for IO to finish until process inbox
takes a lot of memory and then erlang crash. A case study about this
is in [1].

Unfortunately, this will almost always happens in a concurrent
language like erlang and log4erl, and error_logger for that matter,
will always suffer from this.

I've actually been thinking about this for sometime and I think the
best option is to add persistent queue with something like memcacheq
as the backend. This will slow down logging but it will not cause an
"out of memory" crash because of many requests. If anyone has a better
idea, please let me know.

[1] http://blogtrader.net/dcaoyuan/entry/a_case_study_of_scalable

Regards,

Ahmed

2009/4/7 odi <kuba....@gmail.com>:

Ahmed Ali

unread,
Apr 7, 2009, 2:30:59 PM4/7/09
to log...@googlegroups.com
Hi Kuba,

I forgot to mention that the best thing to do right now is to raise
logging level to something like warn and shorten log messages.
However, this is not a solution.

Regards,

Ahmed

2009/4/7 Ahmed Ali <ahmed....@gmail.com>:

Ahmed Ali

unread,
Apr 8, 2009, 3:30:35 AM4/8/09
to log...@googlegroups.com

odi

unread,
May 5, 2009, 8:06:27 AM5/5/09
to Log4erl
hi,
Thanks for the fixes in version 0.8.6;) However I'm very curious about
changing gen_event:notify to sync_notify in log_manager.erl. Actually
it works quite good - no memory crashes and it works faster - during 5
minute test I created 38000 logging processes compared to 26000 using
older version of log4erl. Could somebody explain to me how it is
possible that introducing sync logging caused Erlang to run faster?

regards,
Kuba

On 8 Kwi, 09:30, Ahmed Ali <ahmed.naw...@gmail.com> wrote:
> Hi,
>
> Please check thishttp://log4erl.blogspot.com/2009/04/log4erl-issue-when-high-load.html.
>
> Regards,
>
> Ahmed
>
> 2009/4/7 Ahmed Ali <ahmed.naw...@gmail.com>:
>
> > Hi Kuba,
>
> > I forgot to mention that the best thing to do right now is to raise
> > logging level to something like warn and shorten log messages.
> > However, this is not a solution.
>
> > Regards,
>
> > Ahmed
>
> > 2009/4/7 Ahmed Ali <ahmed.naw...@gmail.com>:
> >> Hi Kuba,
>
> >> Obviously, the crash is because many requests comming to file appender
> >> and requests accumulate waiting for IO to finish until process inbox
> >> takes a lot of memory and then erlang crash. A case study about this
> >> is in [1].
>
> >> Unfortunately, this will almost always happens in a concurrent
> >> language like erlang and log4erl, and error_logger for that matter,
> >> will always suffer from this.
>
> >> I've actually been thinking about this for sometime and I think the
> >> best option is to add persistent queue with something like memcacheq
> >> as the backend. This will slow down logging but it will not cause an
> >> "out of memory" crash because of many requests. If anyone has a better
> >> idea, please let me know.
>
> >> [1]http://blogtrader.net/dcaoyuan/entry/a_case_study_of_scalable
>
> >> Regards,
>
> >> Ahmed
>
> >> 2009/4/7 odi <kuba.od...@gmail.com>:
>
> >>> Kliknijhttp://groups.google.com/group/log4erl/web/file-appender-crash-during...

Ahmed Ali

unread,
May 5, 2009, 2:48:23 PM5/5/09
to log...@googlegroups.com
Hi Kuba,

Actually, you can find the explanation in erlang-list discussion I've
added link to in log4erl blog.

The slow performance was because Erlang is slow at handling process
inbox with large messages, not due to IO or anything else. When I
added sync_notify, I basically eliminated the long queue in gen_event
processes and hence the speedup. I felt the same way once I figured
this out :)

I'm glad to finally see some kind of performance benchmark for
log4erl. Until now, I haven't put a lot of effort on raw speed.
However, I'm aiming to increase the speed even further. I'm reading
more about Log4erl IO speed tricks and will include that in future
releases.

Have a nice day.

Ahmed

2009/5/5 odi <kuba....@gmail.com>:
Reply all
Reply to author
Forward
0 new messages