lighttpd 1.5 + mod_cache memory leak

[carrotis]

Hello Hongyu~

Test Envrionment like this :

Server : Xeon 2 CPU, Memory 6Gb, Ubuntu, lighttpd 1.5.r1992 +
mod_cache
Pyshical Device : 1 Gb
Max Traffic : 120 Mb ( LIghttpd down. no error )

while true ; do ps axfu | grep lighttpd >> ps1.txt ; sleep 10 ; done
result :
root 5510 13.0 19.1 1259780 1191872 ? Sl 11:10 1:43
lighttpd -f /etc/lighttpd.conf
root 5510 10.9 37.4 2632808 2334940 ? Sl 11:09 3:39
lighttpd -f /etc/lighttpd.conf
root 5510 11.0 40.3 2791444 2512144 ? Sl 11:10 3:48
lighttpd -f /etc/lighttpd.conf

and LIghttpd down..

How can I solve this problem ?.

Thank you.

[Hongyu]

Two ways to check whether modcache leaks memory

1) Disable mod_cache and see how much memory lighttpd eats.

or

2) load "mod_status" and add
         status.statistics-url = "/cache_status"
    to your lighttpd.conf

   Then visit http://ip/cache_status after restart lighttpd to see how much memory mod_cache used

[carrotis]

2) load "mod_status" ..

cache.cached-items: 25
cache.hitrate(%): 9
cache.memory-inuse(KB): 1
proxy-core.0.backends."1.1.1.1:80".load: 1
proxy-core.0.backends."1.1.1.1:80".pool_size: 1
proxy-core.0.backends."1.1.1.1:80".requests: 36
proxy-core.0.backends."1.1.1.1:80".requests_failed: 0
proxy-core.0.backlogged: 2
proxy-core.requests: 36

cache.cached-items: 30
cache.hitrate(%): 7
cache.memory-inuse(KB): 1
proxy-core.0.backends."1.1.1.1:80".load: 1
proxy-core.0.backends."1.1.1.1:80".pool_size: 1
proxy-core.0.backends."1.1.1.1:80".requests: 41
proxy-core.0.backends."1.1.1.1:80".requests_failed: 0
proxy-core.0.backlogged: 19
proxy-core.requests: 41


cache.cached-items: 36
cache.hitrate(%): 6
cache.memory-inuse(KB): 2
proxy-core.0.backends."1.1.1.1:80".load: 1
proxy-core.0.backends."1.1.1.1:80".pool_size: 1
proxy-core.0.backends."1.1.1.1:80".requests: 49
proxy-core.0.backends."1.1.1.1:80".requests_failed: 0
proxy-core.0.backlogged: 11
proxy-core.requests: 49


cache.cached-items: 44
cache.hitrate(%): 5
cache.memory-inuse(KB): 2
proxy-core.0.backends."1.1.1.1:80".load: 1
proxy-core.0.backends."1.1.1.1:80".pool_size: 1
proxy-core.0.backends."1.1.1.1:80".requests: 62
proxy-core.0.backends."1.1.1.1:80".requests_failed: 0
proxy-core.0.backlogged: 15
proxy-core.requests: 62


Hmm...mod_cache has not memory leak problem

But Lighttpd killed without log.

I try to make flv streaming + cache with lighttpd

Help me plz..

additional config..

cache Storage : /cache1 , 1.1 TB ( reiserfs , noatime, nodev ...
etc )
cache Object : flv file ( 5 MB ~ 760 MB )

lighttpd.conf

server.modules = (
#"mod_simple_vhost",
"mod_access",
"mod_accesslog",
"mod_cgi",
"mod_fastcgi",
"mod_rewrite",
"mod_auth",
"mod_status",
"mod_cache",
"mod_flv_streaming",
"mod_proxy_core",
"mod_proxy_backend_http",
"mod_proxy_backend_fastcgi",
"mod_evasive",
"mod_secdownload",
"mod_accesslog"
)


server.document-root = "/cache1/1.1.1.1/"
flv-streaming.extensions = ( ".flv" )
log/pid/.. skip.
server.max-write-idle = 360
cache.bases = ("/cache1") #write cached files in /data/cache
directory
cache.refresh-pattern = (
"\.(?i)(js|css|xml)$" => "240", # update js/css/xml every 4
hours and on refresh requests
"\.(?i)(htm|html|shtml)$" => "30", # update html/htm/shtml
every 30 minutes and on refresh requests
"\.(?i)(jpg|bmp|jpeg|gif|png)$" => "2880", # update graphics
files every 2 days
"\.(?i)(flv)$" => "0 fetchall-for-range-request flv-
streaming",
"\.(?i)(rar|zip|wmv|avi|mp3|ape|rm|mpeg|mpg|wma|asf|rmvb|flv)
$" => "0 fetchall-for-range-request",
"." => "30 update-on-refresh" # default to update every 30
minutes and on refresh requests
)
#cache.support-queries = "disable" #ignore '?' in url
cache.support-queries = "enable" #ignore '?' in url
cache.debug = "enable"
proxy-core.backends = ( "1.1.1.1")
#proxy-core.balancer = "round-robin" # or "sqf" or "carp"
proxy-core.protocol = "http"
proxy-core.worked-with-modcache = "enable"

[Hongyu]

1) unload unused module first

2) change settings.h

remove
"#define BUFFER_MAX_REUSE_SIZE  (4 * 1024) "
add
"#define BUFFER_MAX_REUSE_SIZE  0 "

then recompile/reinstall lighttpd

probably help you

[rebelvideo]

The problem is mod_proxy_core I have the same problem and there is no
resolution that I can find

Mod_proxy_core appears to cache the entire file in RAM before handing
the results to mod_cache
thus if you have multiple requests that are referred to the proxy
backend it quickly consumes ram until the process is killed off

Therefore no errors no logs and no clues

I do not have a resolution at present

I am opening another thread to discuss the issue and possible
workarounds

Chris

[Hongyu]

As I understand, mod_proxy_core for lighttpd 1.5 don't cache full content before writing to clients.
Actually mod_proxy_core writes the content of backends as soon as it received from backends.

However, mod_proxy for lighttpd 1.4 cache full content in ram before writing to clients, which eats a lot of memory.

[Chris Hamono]

Perhaps mod_proxy_backend_http is to blame then

Once I disable mod_proxy_core memory usage drops to about 53Mb for the process
if I re-enable mod_proxy_core memory increases to 1.5Gb within a short period and after a while hovers around 2Gb

That is with "#define BUFFER_MAX_REUSE_SIZE  0 " and recompiled

2008/5/18 Hongyu <shel...@gmail.com>:

[Chris Hamono]

You could try using

server.network-backend = "writev"

I found using other backends unreliable particularly if you are throttling by setting connection.kbytes-per-second = n > 0

using sendfile (I think thats the default) caused a large increase in memory usage and resulted in throttling being very unreliable.

sendfile sends a chunk of the file to the user, As much as it can squeeze into the pipe in one go. Eventually the throttling mechanism catches up to it and throttles by stopping the socket.

Unfortunately this can cause the client to disconnect and reconnect, If your wait timeout is too low it may also cause lighty to disconnect.

2008/5/18 Chris Hamono <cha...@gmail.com>: