E2gardian 4.1.4 & 5.0.2 - Can't increase httpworkers due to soft file limit 1024

790 views
Skip to first unread message

wil...@gmail.com

unread,
May 17, 2018, 6:24:49 PM5/17/18
to e2guardian
Guys,

I need some help. I compiled both 4.1.4 and later 5.0.2 for an ARM device (pi) hoping it would work better. I hit max httpworkers often with only a handful of devices running. If I increase to 300,400,500 for example the service fails without any errors reported. If I use 800 per an online post It fails but at least gives an error:

httpworkers option in e2guardian.conf has a value too high for current file id limit (1024)
httpworkers 800 must not exceed 50% of 1017

I put hard and soft file limits in /etc/security/limits.conf for 8192 for e2guardian, then root, and finally *
I put hard and soft limits for 8192 in the /etc/init.d/e2guardian file w/ ulimit
I ensured usePAM is yes
I changed the following to:
/usr/include/arm-linux-gnueabihf/bits/typesizes.h:#define __FD_SETSIZE 8192
/usr/include/linux/posix_types.h:#define __FD_SETSIZE 8192


I tried recompiling with --with-filedescriptors=8192 which doesn't match these versions (4 & 5) compiler options and gets ignored.

I edited e2guardardian.cpp's FD_SETSIZE from 1024 to 8192 manually

Off another post I tried adding CXXFLAGS=-FD_SETSIZE=8192, even what might have been a typo on another site adding CXXFLAGS=-DFD_SETSIZE=8192

no matter what I do after it's compiled either version of e2guardian runs with a soft limit of 1024 and hard of 4096....

I'm confirming this via cat /proc/PID/limits

If I log in as user e2guardian on the system my limits are 8192 as they should be per ulimit

I suspect either sysV isn't giving the proper limits when running the service or else e2guardian is limiting it itself? I'm not savvy enough to take it any further, maybe I'm just missing a compiler option?

one was compiled thus:
e2guardian 4.1.4

Built with: 'CXXFLAGS=-DFD_SETSIZE=8192' '--prefix=/usr' '--enable-clamd=yes' '--with-proxyuser=e2guardian' '--with-proxygroup=e2guardian' '--sysconfdir=/etc' '--localstatedir=/var' '--enable-icap=yes' '--enable-commandline=yes' '--enable-email=yes' '--enable-ntlm=yes' '--enable-trickledm=yes' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' 'CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-z,relro' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' '--enable-pcre=yes' '--with-filedescriptors=8192' '--enable-sslmitm=yes'

my current version 5 is like so:
e2guardian 5.0.2

Built with: '--prefix=/usr' '--enable-clamd=yes' '--with-proxyuser=e2guardian' '--with-proxygroup=e2guardian' '--sysconfdir=/etc' '--localstatedir=/var' '--enable-icap=yes' '--enable-commandline=yes' '--enable-email=yes' '--enable-ntlm=yes' '--enable-pcre=yes' '--enable-sslmitm=yes' 'CXXFLAGS=-FD_SETSIZE=8192'

currently I have the following on the e2guardian running process:
cat /proc/1506/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 7345 7345 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 7345 7345 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

any help would be greatly appreciated, thanks! :)

FredB

unread,
May 18, 2018, 5:49:02 AM5/18/18
to e2guardian
Can you make a try with this package ? http://numsys.eu/e2guardian/

/etc/init.d/e2guardian is deprecated now, if the package doesn't works at least use this file with systemd
https://github.com/e2guardian/e2guardian/blob/develop/data/scripts/e2guardian.service.in

wil...@gmail.com

unread,
May 18, 2018, 8:11:42 PM5/18/18
to e2guardian
Kudos FredB for providing the e2guardian.service script for me. I removed the sysV style script from /etc/init.d and added e2guardian.service to /etc/systemd/system, set permissions and made it executable with the following:


[Unit]
Description=E2guardian Web filtering
After=network.target

[Service]
TasksMax=infinity
LimitNOFILE=8192
Type=forking
IgnoreSIGPIPE=no
GuessMainPID=no
ExecStart=/usr/sbin/e2guardian
ExecReload=/usr/sbin/e2guardian -r
UMask=027

[Install]
WantedBy=multi-user.target


Now I've got my 8192 nofile limit!!!

cat /proc/715/limits


Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 7345 7345 processes

Max open files 8192 8192 files

Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 7345 7345 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us


Bad news though, I increased httpworkers to 800 and restarted it and the service is inactive with zero errors reported in the journal/system messages. Tried 1000, 2000 and no luck, just won't start with anything other than 200 httpworkers...

FredB

unread,
May 19, 2018, 2:20:02 AM5/19/18
to e2gua...@googlegroups.com
Did you make a test with my package ?
--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.

wil...@gmail.com

unread,
May 19, 2018, 11:28:40 AM5/19/18
to e2guardian
Honestly no, I'm interested in being able to compile my own builds with it. I run MITM, and am interested in have Transparent HTTPS, and not needing squid, having more speed and such. I'm also interested in deploying it with specific compilation settings to multiple architectures, x86, x64, arm, etc. So ultimately, I want to be able to compile the version and features desired for any device, the pi is just my starting test. So using a pre-built package, while greatly appreciated, isn't my goal;( I hope you understand.

FredB

unread,
May 19, 2018, 11:48:03 AM5/19/18
to e2gua...@googlegroups.com
Yes, but I'm searching ...
Please just make just a test maybe there is a also an issue maybe not ...
>--
>E2guardian:
>https://groups.google.com/d/forum/e2guardian
>Github:
>https://github.com/e2guardian/e2guardian
>---
>You received this message because you are subscribed to the Google
>Groups "e2guardian" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to e2guardian+...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

wil...@gmail.com

unread,
May 19, 2018, 7:15:53 PM5/19/18
to e2guardian
I tried to install your package, but there were too many unresolvable dependency issues for me to want to fix manually, I'm sorry. I have libtommath1, yours needs 0, I a different libssl and libevent-pthreads, etc. On your pi though you can definitely change the httpworkers to whatever without issues? What's the output of your e2guardian --version...compiling options please? I might just try to compile mine in the like perhaps and see what happens...

FredB

unread,
May 20, 2018, 1:31:12 AM5/20/18
to e2gua...@googlegroups.com
My package is for raspbian 8, I will push my options and files next week


Le 20 mai 2018 01:15:52 GMT+02:00, wil...@gmail.com a écrit :
I tried to install your package, but there were too many unresolvable dependency issues for me to want to fix manually, I'm sorry.  I have libtommath1, yours needs 0, I a different libssl and libevent-pthreads, etc.  On your pi though you can definitely change the httpworkers to whatever without issues?  What's the output of your e2guardian --version...compiling options please?  I might just try to compile mine in the like perhaps and see what happens...

Message has been deleted

FredB

unread,
May 20, 2018, 9:57:14 AM5/20/18
to e2gua...@googlegroups.com

No yet tested, but 4.1.4 package is compiled with:

ulimit -n 16384

./autogen.sh

'--prefix=/usr' '--enable-clamd=yes' '--with-proxyuser=e2guardian' '--with-proxygroup=e2guardian' '--sysconfdir=/etc' '--localstatedir=/var' '--enable-icap=yes' '--enable-commandline=yes' '--enable-email=yes' '--enable-ntlm=yes' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--enable-pcre=yes' '--enable-sslmitm=yes' 'CPPFLAGS=-mno-sse2 -frecord-gcc-switches -g3 -O0'

make

On Raspbian (Debian 8)


Attached please find my files


Fred

datas.zip

wil...@gmail.com

unread,
May 29, 2018, 9:43:21 AM5/29/18
to e2guardian
FredB,

I tried your compile options minus 'CPPFLAGS=-mno-sse2 -frecord-gcc-switches -g3 -O0' and had no luck. Using Systemd init scripts obviously solved the initial problem of file descriptor limits, but I still can't increase http_workers without the service failing without any useful errors/warnings. Were you able to try to increase http_workers on your build on a pi and confirm you can increase them? As a band-aid I made a cronjob to check dsats and when I hit 180 workers it restarts e2guardian, checks every five minutes....

Thanks

Renato C. Pacheco

unread,
May 29, 2018, 9:49:37 AM5/29/18
to e2guardian
wiltdu,

Add this  options in your systemd script, on [Service] section:

TasksMax=infinity
LimitNOFILE=65535

Then, execute this to update daemon:

# systemctl daemon-reload

Try again. I hope this works.

--
E2guardian:
https://groups.google.com/d/forum/e2guardian
Github:
https://github.com/e2guardian/e2guardian
---
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--

Renato Carneiro Pacheco
Pós-Graduado em Segurança em Redes de Computadores
Graduado em Redes de Comunicação

https://www.linkedin.com/in/renatocarneirop/
https://www.facebook.com/renatocarneirop

FredB

unread,
May 29, 2018, 10:01:48 AM5/29/18
to e2guardian
Can you try two things:

In e2guardian.ccp add

#define MAXTHREADS 1000000
#define THREADSTACK 65536

Before compilation

And/or

Start e2guardian manually like this

ulimit -u && /usr/sbin/e2guardian


Fred

FredB

unread,
May 29, 2018, 10:31:00 AM5/29/18
to e2guardian
And also

try ulimit -s 8192 right before your compilation

ulimit -s 8192
./autogen.sh
./configure balabala
make

wil...@gmail.com

unread,
May 29, 2018, 11:00:34 AM5/29/18
to e2guardian
Renato,

Thanks, I tried TasksMax=infinity and hiked the http_workers to 800 and restarted the service, still it shows the service innactive(dead). I also per FredB tried manually setting a high ulimit and ran e2guardian from the command prompt and it didn't give any error, but didn't run, just returned my prompt and grepping ps didn't show it running either. I'll try the compilation ideas from FredB as soon as I'm able, I'd really like confirmation from anybody though that they have E2 running on a pi and can increase the http_workers above 200 with mitm and have the service running...anybody?

Thanks

wil...@gmail.com

unread,
May 29, 2018, 12:12:52 PM5/29/18
to e2guardian
FredB,

I tried adding:

#define MAXTHREADS 1000000
#define THREADSTACK 65536

and increased my ulimit prior to autogen/compiliation, still can't increase http_workers.

What I find interesting is that I can't increase them even to 300 or 400, let alone anything large...

Renato C. Pacheco

unread,
May 29, 2018, 1:18:29 PM5/29/18
to e2guardian
wiltdu,

Systemd has a weird behavior: if you started the daemon and it died for some reason, you have to stop the daemon (even though was dead) and start again. In this case, don't use restart function. Did you put all of two options that I mentioned before in [Service] section? I'm asking because you mentioned only one of them.


--
E2guardian:
https://groups.google.com/d/forum/e2guardian
Github:
https://github.com/e2guardian/e2guardian
---
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

FredB

unread,
May 29, 2018, 1:36:35 PM5/29/18
to e2gua...@googlegroups.com

wiltdu,

Can you send me your config files?To be sure we make the same test


wil...@gmail.com

unread,
May 29, 2018, 2:25:29 PM5/29/18
to e2guardian
Renato,

The service file already has LimitNOFILE=65535 so I omitted it from the reply. I also no longer get complaints about the file descriptor limits, that was resolved simply by switching from the sysV init script to the systemd init script if there was any question about that. I did go ahead and redid the test stopping the service completely and then starting it, same result, no error on screen, says inactive(dead) under status, journalctl -xe -u e2guardian shows no errors, dmesg has no errors, starting it from the command line gives no errors;(

wil...@gmail.com

unread,
May 29, 2018, 2:44:29 PM5/29/18
to e2guardian
FredB,

I'm using version 5.0.2 currently, though can provide the configs for 4.1.4 too if required.

the .conf file looks like this:

loglevel=2
logexceptionhits=0
proxytimeout=5
showweightedfound=off
weightedphrasemode=1
phrasefiltermode=2
reverseaddresslookups=off
maxcontentfiltersize=256
httpworkers=200
nologger=off
logadblocks=off
enablessl=on


preauthstoryboard = '/etc/e2guardian/preauth.story'
transparenthttpsport = 8443
maxheaderlines=60
dstatlocation = '/var/log/e2guardian/dstats.log'
connecttimeout = 1

languagedir = '/usr/share/e2guardian/languages'
language = 'ukenglish'
logfileformat = 1
filterip =
filterports = 8080
proxyport = 3128
proxyexchange = 61
pcontimeout = 55
usecustombannedimage = on
custombannedimagefile = '/usr/share/e2guardian/transparent1x1.gif'
usecustombannedflash = on
custombannedflashfile = '/usr/share/e2guardian/blockedflash.swf'
filtergroups = 1
filtergroupslist = '/etc/e2guardian/lists/filtergroupslist'
preservecase = 0
hexdecodecontent = off
forcequicksearch = off
reverseclientiplookups = off
logclienthostnames = off
maxcontentramcachescansize = 2000
maxcontentfilecachescansize = 20000
filecachedir = '/tmp'
deletedownloadedtempfiles = on
initialtrickledelay = 20
trickledelay = 10
downloadmanager = '/etc/e2guardian/downloadmanagers/default.conf'
contentscannertimeout = 60
contentscanexceptions = off
recheckreplacedurls = off
forwardedfor = off
usexforwardedfor = off
logconnectionhandlingerrors = off
maxips = 0 **NOT YET IMPLEMENTED
nodaemon = off
loguseragent = off
cacertificatepath = '/etc/e2guardian/ssl/my_rootCA.crt'
caprivatekeypath = '/etc/e2guardian/ssl/private_root.pem'
certprivatekeypath = '/etc/e2guardian/ssl/private_cert.pem'
generatedcertpath = '/etc/e2guardian/ssl/generatedcerts'

iplist = 'name=bannedclient,messageno=100,logmessageno=103,path=/etc/e2guardian/lists/bannediplist'
sitelist = 'name=bannedclient,messageno=100,logmessageno=104,path=/etc/e2guardian/lists/bannedclientlist'
iplist = 'name=exceptionclient,messageno=600,path=/etc/e2guardian/lists/exceptioniplist'
sitelist = 'name=exceptionclient,messageno=631,path=/etc/e2guardian/lists/exceptionclientlist'
iplist = 'name=authexception,messageno=602,path=/etc/e2guardian/lists/authexceptioniplist'
sitelist = 'name=authexception,messageno=602,path=/etc/e2guardian/lists/authexceptionsitelist'
urllist = 'name=authexception,messageno=603,path=/etc/e2guardian/lists/authexceptionurllist'

searchsitelistforip = off


the f1.conf looks like this:


ssllegacylogic=off
blockdownloads=off
naughtynesslimit=50
searchtermlimit=3
smartsearchterm=off
categorydisplaythreshold=0
bypass=0
bypasskey=''
reportinglevel=3
sslcertcheck=off
sslmitm=on
mitmcheckcert=off


storyboard = '/etc/e2guardian/f1.story'
notifyav = off
contentscanexceptions = off

onlymitmsslgrey = off
nocheckcertsitelist = '/etc/e2guardian/lists/nocheckcertsitelist'
groupname = 'Default_Profile'
bannedphraselist = '/etc/e2guardian/lists/bannedphraselist'
weightedphraselist = '/etc/e2guardian/lists/weightedphraselist'
exceptionphraselist = '/etc/e2guardian/lists/exceptionphraselist'
maxuploadsize = -1
embeddedurlweight = 0
infectionbypass = 0
infectionbypasskey = ''
infectionbypasserrorsonly = on
disablecontentscan = on
disablecontentscanerror = off
deepurlanalysis = off
mailfrom = ''
avadmin = ''
contentadmin = ''
avsubject = 'e2guardian virus block'
contentsubject = 'e2guardian violation'
notifycontent = off
thresholdbyuser = off
violations = 0
threshold = 0

sitelist = 'name=banned,messageno=500,path=/etc/e2guardian/lists/bannedsitelist'
ipsitelist = 'name=banned,messageno=510,path=/etc/e2guardian/lists/bannedsiteiplist'
urllist = 'name=banned,messageno=501,path=/etc/e2guardian/lists/bannedurllist'
regexpboollist = 'name=banned,messageno=503,path=/etc/e2guardian/lists/bannedregexpurllist'
regexpboollist = 'name=banneduseragent,messageno=522,path=/etc/e2guardian/lists/bannedregexpuseragentlist'

sitelist = 'name=bannedssl,messageno=520,path=/etc/e2guardian/lists/bannedsslsitelist'
ipsitelist = 'name=bannedssl,messageno=520,path=/etc/e2guardian/lists/bannedsslsiteiplist'

sitelist = 'name=grey,path=/etc/e2guardian/lists/greysitelist'
ipsitelist = 'name=grey,path=/etc/e2guardian/lists/greysiteiplist'
urllist = 'name=grey,path=/etc/e2guardian/lists/greyurllist'
sitelist = 'name=greyssl,path=/etc/e2guardian/lists/greysslsitelist'
ipsitelist = 'name=greyssl,path=/etc/e2guardian/lists/greysslsiteiplist'

sitelist = 'name=exception,messageno=602,path=/etc/e2guardian/lists/exceptionsitelist'
ipsitelist = 'name=exception,messageno=602,path=/etc/e2guardian/lists/exceptionsiteiplist'
urllist = 'name=exception,messageno=603,path=/etc/e2guardian/lists/exceptionurllist'
regexpboollist = 'name=exception,messageno=609,path=/etc/e2guardian/lists/exceptionregexpurllist'
regexpurlboollist = 'name=exceptionuseragent,messageno=610,path=/etc/e2guardian/lists/exceptionregexpuseragentlist'

sitelist = 'name=refererexception,messageno=620,path=/etc/e2guardian/lists/refererexceptionsitelist'
ipsitelist = 'name=refererexception,messageno=620,path=/etc/e2guardian/lists/refererexceptionsiteiplist'
urllist = 'name=refererexception,messageno=620,path=/etc/e2guardian/lists/refererexceptionurllist'
sitelist = 'name=embededreferer,path=/etc/e2guardian/lists/embededreferersitelist'
ipsitelist = 'name=embededreferer,path=/etc/e2guardian/lists/embededreferersiteiplist'
urllist = 'name=embededreferer,path=/etc/e2guardian/lists/embededrefererurllist'

sitelist = 'name=refererexception,messageno=620,path=/etc/e2guardian/lists/refererexceptionsitelist'
ipsitelist = 'name=refererexception,messageno=620,path=/etc/e2guardian/lists/refererexceptionsiteiplist'
urllist = 'name=refererexception,messageno=620,path=/etc/e2guardian/lists/refererexceptionurllist'

regexpreplacelist = 'name=change,path=/etc/e2guardian/lists/urlregexplist'
regexpreplacelist = 'name=sslreplace,path=/etc/e2guardian/lists/sslsiteregexplist'

regexpreplacelist = 'name=redirect,path=/etc/e2guardian/lists/urlredirectregexplist'

contentregexplist = '/etc/e2guardian/lists/contentregexplist'


searchlist = 'name=localbanned,messageno=581,path=/etc/e2guardian/lists/localbannedsearchlist'

fileextlist = 'name=exceptionextension,messageno=900,path=/etc/e2guardian/lists/exceptionextensionlist'
mimelist = 'name=exceptionmime,messageno=750,path=/etc/e2guardian/lists/exceptionextensionlist'
fileextlist = 'name=bannedextension,messageno=999,path=/etc/e2guardian/lists/bannedextensionlist'
mimelist = 'name=bannedmime,messageno=750,path=/etc/e2guardian/lists/bannedmimeitypelist'
sitelist = 'name=exceptionfile,path=/etc/e2guardian/lists/exceptionfilesitelist'
ipsitelist = 'name=exceptionfile,path=/etc/e2guardian/lists/exceptionfilesiteiplist'
urllist = 'name=exceptionfile,path=/etc/e2guardian/lists/exceptionfileurllist'

regexpreplacelist = 'name=headermods,path=/etc/e2guardian/lists/headerregexplist'
regexpboollist = 'name=bannedheader,path=/etc/e2guardian/lists/bannedregexpheaderlist'
regexpboollist = 'name=exceptionheader,path=/etc/e2guardian/lists/exceptionregexpheaderlist'

mimelist = 'name=exceptionvirus,path=/etc/e2guardian/lists/contentscanners/exceptionvirusmimetypelist'
fileextlist = 'name=exceptionvirus,path=/etc/e2guardian/lists/contentscanners/exceptionvirusextensionlist'
sitelist = 'name=exceptionvirus,path=/etc/e2guardian/lists/contentscanners/exceptionvirussitelist'
ipsitelist = 'name=exceptionvirus,path=/etc/e2guardian/lists/contentscanners/exceptionvirussiteiplist'
urllist = 'name=exceptionvirus,path=/etc/e2guardian/lists/contentscanners/exceptionvirusurllist'

regexpreplacelist = 'name=searchterms,path=/etc/e2guardian/lists/searchregexplist'

searchlist = 'name=banned,path=/etc/e2guardian/lists/bannedsearchlist'
searchlist = 'name=override,path=/etc/e2guardian/lists/bannedsearchoveridelist'

sitelist = 'name=bannedbypass,messageno=500,path=/etc/e2guardian/lists/bannedsitelistwithbypass'
ipsitelist = 'name=bannedbypass,messageno=500,path=/etc/e2guardian/lists/bannedsiteiplistwithbypass'
urllist = 'name=bannedbypass,messageno=501,path=/etc/e2guardian/lists/bannedurllistwithbypass'

sitelist = 'name=nocheckcert,path=/etc/e2guardian/lists/nocheckcertsitelist'
ipsitelist = 'name=nocheckcert,path=/etc/e2guardian/lists/nocheckcertsiteiplist'


Ryan Stewart

unread,
May 29, 2018, 4:38:11 PM5/29/18
to e2guardian
I don't know much about raspberry pi's. I do most testing and developing in vm's via vagrant and virtualbox. Do you know if there's a handy pi vagrant box or virtualbox-compatible image?

FredB

unread,
May 30, 2018, 4:10:24 AM5/30/18
to e2guardian

>
> I don't know much about raspberry pi's. I do most testing and
> developing in vm's via vagrant and virtualbox. Do you know if
> there's a handy pi vagrant box or virtualbox-compatible image?
>
>

I guess, but I'm not sure, that vbox is only x86 compatible

Ryan Stewart

unread,
May 30, 2018, 4:29:49 PM5/30/18
to e2guardian
Hmm, true. No ARM support, which looks like what pi is based on.

FredB

unread,
May 31, 2018, 6:34:27 AM5/31/18
to e2guardian
Yes the limit is 245 in config file (exactly 250 e2 process) no matter your ulimit, systemctl value
I saw some messages about this, it's seems not related with e2guardian.

I don't know if we can or not remove this limit, I guess the question should be posed here https://www.raspberrypi.org/forums/

wil...@gmail.com

unread,
May 31, 2018, 8:57:11 AM5/31/18
to e2guardian
FredB,

You lost me here, I don't understand where you got the 245/250 and from what config file?

FredB

unread,
May 31, 2018, 9:14:04 AM5/31/18
to e2guardian
245 in e2guardian = 250 real process/threads = OK it works
246 in e2guardian = 251 = Ko segfault

wil...@gmail.com

unread,
May 31, 2018, 9:40:02 AM5/31/18
to e2guardian
Guys,

Might have stumbled onto something useful. Was reading how linux limits the max thread counts based on a computation of the available memory in relation to stack size times the number of threads. It seems that a large stack size means less threads on lower memory systems irregardless of what you set for thread counts and limits. My stack size was 8192kb, and it allowed me to run 200 http_workers, but not 300. I changed my stack size to 1024kb via adding to systemd [Service] section the following:

LimitSTACK=1048576

Now i've successfully restarted e2guardian with up to 800 http_workers successfully! Sadly I'm at work and not at home to test browsing and functionality to see if this hurts anything currently, but I'm hopeful, fingers crossed anyway... If it doesn't hurt anything it'd be interesting to know how low the stack size can be set without hurting anything as it'll help determine the upper limit of http_workers on the pi it seems ???

FredB

unread,
May 31, 2018, 1:29:49 PM5/31/18
to wil...@gmail.com, e2guardian
Very interesting, I have a stress test script, if it doesn't hurt anything I will push the rasp to this limit and find a good value.

FredB

unread,
Jun 1, 2018, 5:31:05 AM6/1/18
to e2guardian
It works with httpworkers 1900 when this limit is reached e2guardian crashes (v5.1), I will investigate why

With 1900 busy workers, there is not enough horsepower and I have network problems, we reach some limitations

Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw867:  retry 4 to connect to numsys.eu
Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw866:  retry 4 to connect to www.google.fr
Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw922:  retry 4 to connect to www.google.fr
Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw908:  retry 3 to connect to linuxfr.org
Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw883:  retry 1 to connect to www.google.fr
Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw923:  retry 4 to connect to www.google.fr
Jun  1 09:29:14 raspberrypi e2guardian[3889]: hw914:  retry 3 to connect to support.google.com

wil...@gmail.com

unread,
Jun 1, 2018, 8:09:11 AM6/1/18
to e2guardian
FredB,

Wow, excellent job;) I'm really excited to no longer have a stringent <300 workers limit on the pi! Keep me posted on anything useful you find, I wouldn't mind tweaking it more if possible. My network seemed to operate at home last night perfectly fine with the reduced stack size, I'm curious if it can be reduced further yet without affecting things and allowing more threads, though I'm not sure I actually need 2000+, lol. Thanks again for your assistance!

FredB

unread,
Jun 3, 2018, 12:53:34 PM6/3/18
to e2gua...@googlegroups.com

> FredB,
>
> Wow, excellent job;) I'm really excited to no longer have a stringent <300 workers limit on the pi!

What's happen for you now when the limit is reached ? A crash ?

wil...@gmail.com

unread,
Jun 5, 2018, 12:29:40 PM6/5/18
to e2guardian
FredB,

It's the same, if I try to start the service with too many threads, i.e. 2000, I get no errors, just won't run just like when I was stuck with 200 threads. I think I'm OK with where we've got it to for now. My only question at this point would be, what can the stack size be decreased to without harming e2's functionality I wonder? If I can make it smaller than 1024 then in theory I can push more threads out of it if desired....

Reply all
Reply to author
Forward
0 new messages