prunsrv.exe

[Iyad Odeh]

hello

prunsrv.exe consumes after awhile 100% of the cpu usage.

this affects the server and every thing goes down

what is the problem

[Jinhee]

prunsrv.exe is for starting NxFilter as a Windows service. What's your version of NxFilter? Did you overwrite prunsrv.exe when you date NxFilter?

[Iyad Odeh]

yes i have the latest nxfilter and when i update i overwrite every thing.

now I downloaded 

http://mirror.apache-kr.org//commons/daemon/binaries/windows/commons-daemon-1.0.15-bin-windows.zip

and i have overwritten the existing prunsrv.exe in nxfilter

i seems to work untill now

[Jinhee]

If it works that's good but I think we have the same one in the latest package.

[Iyad Odeh]

you are right. i still have the same problem

i have windows 2003 server and only nxfilter is installed on it

[Jinhee]

Do you see anything special on /log/nxfilter.log while you have 100% CPU load? Or can you show me your log file when you have that problem?

Alternatively you could use Linux on VM instead of your Windows 2003 server.

[Iyad Odeh]

some of the logfile

 INFO [10-26 09:41:06] - DomainDic.set_queue_full, We don't do domain existence checking for 1 minute!

ERROR [10-26 09:41:06] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:06] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:06] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:07] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:07] - RequestQueue.add: java.lang.IllegalStateException: Queue full

 INFO [10-26 09:41:01] - Writing logs, log_cnt = 32, signal_cnt = 0, flow_cnt = 0, recv_flow = 0.

ERROR [10-26 09:40:26] - DomainDic._find: Local domain bypass. - file.pass.ps

ERROR [10-26 09:40:25] - DomainDic._find: Local domain bypass. - _ldap._tcp.default-first-site-name._sites.dc._msdcs.pssdcom

 INFO [10-26 09:40:20] - blocked by admin, superyou.zapto.org

 INFO [10-26 09:41:08] - blocked by admin, dns.msftncsi.com

ERROR [10-26 09:40:19] - DomainDic._find: Local domain bypass. - _ldap._tcp.default-first-site-name._sites.dc._msdcs.pssd.com

ERROR [10-26 09:40:14] - LoginListener._deal_login, Couldn't find the user 10.0.0.75, 6dd95dbe0bdb3b70df75622107a520ac

ERROR [10-26 09:40:13] - Zone trasfer, java.net.SocketTimeoutException, pssd.com, 10.0.0.1

ERROR [10-26 09:40:13] - DomainDic._find: Local domain bypass. - file.pssd.com

ERROR [10-26 09:40:11] - DomainDic._find: Local domain bypass. - wpad.pssd.com

ERROR [10-26 09:40:11] - DomainDic._find: Local domain bypass. - file.pssd.com

ERROR [10-26 09:40:11] - DomainDic._find: Local domain bypass. - _ldap._tcp.default-first-site-name._sites.dc._msdcs.pssd.com

ERROR [10-26 09:41:07] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:08] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - DomainDic._find: Local domain bypass. - wpad.pass.ps

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:09] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

 INFO [10-26 09:41:10] - blocked by admin, superyou.zapto.org

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

ERROR [10-26 09:41:10] - RequestQueue.add: java.lang.IllegalStateException: Queue full

[Jinhee]

Queue full means it bypasses DNS queries as it can't handle all the queries already in its process queue. This is mostly caused by Internet connection lost. On Windows server I guess Java doesn't get the connection once it lost it. If you have this kind problem consistently you'd better change your OS. Maybe Ubuntu Linux on VMware. It works well.

However I don't know how it's related to your 100% CPU load problem. I am not saying that it's not related. Just don't know as I couldn't recreate your problem on my side.

[Jinhee]

Which upstream DNS server do you use? From Internet or is it on your local network? Actually your server lost connection to its upstream server. So it doesn't get hangs to get the response from your upstream server and its queue gets full and it drops qeuries from your clients. But it didn't get the connectino restored.

[Iyad Odeh]

I am using opendns ip's 208.67.222.222, 208.67.220.220

if the problem still i will try changing the OS

Thanks

[Jinhee]

Yeah, you could do that. And in my previous post there's typo.

  So it doesn't get hangs to get the response from your upstream server

    -> So it hangs to get the response from your upstream server

[greattohe...@gmail.com]

Hi Jahastech, we seem to have this problem as well on a regular bases (although I believe it's related to updating (adding domains) to the Custom Category), so we would like to change over OS as well. However, how do I go about transferring all the Windows NxFilter settings to Linux? Can I just copy over the Windows NxFilter folder to Linux?

[Jahastech]

We use our own Windows service wrapper for Java at the moment. You can try to update it to the latest one if it's about prunsrv.exe.

I don't know why you think it's related to your custom category. I think it shouldn't be but I don't know all the conditions you have.

Yes. You can copy whole directory into Linux. Or install a new one and copy your config backup.

[greattohe...@gmail.com]

Thanks for your quick reply Jahastech. The last time (on Feb 8) that I updated our Custom Category it went down very quickly after and noticed that prunsrv was running at 99%. I just updated it again and saw prunsrv spike to 99% again for about 10-15 seconds, but then recovered. I will keep an aye on it, just to make sure. We are running version 4.2.1, I assume the wrapper is included in that version, otherwise I will update to 4.2.2. Thanks for the Linux info, I'll be working on that.

[Jahastech]

Maybe you are updating it using our zip package. In that case, you should run,

  unstsvc.bat

  instsvc.bat

And try allocate more memory.

  http://nxfilter.org/tutorial.html#performance-tuning

It could be from disk swapping.

[greattohe...@gmail.com]

Yes, I update through zip file. However, I stop the NxFilter service (in Windows Services app), copy the unzipped files to C:\NxFilter and after copy I start the NxFilter service. I basically follow the instructions on the tutorial page. Only instead of using the command lines 'net stop NxFilter' and 'net start NxFilter' I go directly into the Services app to stop and start.

So, it is actually better to run unstsvc.bat to uninstall the service, copy the zipped files, and then run instsvc.bat to re-install the service?

We have 3GB memory allocated of which 1.11GB is actively used. The system only runs NxFilter, should be enough, right?

[Jahastech]

That is for a normal update. Read this,

  https://groups.google.com/forum/?fromgroups=#!searchin/nxfilter200/wrapper%7Csort:date/nxfilter200/aBU8acs5HUw/QGx6TdgOAQAJ

To increase memory allocation, you need to edit startup.bat file. 2048 would be better for you. Read that performance tuning part again.

[greattohe...@gmail.com]

I finally had some time to look at your link and that makes things a lot more clear. I now doubled the default 768mb to 1536mb and I now also understand the reason to run unstsvc.bat and instsvc.bat that to install the newer version of the service :-)