db file getting bigger, nxfilter hang

[Muhammad Nur Hidayat]

Dear all,

I have upgrade nxfilter from vesion 1.4.6 to version 1.6.1 and 2.1.0.
Reason for upgrade because sometimes nxfilter cannot respond or just hang. And I cannot stopped it using bin/shutdown.sh
Then I move nxfilter to another server, perhaps better to split squid and nxfilter.
After moving and upgrade in new server, using top, I can't belive i see cpu usage for java is very high, 354%. Perhaps this is the problem.
After upgrade, nothing changes, in the morning nxfilter started, and at noon it stop responding/hang.
When I move nxfilter files to another server for upgrade, I see files in db folder is very big. Especially traffic.h2.db

-rw-r--r--  1 root root 414M May 30 16:38 traffic.h2.db
-rw-r--r--  1 root root 3.3G May 22 11:35 traffic.h2.db.bak

Then I rename the file to .bak, and restart nxfilter.
Voilla, it works great.

So, what is this traffic db file really, and what should I do, log rotate it ?
Why should I keep it ? Please help so I don't have to monitor the file size.

Thank you,

[Jinhee]

traffic.h2.db file is the DB file for logging and reporting.
When you delete it it creates a new one.
So you delete that bak file.
You already have a new one.

To keep it small adjust 'Config > Misc > Log retention days'.
At default it keeps the log data for 60 days.
But if you don't need it just keep it for 5 days.

Jinhee

[Muhammad Nur Hidayat]

What if reporting is separated to another web/process. Like squid cache, there are log analyzers like SARG or lightsquid.
So nxfilter process not loaded by reporting.

[Jinhee]

No, all in one box is better. Otherwise it takes too much effort to maintain it. And I don't think it's too big. There are 2 choices when you design a reporting system. Saving disk space or better performance. I went with saving disk space as I don't know who would use it with what kind of hardware. Plus NxFilter's reporting is somewhat higher standard even if you compare it to a commercial one. It's not that special this kind of reporting requires several gigs of disk space. And inside the main thread doesn't get affected by the size of the DB. Even if you have a huge reporting DB it performs well. So if you have slowness of the performance it's a different issue. Maybe you had a disk full or disk related issue.

[Jinhee]

Anyway what I am saying is that NxFilter is a multi-threaded program. And its main thread is separated from reporting or logging thread.