Reduce session data filesize

[Dimitris Balaouras]

Hi,

I'm experimenting with integrating ZAP into our CI systems and the process requires copying ZAP session files over the wire.
I noticed that the HSQLDB data file grows to a few Gigs and I'm wondering if it's possible to reduce the size of this file?

For example, I tried reclaiming some of the allocated space by deleting certain sites and requests, but the size of that db file remains the same.

Any advice?

Thanks much,
- Dimi

[Simon Bennetts]

At the moment ZAP records _everything_ by default, and minimizing the db size is not a real priority :/
Once the db grows it may well be difficult to reduce the size, but I'm no HSQLDB expert ;) Their docs may help: http://hsqldb.org/web/hsqlDocsFrame.html
A better approach would be to exclude any sites and/or requests you're not interested from 'the proxy'. If they're excluded they shouldnt get recorded in the db.
You can exclude urls via regexes using the core API action 'excludeFromProxy'.
Give that a try and let us know if it helps.

Cheers,

Simon

[Dimitris Balaouras]

Thanks for your prompt response Simon. I'll try to experiment a bit with HSQLDB first then...I'll keep you posted.

Cheers,
- Dimi

[thc...@gmail.com]

Hi.

Did you try enable "Database" > "Compact (on exit)"? [1]
That should ensure that the database files have the minimum size possible.


[1]
https://github.com/zaproxy/zap-core-help/wiki/HelpUiDialogsOptionsDatabase#compact-on-exit

Best regards.

> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Dimitris Balaouras]

Thanks much for the tip.. It didn't really help unfortunately. The DB size remains at 2 GB.

I used a client to inspect the HSQLDB database flle, and the history table seems to have so much information.

I'm wondering if I could delete some entries, for example those with HISTTYPE 14 which seem to occupy the largest part of the DB (https://www.dropbox.com/s/68ncz4j7gzmvnyg/Screenshot%202015-09-04%2017.53.44.png?dl=0).

Simon: why isn't this type of record picked up by ParosTableHistory.deleteTemporary()? According to org.parosproxy.paros.model.HistoryReference: 

    /**
     * A HTTP message sent by the (active) scanner which is set as temporary (i.e. deleted when the session is closed).
     */
    public static final int TYPE_SCANNER_TEMPORARY = 14;

Thanks,

Dimi

> <mailto:zaproxy-users+unsub...@googlegroups.com>.

[thc...@gmail.com]

Hi.

Yeah, I suspected that, if no messages are deleted then there's not much
to compact.

You can safely delete the history records with type 9, 11 and 14.

Please raise an issue. [1]


[1] https://github.com/zaproxy/zaproxy/issues/new

Best regards.

> > <mailto:zaproxy-user...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

> <mailto:zaproxy-user...@googlegroups.com>.

[Dimitris Balaouras]

Hi thc202,

Thanks much for the followup. I’ve opened issue #1875 and created a pull request too; hope it helps.

This is quite important to my team, so I’ll use my fork for now.

Thanks much,

- Dimi

You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-users/yFVhrSHsBHw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zaproxy-user...@googlegroups.com.

[thc...@gmail.com]

Hi.

Thank you for raising the issue!

Best regards.

On 07/09/15 18:49, Dimitris Balaouras wrote:
> Hi thc202,
>
> Thanks much for the followup. I’ve opened issue #1875

> <https://github.com/zaproxy/zaproxy/issues/1875> and created a pull
> request <https://github.com/zaproxy/zaproxy/pull/1876> too; hope it helps.

>>>> an email to zaproxy-user...@googlegroups.com <http://googlegroups.com>