Loading a large ZAP session

[TW]

I have a 16GB session file that I would like to load into ZAP (ZAP had no problem creating the session).  Whenever I try, however, I see the following error in "zap.log":

.
.
.
2016-04-28 12:02:59,581 [Thread-10] INFO  ENGINE - dataFileCache open start
2016-04-28 12:02:59,582 [Thread-10] FATAL ENGINE - dataFileCache open failed
org.hsqldb.HsqlException: wrong database file version: requires large database support
.
.
.

Interestingly, I'm able to load a 14GB session file just fine.  Perhaps I'm hitting a celiing at 16GB?  However, in looking over HyperSQL project's FAQ, it seems like hsqldb shouldn't have any problems (http://hsqldb.org/web/hsqlFAQ.html):

"The current size limit of an HSQLDB database is 8 TB for all CACHED tables and 256GB for each TEXT table. In addition, maximum totall lob size is 64TB. If you use large MEMORY tables, memory is only limited by the allocated JVM memory, which can be several GB on modern machines and 64bit operating systems. We have performed extensive tests with the latest versions using the TestCacheSize and other test programs inserting millions of rows and resulting in data files of up to 16 GB and larger LOB sizes. Users have reported the use of databases with up top 900 million rows."

Thanks in advance for any thoughts on how I might be able to resolve this issue.

(You may be wondering how I ended up with 16 and 14GB session files....  Although I make liberal use of ZAP contexts and URI regular expressions to include/exclude scan targets, the system being scanned is enormous.)

[Simon Bennetts]

Which version of ZAP are you using?
I'm pretty sure the latest version should be able to support files of that size.

Cheers,

Simon

[TW]

I'm using version 2.4.3.  If it makes any difference, I'm using ZAP via a fully patched Kali Linux ("Rolling Edition") deployment.  Also, specs for the underlying workstation are beefy:  16 cores, 32GB of RAM...plenty of resource to go around.  Finally, I start ZAP with -Xmx16g and -Xms8g.

Thanks!

[thc...@gmail.com]

In the file <session_name>.properties which version does it say?

And in <session_name>.script what value does "SET FILES SCALE" have?

Best regards.

> <http://hsqldb.org/web/hsqlFAQ.html>):

>
> "The current size limit of an HSQLDB database is 8 TB for all
> CACHED tables and 256GB for each TEXT table. In addition,
> maximum totall lob size is 64TB. If you use large MEMORY tables,
> memory is only limited by the allocated JVM memory, which can be
> several GB on modern machines and 64bit operating systems. We
> have performed extensive tests with the latest versions using
> the TestCacheSize and other test programs inserting millions of
> rows and resulting in data files of up to 16 GB and larger LOB
> sizes. Users have reported the use of databases with up top 900
> million rows."
>
> Thanks in advance for any thoughts on how I might be able to
> resolve this issue.
>
> (You may be wondering how I ended up with 16 and 14GB session
> files.... Although I make liberal use of ZAP contexts and URI
> regular expressions to include/exclude scan targets, the system
> being scanned is enormous.)
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[TW]

Thanks; here is the information:

<session_name>.properties --> version=2.2.9

<session_name>.script --> SET FILES SCALE 8

> <mailto:zaproxy-users+unsub...@googlegroups.com>.

[thc...@gmail.com]

Try changing "SET FILES SCALE" value to 16 or 32. [1]

(Worth back up the session files before doing that.)

[1] http://hsqldb.org/doc/guide/management-chapt.html#N14DCE

Best regards.

> > <mailto:zaproxy-user...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

> <mailto:zaproxy-user...@googlegroups.com>.

[TW]

Thanks for the suggestion!

Unfortunately, the circumstance hasn't changed, although the error messages have.  Here is a sanitized excerpt (I see the same thing whether "SET FILES SCALE" is 16 or 32)--I'm guessing that changing the database file scale has thrown off some indexing...?

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
.
.
.
2016-04-29 10:29:52,903 [AWT-EventQueue-1] INFO  MenuFileControl - opening session file Scan.session
2016-04-29 10:29:52,907 [AWT-EventQueue-1] INFO  Control - Open Session
2016-04-29 10:29:52,913 [Thread-13] ERROR ParosDatabase - java.sql.SQLNonTransientConnectionException: connection exception: closed
org.parosproxy.paros.db.DatabaseException: java.sql.SQLNonTransientConnectionException: connection exception: closed
    at org.parosproxy.paros.db.paros.ParosTableHistory.deleteTemporary(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabase.close(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabase.close(Unknown Source)
    at org.parosproxy.paros.model.Session.open(Unknown Source)
    at org.parosproxy.paros.model.Session$1.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLNonTransientConnectionException: connection exception: closed
    at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
    at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source)
    ... 6 more
Caused by: org.hsqldb.HsqlException: connection exception: closed
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.Session.execute(Unknown Source)
    ... 8 more
2016-04-29 10:29:52,934 [Thread-13] INFO  ENGINE - dataFileCache open start
2016-04-29 10:29:52,934 [Thread-13] INFO  ENGINE - dataFileCache open end
2016-04-29 10:29:52,942 [Thread-13] WARN  ENGINE - failed to read a byte array
java.io.EOFException
    at org.hsqldb.persist.ScaledRAFile.read(Unknown Source)
    at org.hsqldb.persist.ScaledRAFile.readInt(Unknown Source)
    at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
    at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
    at org.hsqldb.persist.DataFileCache.get(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
    at org.hsqldb.Table.setIndexRoots(Unknown Source)
    at org.hsqldb.Table.setIndexRoots(Unknown Source)
    at org.hsqldb.StatementCommand.getResult(Unknown Source)
    at org.hsqldb.StatementCommand.execute(Unknown Source)
    at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
    at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
    at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
    at org.hsqldb.persist.Log.processScript(Unknown Source)
    at org.hsqldb.persist.Log.open(Unknown Source)
    at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
    at org.hsqldb.Database.reopen(Unknown Source)
    at org.hsqldb.Database.open(Unknown Source)
    at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
    at org.hsqldb.DatabaseManager.newSession(Unknown Source)
    at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
    at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
    at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at org.parosproxy.paros.db.paros.ParosDatabaseServer.start(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabaseServer.<init>(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabase.open(Unknown Source)
    at org.parosproxy.paros.model.Session.open(Unknown Source)
    at org.parosproxy.paros.model.Session$1.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
2016-04-29 10:29:52,942 [Thread-13] FATAL ENGINE - readObject 1150306291
java.io.EOFException
    at org.hsqldb.persist.ScaledRAFile.read(Unknown Source)
    at org.hsqldb.persist.ScaledRAFile.readInt(Unknown Source)
    at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
    at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
    at org.hsqldb.persist.DataFileCache.get(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
    at org.hsqldb.Table.setIndexRoots(Unknown Source)
    at org.hsqldb.Table.setIndexRoots(Unknown Source)
    at org.hsqldb.StatementCommand.getResult(Unknown Source)
    at org.hsqldb.StatementCommand.execute(Unknown Source)
    at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
    at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
    at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
    at org.hsqldb.persist.Log.processScript(Unknown Source)
    at org.hsqldb.persist.Log.open(Unknown Source)
    at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
    at org.hsqldb.Database.reopen(Unknown Source)
    at org.hsqldb.Database.open(Unknown Source)
    at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
    at org.hsqldb.DatabaseManager.newSession(Unknown Source)
    at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
    at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
    at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at org.parosproxy.paros.db.paros.ParosDatabaseServer.start(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabaseServer.<init>(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabase.open(Unknown Source)
    at org.parosproxy.paros.model.Session.open(Unknown Source)
    at org.parosproxy.paros.model.Session$1.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
2016-04-29 10:29:52,942 [Thread-13] FATAL ENGINE - Scan.session.data getFromFile 1150306291
org.hsqldb.HsqlException: java.io.EOFException
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.persist.DataFileCache.readObject(Unknown Source)
    at org.hsqldb.persist.DataFileCache.getFromFile(Unknown Source)
    at org.hsqldb.persist.DataFileCache.get(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.get(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.setAccessor(Unknown Source)
    at org.hsqldb.Table.setIndexRoots(Unknown Source)
    at org.hsqldb.Table.setIndexRoots(Unknown Source)
    at org.hsqldb.StatementCommand.getResult(Unknown Source)
    at org.hsqldb.StatementCommand.execute(Unknown Source)
    at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
    at org.hsqldb.scriptio.ScriptReaderText.readDDL(Unknown Source)
    at org.hsqldb.scriptio.ScriptReaderBase.readAll(Unknown Source)
    at org.hsqldb.persist.Log.processScript(Unknown Source)
    at org.hsqldb.persist.Log.open(Unknown Source)
    at org.hsqldb.persist.Logger.openPersistence(Unknown Source)
    at org.hsqldb.Database.reopen(Unknown Source)
    at org.hsqldb.Database.open(Unknown Source)
    at org.hsqldb.DatabaseManager.getDatabase(Unknown Source)
    at org.hsqldb.DatabaseManager.newSession(Unknown Source)
    at org.hsqldb.jdbc.JDBCConnection.<init>(Unknown Source)
    at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
    at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at org.parosproxy.paros.db.paros.ParosDatabaseServer.start(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabaseServer.<init>(Unknown Source)
    at org.parosproxy.paros.db.paros.ParosDatabase.open(Unknown Source)
    at org.parosproxy.paros.model.Session.open(Unknown Source)
    at org.parosproxy.paros.model.Session$1.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
    at org.hsqldb.persist.ScaledRAFile.read(Unknown Source)
    at org.hsqldb.persist.ScaledRAFile.readInt(Unknown Source)
    ... 30 more
2016-04-29 10:29:52,943 [Thread-13] WARN  ENGINE - java.io.EOFException in statement [SET TABLE PUBLIC.HISTORY INDEX '1150306291 203679636 989930599 1040512368 0 0 0 0 273411']
.
.
.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

>     > <mailto:zaproxy-users+unsub...@googlegroups.com>.

>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

> <mailto:zaproxy-users+unsub...@googlegroups.com>.

[TW]

I _may_ have a fix...

Looks like upgrading ZAP's hsqldb.jar to the latest iteration (2.3.3) of HyperSQL is working.  I'll post a follow-up in 3.5 hours (the time it will take to load the 16GB session file).  Of course, this approach presumes there won't be any weird compatibility issues with ZAP 2.4.3 and HyperSQL 2.3.3...

[TW]

Not sure what's happening, now...  The session file loaded very quickly:  perhaps in half the time.  Everything appeared to be working just fine, and, as far as I could tell by clicking about, (most of?) the session data were there.  So, I closed out the session and loaded it again.  Interestingly, it loaded in about 2 minutes.  Kinda seems too good to be true!

In looking over the session file, it has mysteriously grown from 16 to 17GB (keep in mind, all I did was load the session, close it out, then reload it).

So, I'm experimenting around with the original 16 and 14GB session files.  I'll post another update if I can get a fix on what's happening.

On Thursday, April 28, 2016 at 4:15:47 PM UTC-4, TW wrote:

[TW]

Okay, I now believe the 16GB session file that appears to load in a couple of minutes is corrupt.  I suspect this may be why the file increases by a gigabyte after it has been opened by ZAP.  Also, the 14GB session file takes 2.5 - 3 hours to load and never increases in size as a result of being loaded.  With the 16GB file, it looks like only seven lines of history are available in the History tab (although all the session information and Sites tab nodes are there).  Finally, even though I can get the 16GB session file open very quickly, a good 30 minutes or so seems to be required to close out this session and exit ZAP (around the same amount of time is required to close out the 14GB session).

So, my current hypothesis is:

* The 16GB file is bad

* The newest version of HyperSQL is able to digest the bad file, and may be padding the file out (by 1GB) as a side effect of the file's state

It's a bummer about losing that 16GB session, but life goes on.

Many thanks to the ZAP team for their work on this tool, though!  I find it extremely valuable!