Initialization of MySQL database taking exceptionally long

Amedee Van Gasse

unread,

Nov 20, 2020, 4:02:36 AM11/20/20

to Dependency Check

I'm trying to create a MySQL database based on (https://jeremylong.github.io/DependencyCheck/data/database.html) and using 5.3.1 of the maven plugin. I expect it to take a while to create the database but it's running excessively long.

<build>

<groupId>org.owasp</groupId>

<artifactId>dependency-check-maven</artifactId>

<groupId>mysql</groupId>

<artifactId>mysql-connector-java</artifactId>

</dependency>

</dependencies>

<databaseDriverName>com.mysql.cj.jdbc.Driver</databaseDriverName>

</configuration>

<goals>

<goal>update-only</goal>

</goals>

</execution>

</executions>

</plugin>

</plugins>

</build>

mvn -Dmaven.repo.local=.repository org.owasp:dependency-check-maven:update-only -DconnectionString=jdbc:mysql://dependency-check.****.eu-central-1.rds.amazonaws.com:3306/dependency_check_531 -DdatabaseUser=**** -DdatabasePassword=****

After running for 5 hours, someone restarted Jenkins to update plugins, which interrupted the job.

00:00:12.441 [INFO] Checking for updates

00:00:19.913 [INFO] NVD CVE requires several updates; this could take a couple of minutes.

00:00:19.913 [INFO] Download Started for NVD CVE - 2008

00:00:19.913 [INFO] Download Started for NVD CVE - 2010

00:00:20.260 [INFO] Download Complete for NVD CVE - 2010 (929 ms)

00:00:20.260 [INFO] Download Started for NVD CVE - 2011

00:00:20.260 [INFO] Processing Started for NVD CVE - 2010

00:00:20.260 [INFO] Download Complete for NVD CVE - 2008 (1125 ms)

00:00:20.260 [INFO] Download Started for NVD CVE - 2012

00:00:20.260 [INFO] Processing Started for NVD CVE - 2008

00:00:21.459 [INFO] Download Complete for NVD CVE - 2011 (996 ms)

00:00:21.459 [INFO] Download Complete for NVD CVE - 2012 (995 ms)

00:00:22.667 [INFO] Download Started for NVD CVE - 2013

00:00:22.667 [INFO] Processing Started for NVD CVE - 2012

00:00:22.667 [INFO] Download Started for NVD CVE - 2014

00:00:22.667 [INFO] Processing Started for NVD CVE - 2011

00:00:23.410 [INFO] Download Complete for NVD CVE - 2013 (1054 ms)

00:00:23.756 [INFO] Download Complete for NVD CVE - 2014 (1060 ms)

00:00:25.486 [INFO] Download Started for NVD CVE - 2015

00:00:25.833 [INFO] Download Started for NVD CVE - 2016

00:00:26.577 [INFO] Download Complete for NVD CVE - 2015 (1011 ms)

00:00:26.924 [INFO] Download Complete for NVD CVE - 2016 (1056 ms)

00:00:50.371 [INFO] Download Started for NVD CVE - 2017

00:00:50.371 [INFO] Download Complete for NVD CVE - 2017 (1407 ms)

00:00:53.452 [INFO] Download Started for NVD CVE - 2018

00:00:54.652 [INFO] Download Complete for NVD CVE - 2018 (1349 ms)

00:01:28.306 [INFO] Download Started for NVD CVE - 2019

00:01:28.306 [INFO] Download Started for NVD CVE - 2020

00:01:28.306 [INFO] Download Complete for NVD CVE - 2019 (1145 ms)

00:01:28.306 [INFO] Download Complete for NVD CVE - 2020 (981 ms)

05:01:54.450 Sending interrupt signal to process

05:01:58.069 Sending interrupt signal to process

05:02:00.435 [INFO] Shutdown hook activated. Shutdown was not called. Shutting down JCS.

05:02:00.435 [INFO] Element event queue destroyed: org.apache.commons.jcs.engine.control.event.ElementEventQueue@e197c2

05:02:00.435 [INFO] In DISPOSE, [NODEAUDIT] fromRemote [false]

05:02:00.435 [INFO] In DISPOSE, [NODEAUDIT] auxiliary [NODEAUDIT]

05:02:00.435 [INFO] In DISPOSE, [NODEAUDIT] put 0 into auxiliary NODEAUDIT

05:02:00.435 [INFO] No longer waiting for event queue to finish: Pooled Cache Event Queue

This is the current content of the database:

$ mysql --defaults-group-suffix=dependencycheck -D dependency_check_531 -e "SHOW TABLE STATUS;"

+---------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+----------------+---------+

+---------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+----------------+---------+

| cpeEntry | InnoDB | 10 | Dynamic | 86236 | 79 | 6832128 | 0 | 12615680 | 4194304 | 92220 | 2020-11-19 09:52:39 | 2020-11-19 23:39:25 | NULL | utf8mb4_0900_ai_ci | NULL | | |

| cweEntry | InnoDB | 10 | Dynamic | 38693 | 68 | 2637824 | 0 | 1589248 | 4194304 | NULL | 2020-11-19 09:52:39 | 2020-11-20 00:58:01 | NULL | utf8mb4_0900_ai_ci | NULL | | |

| properties | InnoDB | 10 | Dynamic | 8 | 2048 | 16384 | 0 | 0 | 0 | NULL | 2020-11-19 09:52:39 | NULL | NULL | utf8mb4_0900_ai_ci | NULL | | |

| reference | InnoDB | 10 | Dynamic | 260032 | 143 | 37289984 | 0 | 6832128 | 6291456 | NULL | 2020-11-19 09:52:39 | 2020-11-20 00:58:30 | NULL | utf8mb4_0900_ai_ci | NULL | | |

| software | InnoDB | 10 | Dynamic | 358075 | 45 | 16269312 | 0 | 19939328 | 6291456 | NULL | 2020-11-19 09:52:39 | 2020-11-20 00:58:30 | NULL | utf8mb4_0900_ai_ci | NULL | | |

| vulnerability | InnoDB | 10 | Dynamic | 37137 | 381 | 14172160 | 0 | 3178496 | 4194304 | 38612 | 2020-11-19 09:52:39 | 2020-11-20 00:58:01 | NULL | utf8mb4_0900_ai_ci | NULL | | |

+---------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+----------------+---------+

How long is the first update of the database expected to take?

Is more than 5 hours expected?

mark

unread,

Nov 20, 2020, 1:15:54 PM11/20/20

to dependen...@googlegroups.com

On 11/20/20 10:02 AM, Amedee Van Gasse wrote:
> I'm trying to create a MySQL database based on
> (https://jeremylong.github.io/DependencyCheck/data/database.html) and
> using 5.3.1 of the maven plugin. I expect it to take a while to
> create the database but it's running excessively long.

please try/use a current/supported release such as 6.0.3, many
improvements were made

Hans Aikema

unread,

Nov 20, 2020, 1:22:50 PM11/20/20

to Amedee Van Gasse, Dependency Check

Amedee,

Initial seeding can take a lot of time indeed for hosted databases compared to the initial load for a local h2 database. See e.g Jeremys remark (5-24 hours was not uncommon to be reported in earlier feedback)

https://github.com/jeremylong/DependencyCheck/issues/2755#issuecomment-706086839

An item specifically dealing with MySQL seems to suggest to expect at least around 6 hours for loading (https://github.com/jeremylong/DependencyCheck/issues/1134)

At the same time... it also depends significantly on the network latency between your database and the system that runs the initial load.

As you can see in that thread for an Oracle DB connected on a sub 1ms latency from the system running dependency-check (a docker container running Oracle Express edition running on a Linux VM (docker-machine) on the same host) I managed to load the DB database loaded in relatively reasonable time (though still significantly slower than a local h2 database took to initialize)

The timings for the Oracle DB were with the 6.x version of DependencyCheck.

Any reason for still seeding a database for the 5.x version? Because for 6.x you have to start all over again as the database hase been updated in a non-upgradable way. And the updated schema results in a significant improvement in the time taken for updates for MySQL

https://github.com/jeremylong/DependencyCheck/commit/6a6d21b83e29d54ee12c5cf4be6d1e5a389a7c09#diff-23975a7378873c27620d7ccee40b72b8

With the obfuscated 'amazonws.com' address from your log: if dependency-check runs on your local system expect indeed significantly larger load times than what I saw for OracleDB (more than 5 hrs is not uncommon based on earlier feedback) as I don't expect sub-ms RTT from your system to amazoncloud. During initial load there is a massive amount of serialized roundtrips to the database.

kind regards,

Hans Aikema

On 20 Nov 2020, at 10:02, Amedee Van Gasse <amedee....@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Dependency Check" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dependency-check/5dcbb892-2eb5-4226-9cbe-a0ce2c87a35bn%40googlegroups.com.

Amedee Van Gasse

unread,

Nov 20, 2020, 1:59:33 PM11/20/20

to Dependency Check

That is on the backlog, but for now it stays 5.3.1.

Anyway, the above pom.xml is on a branch named "update/5.3.1", and there are also branches "update/5.3.2", "update/6.0.0", "update/6.0.1", "update/6.0.2", "update/6.0.3".

These are all the databases on the instance:

$ mysql --defaults-group-suffix=dependencycheck -e "SHOW DATABASES;"

+----------------------+

| Database |

+----------------------+

| dependency_check_531 |

| dependency_check_532 |

| dependency_check_600 |

| dependency_check_601 |

| dependency_check_602 |

| dependency_check_603 |

| information_schema |

| mysql |

| performance_schema |

| sys |

+----------------------+

I'm hesitant to upgrade if I can't get it to work on the current version first.

Another reason to support older versions, is because we have those versions in the pom.xml of released products, so the versions in those branches (tags actually) obviously cannot be changed.

As long as we still support those versions of _our_ products, it makes sense to regularly check those older versions for vulnerabilities.

That being said, the initialization for 6.0.3 has also been running for hours.

Amedee Van Gasse

unread,

Nov 20, 2020, 2:03:25 PM11/20/20

to Dependency Check

Yes, there is a reason for having both 5.x and 6.x databases, see my previous reply.

The database is an RDS instance in the same VPC as the host that runs the seeding.

The database is a db.m5.large, the host is a t3a.xlarge.

Hans Aikema

unread,

Nov 20, 2020, 2:20:06 PM11/20/20

to Amedee Van Gasse, Dependency Check

On 20 Nov 2020, at 19:59, Amedee Van Gasse <amedee....@gmail.com> wrote:

Another reason to support older versions, is because we have those versions in the pom.xml of released products, so the versions in those branches (tags actually) obviously cannot be changed.
As long as we still support those versions of _our_ products, it makes sense to regularly check those older versions for vulnerabilities.

What we use as a strategy for dependency-check is to have the version in he pom.xml als a property that we then override on the build-server with a settings.xml property.

That allows us to upgrade the version of dependency check used diring new builds as long as there is no breaking plugin configuration change.

On a breaking plugin configuration change we would introduce a new property for that version onwards and over time migrate the poms to the new configuration and property.

Not sure if that would fit your build automation scenarios, but wanted to share it in any case.

Another tool you might want to evaluate for such scenarios (monitoring for vulnerabilities in dependencies post-build/release) would be DependencyTrack. It can monitor your vulnerabilities appearing over time without the need for a periodic build from your tag as it does not evaluate the dependencies at build-time, but works from the software composition definition (bill-of-materials) stored in its database. (It's available to us in my company, but I haven't found time to incorporate it in our SDLC yet; colleagues from other teams already started adopting it)

https://dependencytrack.org/

kind regards,

Hans

Amedee Van Gasse

unread,

Nov 20, 2020, 4:25:59 PM11/20/20

to Dependency Check

Thanks for the suggestion about the property, I'll add it to the backlog. Maybe some day, later. :)