Concurrency bug with time zones

[Mindaugas Žakšauskas]

Hi,

I think I've been bitten by a concurrency bug in ebean (version 12.16.1).

The essence is that we get a lot of stacktraces which look like this:

javax.persistence.PersistenceException: Error loading on models.Activity.lastModified
at io.ebeaninternal.server.querydefn.DefaultOrmQuery.handleLoadError(DefaultOrmQuery.java:2008)
at io.ebeaninternal.server.query.CQuery.handleLoadError(CQuery.java:754)
at io.ebeaninternal.server.query.SqlBeanLoad.load(SqlBeanLoad.java:66)
at io.ebeaninternal.server.deploy.BeanProperty.load(BeanProperty.java:514)
at io.ebeaninternal.server.query.SqlTreeLoadBean$Load.loadProperties(SqlTreeLoadBean.java:213)
at io.ebeaninternal.server.query.SqlTreeLoadBean$Load.initialise(SqlTreeLoadBean.java:327)
at io.ebeaninternal.server.query.SqlTreeLoadBean$Load.perform(SqlTreeLoadBean.java:335)
at io.ebeaninternal.server.query.SqlTreeLoadBean.load(SqlTreeLoadBean.java:358)
at io.ebeaninternal.server.query.SqlTreeLoadRoot.load(SqlTreeLoadRoot.java:26)
at io.ebeaninternal.server.query.CQuery.readNextBean(CQuery.java:409)
at io.ebeaninternal.server.query.CQuery.hasNext(CQuery.java:489)
at io.ebeaninternal.server.query.CQuery.readCollection(CQuery.java:518)
at io.ebeaninternal.server.query.CQueryEngine.findMany(CQueryEngine.java:356)
at io.ebeaninternal.server.query.DefaultOrmQueryEngine.findMany(DefaultOrmQueryEngine.java:131)
at io.ebeaninternal.server.core.OrmQueryRequest.findList(OrmQueryRequest.java:404)
at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1463)
at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1442)
at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1483)
<...nonessential lines omitted...>
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.persistence.PersistenceException: Error readSet on models.Activity.lastModified
at io.ebeaninternal.server.deploy.BeanProperty.readSet(BeanProperty.java:539)
at io.ebeaninternal.server.deploy.BeanProperty.readSet(BeanProperty.java:548)
at io.ebeaninternal.server.query.SqlBeanLoad.load(SqlBeanLoad.java:63)
... 81 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 645 out of bounds for length 13
at java.base/sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:457)
at java.base/java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2394)
at java.base/java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2815)
at java.base/java.util.Calendar.updateTime(Calendar.java:3428)
at java.base/java.util.Calendar.getTimeInMillis(Calendar.java:1812)
at java.base/java.util.Calendar.getTime(Calendar.java:1785)
at org.mariadb.jdbc.internal.com.read.resultset.rowprotocol.TextRowProtocol.getInternalTimestamp(TextRowProtocol.java:846)
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.getTimestamp(SelectResultSet.java:1112)
at com.zaxxer.hikari.pool.HikariProxyResultSet.getTimestamp(HikariProxyResultSet.java)
at io.ebeaninternal.server.type.RsetDataReader.getTimestamp(RsetDataReader.java:171)
at io.ebeaninternal.server.type.ScalarTypeUtilDate$TimestampType.read(ScalarTypeUtilDate.java:44)
at io.ebeaninternal.server.type.ScalarTypeUtilDate$TimestampType.read(ScalarTypeUtilDate.java:21)
at io.ebeaninternal.server.deploy.BeanProperty.readSet(BeanProperty.java:533)
... 83 common frames omitted

Similar stacktraces come from all over the place so it's not an isolated place in one model.

Here's what I think is going on:

* The property which is loaded is of TIMESTAMP type in MySQL (we use AWS RDS) DB
* When loading, ebean tries to do the time zone calc in TextRowProtocol.java:846
* It is using `synchronized (calendar) {` in that block, and the reference is passed on from RsetDataReader.java:171. There, it's retrieved from `Calendar cal = dataTimeZone.getTimeZone()`
* The only possible reason for the exception I can think of is coming from the fact that the same `calendar` is used by other threads, without proper synchronization. The class is not thread safe on its own.

I don't think I will be able to provide an isolated test case, due to concurrency.
My question is, would it be possible to get a workaround in this situation without changing the ebean code / version?

The `dataTimeZone` seems to be originated from InternalConfiguration class, where I think I get the MySqlDataTimeZone variety.
As all of our servers are using UTC, we don't really need the tz calc at all (I think), and using NoDataTimeZone could work for us.
It's just that I haven't seen a possibility to use that explicitly via config or otherwise. Any tips, or perhaps better ideas?

Many thanks in advance!

Regards,
Mindaugas

[Rob Bygrave]

If we look at latest version of InternalConfiguration at: https://github.com/ebean-orm/ebean/blob/master/ebean-core/src/main/java/io/ebeaninternal/server/core/InternalConfiguration.java#L451-L462

... we need to set a timezone to bypass the default setup for MariaDB and MySQL.

So with 12.x I think you are looking to set dateTimeZone in ServerConfig. With latest we set dateTimeZone in DatabaseConfig - https://github.com/ebean-orm/ebean/blob/master/ebean-api/src/main/java/io/ebean/config/DatabaseConfig.java#L266

So try setting that to UTC.

BUT ... it looks like you are using MariaDB and not MySQL?  ... and the code I'm looking at would use NoDateTimeZone for MariaDB?  ... so you want to check that with version 12.16.1 

To me it looks like the JDBC driver is mutating the calendar that it's getting. I'd check that you are on the latest version of the JDBC driver that you can get to.

Cheers, Rob.

[Rob Bygrave]

So re-reading this, it looks like the app is using MySQL database and that stack trace is showing a MariaDB JDBC driver (and yes similar but not the same). I'd be checking that JDBC driver and looking to change over to a MySQL JDBC driver if that is the case.

[Mindaugas Žakšauskas]

Hi,

Many thanks for your response.

We are using MySQL db with MariaDB driver for performance/devops reasons.
Tried using MySQL and Amazon drivers, neither had worked for us.

But I think you're right, the problem most likely happens in DB driver, not Ebean.

I'll see if I can somehow force the NoDataTimeZone upon Ebean to avoid the broken code path.

Thanks again.

Regards,
Mindaugas

[Rob Bygrave]

> force the NoDataTimeZone upon Ebean

My quick look at the code suggested that might not be possible - without a change it looked like we end up with SimpleDateTimeZone which might fix the issue.  If you get stuck there and are stuck on 12.x (Java 8, can't move to Java 11) there is a 12.x branch called maintain-v12 and so there is that possibility of putting a change into that branch and releasing a patch to 12.x. [given that I think we are looking for a way to use NoDataTimeZone here]

Cheers, Rob.

[Mindaugas Žakšauskas]

I've tried setting the time zone via System.setProperty which has resulted SimpleDataTimeZone being used. Unfortunately, this class also contains an internal Calendar instance which is then shared through the same code path.

I was able to use the reflection to force NoDataTimeZone unto io.ebeaninternal.server.core.DefaultServer.dataTimeZone. Ugly but works, subject to QA.

Any alternative ideas welcome :-)

Regards,

Mindaugas

--

---
You received this message because you are subscribed to a topic in the Google Groups "Ebean ORM" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ebean/rEjbcYE6K84/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ebean+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/CAC%3Dts-Eo%2B%2BBXb7iovhaFnOi-kKyRxM0HwfawN2ORFSo%3DKtyzUQ%40mail.gmail.com.

[Rob Bygrave]

> I was able to use the reflection to force NoDataTimeZone unto io.ebeaninternal.server.core.DefaultServer.dataTimeZone. Ugly but works, subject to QA.

Excellent news.

> Any alternative ideas welcome :-)

So I think we are looking at (12.x branch) the following method: 

https://github.com/ebean-orm/ebean/blob/maintain-v12/ebean-core/src/main/java/io/ebeaninternal/server/core/InternalConfiguration.java#L446-L459

My thought is to add in something like:

    if ("NoDataTimeZone".equals()) {

      return new NoDataTimeZone();

    }

... so use a magic timezone value of "NoDataTimeZone" ... and that will give us a new NoDataTimeZone() implementation;

and in this way we don't impact anyone else / change the existing logic.

So that method implementation would end up like:

    String tz = config.getDataTimeZone();

    // add check for magic tz value first

    if ("NoDataTimeZone".equals()) {

      return new NoDataTimeZone();

    }

    // continue the existing logic
    if (tz == null) {
      if (isMySql(getPlatform())) {
        return new MySqlDataTimeZone();
      }
      return new NoDataTimeZone();
    }

    if (getPlatform().base() == Platform.ORACLE) {
      return new OracleDataTimeZone(tz);
    } else {
      return new SimpleDataTimeZone(tz);
    }

You received this message because you are subscribed to the Google Groups "Ebean ORM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/CAKTEH80ocmaVyEDFCAnb%3DaV5b%3DK-bNHskFMXOoSnQNs33F%3DV4A%40mail.gmail.com.

[Rob Bygrave]

See https://github.com/ebean-orm/ebean/issues/3393

Released as 12.16.3, used "NoTimeZone"

if ("NoTimeZone".equals(tz)) {
return new NoDataTimeZone();
}

[Mindaugas Žakšauskas]

Amazing news! Many thanks again for looking into this.

Really appreciated.

Regards,

Mindaugas

To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/1122f256-4e2b-4488-b4b3-55a6d5c4ef3en%40googlegroups.com.