Using GLOBAL_INDEX as SEQUENCE_NUMBER to prevent Rollbacks

[Herr Manns]

Hi!

We experimented with the following Table-Definition:

CREATE TABLE "DB2INST1"."DOMAIN_EVENT_ENTRY"  (
    "GLOBAL_INDEX" BIGINT NOT NULL GENERATED ALWAYS AS IDENTITY (  
      START WITH +1  
      INCREMENT BY +1  
      MINVALUE +1  
      MAXVALUE +9223372036854775807  
      NO CYCLE  
      CACHE 20  
      NO ORDER ) , 
    "EVENT_IDENTIFIER" VARCHAR(255 OCTETS) NOT NULL , 
    "META_DATA" BLOB(10000) LOGGED NOT COMPACT , 
    "PAYLOAD" BLOB(10000) LOGGED NOT COMPACT NOT NULL , 
    "PAYLOAD_REVISION" VARCHAR(255 OCTETS) , 
    "PAYLOAD_TYPE" VARCHAR(255 OCTETS) NOT NULL , 
    "TIME_STAMP" VARCHAR(255 OCTETS) NOT NULL , 
    "AGGREGATE_IDENTIFIER" VARCHAR(255 OCTETS) NOT NULL , 
    "SEQUENCE_NUMBER" BIGINT NOT NULL GENERATED ALWAYS AS (GLOBAL_INDEX) , 
    "TYPE" VARCHAR(255 OCTETS) )   
   IN "USERSPACE1"  
   ORGANIZE BY ROW; 

So we would like to simply let the database generate the sequence number for us.
We know that the business logic has to be able to deal with this. But for us, this is the case.

Everything works fine until we enable Snapshotting: Afterwards, it can happen that events are
contained in the snapshot and then be applied once more on top of it.

So it seems like the snapshot gets sorted into the wrong position of the event stream.
Or the snapshot is on the right position but has too many events contained.

We are using the JpaEventStorageEngine with a slightly modified DomainEventEntry:

@Entity
@Table(name = "DOMAIN_EVENT_ENTRY")
@AttributeOverride(name = "sequenceNumber", column = @Column(name = "sequenceNumber", insertable = false, updatable =  false))
public class NonStrictDomainEventEntry extends AbstractSequencedDomainEventEntry<byte[]> {

This way the sequenceNumber isn't written to the store anymore, so that the database can generate
it on their own.

Any idea why the snapshot seems to be misordered in the event stream?

Thanks in advance,
Dirk

[Allard Buijze]

Hi Dirk,

the snapshotting mechanism uses the sequence number to find out where it stopped including events in the snapshot. When multiple threads are inserting data, it is possible that the commit order and insert order doesn't match anymore. This results in 1, 2, and 4 being readable, but 3 maybe not (as it wasn't committed yet). A snapshot will claim it is up to date with nr 4.

When changing the reliable no-gap sequence number in this way, you will also need to ensure that the snapshotter can cope with this, either by registering the gaps (what the TrackingEventProcessor does), or by stopping processing when a gap was detected.

Hope this helps.

Cheers,

Allard

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Herr Manns]

Hi Allard,

thanks for your hints! That would mean that the snapshot would miss some of the events, right?
So over time, we should have more and more events that are lost in the snapshot.

But what we see is the opposite: Over time we have more and more events that are applied as duplicates.
That is why we are thinking the snapshots gets sorted into the wrong position in the event stream.
Could it be that the JPA Caching is returning entities that have the old sequence number and not the
generated sequence number from the database?

Normally this should happen only if the JPA second level cache and query cache is activated and that
shouldn't be the default case when using axon, right?

cu,
Dirk

[Allard Buijze]

Hi Dirk,

valid point, having duplicates is weird. This would mean that sequence numbers of entities change over time. Is that possible with your approach? A snapshot just contains the sequence number of the last event included. When opening a stream, it takes the last snapshot, and then loads all the events that have a sequence number higher than that.

Do you do snapshotting asynchronously (did you configure an Executor on the snapshotter)? If not, it is possible that you're using some form of cache in the EntityManager, instead of raw data from the database. It wouldn't be the first time something like this happens.

Cheers,

Allard

[Herr Manns]

Hi Allard,

we are using the SpringAggregateSnapshotterFactoryBean:

@Bean
public SpringAggregateSnapshotterFactoryBean springAggregateSnapshotterFactoryBean() {

   return new SpringAggregateSnapshotterFactoryBean();
}

But there seems to be no possibility to set the executor on the factory.
What is the expected way to change the executor?

cu,
Dirk

[Herr Manns]

I did a quick try and simply copied the SpringAggregateSnapshotterFactoryBean and extended it with a constructor to pass the executor.
Now my bean looks like:

@Bean
public FactoryBean<SpringAggregateSnapshotter> springAggregateSnapshotterFactoryBean() {

   Executor executor = Executors.newCachedThreadPool();

   return new MySpringAggregateSnapshotterFactoryBean(executor);
}

The problem seems to be really vanished now! Thanks a million, Allard!
I'm not very familiar with hibernate caching, but does that mean the first level cache of hibernate does return cached entities for
the snapshots when running in the same transaction?

cu,
Dirk

[Allard Buijze]

Hi Dirk,

yes, withing a single transaction, Hibernate is pretty aggressive in caching. You can do EntityManager.clear() to clear this cache. It hit me in the face many times in tests....

Cheers,

Allard

--