Tracking Processor fails after application restart. Token owner becomes blank.

[Ajinkya Virkar]

Hello, 

I am noticing a weird behavior, where everytime I restart the application the token owner becomes null. And after that no event is processed until I stop the application, delete the entry and then restart the application. 

1. My application resides in Pivotal Cloud Foundry (PCF).

2. Token DB is MySQL.

In the logs emmitted say that "Could not claim the token hence releasing it". 

[Steven van Beelen]

Hi Ajinkya,

Is the log also stating why it couldn't claim the token?

Or put different, do you have a stack trace with some more info?

Not sure which issue you're encountering, hence why I'm looking for more information.

Cheers,

Steven

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ajinkya Virkar]

Hello Steven,

Is the log also stating why it couldn't claim the token? 

• When you restart the application, it is trying to claim the token using the JpaTokenStore.loadOrCreateToken(). During this call I see that its trying to read the BLOB from SQL and that's where I get the following exception.
• I think because of this exception the JPATokenSotre.releaseClaim() is called and it updates the owner to null. 
  
• Further research shows that the cause might be class MongoTrackingToken. It seems to be missing the default constructor. 

•  [TrackingEventProcessor - eventreplayer-0] WARN  o.a.e.TrackingEventProcessor - Error occurred. Starting retry mode.

  org.axonframework.serialization.SerializationException: Error while deserializing object

  at org.axonframework.serialization.json.JacksonSerializer.deserialize(JacksonSerializer.java:204)

  at org.axonframework.eventhandling.tokenstore.AbstractTokenEntry.getToken(AbstractTokenEntry.java:114)

  at org.axonframework.eventhandling.tokenstore.jpa.JpaTokenStore.fetchToken(JpaTokenStore.java:100)

  at org.axonframework.eventhandling.TrackingEventProcessor.lambda$ensureEventStreamOpened$7(TrackingEventProcessor.java:297)

  at org.axonframework.eventhandling.TrackingEventProcessor$$Lambda$161/1276887870.get(Unknown Source)

  at org.axonframework.common.transaction.TransactionManager.fetchInTransaction(TransactionManager.java:67)

  at org.axonframework.eventhandling.TrackingEventProcessor.ensureEventStreamOpened(TrackingEventProcessor.java:296)

  at org.axonframework.eventhandling.TrackingEventProcessor.processingLoop(TrackingEventProcessor.java:216)

  at org.axonframework.eventhandling.TrackingEventProcessor.lambda$start$3(TrackingEventProcessor.java:187)

  at org.axonframework.eventhandling.TrackingEventProcessor$$Lambda$160/996713740.run(Unknown Source)

  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

  at java.util.concurrent.FutureTask.run(FutureTask.java:266)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at java.lang.Thread.run(Thread.java:745)

  Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.axonframework.mongo.eventsourcing.eventstore.MongoTrackingToken: no suitable constructor found, can not deserialize from Object value (missing default constructor or creator, or perhaps need to add/enable type information?)

   at [Source: [B@2a745ea8; line: 1, column: 2]

  at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)

  at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1456)

  at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1012)

  at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1206)

  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)

  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)

  at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1626)

  at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1237)

  at org.axonframework.serialization.json.JacksonSerializer.deserialize(JacksonSerializer.java:202)

  ... 14 common frames omitted

  10:33:32.281 [TrackingEventProcessor - eventreplayer-0] WARN  o.a.e.TrackingEventProcessor - Releasing claim on token and preparing for retry in 1s
  
  
  
• 
  

Or put different, do you have a stack trace with some more info? - Added above. 

[Allard Buijze]

Hi,

the actual problem here, is Jackson’s inability to deserialize the MongoTrackingToken. Because it can’t deserialize it, it will go into error mode. Resetting helps, because Axon prevents requiring to read the token as much as possible.

The solution is to configure an XStream based serializer for the TokenStore.

Hope this helps.
Cheers,

Allard

--

Allard Buijze
CTO

E: allard...@axoniq.io
T: +31 6 34 73 99 89

[Ajinkya Virkar]

Hello Allard, 

I will implement the Xstream seralizer and test it out. But when I debugged the code I think it's some bug with MongoTrackingToken.java. As Jackson is able to seralizer JSON's correctly in all other scenario's. 

[Allard Buijze]

Hi Ajinkya,

last thing I'd ever recommend is changing from Jackson to XStream based serializers for the Event Store ;-). My suggestion is only to use XStream for the TokenStore. You can do this by defining a bean in your context that returns a JpaTokenStore. In the constructor, you can explicitly pass a "new XStreamSerializer". No other components needs to read from the token store, so it's perfectly fine to use a separate serializer tor this.

In Axon 3.2, you will be able to define a default as well as a message serializer. The default could be an XStream based serializer, while the Message Serializer (used for events, commands and queries) would be better Jackson based.

Hope this clarifies it.

Cheers,

Allard

Op wo 7 feb. 2018 om 20:38 schreef Ajinkya Virkar <ngage....@gmail.com>:

[Ajinkya Virkar]

Thanks Allard, 

Sorry I misunderstood your suggestion. I implemented the change and the replay is working successfully.

Just a question Axon 3.1 has Mongo Support for storing tokens correct.

Secondly when is 3.2 getting released. I wanted plan our tech debt based on your release schedule. 

[Allard Buijze]

Hi,

yes, 3.1 has a MongoTokenStore. However, the MongoTrackingToken apparently isn't compatible with the Jackson serializer. We recommend using XStream as the general purpose serializer, and configure Jackson (or another one) as EventSerializer.

We expect (i.e. hope) to release 3.2 around the end of the month.

Cheers,

Allard

Op do 8 feb. 2018 om 15:38 schreef Ajinkya Virkar <ngage....@gmail.com>: