change serializer from XML to json

[jan.ga...@holisticon.de]

We started our app without any additional serializer setup, so we got XML.

What can we do, when we later decide that JSON is the better choice?

Anyway to migrate the existing event history?

[Koen Verwimp]

Does anyone have a good solution for this? 

Op maandag 23 juli 2018 11:49:10 UTC+2 schreef jan.ga...@holisticon.de:

[Steven van Beelen]

Hi Jan, Koen,

In scenarios like this, it sounds like you'll have to write a migration tool which does the trick for your.

So a single purpose application which reads all entries from your 'xml-serialized-event-store' and writes all those entries to a 'json-serialized-event-store'.

This shouldn't be to hard to write, but understandably not the must fun thing to do.

I have had to do this a couple of times too. From my point of view it wasn't hard to develop, it's mostly a pain for operations.

That's my 2 cents.

Cheers,

Steven

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steven Grimm]

I've just started looking at this exact problem for my company's Axon code base. Still in the prototyping/experimentation stage, but I'll describe what I'm planning to do.

We can't afford to have our application offline for long enough to do a stop-the-world migration of the existing events; we need to do the migration while the application is running with as close to zero downtime as possible. We're still using Axon 2 at the moment, but the same general principles, I think, should apply in Axon 3 as well.

I'm going to say "event store" instead of "event store and snapshot event store and saga store" but you'll almost certainly want to do all of them so you don't have XML-based sagas and JSON-based events in the same application. The same approach applies to all of them.

1. Make sure all your event classes serialize and deserialize correctly with Jackson or whatever JSON serializer you're going to use. XStream can handle some things that Jackson can't.

2. Create an event store class that wraps other underlying event stores. This is pretty straightforward but requires looking through some of Axon's internal APIs. For any write operation, it needs to perform that operation on all the underlying stores, and for read operations, you need to be able to configure which underlying store it uses.

3. Add a new table to hold the JSON-serialized payloads. We're using PostgreSQL, so we create this with payload and metadata columns of type JSONB so we can use PostgreSQL's JSON features for ad-hoc queries against the event store (which is the main point of the exercise for us).

4. Create a new event store class that writes to the newly-added table. Configure it with a Jackson serializer, and then configure the wrapper store from step 1 to wrap both this new event store and the original XML-based one.

5. Like Steven van Beelen says, write a migration utility that goes through the old XML-based event table and, for any event that doesn't already exist in the new table, deserializes the XML, serializes the object as JSON, and inserts the equivalent row in the new table. (Metadata too.)

6. Deploy the code from steps 1-3 and configure the wrapper event store to read from the XML event table. At this point you will be writing two copies of each event, but only the XML version will be read.

7. Run the migration utility from step 4. Other than additional database load, this shouldn't have any impact on the running application since nothing is reading the JSON data yet.

8. Once you're satisfied the events have been migrated correctly, configure the wrapper event store to start reading JSON instead.

9. Once you're satisfied that you won't want to roll back to use the XML-based events, delete the wrapper event store and the XML-based event store, and let Axon directly use the JSON event store.

10. Drop the XML table.

That's the plan. A bit complicated but so far I don't see a reason it won't work.

-Steve

[Allard Buijze]

Something that you could try instead of changing old events, is to create a serializer that delegates to another serializer depending on the first character found in the data. If it's a "<", you could delegate to the XStream one, otherwise delegate to the Jackson one. When serializing, delegate to Jackson, if that's where you want to go.

In the meantime, you could, if you want, migrate events from one format to the other. Once converted, you can change the serializer to the Jackson serializer.

Cheers,

Allard

--

You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Allard Buijze
CTO

E: allard...@axoniq.io
T: +31 6 34 73 99 89

[Steven Grimm]

That's a good idea (and much, much simpler than what I wrote!) if you want to keep the data in the same column type. In our case, part of the motivation for switching to JSON is that we can use the PostgreSQL JSONB type, which is not only more compact on disk than XML-in-BYTEA but will also let us use the database's JSON operators for ad-hoc SQL queries of the event store. With payloads in binary columns you can sort of get there by calling conversion functions in the queries, but it's awkward and much slower.

-Steve


Allard Buijze wrote on 8/3/18 6:13 AM:

You received this message because you are subscribed to a topic in the Google Groups "Axon Framework Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/axonframework/VpB3tvJpcBY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to axonframewor...@googlegroups.com.

[tomhier]

Hi Steve,

I'm trying to do the same, save the event payload as JSON in postgres in the JSONB-type.
How did you manage this?
With a JDBCEventStore this is easier, as you're able to define you own EventSchema.
With JPA I don't really see how I can manage this. The docs show just a very small part of it and I'm not really seeing how to glue it all.

Could you give me some pointers? Or some example config?

Thnx in advance!

Tom

Op dinsdag 31 juli 2018 01:31:38 UTC+2 schreef Steven Grimm:

[Steven Grimm]

Sadly, I can't give you any pointers on JPA; we're using the JDBC event store.

-Steve

[Koen Verwimp]

"You deserializes the XML, serializes the object as JSON, and ..."

Correct me if i'm wrong, but you also have to do upcasting in between ... as the deserializing only works if the XML matches the class structure?

[Steven Grimm]

Yes, that's correct -- should have included that in my writeup. You will need to do whatever upcasting your application would normally do when loading old events from the XML-based event store.

-Steve



Koen Verwimp wrote on 8/12/18 1:45 AM:

"You deserializes the XML, serializes the object as JSON, and ..."

Correct me if i'm wrong, but you also have to do upcasting in between ... as the deserializing only works if the XML matches the class structure?