Single or multiple data storage for Events?

[Psycho Punch]

Hello everyone.

I'm fairly new to CQRS/event sourcing and I'm struggling a bit with many different implementation details. One such detail is about the structure of event storage. While all the events in the system share many similar properties, at the core they contain different details pertaining to a particular occurrence or state transition. This means that essentially each event is modeled differently from the others. Now, I'm quite confused as to how to go about storing such data. My original idea is to define an Event class with timestamp, id, etc. and to represent the actual event details a JSON formatted string. However, I felt as if representing the event data as a blob of JSON-formatted string can make the processing code cumbersome.

Now, what I'm considering is to model events in such a way that it's easy to keep track of them chronologically, while keeping the details in their own buckets so to speak. There will be an Event class that holds timestamp and some essential metada about the event, but ultimately the details will be kept separate. Something like the following (Java code):

class Event {

    private String id;
    private Instant timestamp;
    private String type; //mostly a string representation of the actual event

    private String eventDetailsId;

}

class OrderCreated {

    private String id;

    private String orderId;

    private List<LineItem> lineItems;

}

These are simplified models to convey the general idea. I'm not sure if it's necessary to create a specific value objects to represent the id's, but yeah, this is how I think the model will be. Essentially Event objects are just pointers to actual event details. When rebuilding the query database for example, the first step is to get all the Event objects then one by one process them by retrieving the details from specific buckets. However, I'm not so sure if anyone has tried this approach and found some critical hurdles that I should be aware of this early. How are others approaching this problem?

[Ben Kloosterman]

What your doing is pretty standard , though most implimentations use a separate stream for each  type.  

Have a look at Simple CQRS C# impl its pretty easy to read.

Ben

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Yves Lorphelin]

Is this for the storage of events ? Have you looked at existing implementations/product ?
Building you own abstraction for storage / retrieving requires a lot of work and there are lot's of goodies you won't have.

anway to have some kind of idea for a SQL based storage you could look at https://github.com/SQLStreamStore/

[Kasey Speakman]

I think lots of people have tried this to good effect. Here's the version I am using.

The core table is an event log. Mine looks like this in Postgres.

CREATE TABLE eventlog

(

    sequenceid bigserial NOT NULL,

    streamid uuid NOT NULL,

    eventname text,

    eventmeta jsonb NOT NULL,

    eventdata jsonb NOT NULL,

    logdate timestamp with time zone NOT NULL DEFAULT now(),

    CONSTRAINT pk_sequenceid PRIMARY KEY (sequenceid)

)
TABLESPACE pg_default;

CREATE INDEX ix_eventlog_streamid

    ON eventlog USING btree

    (streamid)

    TABLESPACE pg_default;

Sequence ID is used for ordering.

EventName is the class name, and the code has a mapping (the only reflection code I use) between the class name and a deserializer for it.

EventMeta is a json-serialized dictionary with tracing information like correlation ID, user ID, etc.

The index on Stream ID is for loading all events for a particular stream.

It is common practice from there to create other tables to further categorize the information, like a Stream Type table (to be able to query all streams of a certain type) or a Snapshot table (to save replay time). I have not needed to do either of those things yet, but maybe I will at some point.

[Renato Cavalcanti]

I will back Yves on his point about not building your own abstraction for storage. 

There are a couple of them out there doing exactly what you describe and they solved already a bunch of problems you main encounter if you do it yourself: like snapshotting, serialization, concurrency, replay of events and projections.

Since you are using Java, I invite to you to have a look on how akka-persistence works. 

This doesn't mean you should use akka-persistence directly, there a some good abstractions around it for you, I can point you to Lagom (java and scala api) and Fun.CQRS (scala only). This libraries/frameworks do the hard lifting for you so you can concentrate on domain modelling and leave the dirty bits of persistence out of your way.

Full disclosure: I'm involved with both libraries (as author and/or contributor)