EventStore metadata? Where and how to use?

[Werner Clausen]

Hi,

 

In our own EventStore design we have come to metadata. Information like EventType, UserId etc. is needed to query for events. Why? Because we need this in historical views - and we need this to replay events from a certain context or for a certain aggregate. We might have millions of records, so it is not an option to load all events into memory on demand and select from there in C#.

 

Where should I store these data? In the event table or another table? How did you solve this? E.g. if metadata is stored in another table (which it should be in my mind), do you save the metadata in the same transaction or outside the transaction that stores the events? I guess the common approach is that these metadata points to data in the eventstore (by id) that can then be selected fast. Any pointers on how you did it can help us!

 

--
Werner

[James Nugent]

If you're using Event Store (geteventstore.com) you could have projections which repartition into streams based on the things you mention?

[@yreynhout]

Why not have the meta data persisted next to the event? e.g.

EventRecord { CausationId(IX), CausationOrdinal, CorrelationId(IX), StreamId (PK), StreamRevision(PK), EventId, EventData (blob), EventMetaData (blob), UtcStoreTime }

Then have some process read each event record and index the EventMetaData for ease of querying.

[Hajdrik Attila]

From your description you like to persist 2 "kind" of data which has 1:1 relationship into 2 different places. I agree with Yves that in this case this data should be part of your event data.

- Attila

[@yreynhout]

Meta data is associated with each event. Storing it in a separate table to satisfy some normalization wet dream is the least of my concerns. 

[Marijn Huizendveld]

On Jan 16, 2013, at 5:53 PM, @yreynhout wrote:

Meta data is associated with each event. Storing it in a separate table to satisfy some normalization wet dream is the least of my concerns. 

+1, AFAIK there is no value whatsoever associated with moving event metadata in another table. I'd say it's really easy to add different schema to support querying for events matching metadata conditions. 

[Greg Young]

The event store stores metadata with the event im same transaction. Probably the only real way of handling this.

In terms of querying what the es does is allow you to repartition streams off of metadata. Ex if you wanted to have a stream that returned all events based on the "user" attribute of your metadata you would say....

FromAll().whenAny(function(s,e){linkTo(e.metadata.username, e))})

Then if you went to /streams/{username} you would see all the events done by that user. This is generally how repartitions work In the es (indexing) and sounds like what you want to be doing.

Cheers,

Greg

P.s. streams generally want metadata too not just events

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Werner Clausen]

 

I have downloaded and currently investigating your eventstore. I'm very interested in those projections and the possibilities they give. What exactly happens when you "repartition" streams? Is it "just" a definition of a query? I.e. when you call that url, is it reading the eventstore file? Or does the repartition mean that you now have a new physical file? My thougts here circles around the "you should never query using anything other than ID from an eventstore". Are we deviating from this now?

 

Thanks.

 

--
Werner

[Werner Clausen]

@yreynhout, attilah and Marijn:

 

Thanks for your answers. What you say makes sense - but, this means that you have to query the eventstore. What happened to the "you should never query using anything other than ID"?

 

--
Werner

[Werner Clausen]

Hi James,

 

My thougts were actually on our current implementation (which is homegrown using MS Sql Server). And I'm struggling on how to select events from a table with 10 million rows.

 

I have only known about your ES the last 2 days - reading about it like a wild :). I'm trying to grasp what "projections" are and how they work. The next phase is to actually write some code that inserts events and create and use some projections. I will get back in the ES forum! If I can solve the transactional hubba (dtc) then it could be the solution.

 

--
Werner

[Kijana Woodard]

Hi Werner.
My, very weak, understanding is that you wouldn't query es per se, but you would observe the stream/s and build read models which could then be persisted in a doc db or search engine for queries.

[@yreynhout]

You don't get it.

[@yreynhout]

What he said (almost).

[James Nugent]

You should probably consider carefully whether you really need DTC or not, but there's a *very rough* version of something to make it work with ES here https://gist.github.com/1de2dc287e880fd7f16c

Note that it is by no means production quality or even close to it!!

[Werner Clausen]

 

Awsome reply. Please enligthen me so that I have a chance of getting it...?

 

--
Werner

On Saturday, 19 January 2013 19:55:59 UTC+1, @yreynhout wrote:

You don't get it.