overhead of storing event

[Fredrik Skeel Løkke]

Hi

We're trying to do some back-of-the-envelope calculations on how much storage space we need. How much overhead is there in storing an event in eventstore?

Thanks.

[Greg Young]

Index is 20 bytes over head aside from that mostly depends on :

Length of stream name, size of data, size of metadata.

In total we have about 60 bytes of overhead we use internally for an event including the internal time stamps etc that we store https://github.com/EventStore/EventStore/blob/dev/src/EventStore.Core/TransactionLog/LogRecords/PrepareLogRecord.cs shows what is stored 

Greg

--
You received this message because you are subscribed to the Google Groups "Event Store" group.
To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

[Fredrik Skeel Løkke]

Thank you Greg.

On Wednesday, January 7, 2015 2:32:54 PM UTC+1, Greg Young wrote:

Index is 20 bytes over head aside from that mostly depends on :

Length of stream name, size of data, size of metadata.

In total we have about 60 bytes of overhead we use internally for an event including the internal time stamps etc that we store https://github.com/EventStore/EventStore/blob/dev/src/EventStore.Core/TransactionLog/LogRecords/PrepareLogRecord.cs shows what is stored 

Greg

On Wednesday, January 7, 2015, Fredrik Skeel Løkke <f.lo...@gmail.com> wrote:

Hi

We're trying to do some back-of-the-envelope calculations on how much storage space we need. How much overhead is there in storing an event in eventstore?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Event Store" group.

To unsubscribe from this group and stop receiving emails from it, send an email to event-store+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Greg Young]

The one place to be careful of is what encoding you use on your events/metadata a wrong choice here can double your size.

On Wednesday, January 7, 2015, Fredrik Skeel Løkke <f.lo...@gmail.com> wrote:

Thank you Greg.

On Wednesday, January 7, 2015 2:32:54 PM UTC+1, Greg Young wrote:

Index is 20 bytes over head aside from that mostly depends on :

Length of stream name, size of data, size of metadata.

In total we have about 60 bytes of overhead we use internally for an event including the internal time stamps etc that we store https://github.com/EventStore/EventStore/blob/dev/src/EventStore.Core/TransactionLog/LogRecords/PrepareLogRecord.cs shows what is stored 

Greg

On Wednesday, January 7, 2015, Fredrik Skeel Løkke <f.lo...@gmail.com> wrote:

Hi

We're trying to do some back-of-the-envelope calculations on how much storage space we need. How much overhead is there in storing an event in eventstore?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Event Store" group.
To unsubscribe from this group and stop receiving emails from it, send an email to event-store+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

--
You received this message because you are subscribed to the Google Groups "Event Store" group.

To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fredrik Skeel Løkke]

At the moment we are using protobuf for fast streams and json for slow streams..

[Greg Young]

Should be fine.

What is fast vs slow in terms of messages? Also how long do you hold the messages?

To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fredrik Skeel Løkke]

Fast is max hundred per second for one stream. Time to live is a few hours.. When sharding get's possible we would like to increase this, since the longer data stays around in event store, the more it is of use :) It would greatly simplify our setup if event store was not only used for distribution of these streams, but also permanent storage.. But that would probably also require some sort of log compaction, to make it feasable, either by us or implemented in the event store..

Slow data is maybe one event per minute. With unlimited time to live. 

[James Nugent]

Log compaction is already implemented in the Event Store - if you delete streams or expire events via either max age or max count and then run the scavenge process, you end up with a smaller log (maybe - there are caveats to that).

To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.

[Fredrik Skeel Løkke]

By the way, we are not quite in production yet, but very close..

[Fredrik Skeel Løkke]

Sorry, I'm not expressing my self clearly then. What I mean is compression of stored events.

[Greg Young]

This is actually a pretty simple feature to add. I think we would be open to some discussions especially if your company was willing to help fund it :) eg compress during scavenge.

That said you can basically do this now by running on a compressed file system. As you are not needing high write performance this would work out of the box. To support higher write performance the first option would be needed.

Cheers,

[Fredrik Skeel Løkke]

That's good to know. :)