Slow Projection Reading All Events

[Tyler Gregory]

Hi all,

We created a projection reading from all; it discards system events and events of type Result, and then links what's left to a stream. This projection is going at a rate of around 1k events per second. I've been able to get much higher throughput reading from $all using a multithreaded client reading from different positions in the stream simultaneously, but single threaded reads have similar performance to the projection.

We're running a three node cluster on Azure VMs: 4 cores/16GB RAM, 1TB premium SSD disk for EventStore. CPU and disk (IO queues, average await times, IOPS and disk throughput) are all quite low while the projection is building, and while our client reads are going on. 

Our database is around 500GB, so chewing through the events at 1k events/sec is going to take quite some time. I've tried increasing various combinations of WorkerThreads, ReaderThreads, ProjectionThreads, and ChunksCacheSize, but none seems to make an appreciable difference to the performance of the projection. Is there something else I've missed or could have misconfigured, or should we be using something other than the projection system for this?

Thanks,

Tyler

[Tyler Gregory]

Here's the projection code:

fromAll().

   when({

        $any : function(s,e) {

            if (e.eventType && !e.eventType.startsWith('$') && e.eventType != 'Result')

                linkTo('reporting', e);

   }});

[Steven Blair]

Tyler,

Just out of curiosity, could you not just ditch this proejction, and have the system proejctions create categories for you?

The you could read from those streams (concurrently)

Without knowing anything about the events and streams it's hard to say for certain, but it looks like you are trying to replicate something the eventstore will be doing for you already.