Event Store stopped running with this Exception

Steven Blair

unread,

Jul 14, 2016, 11:16:24 AM7/14/16

to Event Store

Just upgraded to 3.8 and it has been running for the last hour or so, then all of sudden I get an error window and then the Event Store process is terminated.

In the log file, I have the following:

[PID:07468:010 2016.07.14 13:33:14.381 ERROR StorageScavenger ] Failed to write the $scavengeIndexInitialized event to the $scavenges stream. Reason: WrongExpectedVersion

[PID:07468:019 2016.07.14 13:35:11.299 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionMessage+CommittedEventDistributed in queued handler 'Projection Core #0'.

System.ArgumentException: complete TF position required

Parameter name: committedEvent

at EventStore.Projections.Core.Services.Processing.EventByTypeIndexPositionTagger.IsMessageAfterCheckpointTag(CheckpointTag previous, CommittedEventDistributed committedEvent) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EventByTypeIndexPositionTagger.cs:line 36

at EventStore.Projections.Core.Services.Processing.ReaderSubscriptionBase.ProcessOne(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\ReaderSubscriptionBase.cs:line 95

at EventStore.Projections.Core.Services.Processing.HeadingEventReader.DistributeMessage(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\HeadingEventReader.cs:line 204

at EventStore.Projections.Core.Services.Processing.HeadingEventReader.Handle(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\HeadingEventReader.cs:line 97

at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EventReaderCoreService.cs:line 157

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122

Any idea what could have caused this?

Pieter Germishuys

unread,

Jul 14, 2016, 11:23:07 AM7/14/16

to Event Store

That should not be causing the Event Store process to terminate.

Is this a fairly large database?

Does it happen consistently on your database?

What other projections other than the system ones are you running?

If you have sensitive information that you cannot share, you are more than welcome to contact me on pieter at geteventstore dot com

Steven Blair

unread,

Jul 14, 2016, 11:33:32 AM7/14/16

to Event Store

Peter,

Thanks for the quick reply

I am trying to restart the Event Store just now to get that information for you, but it's taking a long time.

messages similar to this:

[02468,04,15:32:55.863] Verifying hash for TFChunk 'C:\Users\Administrator\Desktop\Event Store\db\chunk-000000.000002'...

Anyway, I would say we have somewhere around 500,000 - 1,000,000 events (a bit vague I know)

This is the first time we have seen this issue.

We run one other projection, which checks specific category and writes them to a new stream for today's date.

We have the by category system projection on as well.

Pieter Germishuys

unread,

Jul 14, 2016, 11:36:37 AM7/14/16

to Event Store

If you just want to gather data you can skip verification which would decrease the startup time. Run Event Store with --skip-db-verify

Greg Young

unread,

Jul 14, 2016, 11:36:37 AM7/14/16

to event...@googlegroups.com

That is verifying chunks. You can stop that with --skip-db-verify but
with only 1m events it should be quick.

> --
> You received this message because you are subscribed to the Google Groups
> "Event Store" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to event-store...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Studying for the Turing test

Steven Blair

unread,

Jul 14, 2016, 11:46:34 AM7/14/16

to Event Store

I will try the skip, but it's still doing these checks.

That's 30 minutes now.

Greg Young

unread,

Jul 14, 2016, 11:47:45 AM7/14/16

to event...@googlegroups.com

for a few million events what is this running on?

Steven Blair

unread,

Jul 14, 2016, 11:57:01 AM7/14/16

to Event Store

A very low end server in the cloud:

RAM: 1.7GB

CPU: E5-2650 @ 2.00GHz

Greg Young

unread,

Jul 14, 2016, 11:57:47 AM7/14/16

to event...@googlegroups.com

Ah yes you definitely want --skip-db-verify then

On Thu, Jul 14, 2016 at 6:57 PM, 'Steven Blair' via Event Store

Steven Blair

unread,

Jul 14, 2016, 12:01:03 PM7/14/16

to Event Store

It's on now, and starts within a few seconds!

Greg Young

unread,

Jul 14, 2016, 12:01:54 PM7/14/16

to event...@googlegroups.com

When you have db verify it will hash all your data that you have. The
slow part is likely more so your storage on a-cheap-cloud-server.

On Thu, Jul 14, 2016 at 7:01 PM, 'Steven Blair' via Event Store

Steven Blair

unread,

Jul 15, 2016, 10:48:23 AM7/15/16

to Event Store

Hi,

Unfortunately this has happened another two times today:

The last error:

[PID:08348:022 2016.07.15 12:48:23.963 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionMessage+CommittedEventDistributed in queued handler 'Projection Core #0'.