Projection Faulted: An event emitted in recovery differ from the originally emitted event.

[Sebastian Burgstaller]

Hi there,

we are running Event Store in version 3.0.0 final on Windows and experienced the following problem: the VM our Event Store is running on was unexpectedly killed from the outside (VMWare). Now after the server restart we have a projection that is in the faulted state, showing the following error message:

An event emitted in recovery differ from the originally emitted event. Existing('$>', 'C:613381856/P:613381856'). New('$>', 'C:613378130/P:613378130')

I suspect the writing to the checkpoint was interrupted, so the projection finds an invalid position when trying to start. Is there an easy way we can recover from this situation, or do I have to reset the projection?

In the case of a reset I would have to delete all previously emitted events of the projection, which is possible but still quite an effort.

Thanks, Regards

Sebastian

[Greg Young]

Can we grab your db to take a look at where it is?

> --
> You received this message because you are subscribed to the Google Groups
> "Event Store" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to event-store...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Studying for the Turing test

[Sebastian Burgstaller]

As the DB contains user specific data (like names, emails of our company's employees) this is not so easy. Last time I only got the permission from management to do a remote debugging session, so I could make sure the data was deleted on your side afterwards ;-D

If this is a possibility for you I could create an issue on Zendesk first and then we could talk about the details.

Regards,

Sebastian

[Greg Young]

Probably easier for us to just recreate with dummy data :) What does
your projection look like so we can begin on that

[Sebastian Burgstaller]

I thought so too ;-) This is our projection:

fromAll()
    .when({
        $any: function (s, e) {
            if (e.metadata && e.metadata.$correlationId) {
                linkTo('company.corid-' + e.metadata.$correlationId, e);
            }
        },
    })

We add a metadata field "$correlationId" containing a Guid, to all of our events, so whenever a new conversation starts, we create a new stream "company.corid-05c6c7ea-3331-4948-b79a-f2b5ad0d4ce1' that contains all events of the conversation.

And here is the error log from ES during / after the VM restart (which happend at ~ 9:16:01 UTC).

[PID:06484:025 2014.10.03 09:16:01.518 ERROR MonitoringService   ] Error while collecting stats
System.UnauthorizedAccessException: Access to the registry key '230' is denied.
   at Microsoft.Win32.RegistryKey.Win32Error(Int32 errorCode, String str)
   at Microsoft.Win32.RegistryKey.InternalGetValue(String name, Object defaultValue, Boolean doNotExpand, Boolean checkSecurity)
   at Microsoft.Win32.RegistryKey.GetValue(String name)
   at System.Diagnostics.PerformanceMonitor.GetData(String item)
   at System.Diagnostics.PerformanceCounterLib.GetPerformanceData(String item)
   at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
   at System.Diagnostics.PerformanceCounter.NextSample()
   at System.Diagnostics.PerformanceCounter.NextValue()
   at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetPerfCounterInformation(Dictionary`2 stats, Int32 count) in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 123
   at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 42
   at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\MonitoringService.cs:line 127
[PID:06484:025 2014.10.03 09:16:02.595 ERROR MonitoringService   ] Error while collecting stats
System.UnauthorizedAccessException: Access to the registry key '230' is denied.
   at Microsoft.Win32.RegistryKey.Win32Error(Int32 errorCode, String str)
   at Microsoft.Win32.RegistryKey.InternalGetValue(String name, Object defaultValue, Boolean doNotExpand, Boolean checkSecurity)
   at Microsoft.Win32.RegistryKey.GetValue(String name)
   at System.Diagnostics.PerformanceMonitor.GetData(String item)
   at System.Diagnostics.PerformanceCounterLib.GetPerformanceData(String item)
   at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
   at System.Diagnostics.PerformanceCounter.NextSample()
   at System.Diagnostics.PerformanceCounter.NextValue()
   at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetPerfCounterInformation(Dictionary`2 stats, Int32 count) in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 123
   at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 42
   at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\MonitoringService.cs:line 127
[PID:01548:013 2014.10.03 09:19:07.520 ERROR ProjectionManager   ] The '$by_correlationId' projection faulted due to 'An event emitted in recovery differ from the originally emitted event.  Existing('$>', 'C:613381856/P:613381856'). New('$>', 'C:613378130/P:613378130')'
[PID:01544:014 2014.10.03 09:29:51.094 ERROR ProjectionManager   ] The '$by_correlationId' projection faulted due to 'An event emitted in recovery differ from the originally emitted event.  Existing('$>', 'C:615812407/P:615812407'). New('$>', 'C:615809124/P:615809124')'
[PID:01544:014 2014.10.03 12:05:51.115 ERROR ProjectionManager   ] The '$by_correlationId' projection faulted due to 'An event emitted in recovery differ from the originally emitted event.  Existing('$>', 'C:613381856/P:613381856'). New('$>', 'C:613378130/P:613378130')'
[PID:01788:011 2014.10.03 12:10:17.234 ERROR ProjectionManager   ] The '$by_correlationId' projection faulted due to 'An event emitted in recovery differ from the originally emitted event.  Existing('$>', 'C:613381856/P:613381856'). New('$>', 'C:613378130/P:613378130')'

Hope that helps.

Regards

Sebastian

[Greg Young]

moving off list

On Mon, Oct 6, 2014 at 2:18 PM, Sebastian Burgstaller

[Zurab Sajaia]

Hi,

We're getting the same error on 3.0.1 on Windows. Is there some general resolution or the db will need to be looked at as with Sebastian?

Thanks,

Zurab

[Steven Blair]

We are experiencing the same issue on (version 3.2)

Any solution to this?

From what I can tell, our projection has stopped running as well.

AllSales - Faulted (Enabled)

An event emitted in recovery differs from the originally emitted event. Existing('$>', 'C:243467370/P:243467370'). New('$>', 'C:243350716/P:243350716')

Here is our Projection:

fromAll()

.when({

    '$any' : function (s, e) {

        linkTo('AllSalesCompany', e);

    }

});

[Steven Blair]

We got the same today (after we started from a fresh install)

The only pattern I can see, is this issues appears after the  Event Store Process  has been restarted.

The first time, it was stopped by hand, and restarted, and the second, the Event Store crashed.

We are not sure how to rectify this problem just now, but I am assuming we will have to Reset our Projection (how this would work in a Production environment I have no idea).

[Greg Young]

This is a known issue in projections. There is a card up to resolve it.

Resetting the projection will resolve it. You can also manually edit
the checkpoint.

Projections are at this point in beta (likely to come out later this
fall) and are not intended to be used in critical production
environments.

[Alexey Zimarev]

It still happens in ES4 and there is no documented way to resolve it.

[Julian walls]

I've also experienced this today on EventStore 4.0.1 (3 node cluster) with  the built in projections:

An event emitted in recovery for stream $streams differs from the originally emitted event. Existing('$>', 'C:9722680354/P:9722680354'). New('$>', 'C:9722679820/P:9722679820')

[Pieter Germishuys]

We have been hard at work at root causing and resolving this particular issue. It's a high priority item on the backlog.

If you are willing to share logs, possibly even the DB that would be most useful. If there is sensitive information in your logs or database, please don't hesitate to send them over to pieter at geteventstore dot com and i'll forward to the relevant members of the team.

The way we recommend that customers currently resolve this issue is to either

1. Reset the projection (recommended, events will be re-emitted)

2. Set the projection checkpoint to the existing position which in this case would be 9722680354/P:9722680354

[Serhii Daletskyi]

How can we set a checkpoint for system projections?

[Pieter Germishuys]

The issue has been resolved and the bug fix will be out in 4.0.3

[Siamak Haschemi]

We were hit hard by the issue tonight. Here is how we solved it:

1. Create File "set-by_category-checkpoint.txt"

[

  {

    "eventId": "43770587-AB58-4946-B1F5-2C94AACA846E",

    "eventType": "$ProjectionCheckpoint",

    "data": {

    },

    "metadata": {

      "$v": "3:-1:5:3",

      "$c": 94924010595,

      "$p": 94924010595

    }

  }

]

2. Fire event

curl -i -d @set-by_category-checkpoint.txt "http://${HOST}:${PORT}/streams/%24projections-%24by_category-checkpoint" --user 'admin:${YOURPASSWORD}' -H "Content-Type:application/vnd.eventstore.events+json"

3. I then stopped and started the projection

Hope this helps.

Cheers

   Siamak

[ndangthieu]

I got the same error today, in 4.0.3.0

[Hayley-Jean Campbell]

Hi Ndangthieu,

How long have you been running 4.0.3, and were you running an old version before this?

We want to determine if this is a new occurrence of the issue or if it's a remnant from the previous one.

Also, could you provide us with some logs showing the issue? If you don't want to post them here, you can email them to hayley.campbell at eventstore.org

Thanks

[ndangthieu]

Hi Harley,

It was a fresh 4.0.3.0 installation, using a docker image in my development environment. My PC restarted for some reason and I encountered that error. The event numbers in the error message were 0/2. 

I'm so sorry that I've just re-installed the 4.1 so I lost all the information.

Regards,

Thieu

[ndangthieu]

Hi  Harley,

I encountered the same error in my UAT cluster just now. I'll email you the log.

[mikem...@sussex.team]

We actually just saw this issue on 4.1.1

[mik...@ayrlabs.com]

Just got it on Event Store 4.1.1.0 

[Greg Young]

Can you guys provide more details either here or to support? Most likely it is a random occurrence as not much has changed but would prefer to investigate.

--

You received this message because you are subscribed to the Google Groups "Event Store" group.
To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexey Zimarev]

It can be a part of the FAQ, the reason was mentioned in this group. We have that as well.

Say, we made a projection "xxx" that projects to a stream "xxx-proj". 

Then, we delete this projection and create a new projection "yyy" that writes to "xxx-proj".

As a result, "yyy" fails with "Projection Faulted: An event emitted in recovery differ from the originally emitted event".

In our case, we should have modified the "xxx" with the new code, instead of deleting it and making a new projection that writes to the same stream.

The overall solution was to write to another stream or remove the "xxx-proj" stream and then the new projection starts working.