Single projection but getting "Multiple projections emitting to the same stream detected"

[Ivo M]

Hi all,

We keep getting the "Multiple projections emitting to the same stream detected" message although we have only one projection running and emitting to various streams.

Previously we got in by mistake a projection with the same code and understandably we started seeing the "Multiple projections..." message.

However we performed DELETE using the REST API on the unnecessary projection. The message still shows up and hangs the progress of the existing projection at 0.9%

Any ideas how to fix this?

[Greg Young]

Anything else writing to the stream causes the same error

> --
> You received this message because you are subscribed to the Google Groups
> "Event Store" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to event-store...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Studying for the Turing test

[Ivo M]

Hi Greg,

As always thanks for quick response!

There is only one projection writing to that stream. Multiple events and streams are being tracked ( i.e. fromStreams(...).when(..) ), and we "emit" and "linkTo" from that projection to streams that have specific UUIDs in their names. No other projections write to any of those streams.

We are running the same code in another dev server and prod where there are no issues.

Is there anything that could be related to running in a cluster, or may be anything related to deleting the previous dupe projection?

I noticed that sometimes the projection goes further before it faults when I reset it.

Theoretically shouldn't this be able to run fine if I delete all/any projections, recreate a new one (same or different name) with the same code?

Thanks

-Ivo

[Greg Young]

If anything else not just projections has written to that stream you
will get the same error.

You should not be seeing this if you have nothing else writing to the
streams (and I havnt seen this specific one without this case actually
happening). What version is this and what do logs say?

[Pieter Germishuys]

 When you create a projection that emits/links events to a new stream it will write projection specific link metadata along with each event. 

When another projection is created that links/emits to the same stream it will check the last event in the emitted stream and determine that it's another projection writing to the stream and raise the error you are seeing.

This means that currently even if you delete the projection emitting events to a specific stream that a new projection will not be able to emit to the previous projection's stream.

Expect a fix in 3.6.0.

[Ivo M]

We are using 3.5.0

I will gather logs and post, although what Pieter Germishuys describes sounds like the issue.
Is there any solution currently under these circumstances?

[Ivo M]

Thanks PIeter,

Would changing the emit/linkTo names in the projection fix the issue? 

For example if I was emitting to "newStream-xyz" if I change it to "newStream1-XYZ" ? Would that go around the issue you are describing?

Any ETA on 3.6.0 ?

 

-Ivo

[Pieter Germishuys]

ETA on 3.6.0 is around next week if all goes well.

Changing the stream name is a work around and will solve the issue.

[Ivo M]

Hi Pieter,

Emitting to new stream names worked.

I hope this gets fixed soon!

Thanks for your answer!

-Ivo

[Ivo M]

Hi Pieter,

Want to follow up on this issue. We did upgrade to 3.6.0.

After pushing a projection that is emitting to the previously created/emited streams we are still getting faulted state.

Was the issue addressed as you mentioned or we should wait for a new release at some point?

Here are some logs from after the upgrade:

[PID:24579:036 2016.05.12 15:31:56.796 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.CoreProjectionManagementMessage+Stop in queued handler 'Projection Core #7'.

System.Exception: Current state is Faulted. Expected states are: LoadStateRequested, StateLoaded, Subscribed, Running, CompletingPhase, PhaseCompleted

  at EventStore.Projections.Core.Services.Processing.CoreProjection.EnsureState (State expectedStates) [0x00000] in <filename unknown>:0

  at EventStore.Projections.Core.Services.Processing.CoreProjection.Stop () [0x00000] in <filename unknown>:0

  at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle (EventStore.Projections.Core.Messages.Stop message) [0x00000] in <filename unknown>:0

  at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.CoreProjectionManagementMessage+Stop].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in <filename unknown>:0

  at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in <filename unknown>:0

  at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in <filename unknown>:0

  at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in <filename unknown>:0

[PID:24579:028 2016.05.12 15:31:57.253 ERROR ProjectionManager   ] The 'opportunity_feed' projection faulted due to 'Multiple projections emitting to the same stream detected.  Stream: '$projections-opportunity_feed-order'. Last event projection: '8'.  Emitting projection: '20''

[Pieter Germishuys]

Hi Ivo,

I do apologise for not getting back to you regarding this issue. Unfortunately the work was not completed in 3.6.0 and took a back burner while there was more pressing work that needed to be done.

The PR is destined for 3.7.0

[Ivo M]

Thanks Pieter!

Good to know when to expect it. 

We have temporarily resolved the issue as per our previous discussion and looking forward the fix.

Regards,

-Ivo

[Pieter Germishuys]

Good morning Ivo,

3.7.0 was scheduled to be released yesterday and is currently undergoing some more testing. Although this particular fix was destined to be released with 3.7.0 we have found a number of issues in the working branch for this fix.

We have scheduled 3.8.0 for the 11th of July which will be a projection fix release in which we will focus our efforts to get these issues resolved.

We do apologise for any inconvenience this may have caused you.

[Ivo M]

Hey Pieter,

That's awesome!

I keep getting into this issue despite the renaming workaround we discussed.

Thanks for letting me know!

Best,

-Ivo

[Ivo M]

So the latest related to this I keep seeing , is that when i go through cycles of enable/disable commands, I see the projection moving forward, until its status becomes Running instead of Faulted/enabled. Just FYI

Thanks!

-Ivo

[Ivo M]

Hey Pieter,

Today I installed 3.8.0

I keep seeing the same issue. Deleted and recreated a projection. Still getting the same  error. Enabling/disabling is not fixing it anymore.

Is the fix in this release? Also if it is in would this work only for new projections or messed up old ones too?

Thanks!

-Ivo

[Pieter Germishuys]

Hi Ivo,

We were going to publish a blog post about this but i'll answer you here.

There were 2 PRs that made the `EmittedStreams` a little more tolerable as well as a new feature in projections that tracks emitted streams and on deletion you can choose to delete the emitted streams.

Unfortunately I couldn't find a work around for this release and to resolve the issue you have to delete the emitted streams from the projection and re-create the projection.

The reason is that because when a projection emits events it places metadata in the event. This metadata contains the ID of the projection. When a projection emits events to a stream it will validate the metadata from the last event and determine whether it is the only projection emitting to that stream by checking that it's ID matches that of the last event in the emitted stream.

The PRs addresses this issue in first making the Emitted Streams a little more tolerable in the case when the emitted stream has been deleted. This means that you are able to delete the streams the projection emitted to and able to create a new projection that emits to those same streams.

The second is giving the user the ability to specify in the creation and the deletion of a projection whether they wish to track and delete the streams that the projection emits to.

[Ivo M]

Thanks Pieter,

So just to reiterate and make sure I understand the workaround...

I will issue an HTTP DELETE to the whatever the projected streams are, and after that I should reset the projection and enable it...  And theoretically it should be fine from this point on?

Regards,

-Ivo

[Pieter Germishuys]

Apologies for the late reply Ivo.

Yes, that is correct.

If you have any issues, please don't hesitate to get in touch.

[Ivo M]

Great thanks! Working on it.