Failed to send message to [Machine-Name].[Endpoint] on SQL Transport

[co...@clear-measure.com]

Hello, the system I am working with has services that are operating on NSB 5.2.16 as well as NSB 6.1 We are using SQL Transport and NHibernate persistence. Today I received messages with the following stacktrace. I had seen this once before and I solved it by refreshing the NServiceBus Subscription table, but it was no help this time.

NServiceBus.Unicast.Queuing.QueueNotFoundException: Failed to send message to [Machine-Name].[Fulfillment] ---> System.Data.SqlClient.SqlException: Invalid object name 'Machine-Name.Fulfillment'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader()
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.TableBasedQueue.<SendRawMessage>d__9.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Queuing\TableBasedQueue.cs:line 91
   --- End of inner exception stack trace ---
   at NServiceBus.Transport.SQLServer.TableBasedQueue.ThrowQueueNotFoundException(SqlException ex) in C:\Build\src\NServiceBus.SqlServer\Queuing\TableBasedQueue.cs:line 119
   at NServiceBus.Transport.SQLServer.TableBasedQueue.<SendRawMessage>d__9.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Queuing\TableBasedQueue.cs:line 101
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at NServiceBus.Transport.SQLServer.TableBasedQueueDispatcher.<Send>d__6.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Sending\TableBasedQueueDispatcher.cs:line 96
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.TableBasedQueueDispatcher.<DispatchUsingReceiveTransaction>d__5.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Sending\TableBasedQueueDispatcher.cs:line 82
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.TableBasedQueueDispatcher.<DispatchAsNonIsolated>d__3.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Sending\TableBasedQueueDispatcher.cs:line 46
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.MessageDispatcher.<Dispatch>d__1.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Sending\MessageDispatcher.cs:line 21
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.DispatchTimeoutBehavior.<Invoke>d__1.MoveNext() in Z:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\DelayedDelivery\TimeoutManager\DispatchTimeoutBehavior.cs:line 34
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.ReceiveStrategy.<TryProcessingMessage>d__12.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Receiving\ReceiveStrategy.cs:line 33
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.ReceiveWithTransactionScope.<TryProcess>d__4.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Receiving\ReceiveWithTransactionScope.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQLServer.ReceiveWithTransactionScope.<ReceiveMessage>d__1.MoveNext() in C:\Build\src\NServiceBus.SqlServer\Receiving\ReceiveWithTransactionScope.cs:line 39

These messages also came into service pulse with nothing in the `message body` as well as a blank message type. This is happening after having recently switching from MSMQ to SQL transport. Not all messages are failing, this appears to be intermittent. These messages also fail during retries, which makes sense to me because of their lack of metadata. Does anyone have an idea of what could be happening to these messages to lose both their body as well as the metadata around their type?

Thanks in advance,
Corey

[Marcin Hoppe]

Hi Corey,

It looks like what happens is that you have timeouts that were scheduled when the endpoint was running on MSMQ and the dispatch attempt was made later then the endpoint was already running on the SQL Server transport.

If my guess is correct, what you are seeing in ServicePulse are NServiceBus control messages related to dispatching these timeout messages. These messages do not contain headers and bodies of the original messages, these should be stored in the TimeoutEntity table.

Can you confirm that this is what happened?

Best regards,

Marcin

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/26a7589b-c724-4637-8a5e-4d9ce01e0df7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcin Hoppe | Software Engineer | Particular Software

[co...@clear-measure.com]

I believe your guess was correct. We updated the destination of these messages in the TimeoutEntity table to be "Fulfillment" instead of "Fulfillment@Machine-Name" and the timeouts processed successfully. Something that is interesting to note is that we only received these messages for services running NSB 6 and we had sagas with timeouts in other services as well during the transition. Do you have any insight into why this would be the case?

 

Thanks for the help,
Corey

[Marcin Hoppe]

Hey,

Previous version of the SQL Server transport (NServiceBus 5) ignored the machine name part of the endpoint address. If the address was EndpointQueue@MACHINE-NAME, it would just drop the @MACHINE-NAME part and use EndpointQueue as the queue table name.

The current version of the SQL Server transport (NServiceBus 6) uses the EndpointQueue@[Schema] format for addressing and it tried to use MACHINE-NAME as the database schema. Of course there was no [MACHINE-NAME].[EndpointQueue] table in the transport database and that is why you observed errors during timeout dispatch.

Is it all good now or are there still some timeouts that have not been dispatched? 

Best regards,

Marcin

--

You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/73ff2686-f5fc-4f3e-af7c-3b03d29c69a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simon Timms]

Hi Macin,

I work on the same team as Corey. Your explanation makes perfect sense, thanks so much for taking the time to figure this out. We were able to re-dispatch all our problematic timeouts and clear the errors. No data was lost and our timeouts now seem to be firing as they should. I think we consider this issue closed. 

Thanks,

Simon

On Saturday, January 14, 2017 at 7:30:38 AM UTC-7, Marcin Hoppe wrote:

Hey,

Previous version of the SQL Server transport (NServiceBus 5) ignored the machine name part of the endpoint address. If the address was EndpointQueue@MACHINE-NAME, it would just drop the @MACHINE-NAME part and use EndpointQueue as the queue table name.

The current version of the SQL Server transport (NServiceBus 6) uses the EndpointQueue@[Schema] format for addressing and it tried to use MACHINE-NAME as the database schema. Of course there was no [MACHINE-NAME].[EndpointQueue] table in the transport database and that is why you observed errors during timeout dispatch.

Is it all good now or are there still some timeouts that have not been dispatched? 

Best regards,

Marcin

On 13 January 2017 at 21:47, <co...@clear-measure.com> wrote:

I believe your guess was correct. We updated the destination of these messages in the TimeoutEntity table to be "Fulfillment" instead of "Fulfillment@Machine-Name" and the timeouts processed successfully. Something that is interesting to note is that we only received these messages for services running NSB 6 and we had sagas with timeouts in other services as well during the transition. Do you have any insight into why this would be the case?

 

Thanks for the help,
Corey

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.

Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/73ff2686-f5fc-4f3e-af7c-3b03d29c69a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Marcin Hoppe]

Hi Simon,

That's great news! Thanks for confirmation that your system works fine after the migration.

Best regards,

Marcin

On 14 January 2017 at 21:52, Simon Timms <sti...@gmail.com> wrote:

Hi Macin,

I work on the same team as Corey. Your explanation makes perfect sense, thanks so much for taking the time to figure this out. We were able to re-dispatch all our problematic timeouts and clear the errors. No data was lost and our timeouts now seem to be firing as they should. I think we consider this issue closed. 

Thanks,

Simon

On Saturday, January 14, 2017 at 7:30:38 AM UTC-7, Marcin Hoppe wrote:

Hey,

Previous version of the SQL Server transport (NServiceBus 5) ignored the machine name part of the endpoint address. If the address was EndpointQueue@MACHINE-NAME, it would just drop the @MACHINE-NAME part and use EndpointQueue as the queue table name.

The current version of the SQL Server transport (NServiceBus 6) uses the EndpointQueue@[Schema] format for addressing and it tried to use MACHINE-NAME as the database schema. Of course there was no [MACHINE-NAME].[EndpointQueue] table in the transport database and that is why you observed errors during timeout dispatch.

Is it all good now or are there still some timeouts that have not been dispatched? 

Best regards,

Marcin

On 13 January 2017 at 21:47, <co...@clear-measure.com> wrote:

I believe your guess was correct. We updated the destination of these messages in the TimeoutEntity table to be "Fulfillment" instead of "Fulfillment@Machine-Name" and the timeouts processed successfully. Something that is interesting to note is that we only received these messages for services running NSB 6 and we had sagas with timeouts in other services as well during the transition. Do you have any insight into why this would be the case?

 

Thanks for the help,
Corey

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsubscribe@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/73ff2686-f5fc-4f3e-af7c-3b03d29c69a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcin Hoppe | Software Engineer | Particular Software

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particularsoftware@googlegroups.com.

Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/1bcb22c5-01ce-4626-a263-c7cd23d5115d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.