Slightly worrying ESENT error

[Nic Wise]

Hi there

We have this error coming up when we try to write to Raven (it might
be just one record, as we try to write, it fails, and we go back and
try again later....)

We are using 960 on Windows 2008 R2.

The machine has 12GB of RAM, and Raven commonly uses all of it. The
database is about 6GB or so, with about 170meg of indexes - 1.2m
documents, and about 30 indexes.

Could this be the result of either an out of memory situation, or a
corrupt index (most likely the result of an unscheduled reboot, or an
out of memory). All indexes are showing as stale, tho they are also
being written to quite often, so that may not be a proper data point.

Any ideas?

Thanks

Nic
Universal Music.


Failed to publish orders because
System.Reflection.TargetInvocationException: Exception has been thrown
by the target of an invocation.
---> System.Reflection.TargetInvocationException: Exception has been
thrown by the target of an invocation.
---> System.InvalidOperationException: Url: "/bulk_docs"
Microsoft.Isam.Esent.Interop.EsentNoCurrentRecordException: Currency
not on a record
at Microsoft.Isam.Esent.Interop.Api.JetRetrieveColumn(JET_SESID sesid,
JET_TABLEID tableid, JET_COLUMNID columnid, Byte[] data, Int32
dataSize, Int32 dataOffset, Int32& actualDataSize, RetrieveColumnGrbit
grbit, JET_RETINFO retinfo)
at Microsoft.Isam.Esent.Interop.Api.RetrieveColumn(JET_SESID sesid,
JET_TABLEID tableid, JET_COLUMNID columnid, RetrieveColumnGrbit grbit,
JET_RETINFO retinfo)
at Microsoft.Isam.Esent.Interop.Api.RetrieveColumn(JET_SESID sesid,
JET_TABLEID tableid, JET_COLUMNID columnid)
at Raven.Storage.Esent.StorageActions.DocumentStorageActions.EnsureDocumentEtagMatchInTransaction(String
key, Nullable`1 etag)
at Raven.Storage.Esent.StorageActions.DocumentStorageActions.AddDocumentInTransaction(String
key, Nullable`1 etag, RavenJObject data, RavenJObject metadata,
TransactionInformation transactionInformation)
at Raven.Database.DocumentDatabase.<>c__DisplayClass37.<Put>b__30(IStorageActionsAccessor
actions) at
Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action)
at Raven.Database.DocumentDatabase.Put(String key, Nullable`1 etag,
RavenJObject document, RavenJObject metadata, TransactionInformation
transactionInformation)
at Raven.Database.Extensions.CommandExtensions.Execute(ICommandData
self, DocumentDatabase database)
at Raven.Database.DocumentDatabase.<>c__DisplayClassca.<Batch>b__c6(IStorageActionsAccessor
actions)
at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action)
at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action)
at Raven.Database.DocumentDatabase.Batch(IEnumerable`1 commands)
at Raven.Database.Server.Responders.DocumentBatch.Batch(IHttpContext
context)
at Raven.Database.Server.Responders.DocumentBatch.Respond(IHttpContext
context)
at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx)
at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext
ctx)
at Raven.Client.Connection.HttpJsonRequest.ReadStringInternal(Func`1
getResponse) in
c:\Builds\raven\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line
362
at Raven.Client.Connection.HttpJsonRequest.ReadResponseString() in
c:\Builds\raven\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line
194
at Raven.Client.Connection.HttpJsonRequest.ReadResponseJson() in
c:\Builds\raven\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line
594
at Raven.Client.Connection.ServerClient.DirectBatch(IEnumerable`1
commandDatas, String operationUrl) in
c:\Builds\raven\Raven.Client.Lightweight\Connection\ServerClient.cs:line
801
at Raven.Client.Connection.ServerClient.<>c__DisplayClass4f.<Batch>b__4e(String
u) in c:\Builds\raven\Raven.Client.Lightweight\Connection\ServerClient.cs:line
786
at Raven.Client.Connection.ServerClient.TryOperation[T](Func`2
operation, String operationUrl, Boolean avoidThrowing, T& result) in
c:\Builds\raven\Raven.Client.Lightweight\Connection\ServerClient.cs:line
196
at Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String
method, Func`2 operation) in
c:\Builds\raven\Raven.Client.Lightweight\Connection\ServerClient.cs:line
166
at Raven.Client.Connection.ServerClient.Batch(IEnumerable`1
commandDatas) in
c:\Builds\raven\Raven.Client.Lightweight\Connection\ServerClient.cs:line
786
at Raven.Client.Document.DocumentSession.SaveChanges() in
c:\Builds\raven\Raven.Client.Lightweight\Document\DocumentSession.cs:line
413
at RavenDb.Repository.RavenDbRepository.Save[T](T instance) in
d:\BuildAgent\work\665c94433f83b8d0\src\CommonProjects\RavenDb.Repository\RavenDbRepos





--
Nic Wise
t. +44 7788 592 806 | @fastchicken | http://www.linkedin.com/in/nicwise
b. http://www.fastchicken.co.nz/

mobileAgent (for FreeAgent): get your accounts in your pocket.
http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
Earnest: Self-employed? Track your business expenses and income.
http://earnestapp.com
Nearest Bus: find when the next bus is coming to your stop. http://goo.gl/Vcz1p
London Bike App: Find the nearest Boris Bike, and get riding! http://goo.gl/Icp2

[Daniel Dar]

Did you try to reset the indexes?

[Nic Wise]

We had thought about that, but it can take 24 hours to rebuild the
indexes, so we are quite reluctant to do that.

We also had this error (tho with actual words - I'm at home, so I dont
have the exact error here)

<process name> (<PID>) The version store for instance <instance>
("<name>") has reached its maximum size of <value>Mb. It is likely
that a long-running transaction is preventing cleanup of the version
store and causing it to build up in size. Updates will be rejected
until the long-running transaction has been completely committed or
rolled back. Possible long-running transaction: SessionId: <value>
Session-context: <value> Session-context ThreadId: <value>.

I suspect that we need to restart Raven to clear that out, which will
rebuild, or atleast check, the indexes.

Have you seen any causes etc for this?

Thanks

Nic

[Daniel Dar]

I have not ran into something like that and i'm not sure how to reproduce it.

Could you try and create a duplicate database and see if it works there?

[Nic Wise]

Thanks Daniel

We ended up restarting Raven, and it's now working. I suspect an index
got out of sync when the disk filled up, and the dev's were not overly
willing to restart the service incase it made it worse.

Needless to say, we are about to add another 200GB to the server :)
And possibly move Raven onto it's own box.

Thanks for the help.

Nic

[Daniel Dar]

Glad it helped.

[Oren Eini (Ayende Rahien)]

FWIW, I am pretty sure we fixed that in 1.2

[Nic Wise]

Excellent, thanks :)

[Gaz]

Hi Oren,

We're running build 2017 and just ran into this error:

[3828] System.NullReferenceException: Object reference not set to an instance of an object. 

[3828]    at Errordite.Client.ErrorditeClient.ReportException(ErrorditeConfiguration configuration, Exception exception) in e:\Repositories\Errordite\Client\Trunk\client\Errordite.Client\ErrorditeClient.cs:line 212 

[3828]    at Errordite.Client.ErrorditeClient.ReportException(Exception exception, Boolean async) in e:\Repositories\Errordite\Client\Trunk\client\Errordite.Client\ErrorditeClient.cs:line 39

[3828] Errordite.Reception: 2012-09-24 08:19:32,155 - CodeTrip.Core.Auditing.Entities.ExceptionAuditEvent 

[3828]  

[3828] ================================================================================================ 

[3828] Exception Info [Start] 

[3828]  

[3828] Machine Name     : AMAZONA-LKQK8RS 

[3828] Timestamp        : 9/24/2012 8:19:32 AM 

[3828] AppDomain Name   : NServiceBus.Hosting.Windows.WindowsHost, NServiceBus.Host, Version=3.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c 

[3828] Windows Identity : AMAZONA-LKQK8RS\errordite_svc_rcp 

[3828] Type             : System.InvalidOperationException 

[3828] Message          : Url: "/bulk_docs" 

[3828]  

[3828] Microsoft.Isam.Esent.Interop.EsentNoCurrentRecordException: Currency not on a record 

[3828]    at Microsoft.Isam.Esent.Interop.Api.JetRetrieveColumn(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, Byte[] data, Int32 dataSize, Int32 dataOffset, Int32& actualDataSize, RetrieveColumnGrbit grbit, JET_RETINFO retinfo) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\InternalApi.cs:line 106 

[3828]    at Microsoft.Isam.Esent.Interop.Api.RetrieveColumn(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, RetrieveColumnGrbit grbit, JET_RETINFO retinfo) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\RetrieveColumnHelpers.cs:line 147 

[3828]    at Microsoft.Isam.Esent.Interop.Api.RetrieveColumn(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\RetrieveColumnHelpers.cs:line 189 

[3828]    at Raven.Storage.Esent.StorageActions.DocumentStorageActions.EnsureDocumentEtagMatchInTransaction(String key, Nullable`1 etag) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Storage.Esent\StorageActions\Util.cs:line 60 

[3828]    at Raven.Storage.Esent.StorageActions.DocumentStorageActions.AddDocumentInTransaction(String key, Nullable`1 etag, RavenJObject data, RavenJObject metadata, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Storage.Esent\StorageActions\Documents.cs:line 340 

[3828]    at Raven.Database.DocumentDatabase.<>c__DisplayClass37.<Put>b__30(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 486 

[3828]    at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Storage.Esent\TransactionalStorage.cs:line 327 

[3828]    at Raven.Database.DocumentDatabase.Put(String key, Nullable`1 etag, RavenJObject document, RavenJObject metadata, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 468 

[3828]    at Raven.Database.Extensions.CommandExtensions.Execute(ICommandData self, DocumentDatabase database) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Extensions\CommandExtensions.cs:line 25 

[3828]    at Raven.Database.DocumentDatabase.<>c__DisplayClassc9.<Batch>b__c5(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 1190 

[3828]    at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Storage.Esent\TransactionalStorage.cs:line 373 

[3828]    at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Storage.Esent\TransactionalStorage.cs:line 361 

[3828]    at Raven.Database.DocumentDatabase.Batch(IEnumerable`1 commands) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 1186 

[3828]    at Raven.Database.Server.Responders.DocumentBatch.Batch(IHttpContext context) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\Responders\DocumentBatch.cs:line 94 

[3828]    at Raven.Database.Server.Responders.DocumentBatch.Respond(IHttpContext context) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\Responders\DocumentBatch.cs:line 38 

[3828]    at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\HttpServer.cs:line 581 

[3828]    at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\HttpServer.cs:line 347 

Should we upgrade to a later build to get around the problem?

Thanks,

Gareth

[Fitzchak Yitzchaki]

Yes.

[Oren Eini (Ayende Rahien)]

Can you try on a recent build?

On Mon, Sep 24, 2012 at 10:29 AM, Gaz <garetht...@gmail.com> wrote:

[Gaz]

I think restarting Raven fixed it on that build.  We will update the build at some point and will keep you posted.  FYI this error came after we rebuilt all our indexes on a 2 million document db.

[Nic Wise]

Thanks Gaz :)

[Gaz]

Hi Oren,

Sorry for the delay but we're still seeing similar exceptions on build 2137; this is with a totally new database.  Any ideas?  

Full exception below:

Error=Microsoft.Isam.Esent.Interop.EsentNoCurrentRecordException: Currency not on a record

   at Microsoft.Isam.Esent.Interop.Api.JetRetrieveColumn(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, Byte[] data, Int32 dataSize, Int32 dataOffset, Int32& actualDataSize, RetrieveColumnGrbit grbit, JET_RETINFO retinfo) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\InternalApi.cs:line 106

   at Microsoft.Isam.Esent.Interop.Api.RetrieveColumn(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, RetrieveColumnGrbit grbit, JET_RETINFO retinfo) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\RetrieveColumnHelpers.cs:line 147

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.EnsureDocumentEtagMatchInTransaction(String key, Nullable`1 etag) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Storage\Esent\StorageActions\Util.cs:line 62

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions.AddDocumentInTransaction(String key, Nullable`1 etag, RavenJObject data, RavenJObject metadata, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 409

   at Raven.Database.DocumentDatabase.<>c__DisplayClass42.<Put>b__3b(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 633

   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 432

   at Raven.Database.DocumentDatabase.Put(String key, Nullable`1 etag, RavenJObject document, RavenJObject metadata, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 592

   at Raven.Database.Extensions.CommandExtensions.Execute(ICommandData self, DocumentDatabase database) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Extensions\CommandExtensions.cs:line 25

   at Raven.Database.DocumentDatabase.<>c__DisplayClassf3.<Batch>b__ef(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 1532

   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 481

   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 468

   at Raven.Database.DocumentDatabase.Batch(IEnumerable`1 commands) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\DocumentDatabase.cs:line 1527

   at Raven.Database.Server.Responders.DocumentBatch.Batch(IHttpContext context) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\Responders\DocumentBatch.cs:line 101

   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\HttpServer.cs:line 790

   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\Builds\RavenDB-Unstable-v1.2\Raven.Database\Server\HttpServer.cs:line 534

Gareth

[Oren Eini (Ayende Rahien)]

How do you reproduce this?

[Gaz]

It seems it's specific to the environment we're seeing the problems on.  I exported the db, imported it on my machine, made the exact same HTTP request (using Fiddler) and there was no error.

[Oren Eini (Ayende Rahien)]

Okay, strange.

Did you copy the original db from another machine?

[Gareth Thackeray]

Previously we had a single-tenanted solution.  This database had been grown from scratch.  When we moved to a multi-tenanted solution, I tried to export / import the single database to "tenant 1".  This failed (I think mainly due to the too-weedy EC2 instance it was running on) and after a few attempts I just copied the data directory instead.

However: This caused problems (I was aware you advise against this but wanted to try my luck) and since then I have deleted the whole data directory and started again.  So the current db is all new.

[Gareth Thackeray]

Hi Oren,

How would you suggest we start trying to debug / fix this issue?

- step through Raven in our production environment?

- trash the whole thing and start again, tracing until we run into the issue again?

Or is there is a chance that Esent on this disk is in some way irrevocably broken and we should just scrap the disk and start anew (I realise this is basically witchcraft).

Gareth

Okay, strange.

>> >> t.  +44 7788 592 806 begin_of_the_skype_highlighting FREE +44 7788 592 806end_of_the_skype_highlighting | @fastchicken |

>> >> http://www.linkedin.com/in/nicwise
>> >> b. http://www.fastchicken.co.nz/
>> >>
>> >> mobileAgent (for FreeAgent): get your accounts in your pocket.
>> >> http://goo.gl/IuBU
>> >> Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
>> >> Earnest: Self-employed? Track your business expenses and income.
>> >> http://earnestapp.com
>> >> Nearest Bus: find when the next bus is coming to your stop.
>> >> http://goo.gl/Vcz1p
>> >> London Bike App: Find the nearest Boris Bike, and get riding!
>> >> http://goo.gl/Icp2
>> >
>> >
>>
>>
>>
>> --
>> Nic Wise

>> t.  +44 7788 592 806 begin_of_the_skype_highlighting FREE +44 7788 592 806end_of_the_skype_highlighting | @fastchicken | http://www.linkedin.com/in/nicwise

>> b. http://www.fastchicken.co.nz/
>>
>> mobileAgent (for FreeAgent): get your accounts in your pocket.
>> http://goo.gl/IuBU
>> Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
>> Earnest: Self-employed? Track your business expenses and income.
>> http://earnestapp.com
>> Nearest Bus: find when the next bus is coming to your stop.
>> http://goo.gl/Vcz1p
>> London Bike App: Find the nearest Boris Bike, and get riding!
>> http://goo.gl/Icp2
>
>



--
Nic Wise

t.  +44 7788 592 806 begin_of_the_skype_highlighting FREE +44 7788 592 806end_of_the_skype_highlighting | @fastchicken | http://www.linkedin.com/in/nicwise

[Oren Eini (Ayende Rahien)]

Witchcraft is a good way to describe this, yes.

If you can give me a repro, that would be best.

Basically, I went over the code, and I don't see the problem. If you moved the db from one machine to the other, it is possible that you have wrong internal indexes, and a defrag should fix it.

[Gareth Thackeray]

Defrag meaning standard Windows defrag? Or something else?

[Oren Eini (Ayende Rahien)]

esentutl /d Data

Is the defrag command.

[Gareth Thackeray]

Hi Oren,

I did this, and it didn't help.  If I export all the data, delete the db then import the data again, the problem goes away for a bit but then comes back later.  If I try to import over the existing db I get the same Esent currency exception.

I can give you a copy of a database directory that is in this errored state, but I don't yet know how to take a working db and break it.  Would a copy of a broken directory be helpful to you?

Thanks,

G

[Oren Eini (Ayende Rahien)]

Yes, that would be great.

[Gaz]

Update for the group and posterity:

I sent a broken database and a Fiddler trace showing trace of HTTP requests going from a healthy database to a broken one.  Oren couldn't see how this could be happening.

However, I noticed that the exception appeared to happen whenever a write attempt was made to a particular document. I also noticed that running up to the first instance of this exception there would be a series of 409 responses, all saying the write could not take place on the document due to it being locked by a transaction.  This despite a call to commit the transaction being made prior (the transactions were enlisted in the DTC, as our calls were coming from NServiceBus handlers).

Oren explained that the DTC is async http://ayende.com/blog/4528/who-stole-my-transaction, so I asked him how I can prevent my calls enlisting in the DTC.  Turns out it's as easy as setting EnlistInDistributedTransactions = false on the Document Store.  So we did this and the problem appears to have gone away.

I'd just like to put on record at this point that I hate the DTC.  For something so potentially critical it seems opaque, hard to configure, badly documented and it seems implemented quite strangely too.

Thanks for you help Ayende,

Gareth

[Oren Eini (Ayende Rahien)]

For the record, we hate DTC too.

[nightwatch]

On Wednesday, November 28, 2012 12:31:32 PM UTC+1, Oren Eini wrote:

For the record, we hate DTC too.

Me too, but without DTC message processing is almost, but not quite, transactional.

[Troy]

With this... what are the impacts of: EnlistInDistributedTransactions = false

[Gareth Thackeray]

Basically your Raven transactions will not be part of any larger distributed transaction that may be taking place.  So if for example you have some SQL Server updates as part of the transaction, the Raven changes will be committed even if the distributed transaction rolls back and the SQL Server changes discarded.

[Troy]

Ok cool... I am bringing up NSB myself but will only be touching RavenDB ... wanted to make sure I would not lose any ACIDness with RavenDB by setting that to false in the DocumentStore.

Thanks for posting the solution!

[Chris Marisic]

I'm pretty sure it also means

using (var scope=  new TransactionScope())
                {
                    session.Store(....)
                    session.SaveChanges()

                    scope.RollBack()
                }

That your data is stored because of savechanges. But i could be wrong about that enlisting a distributed transaction.

[nightwatch]

If you are opening a session inside transaction scope, it should enlist in the ambient transaction (unless enlisting is disabled)

But if you open the session outside of TransactionScope, it will not be automatically enlisted.

So your code behavior depends on where the session is created

Usually there's no problem with not using distributed transactions with message bus, but sometimes you really want your messages to be sent as a part of a transaction, together with database update. Without DTC it's possible that messages will be published even if the database transaction is rolled back and so you'll be sending inconsistent information to your subscribers. This situation is quite common if you build a transactional system around message bus so it's imho a major issue if you can't rely on the framework being transactional.

R

[Gareth Thackeray]

For what it's worth, on successive projects I have built my own quasi-transactional "scope" to avoid running into DTC problems that I don't understand and can't fix.  Basically instead of publishing / sending a message at the point in the code I want to send it I call something like:

_appScope.QueueMessageAction(bus => bus.Publish(message));

Then in the _appScope commit (which basically happens at the end of a successful unit of work) I run all the actions.

Clearly it's not properly transactional in the sense that if the action fails the other (Raven / SQL or whatever) txn has already committed.  I wouldn't want to use it as is for something critical and sensitive, but it makes what's going on more transactional and in practice has not caused us any problems.

[Kijana Woodard]

I see the risk there is that the state could be committed to the db, but the event is never published. The chance of that happening is very very low and probably won't be noticed if it ever does happen. But, like you say, it's a hole "for something critical and sensitive".

[Gareth Thackeray]

Incidentally, I'm about to do something similar to this for Raven batch operations, so that I can effectively include these in the same unit of work as standard operations.