Cluster-wide transactions failing after upgrade to 5.2.3 from 5.1.5

[Andrej Krivulčík]

After upgrading both servers and client libraries to 5.2.3 yesterday, the cluster-wide transactions started failing with the following errors:

Failed to execute cluster transaction due to the following issues: Guard compare exchange value 'rvn-atomic/DOC-ID' index does not match the transaction index's 0 change vector on DOC-ID
Concurrency check failed for putting the key 'rvn-atomic/DOC-ID'. Requested index: 0, actual index: 1719072

The first operation on a given document succeeds, but all subsequent attempts fail with the above errors.

I'm unable to replicate this on an empty database with a single document (I tried a standalone server and a 3-node cluster).

Any suggestions about how to investigate this further?

Using DisableAtomicDocumentWritesInClusterWideTransaction prevents this issue from happening.

[Igal Merhavia]

Hi,

Did you load the document before trying to modify it?
In v5.2 the atomic guard is turned on by default and you have to load the document to modify the document while using this feature.

class TestObj
{
    public string Id { get; set; }
    public int Version { get; set; }
}

[Fact]
public async Task TestCase()
{
    const string id = "TestObj0";

    using var store = GetDocumentStore();
    using (var session = store.OpenAsyncSession(new SessionOptions{TransactionMode = TransactionMode.ClusterWide}))
    {
        var testObj = new TestObj{Version = 1};
        await session.StoreAsync(testObj, id);
        await session.SaveChangesAsync();
    }

    using (var session = store.OpenAsyncSession(new SessionOptions { TransactionMode = TransactionMode.ClusterWide }))
    {
        await session.StoreAsync(new TestObj{Version = 2}, id);
        await Assert.ThrowsAnyAsync<ConcurrencyException>(() => session.SaveChangesAsync());
    }

    using (var session = store.OpenAsyncSession(new SessionOptions { TransactionMode = TransactionMode.ClusterWide }))
    {
        var testObj = await session.LoadAsync<TestObj>(id);
        testObj.Version = 3;
        await session.StoreAsync(testObj, id);
        await session.SaveChangesAsync();
    }
}

Best regards,
Igal

--
You received this message because you are subscribed to the Google Groups "RavenDB - an awesome database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/b0f1c9ba-0c6d-4be5-847f-4e3a218e7214n%40googlegroups.com.

[Oren Eini (Ayende Rahien)]

Are you running a mix of clients with 5.2 and earlier than 5.2 ? 

On Fri, Oct 1, 2021 at 11:14 AM Andrej Krivulčík <kriv...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "RavenDB - an awesome database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/b0f1c9ba-0c6d-4be5-847f-4e3a218e7214n%40googlegroups.com.

--

Oren Eini CEO   /   Hibernating Rhinos LTD Mobile:  972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

[Andrej Krivulčík]

Sorry for not replying to this earlier. I meant to investigate in more detail, but since disabling the atomic writes handles our use case quite nicely, I'll probably not check this more closely.

We load the document in the session where it is saved, so the issue should not be caused by reusing data from another session or storing new data with an existing id.

Regarding using a mix of clients, I can't say with complete certainty, but only one application deals with the collection where this happened, and that application (website) was using the new client library. It's possible that there were other applications with the 5.1.5 client library, but they all work with different entities, or are read-only for this collection.

[Igal Merhavia]

Hi,

I am adding here code example just to clarify the behavior and the possible cause of the issue.
The atomic guard relay on the compare exchange feature and handled on the server-side.
The exception you got means we tried to modify the compare exchange with the wrong index.
We get this information from the RAFT part of the document change vector.
From the exception, you sent you tried to store the document with no RAFT part.
Can happen:

• If you store a new instance without using a loaded document
• If you delete and recreate it without TransactionMode.ClusterWide and then you try to store it with TransactionMode.ClusterWide (In this case, the compare exchange will remain with its index and that will cause the issue).

class TestObj
{
    public string Id { get; set; }
    public string Prop { get; set; }
}

[Fact]
public async Task TestCase()
{
    const string id = "testObjs/0";

    using var store = GetDocumentStore();
    using (var session = store.OpenAsyncSession(new SessionOptions{TransactionMode = TransactionMode.ClusterWide}))
    {
        await session.StoreAsync(new TestObj(), id);
        await session.SaveChangesAsync();
    }

    using (var session = store.OpenAsyncSession(new SessionOptions { TransactionMode = TransactionMode.ClusterWide }))
    {
        var loaded = await session.LoadAsync<TestObj>(id);
        loaded.Prop = "Changed2";
        await session.SaveChangesAsync();
    }

    using (var session = store.OpenAsyncSession(new SessionOptions { TransactionMode = TransactionMode.ClusterWide }))
    {
        await session.StoreAsync(new TestObj{Prop = "Changed"}, id);
        //Should load the 
        var e = await Assert.ThrowsAnyAsync<ConcurrencyException>(async () => await session.SaveChangesAsync());
        Output.WriteLine(e.Message);
    }

    using (var session = store.OpenAsyncSession())
    {
        session.Delete(id);
        await session.SaveChangesAsync();
    }

    using (var session = store.OpenAsyncSession(new SessionOptions { TransactionMode = TransactionMode.ClusterWide }))
    {
        await session.StoreAsync(new TestObj { Prop = "Changed2" }, id);
        var e = await Assert.ThrowsAnyAsync<ConcurrencyException>(async () => await session.SaveChangesAsync());
        Output.WriteLine(e.Message);
    }

    using (var session = store.OpenAsyncSession())
    {
        await session.StoreAsync(new TestObj { Prop = "Changed3" }, id);
        await session.SaveChangesAsync();
    }
    using (var session = store.OpenAsyncSession(new SessionOptions { TransactionMode = TransactionMode.ClusterWide }))
    {
        var loaded = await session.LoadAsync<TestObj>(id);
        loaded.Prop = "Changed2";
        var e = await Assert.ThrowsAnyAsync<ConcurrencyException>(async () => await session.SaveChangesAsync());
        Output.WriteLine(e.Message);
    }
}

Best regards,
Igal

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/6b11b6d9-4cba-46ed-9aaa-e5b66eab506en%40googlegroups.com.