Elegant way of dealing with ScriptedPatchRequest

[Vlad Kosarev]

Hi there, is there a better way of dealing with a situation like this -

            using (var session = documentStore.OpenSession())

            {

                session.Advanced.Defer(new ScriptedPatchCommandData

                {

                    Key = "products/1",

                    Patch = new ScriptedPatchRequest

                    {

                        Script = @"if (this.UnitsInStock-orderTotal < 0) throw 'Not enough inventory';

                                   this.UnitsInStock-=orderTotal",

                        Values = { {"orderTotal", 50} }

                    }

                });

                try

                {

                    session.SaveChanges();

                }

                catch (Exception ex)

                {

                    Console.WriteLine("Not enough inventory");

                }                

            }

In Redis I can just return and then check that return (true/false). Is there a way to do something like that in Raven?

Also, does ScriptedPatchCommand participate in transactions at all?

Thanks.

[Vlad Kosarev]

I think I got it, something like this -

var patchOutput = documentStore.DatabaseCommands.Patch(

                        "products/1",

                        new ScriptedPatchRequest

                        {

                            Script = @"

                                    if (this.UnitsInStock-orderTotal < 0) 

                                    {

                                        output(-1);

                                    } else {

                                        this.UnitsInStock -= orderTotal;

                                        output(this.UnitsInStock);

                                    }

                           ",

                            Values = { { "orderTotal", 1 }, { "orderNumber", 1000 } }

                        });

                    

                    var inventoryLeft = patchOutput.Value<RavenJArray>("Debug").Single().Value<int>();

                    if (inventoryLeft < 0)

[Vlad Kosarev]

Ok, how would one get a document Id created by PutDocument?

So for example -

                    var patchOutput = documentStore.DatabaseCommands.Patch(

                        "products/1",

                        new ScriptedPatchRequest

                        {

                            Script = @"

                                    if (this.UnitsInStock-requestTotal < 0) 

                                    {

                                        output('');

                                    } else {

                                        this.UnitsInStock -= requestTotal;                                        

                                        PutDocument('baskets/', {'total': requestTotal});

                                        output('baskets/1');

                                    }

                           ",

                            Values = { { "requestTotal", 1 }}

                        });

Instead of 'baskets/1' how would I get actual document that PutDocuments just created?

And just to confirm, ScriptedPatchRequest on a particular document is processed in sequence on one thread (like a queue), correct? According to the book that is the case but just want to make sure that there can't be concurrency exceptions. This will actually massively speed up our app vs current use of Transactions.

[Oren Eini (Ayende Rahien)]

Scripted Patch Request are processed serially for each document, yes.

I'm afraid that there isn't currently any way to get the id of a document created in this manner:

http://issues.hibernatingrhinos.com/issue/RavenDB-3258

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vlad Kosarev]

Great, thanks.

How does something like this behave in replica scenario?

[Oren Eini (Ayende Rahien)]

Normally? I'm not sure that I understand the question

[Vlad Kosarev]

Sorry, I mean how will the sequence work in Master-Master replication.

So if I have two clients and two servers and one client sends in one script patch and another sends in another (both decrementing value on the same document) what exactly happens server side?

[Oren Eini (Ayende Rahien)]

There would be a conflict. And you'll need to resolve that.

Note that normally in replication scenarios only one master accepts writes.

[Vlad Kosarev]

Got it, thank you.

[Vlad Kosarev]

Another question on this - what is the transaction scope on the script, is it the document being modified? Meaning, how is the sequence built, is it based on a particular document (by document id) being modified?

So if multiple scripts are modifying the same document will they all wait in sequence to modify it?

documentStore.DatabaseCommands.Patch(

                    "products/1",

                    new ScriptedPatchRequest

                    {

                        Script = @"this.UnitsInStock--",                        

                    });

                    documentStore.DatabaseCommands.Patch(

                    "products/1",

                    new ScriptedPatchRequest

                    {

                        Script = @"this.UnitsReserved++",

                    });

It looks like if I run those in parallel they still work properly. Am I understanding this correctly?

[Oren Eini (Ayende Rahien)]

All modifications inside a single db goes into a serialized queue.

So everything is modified in a single threaded fashion

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Vlad Kosarev]

Oh wow, so each and every ScriptedPatchRequest goes into that queue then?

That begs the question - what are the limitations on the queue. Size? Time? etc

[Oren Eini (Ayende Rahien)]

All writes goes into the same queue. 

There isn't a limit to it.

[Vlad Kosarev]

So a patch can still fail if we have a transaction running in parallel on the same document, right? If I want a document to never have concurrency/etag staleness issues then I have to only allow access to it through scripted patches, correct?

[Oren Eini (Ayende Rahien)]

No, it will not fail, they will be serialized.

[Vlad Kosarev]

I mean something like this (not a proper test but just wanted to give an idea)

Without bolded part it's all good, with bolded part test just spins for a while and then - Raven.Abstractions.Exceptions.ConcurrencyExceptionPUT attempted on : products/1 while it is being locked by another transaction

That's what I mean. Same document being modified by scripts and transactions. It does make sense that transaction rules apply and make this a concurrent write issue.

        [Fact]

        public void CanUseScriptAndTransactionInParallel()

        {

            using (var documentStore = NewDocumentStore(requestedStorage: "esent"))

            {

                using (var session = documentStore.OpenSession())

                {

                    session.Store(new Product

                    {

                        UnitsInStock = 50,

                        UnitsReserved = 0

                    });

                    session.SaveChanges();

                }

                Parallel.For(0, 50, new ParallelOptions() { MaxDegreeOfParallelism = 50 }, a =>

                {

                    documentStore.DatabaseCommands.Patch(

                                           "products/1",

                                           new ScriptedPatchRequest

                                           {

                                               Script = @"

                                    if (this.UnitsInStock-requestTotal < 0) 

                                    {

                                        output('');

                                    } else {

                                        this.UnitsInStock -= requestTotal;                                        

                                        PutDocument('baskets/', {'total': requestTotal});

                                        output('baskets/1');

                                    }

                           ",

                                               Values = { { "requestTotal", 1 } }

                                           });

                    documentStore.DatabaseCommands.Patch(

                       "products/1",

                       new ScriptedPatchRequest

                       {

                           Script = @"

                                    if (this.UnitsReserved+requestTotal > this.UnitsInStock) 

                                    {

                                        output('');

                                    } else {

                                        this.UnitsReserved += requestTotal;                                        

                                        PutDocument('baskets/', {'total': requestTotal});

                                        output('baskets/1');

                                    }",

                           Values = { { "requestTotal", 1 } }

                       });

                    using (var tx = new TransactionScope())

                    {

                        using (var session = documentStore.OpenSession())

                        {

                            var product = session.Load<Product>(1);

                            product.UnitsInStock--;

                            product.UnitsReserved++;

                            session.SaveChanges();

                        }

                        tx.Complete();

                    }

                });

                using (var session = documentStore.OpenSession())

                {

                    //Assert.Equal(0, session.Load<Product>(1).UnitsInStock);

                    //Assert.Equal(50, session.Load<Product>(1).UnitsReserved);

                }

            }

[Oren Eini (Ayende Rahien)]

Don't use 

                    using (var tx = new TransactionScope())

DTC play by different roles and force us to lock the document completely, we can't do anything to it (and we'll reject any writes, including patching), until the document has been committed.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Oren Eini (Ayende Rahien)]

In general, avoid DTC if you can possible can.

[Vlad Kosarev]

Yeah, I know. DTC has been a nightmare to deal with for us.

That's why this sequential scripting thing is a massive feature. I wasn't even planning to upgrade to 3.0 but if PutDocument returns docId then I can switch from transactions to scripts completely and that will have a colossal impact on speed and reliability. 

Instead of using DTC we will be able to create/update documents using a script which completely removes the complexity of a transaction.

My question had more to do with if I change some of my code to scripts would it work well with other code that uses transactions. The answer is as I expected, I have to rewrite all of that code to use scripts to get what I want.

Thanks.

[Oren Eini (Ayende Rahien)]

Note that DTC transactions != transactions.

RavenDB is fully transactional with respect to all write operations.

So you still have transactions without DTC

[Vlad Kosarev]

You mean on session.SaveChanges level, right? The whole batch will get queued.

The issue with that is things like

Load inventory

Subtract inventory

Update inventory

Update Basket

Update Order

All of this should be done as a transaction and inventory has to be consistent. If anything fails, everything has to fail. Inventory between Load and Update should be the same inventory.

I can see how to get that done with server scripts, I don't quite see how to get that done on client side without getting concurrency issues in inventory.

[Oren Eini (Ayende Rahien)]

You can do that from the client side by setting UseOptimisticConcurrency = true;

Note that something like inventory is _not_ a good example usually.

We usually still want to accept the order even if we have no inventory.

[Vlad Kosarev]

We definitely do not want to accept the order if there is no inventory. Different business (tickets).

We do have UseOptimisticConcurrency and when you have a thousand people hammering the site at the same time that becomes a problem. That's why I'm thinking that server side script will deal with that issue since I can check inventory, subtract it if it checks out, create order, update basket all in the same atomic sequential operation. I can't do that on the client.

[Kijana Woodard]

Up to this point, I've been reading this thread for it's technical merits.

Now that you mention tickets and inventory, I would suggest backing up.

Accepting the order is different from granting the tickets.

You would reduce technical complexity and enable interesting business strategies by separating the concepts.

For instance, given three orders that arrive in the same 30 second window, prioritize the tickets to orders with significant order history, or a confirmed credit card, or more seats, or etc etc etc.

If you really just want serial, throw the orders on a queue with 1 agent.

[Vlad Kosarev]

Kijana, thanks for the input.

None of those things apply to us.

Yes, queue is exactly the solution for this but it increases complexity and surface of the whole setup. We have it running but have had some IT reliability related issues so we still haven't pushed it live.

Apparently raven already has built in queues in the form of patch scripts. I had no clue about that and it's a fantastic solution to this problem. It adds zero complexity and removes a bunch (transactions). I just need PutDocument to return document Id and unless I am missing something this will be my preferred solution. From code change perspective it also looks very simple. Moving from 2.5 to 3.0 seems to be a much more involved exercise. Too many things changed.

[Kijana Woodard]

Can the Id be "calculable" so you don't need it returned?

[Vlad Kosarev]

I think that was a modeling mistake in the first place. We relied on raven's Id generator when we should've created other counters. I think the reason was the same reason as everything else, Load->Read->Save doesn't work well with concurrent calls. Now I would've just used increment/decrement scripts. This product has been around since alphas of raven 2.0 so some of the stuff no longer makes sense but is still there and I want to make things work with the least amount of breaking changes.

[Vlad Kosarev]

You know what, I think I might just try to get the counters going. After thinking about it for a bit I think it will actually be way less impact on everything vs updating to 3.0

I am not sure what features are missing in 2.5 when it comes to scripting but I guess I'll find out.

Thanks.

[Oren Eini (Ayende Rahien)]

RavenDB doesn't actually have a queue. The effect is the same, but in practice what is going on is that we are taking a lock, making a modification and then moving on, so another request takes the lock, runs, etc.

[Vlad Kosarev]

Right, I figured that was the case. Should be good enough for my scenario. Thanks.

Another question, any way to use NextIdentityFor in the script? That would satisfy PutDocument returning document Id I believe.

[Oren Eini (Ayende Rahien)]

No, that isn't exposed in any way, I'm afraid. We'll fix the PutDocument so you'll get it.

Alternatively, just have do a LoadDocument / PutDocument to Config/Sequence document.

[Vlad Kosarev]

Doing exactly that right now :) Thanks.

[Vlad Kosarev]

So I've rewritten the app using scripts instead of transactions and now I see the following - 

When under load all client nodes now time out and fail with lots of these -

The operation has timed out ---> System.Net.WebException: The operation has timed out

   at System.Net.HttpWebRequest.GetResponse()

   at System.Net.HttpWebRequest.GetResponse()

   at Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse)

   at Raven.Client.Connection.HttpJsonRequest.ReadResponseJson()

   at Raven.Client.Connection.ServerClient.DirectBatch(IEnumerable`1 commandDatas, OperationMetadata operationMetadata)

   at Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, OperationMetadata operationMetadata, OperationMetadata primaryOperationMetadata, Boolean avoidThrowing, T& result, Boolean& wasTimeout)

   at Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, OperationCredentials primaryCredentials, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation)

   at Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation)

   at Raven.Client.Document.DocumentSession.SaveChanges()

On the server though I can see changes happening long after all the clients already failed. So it does seem like there is a queue but if it takes a long time to process then all clients get timeouts and server keeps on chugging along. This is not the behaviour I saw with transactions.

I looked at performance counters for raven, nothing seems to jump at me, it looks orderly enough. CPU utilization is low. Disk queue length never flatlines and I would assume that if raven was writing nonstop then disk queue would be pinned high until raven is done. This might be wrong assumption.

Also, ConcurrentRequests counter seems to be overflowing. Both in admin/stats and in perf counter - ConcurrentRequests: -2147483648

This is on Azure with 8 striped drives.

I'm not sure anything can be done about this but will listen to ideas if anyone has any.

Thanks.

[Oren Eini (Ayende Rahien)]

Yes, perf counters really don't work for us. We moved away from then in 3.0

Do you see anything in the logs?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Vlad Kosarev]

Ok here's something that jumps out from the logs. I bolded the time taken. This number starts reasonably low but becomes larger and larger during load.

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'inventoryrecords/107' was found, etag: 01000000-0000-009F-0000-000000000C9B",

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'inventoryrecords/107' was found, etag: 01000000-0000-009F-0000-000000000C9B",

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'inventoryrecords/107' was found, etag: 01000000-0000-009F-0000-000000000C9B",

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Inserted a new document with key 'inventoryrecords/107', update: True, ",

2015-02-27 17:15:07.3139,Raven.Database.DocumentDatabase,Debug,Checking references for inventoryrecords/107,

2015-02-27 17:15:07.3139,Raven.Database.DocumentDatabase,Debug,Put document inventoryrecords/107 with etag 01000000-0000-009F-0000-000000000C9F,

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'LEAP/BasketsCounter' was found, etag: 01000000-0000-009F-0000-000000000C9C",

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'LEAP/BasketsCounter' was found, etag: 01000000-0000-009F-0000-000000000C9C",

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'LEAP/BasketsCounter' was found, etag: 01000000-0000-009F-0000-000000000C9C",

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Inserted a new document with key 'LEAP/BasketsCounter', update: True, ",

2015-02-27 17:15:07.3139,Raven.Database.DocumentDatabase,Debug,Checking references for LEAP/BasketsCounter,

2015-02-27 17:15:07.3139,Raven.Database.DocumentDatabase,Debug,Put document LEAP/BasketsCounter with etag 01000000-0000-009F-0000-000000000CA0,

2015-02-27 17:15:07.3139,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'inventoryrecords/107' was found, etag: 01000000-0000-009F-0000-000000000C9F",

2015-02-27 17:15:07.3239,Raven.Database.Indexing.WorkContext,Debug,"Incremented work counter to 3153 because: PUT inventoryrecords/107, PUT LEAP/BasketsCounter, PATCH inventoryrecords/107",

2015-02-27 17:15:07.3239,Raven.Database.DocumentDatabase,Debug,Successfully executed 1 patch commands in 00:00:00.3104958,

2015-02-27 17:15:07.3239,Raven.Database.Server.HttpServer,Debug,"Request #11,248: POST    - 14,679 ms - LEAP       - 200 - /bulk_docs",

2015-02-27 17:15:07.3239,Raven.Database.Server.HttpServer,Debug," EVAL inventoryrecords/107

",

and then

2015-02-27 17:15:07.3239,Raven.Database.Indexing.WorkContext,Debug,Incremented work counter to 3154 because: PUT baskets/156610,

2015-02-27 17:15:07.3239,Raven.Database.Server.HttpServer,Debug,"Request #11,249: POST    - 14,692 ms - LEAP       - 200 - /bulk_docs",

2015-02-27 17:15:07.3239,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'Raven/SqlReplication/Status' was found, etag: 01000000-0000-006E-0000-000000000003",

2015-02-27 17:15:07.3239,Raven.Database.Server.HttpServer,Debug," PUT baskets/156610

",

Here is the code for first part of the log

var patchOutput = _documentStore.DatabaseCommands.Patch(

                        ticketSection.InventoryRecordId,

                        new ScriptedPatchRequest

                        {

                            Script = @"

                                    if (this.Reserved+requestTotal > this.Quantity) 

                                    {

                                        output('');

                                    } else {

                                        this.Reserved += requestTotal;

                                        var basketsCounter = LoadDocument('LEAP/BasketsCounter');

                                        var id = basketsCounter.Counter;

                                        PutDocument('LEAP/BasketsCounter', {'Counter':++id});

                                        output('baskets/'+id);

                                    }",

                            Values = { { "requestTotal", quantity } }

                        });

Second part is just basket creation in case the script returns the Id (Basket = {bla} and Basket.Store/SaveChanges). Nothing fancy in the basket and index is very simple map only.

Anything I can improve or is this just straight up write speed issue?

Thanks.

[Oren Eini (Ayende Rahien)]

Can you send us a way to reproduce this?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Vlad Kosarev]

It would take me a while to strip my app to just this level. I'll see what I can do next week.

Meanwhile I converted that first part (the one that deals with inventory changes) to Azure Table Storage and got massive speed improvements. What takes minutes in raven and times out, takes seconds in table storage. I hate having to use multiple storage mechanisms for similar things but the difference is non-responsive app vs super snappy app.

It has to be storage speed on Azure, otherwise it makes zero sense why raven becomes so slow at writes. I got into storage preview so will build a raven box with ssd storage and we'll see if that behaves any different.

[Oren Eini (Ayende Rahien)]

Azure I/O is _horrible_ for us. We saw write latencies in _seconds_.

[Vlad Kosarev]

I believe I've read EVERYTHING there is about azure storage. I did everything by the book. How can a stripe of 8 drives be that slow? It's ridiculous. Hopefully their Premium storage is much better.

I added performance monitoring to storage and average end 2 end latency is 14ms which sounds quite reasonable. Does not explain raven behaviour unless on a disk flush it all goes to hell. Also raven behaviour seems to indicate that it just sits there and waits for writes to complete.

The crappy thing is that azure sql is plenty fast and even lower tier behaves faster than raven in a situation like this. I'm sure the underlying hardware is very different there though.

[Oren Eini (Ayende Rahien)]

Try the IO Tester, it should give you some indication on what is going on.

Note that RavenDB _does_ flush. It has to, because it need to ensure that the data is persisted on disk.

[Vlad Kosarev]

I ran crystal disk mark and it did show pretty horrible speeds. What would be the best approximation disk test for raven load? I'm sure it's not sequential read.

As to flushing I absolutely get that, I am not saying it shouldn't, I'm saying that my guess is that flush buffer might be getting bigger and bigger or something like that.

[Kijana Woodard]

Fwiw, I've recently learned that going to a larger instance makes a noticeable difference in disk io, as non-intuitive as that seems. Made an order of magnitude difference, iirc.*

* told to me by someone else.

[Vlad Kosarev]

How much larger? I'm running ExtraLarge. Past it seems to just be memory scaling.

[Kijana Woodard]

I'll have to ask.

It was nonsensical that scaling up the instance made any difference in disk perf.

[Vlad Kosarev]

I am aware of disk perf differences between sizes - https://msdn.microsoft.com/en-us/library/azure/dn197896.aspx

The only dick move they did was when they announced basic VMs they didn't mention that they had lower throughput in all their 'hey, cheaper VMs!!!' advertising.

As you can A4 see extra large has full 16x500 max iops. My stripe is 8 drives and the speed SUCKS. Also, 4 vs 8 vs 16 is not the difference you'd think. They all suck about equally.

I am rewriting the pipeline now in Table Storage and so far I am amazed at the speed of that thing. I might move everything that doesn't need fancy queries there. The speed is just staggering compared to the raven box and the cost is peanuts.

[Oren Eini (Ayende Rahien)]

Vlad,

You might want to go with RavenHQ for Azure, they did a lot of work on the I/O side to make it fast.

[Mircea Chirea]

Unfortunately RavenHQ is only available in the East US region and is extremely expensive for storage.

[Vlad Kosarev]

I thought about ravenhq and while I can probably deal with high prices I just don't see how I will be able to sleep at night without having control over raven version. Too many times we hit a critical bug and then you guys fix it so we update right away. I can't imagine RavenHQ being on such aggressive update schedule.

Just to update on this topic though.

I removed Counter updating script that I mentioned before and moved all of that functionality to Azure Table Store, it's a few of orders of magnitude faster now.

Apparently this is very expensive for Raven on Azure (that's the script that takes progressively longer to run and then client calls start timing out) -

this.Reserved += requestTotal;

var basketsCounter = LoadDocument('LEAP/BasketsCounter');

var id = basketsCounter.Counter;

PutDocument('LEAP/BasketsCounter', {'Counter':++id});

I think I'm just going to go down this path since I can see improvement that I can't imagine will happen on raven azure even if I stripe 8 SSD disk with premium storage and pay the price for that. The only downside is obviously a disconnect between two data stores but I can mitigate that for the most part. I think I'll move more time critical things to table store and then sync them back to raven when load is over so I can query over them.

If I had time I would love to keep digging into azure+raven optimization but at this point I have to fix load issues asap so that I can get some free time at all.

I've gotten a new appreciation for raven after reading your book (I love how in depth you go on some topics) but it is still the flakiest part of our whole infrastructure (it obviously wasn't designed to run well on Azure store but we used it before we moved to Azure). For now I'll just deal with hybrid solution and maybe soon enough azure storage layer will become much batter at satisfying raven's write speed needs.