Updating large number documents in collection

[Daniel Zolnjan]

Suppose I have 500k documents in collection.

I need to update around 100k of them which would be filtered/returned by query (Where).

What is best way to do this with Raven considering the 1024 read limit and session requests limit?

[Oren Eini (Ayende Rahien)]

Do you want to read them, modify and send them back? Or modify them all on the server?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Marisic]

How about explain why you suddenly need to modify 20% of your persisted data?

[Daniel Zolnjan]

I need to modify a single property on all of these filtered documents. Replace nested object with new one, but not the same one. 

Not sure what you mean by 'modify all on the server'?

Usually there are up to 1k updates daily so standard retrieving, modifying and sending back to Raven just works.

Once in a while there will likely be at least 10k docs to update. This runs in background process. Its not like user can initiate it and wait for response, just needs to be reliable.

[Oren Eini (Ayende Rahien)]

Did you see this?

http://ravendb.net/docs/article-page/3.0/csharp/client-api/commands/documents/how-to/delete-or-update-documents-using-index

[Daniel Zolnjan]

This is useful if I need change sets of documents with same value, which is not case my case.

I'm currently using this helper method that accepts filtering expression and generic action. Updates are sent to server in batches of 1024 but since I dont know how many there might be I'm raising limit for max request per session. 

        public static async Task<int> BatchUpdateAsync<T>(this IAsyncDocumentSession session, Expression<Func<T, bool>> expression, Action<IList<T>> action)
        {
            session.Advanced.MaxNumberOfRequestsPerSession = int.MaxValue;
            RavenQueryStatistics stats = null;
            int start = 0;

            while (true)
            {
                if (stats != null && stats.TotalResults <= start)
                    break;

                var query = session.Query<T>()
                    .Where(expression)
                    .Take(1024)
                    .Skip(start);

                if (stats == null)
                {
                    query = query.Statistics(out stats);
                }

                var entities = await query.ToListAsync();

                if (entities.Count == 0)
                    break;

                start += entities.Count;

                action(entities);      
               
                await session.SaveChangesAsync();        
            }


            return stats.TotalResults;
        }

and use it like so:

 

var total = await session.BatchUpdateAsync<Prediction>(x => x.Status == PredictionStatus.Open, predictions =>
                {
                    foreach (var prediction in predictions)
                    {
                        // do some updates on any of Prediction properties
                        prediction.SetLastStockQuote(stockQuote);
                    }
                });

I also just dicovered Batch api which might be better suited for this.

http://ravendb.net/docs/article-page/3.0/csharp/client-api/commands/batches/how-to-send-multiple-commands-using-a-batch

Is there any limit on how many commands can be sent in same bundle?

[Oren Eini (Ayende Rahien)]

You are better off using a stream + bulk insert for this.