Dealing with large number of queries/updates/inserts

[mare]

Suppose we are importing a large number (in hundreds) of thirdparty items into RavenDB (from SQL Server). We might already have thousands of products in Raven. We want to expose an UI for the user to do the imports, in an ASP.NET MVC application (imports via Raven Studio won't do). We will map the imported products to the third party products by ThirdPartyProductId. When doing imports the second time, updates also come in play. So we might do something like

 

if(existingProducts.Any(thirdPartyProductId)) ...do update...

else

...insert new product...

 

My questions are:

1) How do you ever get all the existing products out so that you can do Any() (I guess I have to do Query() with Statistics(), page it by 128 and then try to load all those pages)

2) Is this the right approach? I guess I'll have to set the maximum number of requests to the int max?

 

In reality, the numbers will be lower (but still few hundreds) because the UI will expose filtering by categories, so user will be limited by selecting a category (he won't be able to just import the whole third party Products table). Still those numbers are way higher than what Raven definies as SAFE.

[Felipe Leusin]

You could create an Index Product_ByThirPartyProductId and query it waiting for NonStale results in this case. Or you keep a constraint doc to do this check so you can use Raven.Load<ConstraintDoc>(arrayOfProductIds) then loop the returinng objetcts, if it`s null the corresponding Id doc doesn`t exist.

public class ConstraintDoc {

    public string Id { get; set; } // this is the third party Id

    public string ProductId { get; set; }

}

var arrayOfThirdPartyIds = [ 11111,22222,33333];

var existingDocs = Raven.Load<ConstraintDoc>(arrayOfTHisPartyIds.Select(x => "thirdpartydocs/" + x));

for(var i=0;i<arrayOfTHisPartyIds.Count;i++) {

    if ( existingDocs[i] == null ) {

        // constraint doc doesnt exist, create a new one!

    } else {

        // update the doc!

    }

}

I`ve set up a nice system to run long running tasks while giving feedback to the user, i`ll try to put it on git later and post here since it might help you.

[mare]

thanks Felipe, for this, I will try it out. When you find time, please do publish it (and post a link to it here if possible).