raven db - client application & backend synchronization (briefcase pattern, continuation token))

[frederikm]

Hi

I'm building an application that is online only part of the time, and thus uses a briefcase pattern.  All data is synced when the client application is online, available for use when offline.

moving parts

- clientapp = .net maui application sitting on top of an embedded raven db

- server = website that sits on top of a cloud instance of raven db.

the data I need to sync is basically

class Product{
  public string Id => "products|";

  public DateTime LastModified => DateTime.UtcNow;

  // lots of business data here

}

the product catalog contains 1000s of products, some of which are bulk uploaded into the database (and thus have the same LastModified date).

The API to read out the products is paged - returning 100 or so products per call

The problem i'm facing now is building a reliable paging system for the data.

The most naive version of this is something along the lines of

     var products = await session.Query<Product>()
         .Where(p => p.LastModified > previousLastModified)
         .OrderBy(e => p.LastModified)
         .Take(pageSize)
         .ToListAsync();

which obviously doesn't work due to bulk updates and many items having the same last modified. 

In a world with numerical document Ids I could do something like this 

     var productsWithSameLastModified  = await session.Query<Product>()
         .Where(p => p.LastModified = previousLastModified && p.Id > previousLastId)
         .OrderBy(e => p.Id)
         .Take(pageSize)
         .ToListAsync();

        var products = await session.Query<Product>()

         .Where(p => p.LastModified > previousLastModified)
         .OrderBy(e => p.LastModified).ThenBy(p => p.Id)
         .Take(pageSize -  productsWithSameLastModified.Count() )
         .ToListAsync();

        var result = productsWithSameLastModified.Union(products);

But given that all documents are string Ids - seems a bit counter intuitive?

Long story short - is there a better / simpler way of doing this?

[Oren Eini]

I think that you are mixing a few things in the question here that aren't related to one another. 

The backend sync and offline work aren't related to the question you have, unless I miss something.

As an aside, I would recommend moving the discussion here: https://github.com/ravendb/ravendb/discussions/

Your issue is how to page through a (potentially large) list of items, where your `LastModified` value is the same across many values.

I also assume that the actual id of the document is not related to its LastModified date, since you can update the products documents in any order.

Given that you don't care about the order between documents, it there a reason you cannot use:

     var products = await session.Query<Product>()

         .Where(p => p.LastModified > previousLastModified || (p.LastModified = previousLastModified  && p.Id > previousId)) 

         .OrderBy(e => p.LastModified).ThenBy(x => x.Id)
         .Take(pageSize)
         .ToListAsync();

That would achieve the exact same behavior, in a single query. 

[Frederik Maertens]

Hi Oren,

since i'm using "Products|" as a prefix, the C# ' p.Id > previousId'  doesn't work.
The query ends up seeing it as a c# string comparison.

I mean i can see ways around this 

- using an index that strips the prefix from the document Id
  as long as all the objects to sync down to the client use the identity strategy that works   

- manually create a change tracking table 

  avoids the need for all objects to be identity (just the change tracking table would need to be)

- ...
 
Those work but are a bit cumbersome as I need to sync a lot of types down to client. 

It's one of those 'there has to be a simpler way'

Kind regards

Frederik

 

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - an awesome database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/GaBBgKmAwtE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/3ed7d3dc-c5f4-4bd8-b7b7-fdc4efd311e4n%40googlegroups.com.