CBL .Net 4.5 :: What is document limit size in Couchbase Lite?

[Sherry Ummen]

Hello,

 What is document limit size in Couchbase Lite? Does it have the same limit what Couchbase Server has?

-Sherry 

[Jens Alfke]

On Aug 20, 2014, at 12:54 AM, Sherry Ummen <sherry...@gmail.com> wrote:

 What is document limit size in Couchbase Lite? Does it have the same limit what Couchbase Server has?

There's effectively no limit. The documents are stored in a SQLite database, which can handle terabytes of data, and attachments are stored as individual files in the filesystem.

I don't recommend putting huge amounts of JSON data directly into a document; it's slow to parse, which especially slows down the indexer. Use an attachment instead.

—Jens

[Sherry Ummen]

Thanks Jens,

 as per our requirement our software (.net desktop app) should have a local database which can handle huge data and should handle intense read and write since its about various calculations.

And in multi user scenario the users should be to sync the data with each other. Thats the reason we are looking for Couchbase lite.

Now in our case we are writting as Json objects and dumping it to the couchbase lite sqllite and this json object can sometimes be huge depending on the size of the project which as user is workiing on. So do you say or I am wrongly interpretting that Couchbase lite is not a good fit for this purpose?

COuld you please help to give me more insight on this so that I can better understand the COuchbase lite and understand whether we can actually use this.

Please ask me if you want more details about what I am trying to achieve.

-Sherry

[Jens Alfke]

On Aug 20, 2014, at 10:44 AM, Sherry Ummen <sherry...@gmail.com> wrote:

Now in our case we are writting as Json objects and dumping it to the couchbase lite sqllite and this json object can sometimes be huge depending on the size of the project which as user is workiing on. So do you say or I am wrongly interpretting that Couchbase lite is not a good fit for this purpose?

Does the JSON data need to be indexed/queried? If not, you can store it in an attachment. Then your app can just read the attachment as a blob and run it through a JSON parser. (Or use some other encoding format; JSON isn't very efficient.)

If it has a small subset that needs to be queried, you can store that subset as actual document properties and keep the rest as an attachment.

—Jens

[Sherry Ummen]

Does the JSON data need to be indexed/queried?

-> Some background about the data what we store; till now we were storing the data in a specific native data structure in a binary file. Now to turn to modern databases we are testing with databases for example couchbase lite and convert this data to json object for now and store it.

And when we read then we again convert it to our native data structure format. (legacy system pain sob sob..)

Coming to question; the data what we store has some meta data like name,type,datemodified so these meta data are queried often; rest of the data should be in its native format so that system can understand. So yes some amount of query required.

indexing not sure; may be if we have it then we can have better read performance?

 If not, you can store it in an attachment.

how the system will behave in terms of syncing? since we have multi user scenario (mainly cause of which we are looking at couchbase lite) and if we store it as attachments how will it affect it? How will be the speed or performance? Main challenge here is the performance

since the software of ours is heavy calculation intensive software so need to be fast.

your app can just read the attachment as a blob and run it through a JSON parser. (Or use some other encoding format; JSON isn't very efficient.)

Hmm right now we are always converting our native format to json format (a bit of overhead); so if not JSON do you suggest something else? I had it so that we can only store it as JSON only.

And I have a doubt now:

 I see the .net API and the PutLocalDocument take paramter as Dictionary<string,object> so this object can be any C# object type? or should be in a specific json format? (may be a dumb question)

Thanks

[Jens Alfke]

On Aug 21, 2014, at 12:22 AM, Sherry Ummen <sherry...@gmail.com> wrote:

Coming to question; the data what we store has some meta data like name,type,datemodified so these meta data are queried often; rest of the data should be in its native format so that system can understand. So yes some amount of query required.

Then all you need to do is store the queryable attributes in JSON; the rest can go in an attachment. This helps performance because when a document is indexed only the JSON part is parsed.

how the system will behave in terms of syncing? since we have multi user scenario (mainly cause of which we are looking at couchbase lite) and if we store it as attachments how will it affect it? How will be the speed or performance? Main challenge here is the performance

Attachments are a part of the document, so they're always transferred when the document is replicated. Again, using attachments for large data will speed up replication because the attachments never get parsed, they're just transmitted directly as raw data.

Hmm right now we are always converting our native format to json format (a bit of overhead); so if not JSON do you suggest something else? I had it so that we can only store it as JSON only.

Attachments can be any format you want; Couchbase Lite doesn't care what's in them. You can use your old binary format if you like.

 I see the .net API and the PutLocalDocument take paramter as Dictionary<string,object> so this object can be any C# object type? or should be in a specific json format? (may be a dumb question)

I don't know the C# API, but I'm pretty sure it only types that parameter as 'object' because there's no specific way to say "boolean, number, string, array or dictionary". It's going to get converted to JSON so it can only contain types that can be represented as JSON.

—Jens