Compaction progress?

[Foosh]

Is there a way to monitor compaction status and when it completes? Or even a way to simply tell whether a database is in the process of compacting? Our iOS app purges data and runs compaction daily but the app locks up long enough that I'd like to display an activity indicator while it's compacting and dismiss it when compaction finishes. 

Thanks!

Bill

[Jens Alfke]

On Sep 5, 2014, at 9:31 AM, Foosh <bill....@kiddienotes.com> wrote:

Is there a way to monitor compaction status and when it completes?

No :( I don't think I could add any effective progress indicator, because most of the time seems to be spent in one or two SQL statements.

Or even a way to simply tell whether a database is in the process of compacting?

It's compacting if you called -compact and the call hasn't returned yet :) You could define a global flag and set it before the call and clear it after.

I know compaction performance isn't very good in 1.0 … ForestDB will take care of this problem, because it does auto-compaction in the background without blocking access to the database. (A preview of CBL+ForestDB should be available very soon. I'm just waiting on the ForestDB team to declare beta and open their repo to the public.)

—Jens

[Frederic Yesid Peña Sánchez]

Hi.

I'm trying to run compact on Sync, but seems only limiting to 1000 revisions

I see you say on some Github issue to use the "revs_limit" database config, but even setting that to 50 and restarting sync, compaction keeps 1000 revisions instead of the desired 50

this is my sync config.json

{
        "interface":":4984",
        "adminInterface":":4985",
        "maxConnections":1024,
        "log":["REST","CRUD"],
        "databases":{
                "pharmadvisor-mobile2":{
                        "server":"http://x:8091",
                        "bucket":"pharmadvisor-mobile2",
                        "username":"xxxx-mobile2",
                        "password":"xxxx",
                        "revs_limit":50,
                        "sync":`
function(doc) {
        channel(doc.pmsync_channels);
        access(doc.pmsync_usuario, doc.pmsync_channels);
        access(doc.pmsync_usuarios, doc.pmsync_channels);
}`

[Jens Alfke]

On Apr 16, 2015, at 5:31 PM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

I see you say on some Github issue to use the "revs_limit" database config, but even setting that to 50 and restarting sync, compaction keeps 1000 revisions instead of the desired 50

This is on Sync Gateway?

How do you know how many revisions SG is keeping?

(In the one case I’ve looked at where too many revs were being kept, it was because a lot of them were conflicts. Compaction doesn’t delete any ‘leaf’ revisions, including those from unresolved conflicts.)

—Jens

[Frederic Yesid Peña Sánchez]

Well, i have queried a document and has about 1000 revisions, i dont know if those aré conflicts, how to be sure?

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/GnT4-qYJwgA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/00482D2D-DDFA-4639-9098-691ABA35E91A%40couchbase.com.
For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

On Apr 16, 2015, at 6:58 PM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

Well, i have queried a document and has about 1000 revisions, i dont know if those aré conflicts, how to be sure?

There are a couple of ways; probably the easiest is to GET the doc with ?open_revs=all. This will return a JSON array with one element for each leaf/conflict revision.

—Jens

[Frederic Yesid Peña Sánchez]

Thanks Jens

But SyncGateway responds:

"open_revs=all unimplemented"

[Jens Alfke]

On Apr 20, 2015, at 9:07 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

But SyncGateway responds:
"open_revs=all unimplemented”

Oops, looks like that was just implemented recently (Feb 19). Well, as an alternative you can get the _changes feed with ?style=all_docs and it will include all the conflicting revisions for every document.

—Jens

[Frederic Yesid Peña Sánchez]

Ah so that commit is not on any release yet..

I'm figuring how SyncGateway removes old revisions (reading go-language from repo)

Something about "_sync:rev:" documents, may be trying to do some "voodoo" and remove all those old revisions...

--

You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/GnT4-qYJwgA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/10A0D257-73C4-493F-A195-32F4606FD67C%40couchbase.com.

[Frederic Yesid Peña Sánchez]

While reading the compaction code, i notice it removes the revisions who reside on separate documents.

How are the revisions stored in Sync/Couchbase?

El lunes, 20 de abril de 2015, 12:09:28 (UTC-5), Frederic Yesid Peña Sánchez escribió:

Ah so that commit is not on any release yet..

I'm figuring how SyncGateway removes old revisions (reading go-language from repo)

Something about "_sync:rev:" documents, may be trying to do some "voodoo" and remove all those old revisions...

2015-04-20 11:55 GMT-05:00 Jens Alfke <je...@couchbase.com>:

On Apr 20, 2015, at 9:07 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

But SyncGateway responds:
"open_revs=all unimplemented”

Oops, looks like that was just implemented recently (Feb 19). Well, as an alternative you can get the _changes feed with ?style=all_docs and it will include all the conflicting revisions for every document.

—Jens

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/GnT4-qYJwgA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

[Jens Alfke]

On Apr 20, 2015, at 10:56 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

While reading the compaction code, i notice it removes the revisions who reside on separate documents.
How are the revisions stored in Sync/Couchbase?

The key “foo” in the bucket maps to document “foo” in the database. It contains an extra internal property “_sync” that stores metadata the Gateway uses. Inside that, “revisions” (sp?) stores the revision tree in a weird encoding that was designed to save space.

The revision tree only stores a revision’s body if it’s an active conflict. Other revisions (ones that have been updated) have their bodies moved out into separate docs in the bucket, whose keys are of the form “_sync:rev:” followed by the docID and revID. These docs have an expiration time of 5 minutes so Couchbase Server cleans them up automatically.

The revision tree itself gets ‘pruned’ during every document update. Nodes that are too far from any leaf are deleted. Note that active conflicts are leaves, so if the doc has a lot of conflicts going back through its history, they’ll effectively keep anything from getting pruned.

—Jens

[Frederic Yesid Peña Sánchez]

Hi again.

So the thing i can see in fact is the revision tree without any revisions in fact.

So the issue here is that the "revision" info, that "list" of revisions is getting too long (and not that the document itself is holding a lot of revisions), and that may be collapsing my DB, so i'ts safe to trim that revision tree?? cut from 1000 lines to 20 for example??

Thanks.

[Frederic Yesid Peña Sánchez]

I've reading the wiki con Sync, and came across an idea:

If i trim the revs, parents and deleted arrays at the same time, leaving the last 20 items, do this will trim my history tree?.

Altough i'm asking, right now i'm doing that for research.

You will see, we have a lot of big documents because of these arrays...

--

You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/GnT4-qYJwgA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/649d1bf2-88ca-475a-84f3-1bc1764c1ad4%40googlegroups.com.

[Frederic Yesid Peña Sánchez]

So far, tried with a small tree, all seems good, but a larger tree messed the sync.

Think has to be with not preserving the "deleted" relationship.

Fixing that relation an trying again.

[Jens Alfke]

On Apr 20, 2015, at 12:28 PM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

So the issue here is that the "revision" info, that "list" of revisions is getting too long (and not that the document itself is holding a lot of revisions), and that may be collapsing my DB, so i'ts safe to trim that revision tree?? cut from 1000 lines to 20 for example??

Unless there are conflicts, it shouldn’t be getting longer than the RevsLimit configured for the database. Did you edit your configuration to set a lower RevsLimit? (And note that the change doesn’t apply to a document until the next time it’s saved.)

If i trim the revs, parents and deleted arrays at the same time, leaving the last 20 items, do this will trim my history tree?.

No! This will just break the revision tree and cause Go panics when accessing the document. Please read the comments on the revTreeList struct in revtree.go. "Ordering in the arrays doesn't matter.” (It’s random, determined only by the order in which the hash table is enumerated.)

It is possible to edit the JSON to trim old revisions, but it will take a good knowledge of the schema it’s written in. Hopefully the revTreeList struct and the MarshalJSON method have enough info to figure out the details.

I don’t want to tell you not to hack on this :) but if you do so, you’re entering the territory of a SG developer. It would be great if you found bugs or inefficiencies in managing the rev tree, and even better if you sent us fixes, but please don’t do this to any database you plan to use in production. (Or if you do, don’t ask us for help with resulting problems because they’re going to be impossible for us to troubleshoot without understanding the changes you’ve made to the code…)

—Jens

[Frederic Yesid Peña Sánchez]

Hi Jens

I'm not planning on modifying the SG code, i'm modifying the docs directly with a tiny script made in PHP.

I took measures in case of errors, and made backups of every document before modifiying them.

I'm studying the specs you told me about revsTree and MarshalJSON.

I'm not ever "knowful" of go, so i prefer to use php api.

I'll tell you how i'm doing, without asking you about errors. ;)

--

You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/GnT4-qYJwgA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/48960EBA-CAF6-4DB4-90E8-94E051658C4C%40couchbase.com.

[Frederic Yesid Peña Sánchez]

Hi Jens.

I did it!!, my script works without damaging the documents, not too fast, but decreased document size

Here is the gist: https://gist.github.com/razorblade446/b2e9c3d96adf7c8eb82d

If you could give me your opinion, will be great!!!

To unsubscribe from this group and all its topics, send an email to mobile-couchbase+unsubscribe@googlegroups.com.