How to disable notify changes ([CBLDatabase(Internal) notifyChanges:])?

[Daniel Sonny Agliardi]

I was analyzing performances of my app with Instruments and i noticed that a method needs a lot time to be completed, this method is [CBLDatabase(Internal) notifyChanges:] (see the attached photo below). 

There is a way to disable it? In particular because i assume that method is used for live queries feature and i don't currently use it.

[Jens Alfke]

On Jul 15, 2015, at 6:48 AM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:

I was analyzing performances of my app with Instruments and i noticed that a method needs a lot time to be completed, this method is [CBLDatabase(Internal) notifyChanges:] (see the attached photo below). 

In that profile, it would have been interesting to drill down into -[NSNotificationCenter postNotificationName:…], which is where 97% of the time is spent. The actual notification observer methods are called during this, and it’s probably one or more of those observers that’s consuming the CPU. Those could be observers in your own code, which would indicate that your code should be optimized.

If you profile this again and need help interpreting it, save the profile and email me the file and I’ll take a look.

There is a way to disable it? In particular because i assume that method is used for live queries feature and i don't currently use it.

No, it can’t be disabled. It’s used for a lot of other things too, like

• Posting CBLDatabaseChanged notifications
• Posting CBLDocumentChanged notifications
• Updating CBLModel objects’ properties
• Notifying a continuous push replication that there are new revisions to push

—Jens

[Daniel Sonny Agliardi]

Thanks you for precise replay.

I have recreate the situation and in the screen you can see what operations are done deeper.

This "issue" happens when i do about 30 UPDATE queries in a for.

[Daniel Sonny Agliardi]

[Jens Alfke]

On Jul 16, 2015, at 12:59 AM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:

This "issue" happens when i do about 30 UPDATE queries in a for.

Can you show what your code looks like? I’m not sure exactly what you’re calling because there’s nothing in Couchbase Lite called literally “UPDATE”.

Also, are you saying it takes 12 seconds (the time shown in the profile) to do 30 document updates? That’s way off from our benchmarks; you should be able to update something like 1,000 docs a second (depending on the device and the size of the docs of course.)

—Jens

[Daniel Sonny Agliardi]

With UPDATE i refer to this method (that i wrote to act like an update): 

- (BOOL)saveExeciseTemplate:(ExerciseTemplate *) exerciseTemplate{

   

    NSError *error;

   

    // create an object that contains data for the new document

   

    NSDictionary *myDictionary = [[exerciseTemplate asDictionary] copy];

   

   

    // Create an empty document

    CBLDocument *doc = [[self getDatabase] documentWithID:exerciseTemplate.name];

   

    // Save the ID of the new document

    NSString *docID = [[myDictionary objectForKey:@"template_id"] copy];

   

   

   

    // Write the document to the database

    if(myDictionary && doc){

       //DDLogInfo(@"Saving ID = %@...", docID);

       

        CBLRevision *newRevision = [doc putProperties: myDictionary error:

                                    &error];

        if (newRevision) {

           //DDLogInfo(@"Document created and written to database, ID = %@", docID);

           

            //return true;

        }

       

        if (error) {

           //DDLogInfo(@"Document NOT created, error = %@", error.localizedDescription);

           

           

            //if this exerciseTemplate is already in DB, then update it

            if (error.code == 409) {

                [doc deleteDocument:&error];

               

                doc = [[self getDatabase] documentWithID:docID];

               

                CBLSavedRevision *newRev = [doc putProperties:myDictionary error:&error];

               

                if (!newRev) {

                   //DDLogInfo(@"Cannot update document. Error message: %@", error.localizedDescription);

                } else {

                   //DDLogInfo(@"Document updated and written to database, ID = %@", docID);

                    return true;

                }

            }

        }

    }

   

    return false;

}

[Daniel Sonny Agliardi]

I just call that method in a for, after a HTTP GET (i get JSON that contains some exerciseTemplates), 

NSURLSessionDataTask *exerciseTask = [[SessionManager sharedSessionManager] GET:EXERCISE_LIST_PATH parameters:params success:^(NSURLSessionDataTask *task, id responseObject) {

            NSDictionary *JSON = (NSDictionary *)responseObject;

            [self parseExercises:JSON];

        } failure:^(NSURLSessionDataTask *task, NSError *error) {

            DDLogError(@"Could not load exercises!");

        }];

and here the parsing method: 

- (void)parseExercises:(NSDictionary *)JSON {

    NSArray *tempArray = [JSON valueForKey:@"exercises"];

  for (NSDictionary *dict in tempArray) {

ExerciseTemplate *template = [[ExerciseTemplate alloc] init];

           

            NSString *exerciseName = [dict valueForKey:@"name"];

            NSString *exercise_id = [dict valueForKey:@"id"];

            NSNumber *dynamic = [dict valueForKey:@"dynamic"];

            NSNumber *uniateral = [dict valueForKey:@"unilateral"];

            NSString *content = [dict valueForKey:@"content"];

            NSNumber *repetitionMaximum = [dict valueForKey:@"repetition_maximum"];

            NSDictionary *thresholdDict = [dict valueForKey:@"thresholds"];

            NSDictionary *polygonDict = [dict valueForKey:@"polygon"];

           

            template.name = exerciseName;

            template.template_id = exercise_id;

            template.dynamic = dynamic.boolValue;

            template.unilateral = uniateral.boolValue;

            template.content = content;

            template.repetitionMaximum = repetitionMaximum.floatValue;

            template.thresholdDict = thresholdDict;

            template.polygonExplosiveness = [polygonDict objectForKey:@"ex"];

            template.polygonPower = [polygonDict objectForKey:@"po"];

            template.polygonResistance = [polygonDict objectForKey:@"re"];

            template.polygonStrength = [polygonDict objectForKey:@"st"];

            template.polygonVelocity = [polygonDict objectForKey:@"sp"];

       

            //[transaction setObject:template forKey:template.template_id                      inCollection:EXERCISE_TEMPLATE_COLLECTION];

           

            [[TPADatabaseManager sharedManager] saveExeciseTemplate:template];

        //}];

    }

}

Yes, i know, i can just skip the "parsing"/allocation/init of an exerciseTemplate, and put in DB value from Dictionary, that i do in for's body but i do not think is that my problem. Am i right?

I'm currently trying some other strategies to lighten the algorithm.

Thank you for the help.

[Jens Alfke]

If you wrap all the document updates in a transaction it should go much faster.

--Jens

[Daniel Sonny Agliardi]

I wrapped that update in a transaction, but i have still performance issue in that operation. This is the Instruments' row:

 68556.0ms   82.6% 0,0            -[CBLDatabase(Internal) postChangeNotifications] 

[Jens Alfke]

> On Jul 22, 2015, at 7:58 AM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:
>
> I wrapped that update in a transaction, but i have still performance issue in that operation. This is the Instruments' row:
> 68556.0ms 82.6% 0,0 -[CBLDatabase(Internal) postChangeNotifications]

Is it possible for you to send me the project so I can investigate? (I can sign an NDA if necessary.)

—Jens

[Daniel Sonny Agliardi]

To share with you I have to ask before to my bosses (I will do tomorrow).

Basically my problem is that the given UPDATE method (found here http://developer.couchbase.com/mobile/develop/training/build-first-ios-app/do-crud/index.html#update) raises an error (409 - conflict) so I'm not able to proper UPDATE documents in database with some new versions of properties. I came up with a DELETE + INSERT to archive the same result of an UPDATE as you can in posted code.

I think the the point is: DELETE and INSERT have a way bigger notification time (I refer to my performance issue) then just an update, even if you put both operations in a single transaction.

If can help me first, I can, maybe, fix this problem sooner by remove the DELETE and INSERT operations.

(Ps I write in upper case the SQL's primitive function/operation that I used to handle, to, hopefully better, explay what I want to say)

[Jens Alfke]

On Jul 22, 2015, at 10:58 AM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:

Basically my problem is that the given UPDATE method (found here http://developer.couchbase.com/mobile/develop/training/build-first-ios-app/do-crud/index.html#update) raises an error (409 - conflict)

That means the revision you’re updating isn’t the current revision. Or the properties dictionary you’re giving it doesn’t have a “_rev” property at all. Either way it’s a bug in your code; what does the code look like?

I came up with a DELETE + INSERT to archive the same result of an UPDATE as you can in posted code.

That’s not the same thing. Delete doesn’t work the way it does in a SQL database, because Couchbase Lite tracks the revision histories of documents. So deleting adds a ‘tombstone’ revision, and then creating the document again creates a new revision on top of that.

(If it helps, think of this in terms of a version control system: “git rm” is not the same thing as “rm”.)

—Jens

[Daniel Sonny Agliardi]

// Updates the document

- (BOOL) updateWorkout:(NSDictionary *) workout {

NSError *error;


// retrieve the document from the database

CBLDocument *getDocument = [database documentWithID: [[workout objectForKey:@"workout"] objectForKey:@"workout_id"];


// make a mutable copy of the properties from the
 document we just retrieved

NSMutableDictionary *docContent = [getDocument.properties
 mutableCopy];


// modify the document properties

docContent[@"exercises"] = [[workout objectForKey:@"workout"] objectForKey:@"exercises"];
// save the Document revision to the database

CBLSavedRevision *newRev = [getDocument putProperties:docContent
 error:&error];


if (!newRev) {

NSLog(@"Cannot update document. Error message: %@",
 error.localizedDescription);

}


// display the new revision of the document

NSLog(@"The new revision of the document contains: %@",
 newRev.properties);


return YES;

}

The given workout is a wrapper dictionary with a key @"workout" and the value is a dictionary that contains some keys-values.

This method is pretty the same as the tutorial's one.

Have I to put manually a _rev property in document at the moment of insertion?

[Jens Alfke]

That code is reasonable, except in cases where another thread could be modifying the document. In that case the document might be updated by that thread after you access its properties and before you put the updated properties. The conflict error is to prevent your update from clobbering the one made by the other thread.

The most common situation where a background thread can update the document is if you’ve got a pull replication running, and the replicator (which is in the background) pulls a new revision of the document.

This does happen but since it’s a race condition it isn’t consistent, and in my experience it isn’t common. So I’m wondering why you consistently have this problem. Do you use multiple threads yourself?

—Jens

[Daniel Sonny Agliardi]

No, it's not my case. I have only a thread that access to DB during that UPDATE.

Basically I have to do 2 UPDATEs and in both cases I have only the main thread using DB.

There is a way to find out the reason 409 error in my code? The UPDATE seems pretty easy to understand (getting document, extract properties, update locally the properties, and put properties back in document).
Do the entire UPDATE method in a transaction can help? I haven't tried this way.

[Jens Alfke]

> On Jul 22, 2015, at 10:37 PM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:
>
> There is a way to find out the reason 409 error in my code? The UPDATE seems pretty easy to understand (getting document, extract properties, update locally the properties, and put properties back in document).

A conflict error happens when you try to update a document but the value of the _rev property in the dictionary you provide isn’t equal to the _rev property of the document on disk. That is, when the revision you’re updating isn’t the current revision.

When a document is saved successfully, the _rev property is changed to a new unique revision ID before it’s written to disk.

Hopefully that helps you understand the way things work.

Another thing that can cause conflicts is when you hang onto the properties dictionary in between updates and try to make two updates using it:
props = doc.properties.mutableCopy
props[“x”] = “y”;
[doc putProperties: props]; —> OK
…
props[“a”] = “b”;
[doc putProperties: props]; —> Conflict
The reason is that when the second putProperties is called, props still has the old _rev value, not the updated one from the first save.

—Jens

[Daniel Sonny Agliardi]

How can I send you our project if you are so gentle to help us with DB performance issues?

[Jens Alfke]

On Sep 2, 2015, at 10:48 AM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:

How can I send you our project if you are so gentle to help us with DB performance issues?

Email should be fine. Or if it’s in a private Github repo, you can invite me to join your team there.

Let me know how to run the app to trigger the slow operation.

Before doing this, it’ll help if you can update to the latest 1.1.1 build, so we'll know we’re not dealing with any already-fixed issues.

—Jens

[Daniel Sonny Agliardi]

I sent you a private mail. Thanks you again!

[Daniel Sonny Agliardi]

Any chance to get some help about it? Have you received the private mail?

Thank you.

[Jens Alfke]

> On Sep 21, 2015, at 10:55 PM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:
>
> Any chance to get some help about it? Have you received the private mail?

I did, but I’ve had a bunch of other stuff on my plate. Did you file a bug report on Github? That’s how we schedule what issues to work on during our sprints, so if there’s not an issue filed for this, it’s not going to officially appear in my to-do list.

—Jens

[Daniel Sonny Agliardi]

No, i have not opened a issue on Github, because i still don't understand if this issue is my fault (due to an inefficient algorithm i used to insert or update records in DB or something other) or not. 

I created a Bitbucket repository to show our code and i can invite you anytime, just give me a mail address (here in private mail or on skype). As i said in private, i have encapsulated all CURD methods in class, so i think you can evaluate the situation very quickly.

Take your time, no worries at all.

[Jens Alfke]

On Sep 22, 2015, at 8:59 AM, Daniel Sonny Agliardi <dan...@gmail.com> wrote:

No, i have not opened a issue on Github, because i still don't understand if this issue is my fault (due to an inefficient algorithm i used to insert or update records in DB or something other) or not. 

It’s OK, file an issue. Something’s not behaving right, and it’s not obviously a problem in your code, so we need to investigate whether it’s with CBL itself. And having it in the issue tracker makes it easier for us to schedule time to look at it.

—Jens

[Daniel Sonny Agliardi]

Nice, so I'll open a issue ASAP. Thank you for the support.