Mysterious Core Data fault exceptions

[Paul Cantrell]

Does [NSManagedObjectContext save:] finish up its work asynchronously? If so, is there a way to know when a Core Data save is complete?

Here's my situation:

My app includes a Core Data type (call it ThingerImage) which holds some large, expensive NSData objects. They take a while to generate, and I don't want that slowing the UI, so I do something like this:

1 Main thread: request ThingerImage
2 Background thread: create ThingerImage
3 Background thread: call [NSManagedObjectContext save:]
4 Background thread: pass objectID of newly created ThingerImage back to main thread
5 Main thread: attach ThingerImage to its parent Thinger

As the docs recommend, I have a different NSManagedObjectContext for each thread, sharing the same NSPersistentStoreCoordinator underneath. And as the docs recommend, I only pass NSManagedObjectIDs between threads.

The puzzling thing is, I sometimes get a NSObjectInaccessibleException during step 5:

CoreData could not fulfill a fault for '0x13dbe0 <x-coredata://53821256-6669-4D90-8D99-A3C99185AAC1/ThingerImage/p2053>'

At first I thought perhaps step 3 was failing silently. But then I found that when I get this error, I can simply pause for one second, then retry step 5, and it works. Yikes!

This suggests that there's some kind of race condition in my code. I'm guessing that either Core Data or sqlite itself writes out large rows asynchronously, and [NSManagedObjectContext save:] returns before it's truly done. Anybody know if this is the case? Any other theories?

Cheers,

Paul

[Aaron Kardell]

I can't answer your specific question, but I can say that Apple's engineers seem to suggest not storing large objects with Core Data (large being >100KB).  Though the docs say something like there not being size restrictions or a size restriction of around 2GB, when you start asking around on the Apple forums Apple's engineers suggest the "right way" to do it is to store large objects on the file system and references to the file system location in Core Data.

Probably not the answer you want to hear, but maybe this different approach is worth considering.

Aaron

--
You received this message because you are subscribed to the Google Groups "iphonedevmn" group.
To post to this group, send email to iphon...@googlegroups.com.
To unsubscribe from this group, send email to iphonedevmn...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/iphonedevmn?hl=en.

[Paul Cantrell]

No, not the answer I want of course, since it's so easy the way I'm doing it — but that is my fallback position. I'm not excited about taking on the housekeeping of orphaned files, etc.

Cheers, P

[John Sheets]

Good timing!  I spent all of this past weekend pounding my head against this same problem.  I am downloading stuff in the background with NSOperation and then handing them off to the main thread as they complete.  I finally got it working with a processPendingChanges in the operation, and a refreshObject in the main thread.  Both were important and not quite intuitive, but obvious in hindsight.

Here's the gist of what I did in the NSOperation subclass.  I watch the thread context for notification of the context save, then explicitly flush the pending changes with processPendingChanges (the critical part).  Being used to Hibernate and such, I had expected Core Data to automatically flush the pending changes but I guess that doesn't happen.  Or...may happen later, asynchronously...?

=====

- (void)performOperation

{

    // Main Loop.

    // ...

    // Set up callback to merge changes back into main thread.  After we

    // save the context below, the NC calls contextDidSave: then the op

    // removes itself as an observer.

    NSNotificationCenter *center = [NSNotificationCenter defaultCenter];

    [center addObserver:self

               selector:@selector(contextDidSave:)

                   name:NSManagedObjectContextDidSaveNotification

                 object:self.threadContext];

    // ...

}

- (void)contextDidSave:(NSNotification*)notification

{

    // Flush changes in the thread's context, then merge to main thread.

    [self.threadContext processPendingChanges];

    [self.threadContext performSelectorOnMainThread:@selector(mergeChangesFromContextDidSaveNotification:) withObject:notification waitUntilDone:YES];

    // Trigger the main-thread object loading.  Can't rely on notification center to call this

    // in the proper order, so call it manually, synchronously, here.

    [self.viewThingie performSelectorOnMainThread:@selector(contextDidSave:) withObject:notification waitUntilDone:YES];

    [[NSNotificationCenter defaultCenter] removeObserver:self];

}

=====

Then I synchronously passed on the notification to the owning view to reload its main thread version of the object with the objectID (passed along in the NSNotification dict).  I didn't really need to have the same contextDidSave: method signature in my view too; it's a relic of when I had both the NSOperation and the view listening for the save notification.  Since I need the op to fire first, I just switched to invoke the view's version manually.

=====

// Can now load Core Data object in the view.

- (void)contextDidSave:(NSNotification*)notification

{

    DataManager *dataManager = self.dataManager;

    

    NSArray *updatedObjects = [[notification userInfo] valueForKey:@"updated"];

    for (id updatedObject in updatedObjects)

    {

        // Ignore other Core Data model classes.

        if (![updatedObject isKindOfClass:[MyWidget class]])

        {

            continue;

        }

        

        MyWidget *threadObject = updatedObject;

        

        // Reload MyWidget freshly in local main thread.

        NSManagedObjectID *objectId = [threadObject objectID];

        MyWidget *localObject = (MyWidget*)[dataManager.managedObjectContext objectWithID:objectId];

        // Force the refresh.  Probably necessary to force the fault; otherwise we're

        // still using the version cached in memory, passed from one context to the other.

        [dataManager.managedObjectContext refreshObject:localObject mergeChanges:YES];

        

        // ...

    }

}

=====

I have no idea how well the above follows proper Apple idioms, but it does seem to work in my case.  In particular, it felt wrong to manually call the view from the op's contextDidSave: but I'm not sure how else I should have done it.

Hope this helps!

John

--
You received this message because you are subscribed to the Google Groups "iphonedevmn" group.
To post to this group, send email to iphon...@googlegroups.com.
To unsubscribe from this group, send email to iphonedevmn...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/iphonedevmn?hl=en.

-- 

John Sheets

H: fis...@metacasa.net

W: john....@space150.com

"The Internet is not something you just dump something

on.  It's not a big truck.  It's a series of tubes."

 --Senator Ted Stevens, R-AK

[Paul Cantrell]

John —— Thanks for this extensive suggestion. What you did certainly does seem much harder than it should be (I too hang on to Hibernate as my idea of an ORM), but it makes sense in principle. The details sure are puzzling, though. I tried some experiments in my code, and it seems that:

(1) NSManagedObjectContextDidSaveNotification is always fired during context save, not at some later time after the save completes. I though using the notification might be causing your code to wait for a long-running asynchronous operation to complete, but that seems not to be the case.

(2) Calling processPendingChanges on the receiving thread may well be the key, although I couldn't get that alone to fix my issue. You said the refresh was also necessary, but in may case, the big objects are created for the first time on the background thread, and I'm getting objects that are too *new* and reference big rows that aren't saved yet. So it sure seems lik the refresh shouldn't be necessary.

I suspended investigations in the midst of all that, however, when I found a showstopper: even though my operation is on a background thread, Core Data operations are still blocking the main thread. I'm guessing it's because of the shared NSPersistentStoreCoordinator.

Given that, and all the troubles preceding, I'm going to attempt the file-based solution and see how that goes. It will let the background thread be completely free of Core Data operations; it just writes an image file to a given path and calls it good. That should nix all these headaches.

Thanks for walking me through your approach. It saved me several hours of experimenting.

Cheers,

Paul

_________________________________________________________________

"Verbing weirds language."  —Bill Watterson

[John Sheets]

On Jan 15, 2010, at 11:16 AM, Paul Cantrell wrote:

(1) NSManagedObjectContextDidSaveNotification is always fired during context save, not at some later time after the save completes. I though using the notification might be causing your code to wait for a long-running asynchronous operation to complete, but that seems not to be the case.

It caught me off guard too that the notifications were sort of synchronous.  Also I (wrongly?) assumed that the notification center would invoke its listeners in the same order that I added them.  Maybe they do, maybe they don't.  A lot of stuff was flying around while I was trying to debug all this, so I didn't get a concrete read on that.  Has anyone experimented with the timing/order of notification orders?

(2) Calling processPendingChanges on the receiving thread may well be the key, although I couldn't get that alone to fix my issue. You said the refresh was also necessary, but in may case, the big objects are created for the first time on the background thread, and I'm getting objects that are too *new* and reference big rows that aren't saved yet. So it sure seems lik the refresh shouldn't be necessary.

I wonder if the refresh might still be necessary in the main thread.  Possibly worth a try.  The context merging seems to invoke some strange voodoo, especially when coupled with its aggressive Core Data object caching.  It might be merging and loading your primary objects, but not their associations...?

In my case, I have a Core Data "Photo" model with a one-to-one association to a CoreData "ImageBytes" model whose sole purpose is to (lazily) hold the image data.  The background thread loads the image data then signals the main thread to load it in.  Without both processPendingChanges and refreshObject: I was getting the Photo on the main thread, but not its association.  Is that similar to what you're seeing?

All this complexity, and I've only just scratched the surface of Core Data.  Head-spinny good-ness.  (c:

John

[Paul Cantrell]

That's all remarkably similar to my situation, and the symptoms I'm seeing. Yes, I'm getting the updated parent object but not its association. So maybe refreshObject: is critical, although that sure seems fishy: why should I need to refresh a parent to get its association?

Regardless, I'm now a bit down the road of using files, and having much better luck. I'll post some code once I have it working smoothly.

P

--
You received this message because you are subscribed to the Google Groups "iphonedevmn" group.
To post to this group, send email to iphon...@googlegroups.com.
To unsubscribe from this group, send email to iphonedevmn...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/iphonedevmn?hl=en.

_________________________________________________________________

“Democracies die behind closed doors.” — Judge Damon Keith

[John Sheets]

On Jan 15, 2010, at 12:52 PM, Paul Cantrell wrote:

That's all remarkably similar to my situation, and the symptoms I'm seeing. Yes, I'm getting the updated parent object but not its association. So maybe refreshObject: is critical, although that sure seems fishy: why should I need to refresh a parent to get its association?

Those were my exact thoughts, and part of why it took me so long to figure it out.  Also I was expecting (per Hibernate and per general implications in Core Data docs) that simply referencing the association would load it in at that point.  Yet it always came back nil.  I think in my case I was initially creating the parent object in the main thread, but instantiating and associating the child object in the background thread.

I don't even want to think about all the cargo culting I did to finally get it working.  I felt (and sounded) like a monkey on a xylophone.

Regardless, I'm now a bit down the road of using files, and having much better luck. I'll post some code once I have it working smoothly.

Interesting.  I actually started with a straight file downloading cache, then moved to Core Data binary blobs.  IIRC the file cache was proving awkward to keep in sync with the Core Data objects (some of which are persistent and some of which are transient, per session).

I'm starting to wonder if we're working on the same app.  (c;

[Paul Cantrell]

I don't have the problem of some core data objects being transient, so syncing isn't so hard.

My approach is to make the code gracefully handle the case where I have a ThingerImage but no corresponding file, and always ensure that the ThingerImage is committed before the image gets saved. Then I don't have to worry about orphan files with no corresponding row; I only need to delete the associated file when the row gets deleted.

So in my ThingerImage, I do this:

- (NSString*) imagePath

    {

    if([self.objectID isTemporaryID]) // sanity check

        {

        NSLog(@"WARNING: shouldn't be trying to get image from transient object");

        return nil;

        }

    

    static NSString *imageBaseDir = nil;

    if(!imageBaseDir)

        {

        NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);

        imageBaseDir = [[[paths objectAtIndex:0] stringByAppendingPathComponent:@"Images"] retain];

        if(![[NSFileManager defaultManager] fileExistsAtPath:imageBaseDir])

            [[NSFileManager defaultManager] createDirectoryAtPath:imageBaseDir attributes:nil];

        }

    

    return [imageBaseDir stringByAppendingPathComponent:

        [NSString stringWithFormat:@"%@.png",

            [self.objectID.URIRepresentation.relativePath stringByReplacingOccurrencesOfString:@"/" withString:@"_"]]];

    }

- (UIImage*) image

    {

    NSString *imagePath = self.imagePath;

    if([[NSFileManager defaultManager] fileExistsAtPath:imagePath])

        return [UIImage imageWithData:[NSData dataWithContentsOfFile:imagePath]];

    else

        return nil;

    }

- (void) prepareForDeletion

    {

    NSError *error = nil;

    [[NSFileManager defaultManager] removeItemAtPath:self.imagePath error:&error];

    if(error)

        NSLog(@"WARNING: Unable to delete image file \"%@\": %@", self.imagePath, error);

    [super prepareForDeletion];

    }

Then, after I save a new ThingerImage, I enqueue a task to create it:

    ThumbnailWorkerTask *task = [[[ThumbnailWorkerTask alloc] init] autorelease];

    task.image = image;

    task.imagePath = rf.imagePath;

    [worker enqueueTask:task];

...and on the background thread (note the total absence of anything Core Data, and the absence of any sort of callback to the main thread):

    ThumbnailWorkerTask *task = ...next task from queue...;

    [UIImagePNGRepresentation(task.image) writeToFile:task.imagePath atomically:YES];

I like this approach. It's simple, and I know what the heck is going on.

Paul