Multi-Threading CoreData and establish relationship between different contexts

[Do Hoang Minh Quan]

Dear all,

I think I'm in the most terrible problem ever than before. I'm trying to resolve an app that implement CoreData and multi-threading that fetch and modify data in background. The app also has complicated relationship between objects.

I have some majors questions:

1/ Should I turn to SQlite ? Working with CoreData and Multithread (make them thread-safe) is really pain in the ass.

2/ if I have to keep using CoreData, which method will be the best as provided by Apple
+ single persistent coordinator - many context => easy get trouble with "relationship between object" problem even I tried to update context via saved-notification.
+ use different context and persistent coordinate for each thread

In the end, I appreciate for any best practices with example/demo.

Thanks,

Quan

[Hamish Rickerby]

Hi Quan,

You could look at implementing something like MagicalRecord - https://github.com/magicalpanda/MagicalRecord  - It supports contexts for different threads, merging changes across multiple contexts, background saving and in my experience, does it really well. I use it on pretty much every project that I have that uses Core Data.

However, out of your two options below I'd look at Core Data (as long as you're actually wanting to use it as an object persistence mechanism, rather than as a database) with the single persistence controller and many contexts, and figure out why the merging isn't happening correctly off the back of the notifications. This is essentially what MagicalRecord does as well. I've never tried, but from what I've historically read it's quite tricky to get multithreading working correctly with SQLite as a standalone database.

Cheers,

H.

::: Hamish Rickerby :::

twitter: @rickerbh

--
You received this message because you are subscribed to the Google Groups "Australian Cocoaheads" group.
To post to this group, send email to cocoah...@googlegroups.com.
To unsubscribe from this group, send email to cocoaheadsau...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cocoaheadsau?hl=en.

[Rogerio de Paula Assis]

I'll second MagicalRecord but it might be a good idea to understand what's happening behind the scenes too.

The recommended approach is to use one persistent store coordinator and multiple contexts (one for each thread). 

The way MagicalRecord deals with multiple contexts is by storying a reference to the context for that thread on the thread dictionary.

Here's the relevant code (copy & paste from MagicalRecord NSManagedObjectContext category extension)...

NSMutableDictionary *threadDict = [[NSThread currentThread] threadDictionary];
NSManagedObjectContext *threadContext = [threadDict objectForKey:kMagicalRecordManagedObjectContextKey];
if (threadContext == nil)
{
threadContext = [self MR_contextThatNotifiesDefaultContextOnMainThread];
[threadDict setObject:threadContext forKey:kMagicalRecordManagedObjectContextKey];
}
return threadContext;

What are the issues you are having?

[Chris Miles]

Hi Quan,

1. No, sqlite won't magically solve your multi threading issues. CoreData works great multi-threaded, you just need to follow Apple's rules and use it properly.

2. Use a single persistent coordinator. In each thread, create a managed object context, do the work and save it, then throw the moc away. Make sure each moc is created in the thread where it is needed and only used in the thread it is created. Don't pass managed objects between threads (as each are linked to a specific moc). Register for NSManagedObjectContextDidSaveNotification and merge any background thread changes to main thread moc (-[moc mergeChangesFromContextDidSaveNotification:]).

IMO you will be much better off in the long run learning to use Core Data directly than using an even higher level abstraction like MagicalRecord. Once you get the hang of it, CoreData works really well, I use it for all apps, large and small.

Cheers,
Chris

On 20/03/2012, at 5:12 AM, Do Hoang Minh Quan wrote:

[Tony Arnold]

+1 to everything Chris said, however you mentioned issues with "relationship between object(s)".

One caveat I've found is that you need to resolve the permanent IDs of any objects you create in a secondary thread if you wish to pull them back out and reference them again (not sure if it's a bug, or defined behaviour). Basically, after you create an object (or a bunch of objects) on a background thread, but before you save them, execute:

NSError *obtainPermanentIDError;

if (NO == [threadContext obtainPermanentIDsForObjects:[NSArray arrayWithObject:localManagedObject] error:&obtainPermanentIDError]) {

CBLogError(obtainPermanentIDError);

}

This solved a lot of problems resolving objects properly for me.

It might be helpful if you provide a bit more detail about what the "relationship between object(s)" problem is so that we can provide some more targeted advice.

all the best,

Tony

--
You received this message because you are subscribed to the Google Groups "Australian Cocoaheads" group.
To post to this group, send email to cocoah...@googlegroups.com.
To unsubscribe from this group, send email to cocoaheadsau...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cocoaheadsau?hl=en.

----------
Tony Arnold
http://thecocoabots.com/

[John Gronow]

Quan,

I have to agree totally with Chris - CoreData actually works really well with multi-threading, as long as you stick to the rules. I suspect your comment that there are complicated relationships between objects may point to the source of the problems; if the object model is messy then it's going to be harder for Core Data to merge changes and for you to contain objects to threads.

Have a study of the Core Data documentation from Apple, but if that doesn't make it clear enough, I can recommend Marcus Zarra's core data book by the Pragmatic Programmers - it has a whole chapter on Multithreading.

Regards,

John.

On 20/03/2012, at 8:18 AM, Chris Miles wrote:

[Chris Miles]

Hey Tony,

If you request the objectID of a managed object before you have persisted it ([moc save:]) then you will only get back temporary IDs, which is presumably why you needed to force the creation of permanent IDs (which partially persists the object anyway). I prefer saving the moc first, before fetching any objectIDs, as then they will be permanent IDs.

You probably know all that already. So curious as to the logic that requires you to fetch objectIDs before calling save (I've never had a need for it).

Cheers,

Chris

[Tony Arnold]

Hi Chris,

On 20/03/2012, at 9:55 AM, Chris Miles wrote:

> If you request the objectID of a managed object before you have persisted it ([moc save:]) then you will only get back temporary IDs, which is presumably why you needed to force the creation of permanent IDs (which partially persists the object anyway). I prefer saving the moc first, before fetching any objectIDs, as then they will be permanent IDs.
>
> You probably know all that already. So curious as to the logic that requires you to fetch objectIDs before calling save (I've never had a need for it).

There were two reasons —

1. I was batching up changes to my objects and only saving every nth cycle (potentially unnecessary);
2. I recall there being a nasty bug in iOS 5.0 around relationships in Core Data that this method worked around. I cannot for the life of me find the Stack Overflow thread that outlined the problem and this as a fix, though — so take it with a grain of salt :)

It definitely solved some referential integrity errors I was having with saving my model after a specific bunch of changes.

-t

[Do Hoang Minh Quan]

HI Guys,

Thank you very much for your help. I decided to create a second thread and turn all performSelectorInBackground to performSelector in second thread. Beside, I merged information from change notification between 2 contexts and it works very well. No problem at all.

Thanks,

Quan

[Simon Harris]

Howdy folks,

I too am coming up to speed with multithreaded core data stuff. I've been looking at Magical Record for inspiration but I'm also very keen to actually understand Core Data myself. A few questions I have so far:

I've thus far avoided holding a per-thread context in a thread dictionary as I was worried it would be kept around unnecessarily - containing all the managed objects I had fetched. Instead I've been creating a context as needed, using it, then throwing it away. Is there an advantage to holding a context per thread, need I worry about the consequences of doing so?

If I want to merge changes between contexts, the recommended approach is to listen for notifications to specific context rather than all contexts (presumably because system processes, etc. might also send notifications). My question is therefore, given that I will likely have many contexts in play at once, is there a recommended way to have them all listen to each other?

[Chris Miles]

Hi Simon,

On 27/03/2012, at 9:03 AM, Simon Harris wrote:

> I've thus far avoided holding a per-thread context in a thread dictionary as I was worried it would be kept around unnecessarily - containing all the managed objects I had fetched. Instead I've been creating a context as needed, using it, then throwing it away. Is there an advantage to holding a context per thread, need I worry about the consequences of doing so?

I prefer the pattern of only creating a context when it is needed, then throwing it away when the work is complete (excluding the main thread context). The context should be linked to the task, not the thread IMO. This keeps things tidier than attempting to hold a context per thread globally. This also helps avoid the need to merge changes into a non-main thread context (see below). It is also possible (and feasible) to have more than one context per thread, in some cases.

>
> If I want to merge changes between contexts, the recommended approach is to listen for notifications to specific context rather than all contexts (presumably because system processes, etc. might also send notifications). My question is therefore, given that I will likely have many contexts in play at once, is there a recommended way to have them all listen to each other?

As you say, you can register to receive notifications from all contexts, but the docs recommend against it as you may receive notifications about private system framework Core Data changes. You could still do this and just verify the context making the change is one of yours.

However, I recommend only registering for context changes when you definitely need to merge any background changes into a context. Typically, this is only normally needed for the main thread context, so that UI is in sync with the database. If you create and destroy background task contexts as needed (i.e. keep them relatively short-lived) you will have less need to merge changes into those contexts.

Cheers,
Chris

[Simon Harris]

Hey Chris,

Thanks for getting back to me so quickly.

I prefer the pattern of only creating a context when it is needed, then throwing it away when the work is complete (excluding the main thread context). The context should be linked to the task, not the thread IMO. This keeps things tidier than attempting to hold a context per thread globally. This also helps avoid the need to merge changes into a non-main thread context (see below). It is also possible (and feasible) to have more than one context per thread, in some cases.

Excellent. That was my feeling as well. I tried the other approach and it just felt icky. Too much shared state. The short-lived stuff (which Tony also recommended) feels much nicer.

 

However, I recommend only registering for context changes when you definitely need to merge any background changes into a context. Typically, this is only normally needed for the main thread context, so that UI is in sync with the database. If you create and destroy background task contexts as needed (i.e. keep them relatively short-lived) you will have less need to merge changes into those contexts.

Sounds like excellent advice. Now that I think about it, my actual scenario is much simpler. I have a bunch of background threads that may update or delete a record that has just been processed on screen. What I think I really need to do is refresh it and then update it. In my specific case, that will be just as good. Either the record was deleted - in which case I don't need to do anything - or I apply my changes and save.

Thanks again!

[Mark Aufflick]

Three suggestions:

1. Your best bet for new code is not to use threads directly at all - use GCD queues. If you can afford to target iOS5/Lion only you can use the new Main Queue/Private Queue confinement model to avoid the nasty merge stuff you have to do with thread confinement - I have a talk on that recorded from last year's AUC /dev/world conference : http://mark.aufflick.com/talks/well-managed-objects

2. Whatever confinement model you use you can only use objectIDs to refer to objects between MOCs as other people have pointed out.

3. Managed object contexts are counterintuitively pretty lightweight objects - don't go to any trouble to track an MOC per thread/queue for your background processing - just make and dispose a new one for every unit of background work. This approach also makes dealing with errors (eg. your http/json connection drops out mid-way) much easier -- just trash the MOC without saving/merging, and you don't need to do any tricky rollbacks etc.

--
Mark Aufflick
 http://mark.aufflick.com/about/contact
 http://pumptheory.com/about

--

You received this message because you are subscribed to the Google Groups "Australian Cocoaheads" group.

To view this discussion on the web visit https://groups.google.com/d/msg/cocoaheadsau/-/8vdnc3I6wlwJ.

[Jasper Blues]

Do you have the audio for the talk up there? 

[Sean Woodhouse]

If you're targeting iOS 5.x you should also consider nesting your managed object contexts:

Cheers

Sean

Nested Managed Object Contexts

Rather than specifying a persistent store coordinator for a managed object context, you can now specify a parent managed object context using setParentContext:. This means that fetch and save operations are mediated by the parent context instead of a coordinator. This pattern has a number of usage scenarios, including:

• Performing background operations on a second thread or queue.
• Managing discardable edits, such as in an inspector window or view.

As the first scenario implies, a parent context can service requests from children on different threads. You cannot, therefore, use parent contexts created with the thread confinement type (see “Concurrency Support for Managed Object Contexts”).

When you save changes in a context, the changes are only committed “one store up.” If you save a child context, changes are pushed to its parent. These changes are not saved to the persistent store until the root context is saved. (A root managed object context is one whose parent is nil.) In addition, a parent does not pull changes from children before it saves. You must save a child contexts if you want ultimately to commit the changes.

Nested contexts make it more important than ever that you adopt the “pass the baton” approach of accessing a context (by passing a context from one view controller to the next) rather than retrieving it directly from the application delegate.

[Alan Rogers]

And something it doesn't mention in the documentation ->

Saving a parent context doesn't push changes down to children contexts.  You will need to use the good'ol NSMangedObjectContextDidSaveNotification to handle that.

Al

[Simon Harris]

2A couple of questions on this:

* What is the advantage of having child contexts over just creating contexts as needed that save directly to the store?

* Somewhat related. If I have a shared context with a concurrency type of NSPrivateQueueConcurrencyType, I am lead to believe that all operations must use performWithBlock[AndWait]. Does that mean I can't even read NSManagedObject properties unless it's inside a performWithBlock[AndWait]? What are the advantages, if any, of using a shared context in this way?

[mark-...@aufflick.com]

Belatedly re-visiting this old thread.

After a good chat with a Core Data engineer at WWDC it turns out nested contexts are a bad idea in nearly all cases. There are a number of reasons including high-water memory usage and blocking. There are a few blog posts around about the latter issue.

So while nested contexts sound really awesome conceptually, the implementation (which of course we can only observe via side-effects) does not achieve what you really want in most cases.

Mark.

[Simon Harris]

FWIW, and I'm not for one second suggesting that either you or the Apple engineer are incorrect, but in my particular case, it achieved precisely what I needed/wanted. I've had no blocking issues, no memory issues, and everything has been seemingly much simple. No doubt I'm doing something incredibly stupid in the process and it will eventually come back to bite me.

--
You received this message because you are subscribed to the Google Groups "Australian Cocoaheads" group.

To view this discussion on the web visit https://groups.google.com/d/msg/cocoaheadsau/-/kM94kLqM0PEJ.