Importing large datasets into CoreData.

[Anush Saidmuradov]

The project I am working deals with a lot of data that will be stored on the phone. What are some strategies to do large imports of data into Core Data via the Network?

Are there any guides out there on how to work with background threads with RubyMotion?

[Fernando Espinosa]

Your best chance here is Grand Central Dispatch. Here is a nice tutorial1.

Cheers!

---

Fer
http://twitter.com/ferdev

[sas]

Are you asking about RubyMotion specifically or the import more
generally?

More generally, you should set up a GCD queue to handle your import
task and be careful to either use a single managed object context just
for that queue or make sure you use a new managed object context per
save block (see [1] [2])

You should also set up a main context which is tied to the main thread
for all your core data accessors and merge any changes happening on
your save thread in the handler of the
NSManagedObjectContextObjectsDidChangeNotification by calling
mergeChangesFromContextDidSaveNotification for your main context with
the notification you've been given.

But in order to make the import smoother, only set up this
notification handler after you're done importing in order to avoid
generating a lot notification traffic while the import is still in
progress. This could for example be a problem if you issued frequent
saves during your import stage.

Hope that helps,
Sven

[1] http://www.cimgf.com/2011/05/04/core-data-and-threads-without-the-headache/
[2] https://github.com/magicalpanda/MagicalRecord

[Anush Saidmuradov]

I guess I was curious to see how other people have approached working with large data in Core Data in a Ruby Motion app.

I say this because I think there isn't a full fledged, battle-tested Ruby Motion wrapper for Core Data out there that can handle and optimize mass imports, at least AFAIK.

[Anush Saidmuradov]

colinta did mention using magical record as a core data wrapper. But since my iOS knowledge at the moment only consists of a hodge-podge of pieces of knowledge gained over the last month, I realize I am gonna have a hard time getting my head around using the iOS framework inside Ruby Motion, at least in the beginning.

[Bora Celik]

Yeah I also use GCD as recommended above but in general, is it a good idea to store large datasets on the phone? I first started doing this, importing lots of data on the phone from another API. Maybe I wasn't doing it the most optimal way but the app started acting quite unstable during these data transfers despite the asynchronous processing as it was just taking a long time.

In the end I decided to move the heavy data storage bits to the API and not use the iOS app for storing large amounts of data.

But please don't take this as expert advice because I'm only a few months into iOS / Rubymotion development. Maybe there are others who import large data to CoreData without issues.

[Matt Green]

Does RestKit cover your use cases?

[Anush Saidmuradov]

I just looked through this article, and from a brief glance it looks like we may have to do a bunch of refactoring on the server side. 

We already have a full fledged RoR web app, and we want a native mobile app to interface with it.

Also, RestKit just maps to the native cocoa models, it does not handle data persistence, does it not?

[Matt Green]

CoreData handles the persistence; and RestKit can be configured to map API responses to CoreData entities and save them automatically.

You don't have to use ActiveModel::Serializer like he does; though it makes it very nice. You just need a JSON/XML response.

[radiospiel]

On Wednesday, October 31, 2012 1:16:03 AM UTC+1, Bora Celik wrote:

Yeah I also use GCD as recommended above but in general, is it a good idea to store large datasets on the phone? I first started doing this, importing lots of data on the phone from another API. Maybe I wasn't doing it the most optimal way but the app started acting quite unstable during these data transfers despite the asynchronous processing as it was just taking a long time.

 

In the end I decided to move the heavy data storage bits to the API and not use the iOS app for storing large amounts of data.

Todays smartphones are able to deal with multi-megabyte databases with ease. I know of an app, which holds all flights worldwide in a database and works just great. So it would be a design choice of the app. Does it make sense to have the data available offline? I built a cinema listing app (http://bit.ly/kpilot) which uses offline data for movies and listings. If this was an app for buying movie tickets online having the data offline would probably not make much sense, because the central idea of the app - buying tickets online - would require an online connection already.

Anyways, my data is up to 10.000 records, and changes on a daily basis. I worked on different solutions for fetching and storing the data. In the end I came up with this solution, which was outperforming everything else I tried by far:

- prepare the dataset on the server, producing a JSON response

- The JSON response stores each record in an array notation, because this way it can be directly consumed by a native API without reordering or rebuilding

- On the device I store the data in and retreive it from a sqlite database. Coredata was not good enough for me, because using it involved rebuilding several objects per record, resulting in a >200% overhead. 

- If possible, the data block does not contain the full dataset, but a "diff" to minimize the data transfer rate.

- run a blocking HTTP request in the background (GCD), which fetches the data block, parses it, opens a db transaction, and passes it into the db.

This works perfectly, and I think it really adds value to the app, because it has all data available all the time. Storing a full update takes ~5 secs on an iPhone 4,   storing a daily update often takes less than 1 sec on an iPhone 4, and is done in the background anyways.

Not using Coredata, however, means not having data bindings and such available. On the other hand using a sqlite database is potentially portable, while using Coredata is not. 

/eno

---

iPhone user and movie afficionado? Meet the Kinopilot Berlin app: http://bit.ly/kpilot

Follow me on twitter: @radiospiel

See my stuff on github: http://github.com/radiospiel