Memory intensity in sqlite-persistent-objects

[Sudhir B]

Hello all,

I have recently attempted to use sqlite-persistent objects in my iPhone app. I decided on it because I already had a database structure on my server and I had to create the same inside my app. Since it was a huge db structure I decided against using Core Data.

However, at first all I came across was disappointment. My database was relatively large enough. I had to delete/modify/insert about 1500+ objects in one go. The app ran well on the simulator but when I went onto the device, the app crashed stating memory unavailability issues. I first thought the caching by sqlite-persistent-objects was the reason for the same and cleared the cache whenever a memory warning was encountered. That did not help.

Finally, on performing a full fledged memory profiling, I found out that the app was consuming a lot of memory due to autoreleased strings. These accumulated all the way to consume a lot of memory even before the event loop finished.
To resolve this problem, I made a number of changes in the source of sqlite-persistent-objects. All the changes were primarily to get rid of any autoreleased objects or memory allocation which isn't totally under the control of the programmer. A sample change is as under in the file NSString-SQLiteColumnName.m

Existing code:
- (NSString *)stringAsSQLColumnName
{
    NSMutableString *ret = [NSMutableString string];
    for (int i=0; i < [self length]; i++)
    {
        NSRange sRange = NSMakeRange(i,1);
        NSString *oneChar = [self substringWithRange:sRange];
        if ([oneChar isEqualToString:[oneChar uppercaseString]] && i > 0)
            [ret appendFormat:@"_%@", [oneChar lowercaseString]];
        else
            [ret appendString:[oneChar lowercaseString]];
    }
    return ret;
}

In the above code the functions substringWithRange are too much memory intensive. All this function needs to do is convert any upper case alphabets that occur in the name of the property of the SqlitePersistentObject to an underscore followed by the lowercase equivalent of the same alphabet. I have implemented the same in the following fashion where in I have used basic C code to do the same job. This underlying implementation provides a lot of memory efficiency as well as efficiency wrt speed.

Modified code:
#define MAX_SQL_COLUMN_NAME_LENGTH    128

- (NSString *)stringAsSQLColumnName
{
    static char cString[MAX_SQL_COLUMN_NAME_LENGTH];
    BOOL isSufficient = [self getCString:cString maxLength:MAX_SQL_COLUMN_NAME_LENGTH encoding:NSUTF8StringEncoding];
    NSAssert2(isSufficient, @"Insufficient buffer space %d for sql column name %@", MAX_SQL_COLUMN_NAME_LENGTH, self);
    //char* cString = [self UTF8String];
    int noCapsInBetween = 0;
    int length = strlen(cString);
    for (int i=0; i<length; i++) {
        if (i > 0 && isupper(*(cString+i))) {
            noCapsInBetween++;
        }
    }
    char* ret = (char *)malloc(length+noCapsInBetween+1);
    for (int i=0,j=0; i<length; i++,j++) {
        if (i > 0 && isupper(*(cString+i))) {
            *(ret+j) = '_';
            j++;
        }
        *(ret+j) = tolower(*(cString + i));
    }
    *(ret + length + noCapsInBetween) = '\0';
    NSString *retVal = [NSString stringWithUTF8String:ret];
    free(ret);
    return retVal;
}

I have made similar changes at a number of other places within sqlite-persistent-objects. All together the memory consumption due these changes went all the way down to 20% of where it was initially and the app became stable. I haven't yet implemented such changes all through though. I would like to push these changes onto the google code project. I am planning to do many more similar memory optimization changes in the code base. I will integrate those changes with the project as I finish them. Please let me know how can I push these onto the google code project.

Right now although my app is stable, sqlite-persistent-objects still eats a sizable chunk of all the memory used by my app. Most of those memory crunches are due to -[NSString UTF8String] that has been used all through the code base. I am a little unhappy with the basic implementation strategy of sqlite-persistent-objects. The entire project is primarily based upon two libraries - libsqlite3 and Objective C's runtime Class libary. Both these expose basic C functions. Hence a complete C based implementation of sqlite-persistent-objects would have been far better both wrt memory and speed. The basic idea and architecture are really good, it is only the implementation strategy I am not quite OK with.

Soliciting your early reply,

-Sudhir

[Andrew W. Donoho]

On Dec 28, 2009, at 8:32 AM, Sudhir B wrote:

However, at first all I came across was disappointment. My database was relatively large enough. I had to delete/modify/insert about 1500+ objects in one go. The app ran well on the simulator but when I went onto the device, the app crashed stating memory unavailability issues. I first thought the caching by sqlite-persistent-objects was the reason for the same and cleared the cache whenever a memory warning was encountered. That did not help.

Sudhir,

Welcome to the SQLPO performance tuning club.

As I made the most recent release of SQLPO, are you using it? See this blog post: <http://blog.ddg.com/?p=21>.

Finally, on performing a full fledged memory profiling, I found out that the app was consuming a lot of memory due to autoreleased strings. These accumulated all the way to consume a lot of memory even before the event loop finished.

Yes, SQLPO does that. 

To resolve this problem, I made a number of changes in the source of sqlite-persistent-objects. All the changes were primarily to get rid of any autoreleased objects or memory allocation which isn't totally under the control of the programmer. A sample change is as under in the file NSString-SQLiteColumnName.m

[ Good code snipped.]

I have made similar changes at a number of other places within sqlite-persistent-objects. All together the memory consumption due these changes went all the way down to 20% of where it was initially and the app became stable. I haven't yet implemented such changes all through though.

As these routines are creating column names, I suspect you can cache this info rather than recreate it every time. (I have used this strategy in other parts of SQLPO.) Did you look at this strategy?

I would like to push these changes onto the google code project. I am planning to do many more similar memory optimization changes in the code base. I will integrate those changes with the project as I finish them. Please let me know how can I push these onto the google code project.

As the somewhat de facto "owner" of SQLPO, I haven't been using the historic Google code repository. No one else has wanted to integrate anything with the code base -- either the historic release or my October release. Hence, it was just a pain in the butt to use.

I'm happy to work with you to further performance tune SQLPO. We need to talk about what implementation techniques you considered in your performance tuning work. And we need to start with the same codebase. Could you rebase your changes to the version I published in October?

Right now although my app is stable, sqlite-persistent-objects still eats a sizable chunk of all the memory used by my app. Most of those memory crunches are due to -[NSString UTF8String] that has been used all through the code base.

Are you sure? My quick scan of the code base has probably 80% of the -UTF8String methods are called in setting up a SQLite DB call. IOW, the DB call will dominate the performance profile of the routine. If you are making so many calls to the DB that these UTF8Strings create a memory pressure problem, then you may want to wrap your calls with an autorelease pool and then periodically drain it.

We need to identify if the memory pressure problem is really in SQLPO or in your use of SQLPO. Frankly, creating an autorelease pool is much simpler, and likely more effective, than opening up the SQLPO code base.

I am a little unhappy with the basic implementation strategy of sqlite-persistent-objects.

OK. The SQLPO architecture leaves much to be desired. It is a convenience library. While I believe there are many developers using SQLPO, I don't know of many that are starting new projects with SQLPO. (This is probably due to the fact that the project creators have moved on to other pursuits. I myself am just maintaining the code base to support my existing products.)

Unless you are proposing to single handedly rewrite SQLPO, then its architecture is not going to change. 

The entire project is primarily based upon two libraries - libsqlite3 and Objective C's runtime Class libary. Both these expose basic C functions. Hence a complete C based implementation of sqlite-persistent-objects would have been far better both wrt memory and speed.

Memory use and performance were NOT primary design criteria of SQLPO. If you were expecting high performance and a low memory footprint by using SQLPO, then you may wish to reconsider your choice. Please see Jeff LaMarche's performance comments in the presentation slides also linked in my October code release.

The basic idea and architecture are really good, it is only the implementation strategy I am not quite OK with.

Did you consider other ORMs? FMDB? Migrating your data into Core Data? Those projects are under active development and may meet your needs better than SQLPO. 

I am happy to work with you to tune the library. 

Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
a...@DDG.com, +1 (512) 750-7596

"We did not come to fear the future. 
    We came here to shape it."

-- President Barack Obama, Sept. 2009