GEDitCOM II memory problem?

[Jim Eggert]

Is there a memory freeing problem with GEDitCOM II?

My main genealogy database consumes 12.3 MB on disk in a .gedpkg package. The GEDCOM file in it is 12.1 MB.

If I start GEDitCOM II by double-clicking its icon, it consumes 17.0 MB in real memory and 75.9 MB in virtual memory according to Activity Monitor, with no database open.

If I open the database by double-clicking on it (with GEDitCOM II not running before), GEDitCOM II consumes 232.6 MB of real memory and 279.1 MB of virtual memory.

If I then close the database without modifying it (click the red button), GEDitCOM II's real memory goes down to 227.2 MB and virtual to 278.2 MB.

On reopening the database, real memory is at 355.7 MB and virtual 392.6 MB.
On reclosing the database, real memory goes to 350.6 MB and virtual 392.6 MB.

Every re-open and re-close cycle consumes more memory. After a long session involving many cycles of doing this, GEDitCOM II was up to about 6 GB of real memory and a similar amount of virtual memory.

It seems it isn't freeing memory on database closure very well. Or maybe I'm misinterpreting things and this is not a problem.

=Jim

[John Nairn]

I am not aware of any and very careful to free up memory when not used, but it is easy for subtle things to slip by.

Does it happen to all your files? And do you have a lot of multimedia (their thumbnails make a lot of use of memory cacheing)?

> --
> You received this message because you are subscribed to the Google Groups "GEDitCOM II Discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to geditcom-ii-discu...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Kevin Gartner]

When I get in front of my iMac this evening I will take a look at the behavior on my machine.  I have not noticed a problem to this point, but I have a lot of RAM on it.  My machine runs OS X 10.6.8.

Kevin

[Jim Eggert]

I really only have one file. It doesn't have any multimedia at all. Just 570K lines of GEDCOM.

So I built a test GEDCOM file with 10^6 individuals. All different names, but no families.

GEDitCOM II memory usage real/virtual (MB) as shown in Activity Monitor:
Before loading file 16.7 / 76.9
Open file (1) 1100 / 1150
Close file (1) 1090 / 1140
Open file (2) 2140 / 2180
Close file (2) 2140 / 2180
Open file (3) 3220 / 3250
Close file (3) 3210 / 3250
Open file (4) 4160 / 4200
Close file (4) 4160 / 4200

Looks like GEDitCOM II isn't freeing hardly anything. Loaded four times, got 4 GB of memory usage. The wasted memory doesn't come back if I close the file and load a different file either. I have to close GEDitCOM II.

I'm running:
GEDitCOM II 1.8
Mac OS X Mountain Lion 10.8.3

Here's the program for generating the test GEDCOM file:

#!/usr/bin/perl -w
# Make a test GEDCOM file of 10^6 individuals, no families
print <<'EOL';
0 HEAD
1 SOUR GEDitCOM II
2 NAME GEDitCOM II
2 VERS 1.8
2 CORP John A. Nairn, RSAC Software
3 ADDR 2995 NW Snowberry Pl
4 CONT Corvallis, OR 97330
4 CONT USA
1 SUBM @SUBM@
1 DEST ANSTFILE
1 GEDC
2 VERS 5.5
2 FORM LINEAGE-LINKED
1 CHAR UTF-8
1 LANG English
1 NOTE This file was created by GEDitCOM II - the customizable Macintosh genealogy application. The GEDitCOM II web site is <http://www.geditcom.com>.
0 @SUBM@ SUBM
1 NAME Submitter's Name
EOL
foreach my $i (1 .. 1000000) {
print <<PERSON
0 \@I$i\@ INDI
1 NAME John I$i /Doe/
1 SEX M
1 REFN 9 MAY 2013 22:16:10
2 TYPE Creation Date
PERSON
}
print "0 TRLR\n";


=Jim

[John Nairn]

I see you are using MacOS 10.8 and I got similar results in 10.7. I want to test in 10.6 too. It seems to lock up memory when reading a file and then the memory stays constant while using a file. I got a little more released when my file closed but big increase on the next read too.

It is strange to be in the reading code because I wrote that a long time ago and never noticed any problems. I was also obsessive about clearing and using memory Memory. Back in Mac 10.4, large files could not be read at all until I worked out all memory issues. Apple made big changes in Memory management around 10.6, but they recommended existing apps should continue to use the previous methods (which is what I still do). I wonder if they changed theri minds at some point and a no longer really supporting all memory options.

XCode has some tools for analyzing code for leaks and one tool that is supposed to help you see leaks. The first one flagged about 200 lines in my code. I have looked at abouit half of rhem and so far none of them are actual leaks, but rather just new conventions in Apple documentation (I did find one or two minor leaks, but not in any code that is used by reading, closing, and reading files). I will continue to look at these warnings and then move on to leak tracking (although that one is very complex ro use by ordinary programmers).

I might also try old verison on 10.4 (I still have a PPC machine with 10.4). I can't try 10.5 anymore because my wife's new iPhone 5 required upgrading that machine otherwise the phone would be unusable. But, testing in 10.4 or 10.5 is just for historical perspective because it is almost impossible to support those systems (and acually forbiidden by Apple to support them for anything that might someday be in the App Store).

John

------------
John Nairn
http://www.geditcom.com
Genealogy Software for the Mac

[John Nairn]

Jim,

I have been through all of XCode's code analysis and find no leaks. My "verified" opinion is that GEDitCOM II is not leaking any memory and what you are seeing in memory usage is how the OS handles mamory allocation. It expands space as needed (e.g. when opening a file), but never contracts it when no longer needed (e.g., when closing a file). It is possible the end result could be too much or fragmented memory, but I am not sure what a coder can do about low-level OS memory management. It is also posible the OS will fix that situation if needed. 15 years ago (before Mac OS X), one had to deal with low-level memory access  a lot, such as lock and unlock pieces of memory and move blocks around, or else memory problems would be severe. Those task are now automatically handled by MacOS X.

To verify, I ran GEDitCOM II using a developer tool call Instruments and attached a tool to track all memory allocations. It tells you how many bytes are "live". The results for live bytes when opening and closing a file (with about 10000 records) are as follows:

Start GEDitCOM II - 1.91 MB

Open File (1) - 27.27 MB

Close File (1) - 1.91 MB

Open File (2) - 27.29 MB

Close File (2) - 1.91 MB

Open File (3) - 27.29 MB

Clise File (3) - 1.91 MB

etc.

This same tool reported "Overall Bytes" increasing. It think you are seeing overall bytes and not a measure of memory usage.

I also tried an Instruments tool to detect leaks. It found some that seem to be in core OS methods, but nothing in my code. I was not sure I was interpreting this tool correcting so I went in and caused a leak by not freeing a string that is allocate often when reading a file. This leak showed up clearly in the leaks too and said GEDitCOM II was responsible for the leak. In other words, I would see any leak that happens when opening and closing a file. Since none appear, I think it also confirms no leak is happening.

I think these tests show GEDitCOM II is not leaking anything. What your are seeing in Activity Monitor is something different. My guess it is total virtual address space currently allocated to the process. If anyone knows how to alter that state, I can look into if it is needed. I am not aware of any MacOS methods provided to change OS allocation methods. I will try to look into them.

The above excersice into XCode code analysis tools was useful. I did found a couple of minor leaks, which cleans up the code (but none of them were involved in opening or closing files). Also, that Instriuments developer tool has some powerful options that are slightly more clear to me now.

John

P.S.: I tried using the dummy million-man GEDCOM created using the script you provided. It was exceedingly slow to open when using the Intruments developer tool (presumable because it is logging every memory access). I gave up and used the smaller file above instead.

On May 9, 2013, at 7:51 PM, Jim Eggert wrote:

---------------
John Nairn (1-541-737-4265, FAX:1-541-737-3385)
Professor and Richardson Chair
Web Page: http://www.cof.orst.edu/cof/wse/faculty/Nairn/
FEA/MPM Web Page: http://oregonstate.edu/~nairnj

[Kevin Gartner]

Though the behavior described by Jim and John is baffling, it is apparently normal.  The key is that the term "Overall Bytes" apparently is a running cumulative total of all allocations by an application, including memory that has since been freed.  See 

http://stackoverflow.com/questions/2154219/instruments-objectalloc-explanation-of-live-bytes-overall-bytes/2155806#2155806

for further explanation.

Kevin

[Jim Eggert]

I made a 10^7 individual file using the same script so I could see if the memory allocation would exhaust my system resources This file is about 1GB in size. I have 16GB RAM, so the OS and GEDitCOM II will run fine even with this big file open. Opening the file causes GEDitCOM Ii to consume about 10GB of memory.

On closing and reopening the file, now GEDitCOM II uses about 20 GB virtual memory, and my swap file is several GB big. On closing and opening it a third time, now GEDitCOM II is 30 GB in virtual memory and my swap file is about 15 GB. I don't think that swap should be required for this exercise.

If I do the same experiment using TextEdit instead of GEDitCOM II I get:

TextEdit (no file open) 14.8 MB real, 83.3 MB virtual
Open 10^7 file (1) 981.0 MB real, 1.02 B virtual
Close 10^7 file (1) 18.9 MB real, 85.3 MB virtual
Open 10^7 file (2) 983.8 MB real, 1.02 GB virtual
Close 10^7 file (2) 21.7 MB real, 86.2 MB virtual
Open 10^7 file (3) 983.6 MB real, 1.02 GB virtual

No gigantic memory buildup there. Somehow TextEdit can cause the memory used by the file to be freed and released back to the system just fine.

And if I try a 10^6 individual file with TextWrangler, I get:

TextWrangler (several files open already) 33.1 MB real, 132.4 MB virtual
Open 10^6 file (1) 221.8 MB real, 321.9 MB virtual
Close 10^6 file (1) 34.4 MB real, 133.5 MB virtual
Open 10^6 file (2) 222.3 MB real, 325.6 MB virtual
Close 10^6 file (2) 34.8 MB real, 137.1 MB virtual
Open 10^6 file (3) 223.7 MB real, 325.5 MB virtual
Close 10^6 file (3) 35.1 MB real, 136.1 MB virtual

No gigantic memory buildup there either.

To look at active memory (perhaps the same as the "live" memory?), I looked at Activity Monitor's active memory utilization (for OS and all apps) and using the 10^6 individual file, I get:
GEDitCOM II open with no file open: 4.15 GB active memory
Open 10^6 file (1) 5.23 GB active memory system-wide
Close 10^6 file (1) 5.27 GB
Open 10^6 file (2) 6.28 GB
Close 10^6 file (2) 6.28 GB
Open 10^6 file (3) 7.35 GB
Close 10^6 file (3) 7.36 GB
This is consistent with the memory allocation made during opening the file not being de-allocated on file closure.

I conclude something is wrong with GEDitCOM II, at least on my system, that makes it unfriendly to memory when opening and closing files. Perhaps I am interpreting the information incorrectly, but certainly GEDitCOM II acts differently from TextEdit and TextWrangler, using the same tools.

=Jim

[John Nairn]

What I know is that GEDitCOM II is NOT using that memory. If you monitor allocated memory with Apple Instruments app (which comes free with developer tools) it confirms that the memory is not being used and that all memory is freed when a file is closed.

So how could it be changed? It is not possible to find and fix a memory leak, because GEDitCOM II does not have a leak - there is no code that can be changed. I don't know if there is a different way to allocate and free memory that will end up with different rresults with MacOS. I use only standard tools now. It would be frustrating project to just randomly start rewriting code to do the same thing with different tools with the hope that a some other style will dramatically change the results. If anyone knows a "trick" that causes MacOS core to handle memory different, I am willing to try. I have looked and cannot find any advice.

In other words, maybe it is problem in MacOS and that is where the fix is really needed. They it could help other applications too. Why does MacOS not handle memory well when memory is correctly handled by an application? What assumptions does MacOS make and does it provide any tool to change things? If standard tools should not be used, when should they not be used and what should replace them? Without answers to these types of questions, I have not good place to start. For example, I agree GEDitCOM II should not need 15GB swap file to open a file, but in writing Cocoa applications, there are no calls to choose or set the swap file size. The tasks are handle bythe kernal and not by applications.

The comparison to opening a text file is probably not that close. Opening a text file is far simpler (trival one line of code) and just allocates one large chunk of memory to hold the text. If contrast, GEDitCOM II is opening genealogy data and each line potentially becomes a text edit object. In other words, when opening a file with 10^7 individuals, GEDitCOM II will likely need to allocate 10^8 (or more) small objects in memory. In contrast, when opening a large file in TextEdit ot TextWrangler, it is probably allocating 1 large object (and probable a few others, but not many). Articles on Cocoa and Mac memory claim it was designed to handle lots of small object allocations, but maybe that is not really true in practice if that number gets too large.

A better test would be to have TextEdit open 10^7 ot 10^8 small files at the same time, then close them all and see what happens. An AppleScript could probably try that out, but I would be surprised it could surving opening them all at once.

--
You received this message because you are subscribed to the Google Groups "GEDitCOM II Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geditcom-ii-discu...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[John Nairn, Developer]

I tried some more tests because the real issue is not what Activity Monitor says about memory, but are there any consequences of what it says. If yes, then it would be good to look for a work around potential shortcomings in way MacOS kernel handles certain types of memory allocation.

I took Jim's million individual file and opened it over and over again. If MacOS kernel memory allocation is causing problems, I might expect to see time to open the file get slower and slower, but it doesn't. In fact it got faster in the beginning and then I did not notice any change. I assume it got faster after the first one because now the kernal has already given GEDitCOM II more memory. Here are results of 12 opens on my Mac (10.7.x) in seconds:

1 29.73

2 21.73

3 21.73

4 21.73

5 22.07

6 22.73

7 21.73

8 22.05

9 22.066

10 22.39

11 22.79

Note that one can look up this times in the Console App while running GEDitCOM II.

I did not bother even looking an Activity monitor memory (but remember that is memory allocated and NOT memory allocated) until the 7th open. From 7 through 12, GEDitCOM II memory allocated was 1.12 GB when the file was open and reduced to 330 MB when it was closed. It was rather slow getting back down to 330 MB (10 to 15 seconds), but always went down. At least from 7 through 12 I do not see even Activity Monitor memory going up any more.

So on my Mac (10.7.x) I do not see any evidence of MacOS kernel memory issues or any degradation in time to read a file. Perhaps my development version differs from current GEDitCOM II 1.8, but I don't think I have changed any code for reading and writing files (Jim: I can get you a copy of development version to try if you want, although it has a lot of undocumented stuff (in fact I and changing all the documentation) and should only be tested as a copy for testing and not used yet for genealogy).

This is consistent with the memory allocation made during opening the file not being de-allocated on file closure. 

This is simply impossible and can be eliminated as a hypothesis. You are misinterpreting what those numbers in Activity Monitor mean because  they are no memory allocations. The only answer is to find out if there limitations in MacOS kernel and then if there are consequences see if there is a way to program around those limitations.