Memory usage

[Richard Blake]

Hi

I had a message pop up today saying that my MacBook was running out of application memory. I checked and it was GEDitCOM II that was the culprit.

I have been writing and testing a script that creates an external file and it looks like running that multiple times has caused the issue. Each time I run it, around 2GB is added to the memory usage of GEDitCOM II each time it runs (from Activity Monitor).

As a test I ran the Web Site extension and that added 9GB to the memory footprint.

I guess what these both have in common is that it is building the text of the file which is then written. When the extension finishes, the memory is not released.

I ran another extension that I have written which are quite complex and while this also added to the memory footprint while it was running, most of that was released soon after the extension concluded.

Let me know if you need any more info.

Regards, Richard

[John Nairn]

I will have to check on that. A Web Site script is a good clue. I will try to look for memory leak. I did thorough code analysis before posting version 3.2 and XCode did not find any potential leaks. Having an extension cause it sounds like an internal scripting issue.

If you have a simple extension that demonstrates it, that might make the process faster (Web Site script can take a while) (you could send in email off this list).

How big was your file that caused 9GB memory leak with Web Site script?

John Nairn.

--
You received this message because you are subscribed to the Google Groups "GEDitCOM II Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geditcom-ii-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/d933ba55-0c40-41e1-ba50-e958c21f963fn%40googlegroups.com.

[Richard Blake]

Hi John

My file that added 9GB to the memory footprint when creating web site has around 5,700 individuals.

I knocked together the script below which in a file with a total of about 10,000 records adds 2GB to the memory footprint each time it is run.

Hope that helps.

Regards

Richard

! Script name and version requirement

#scriptName="test"

#fractionStepSize=0.05

! Check version and find front document

PreChecks #scriptName,3.1,1,1,1

gcapp.get gdoc,frontDocument

ifndef gdoc

  exit "No documents are open."

endif

#folder = @gdoc.folder

#output_file = #folder&"/output.txt"

gdoc.get records, "gedcomRecords"

#numRecords = @records.count

#nextFraction=#fractionStepSize

CreateList recordList

#outputString=""

Repeat "#recordCount", 0, #numRecords-1

    #fractionDone = #recordCount/#numRecords

    if #fractionDone>#nextFraction

        NotifyProgress #fractionDone

        #nextFraction+=#fractionStepSize

    endif

    recordList.addString @records.#recordCount.name

EndRepeat

Repeat "#recordCount", 0, @recordList.count-1

    #outputString=#outputString&@recordList.#recordCount.name&return    

EndRepeat

Export #outputString, #output_file

You received this message because you are subscribed to a topic in the Google Groups "GEDitCOM II Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/geditcom-ii-discussions/qfuMLoFjWyA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to geditcom-ii-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/B9EA7EF4-8EC1-4181-B227-783E4DBD44B8%40gmail.com.

[John Nairn]

HI Richard,

Yes that will help a lot. Finding memory leakages can by challenging, but that small script you sent will help me narrow it down much faster. The Web Site scripts is long and therefore hard to know what is causing the problem. My hunch is some in the internal list where you create recordList and then add strings. Or, as you suggest it could be the Export command. But that narrows it down to two things.

John

To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/4ABA576F-DC04-445E-8419-48DE65EA4F87%40blake.uk.com.

[John Nairn]

I found a fix to memory leak in your sample script, but now I need to look in code to see if it can be fixed. In the meantime, it was not the createList feature or the Export command but rather the loop that builds the #outputString

    Repeat "#recordCount", 0, @recordList.count-1

        #outputString=#outputString&@recordList.#recordCount.name&return    

    EndRepeat

If you replace that with built-in internal scripting command to do the same thing

    recordList.join "#outputString",return

the memory leak disappears. This join command concatenates all strings in the list and separates them with a return character. When I build long strings, I try to build them in pieces in a list and concatenate at the end. I think I had a similar issue that was slow, but I did not look for a memory leak then. I recall some AppleScripts that were dismally slow if you built long strings the wrong way and that was solved by creating “references” to variables instead of using the variables (an odd requirement). Hopefully I can make building long strings better for internal scripts.

Until them, try to use the join command.

John

To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/4ABA576F-DC04-445E-8419-48DE65EA4F87%40blake.uk.com.

[Richard Blake]

Thanks John

The script that I noticed this issue with has now evolved so I am no longer concatenating values from a list. I now build a dictionary of dictionaries(!) and then build my output by iterating through that. The memory leak is still present but much less. Down to 150MB rather than 2GB. It's seems to be script specific.

I'll keep an eye on this though. The join command looks like it might be useful elsewhere for me.

Regards

Richard

To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/BBECD2A9-0811-40C5-9F2E-4DD4AD8F11CE%40gmail.com.

[John Nairn]

I found the memory leak (or at least one memory leak) and the sample script you sent runs without any increases in memory. I will post for next build. In the meantime, the loop that concatenated strings (like a join command) did not cause the leak, but made it noticeable. The core problem was in the string expressions such as:

     #outputString=#outputString&@recordList.#recordCount.name&return 

The expression on the write was storing a result of that expression, but not releasing the final expression result. If those results were small strings, you would not notice it. But here output string is growing and each one was leaking to memory leaking increased dramatically. To avoid problems before next build, try to avoid building long strings by such methods. The join command is the fix. Put each piece into a list and them join them once into a long string. The leak will still be there, but it should take hours to use up memory (and a GEDitCOM II restart will clear memory if needed).

As mentioned, finding leaks is challenging and it took a while. Apple provides a developer tool called Instruments that can record applications and look for things and one is to check for leaks. It found the leak caused by string expression (like the line above), but that did not make sense because I could not find any issues in the expression coding. I was suspecting a MacOS bug, but when I googled for that possibility, I found the answer. It turns out Instruments points to where you create and object that leaks but not to why it leaks. I eventually tracked it down to one string that was not released when the expression calculations were done. That fixed it.

I don’t know if that is the only leak. I ran a very long Web Site export (7500 individuals) under Instruments control. It took of 3 hours (Instruments recording slows things down) and did not find any leaks. The memory did increase, but since no links are found, I am not sure the cause. I then tried Web Site export with a smaller file (about 1000 records) and memory went up about 100 MB. So all might not be perfect, but according to Apple Instruments, Internal Scripts don’t seem to have any leaks.

John

To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/7BED1ABE-8E4A-464B-9948-7FC85486682B%40blake.uk.com.

[Richard Blake]

Hi John

Well done in finding this. Sounds like a challenge.

I think the leak issue was highlighted for me as I was developing the script and therefore running it over and over again. In normally day to day genealogy work that wouldn't happen. I'll take into account what you have said about using the join. The current version of the script I'm working on only increases memory by around 90MB each time I run it now. And hopefully that will go to zero once you release the next GEDitCIN II version.

Richard

To view this discussion on the web visit https://groups.google.com/d/msgid/geditcom-ii-discussions/599F8164-3BB1-4314-8564-D44E7E9AEE31%40gmail.com.