Memory management / clearing in Javascript - unable to clear

[Tamlyn Peel]

Hi,

I've been using Javascript to automate analysis of large two-photon datasets and have worked most things out, but have an issue with my script building up memory use and I'm unable to clear it. I have files that are approximately 1.5GB on hard disc and post-processing save as approximately 2GB files, running on a PC with 24GB RAM. I peak around 18GB but end up with 9GB in the memory at the end that won't clear even with forcing a garbage collector event by clicking the memory window.

Everything is wrapped in functions to avoid global variables, and once used variables are set to null.

Files are loaded as:

    f = FileDialog.open()

    var origSeq = new Sequence()
    var origSeq = Loader.loadSequence(f)


And closed as:

    origSeq.close()
    origSeq = null


The original file has each channel extracted, median filtered, some maths, ROI generated with HK Means, spots generated and tracked, results saved in XLS / XML files and then the channels concatenated and ROI added, with the resulting files saved before being set to null:

    Saver.save(globalResult, outputFile, false, true)


I've tried cutting bits of code out to isolate the problem, but not gotten very far. Is there something I'm missing to allow Java to clear everything from memory? Script attached for reference but it's pretty long and not so well written so I don't expect it to be read! Any help appreciated.

Cheers

Tamlyn

[Stephane]

Hi Tamlyn,

I think that a part in your workflow analysis is retaining the memory, the problem is to find where.
Normally doing System.gc() does try to release memory (sometime you have to call it several time).
You probably forgot to attach the JS file as i can't find it in the post, but what you can try is to isolate part independently and try to find which one retain / not retain memory. I know it's a paintful task but i don't see obvious reason for your memory leak problem here. Unfortunately maybe there is an issue with a plugin but at least we have to know which one to start with.
If you attach your JS file i can try to look at it, but i'm not sure to find any relevant things in it from what you told here.

Cheers,

- Stephane

[Tamlyn Peel]

Hi Stephane

Thanks for replying, shame I haven't missed something obvious! I hadn't worked out how to call the garbage collector in Javascript, so I'll throw in a couple of System.gc() for luck between steps, but forcing it once the script finishes hasn't worked so far. I had a go at taking out some sections of code and seeing where the memory was leaking:

*1.6Gb of memory bleed in Median filter plugin of 3 x channels:

(function(){
    var median = new MedianFilter();
   
    gui.closeAllViewers()
   
    for (i = 0; i < Chns.length; ++i) {
        if (median.isHeadLess()) median.createUI() //First instance create UI to generate viewer
        median.Median_filter_3D(Chns[i], 2)
        //WAIT//
        do {} while (gui.getActiveSequence() == null)
        Chns[i] = gui.getActiveSequence()
        gui.closeSequence(Chns[i])
        println('Median Filtered Ch: ' + i)
    }
   
    gui.closeAllViewers()
   
    median = null
})();

*0Gb in maths (using several mathsChns[0] = Functor2.parse(formula).apply(Chns[1], Chns[0]))

*1.5Gb in distance mapping of one channel:

(function(){

...

distmap[i] = new Sequence()
distmap[i] = roimap.newInstance().createDistanceMap(Chns[i], false, false, true)

...

if (!(typeof distmap[i] == 'undefined')) Chns[Chns.length] = SequenceUtil.convertToType(distmap[i], DataType.DOUBLE, null)

...

distmap[i] = null

})();

*0Gb in saving the image

So I can find about 1/3 of the memory leaking in calling two plugins, it's just some steps rely on others so I struggle to get it to run still if I take them out, but I could try and work something out to test. I've attached the script (sorry it wouldn't take it as a .js file before) in case you see anything obvious. I'm not sure if I'm not calling something from those plugins to help them clear the memory or I haven't properly destroyed all references to them, but certainly if you use them properly from the GUI I don't get memory problems.

Any ideas greatly appreciated!

Cheers

Tamlyn

[Tamlyn Peel]

Hi,

I've been playing around trying to find where the memory leaks are; I suspect the plugins I'm calling are keeping hold of sequences I no longer need. For instance, if I throw an empty sequence at the MedianFilter().Median_filter_3D(new Sequence(), 2) after doing my real data, it no longer seems to cause a memory leak (or at least a lot less). 

I can get sequence IDs with sequence.getId(), but are there methods for (a) getting a list of all sequences in Icy (not just those open in viewers) and (b) referring to a sequence by its ID? This way I could assess all the open sequences and close those I no longer have a handle for.

Cheers

Tamlyn

[Stephane]

Hi Tamlyn,

It looks like you're very familiar with javascript to be able to trace the memory leak issue at that point ;)
Indeed some plugin may retains the last used Sequence, the thing is that, normally the plugin instance itself shouldn't be retained, and so the sequence it was referring. For some reason the plugin instance aren't released (probably because of some uncleaned listeners).
The sequence id is just an internal id which guarantee you to have an uniq identifier for a Sequence.
If you don't open a Sequence in a viewer, it should not stay in memory unless a plugin retain it, using a fake empty Sequence is a way to force some plugin to release your data, but normally the plugin should not retain Sequence anyway.

Cheers,

- Stephane