Leveraging Closeable Memory in Structures and ByReference classes

[Daniel B. Widdis]

Now that the "remove finalizers" PR has been merged, I took a brief look into the possibility of improving user ability to release native memory via close() (or try-with-resources).  I was looking at two potential implementations: Structures and the classes extending ByReference.

Structures proved quickly to be too complex, but in the process I realized that the constructor takes a Pointer arg so one can easily create their own Memory object and use that (possibly even recycling it).

Unfortunately, ByReference does not have a Pointer constructor, only an argument specifying the size of memory to allocate.

I got as far as adding the following steps as a potential implication:

 - Added "implements Closeable" to ByReference class

 - Separated the new Memory() call in the constructor to an instance variable

 - in the ByReference close() method, called close() on that instance variable

While this would handle most use cases, I realized it was not clear what to do when setPointer() was called.

 - If the user had previously used getPointer() to retrieve the value before changing it, it would not be appropriate to close it; although the user would intentionally be calling close() so they could be warned of this limitation

 - Could close() just work on the PointerType's stored pointer regardless of what it was (with some null and type checking)?  But then you'd close a resource you didn't open, maybe.

 - If you changed which pointer was stored internally what should you do with the old pointer? Leave it to be cleaned or intentionally close it (but you don't know if it's referenced elsewhere).

My experiments ended with "git reset --hard". :-)

After a bit more coffee, I'm thinking there's a much simpler solution, the same solution I concluded was appropriate for Structure.  Let the user create the memory allocation.  Except: ByReference doesn't have a Pointer constructor.

I think adding a constructor taking a Pointer argument (documented where the user is responsible for determining allocation size) for ByReference and its subclasses could be very useful.  I'd like feedback on this before submitting a PR.   Or any other ideas on using auto-closeable Memory in these very common allocations.

--

Dan Widdis

[Matthias Bläsing]

Hi Daniel,

Am Mittwoch, dem 11.05.2022 um 11:15 -0700 schrieb Daniel B. Widdis:
> Unfortunately, ByReference does not have a Pointer constructor, only
> an argument specifying the size of memory to allocate.

I think you are looking at the wrong ByReference. The description
sounds like:

com.sun.jna.ptr.ByReference

But in context of structures we are talking about the inner interface
in Structure:

com.sun.jna.Structure.ByReference

The latter is a pure interface without any methods defined. It is just
a marker, that causes different behavior when the structure passes
through the Java<->Native border.

Does this help?

Greetings

Matthias

[Daniel Widdis]

I was referring to com.sun.jna.ptr.ByReference.

 

For background, the biggest use of native memory in my project is collecting Windows performance counters.  For someone wanting to replicate a process listing similar the Windows Task Manager, each of 8 counters, for each process is queried. Assuming 100 processes queried every second, that's 800 queries.  Each one involves the creation of a HANDLEByReference and populates counter information in a PDH_RAW_COUNTER structure.  With the current “finalizer” code, these 1600 Memory allocations don’t get promptly released.  The change to use cleaners improves on that.

 

However, I’d like to use the fact that Memory is now closeable to more promptly release the native memory resource, even before the Cleaner gets to it.

 

Here's the current code for the structure:

 

public static long queryCounter(WinNT.HANDLEByReference counter) {

    PDH_RAW_COUNTER counterValue = new PDH_RAW_COUNTER();

    int ret = PDH.PdhGetRawCounterValue(counter.getValue(), PDH_FMT_RAW, counterValue);

    if (ret != WinError.ERROR_SUCCESS) {

        if (LOG.isWarnEnabled()) {

            LOG.warn("Failed to get counter. Error code: {}", String.format(FormatUtil.formatError(ret)));

        }

        return ret;

    }

    return counterValue.FirstValue;

}

 

This can be easily modified to use closeable memory:

 

public static long queryCounter(WinNT.HANDLEByReference counter) {

    try (Memory m = new Memory(RAW_COUNTER_BYTES)) {

        PDH_RAW_COUNTER counterValue = new PDH_RAW_COUNTER(m);

        int ret = PDH.PdhGetRawCounterValue(counter.getValue(), PDH_FMT_RAW, counterValue);

        if (ret != WinError.ERROR_SUCCESS) {

            if (LOG.isWarnEnabled()) {

                LOG.warn("Failed to get counter. Error code: {}", String.format(FormatUtil.formatError(ret)));

            }

            return ret;

        }

        return counterValue.FirstValue;

    }

}

 

So I don't need to make any changes to the Structure code. 

 

However, I can’t do the same with that HANDLEByReference argument.  I allocate it here:

 

// Get a new handle for the counter

HANDLEByReference p = new HANDLEByReference();

if (!PerfDataUtil.addCounter(this.queryHandle, counter.getCounterPath(), p)) {

    LOG.warn("Failed to add counter for PDH counter: {}", counter.getCounterPath());

    return false;

}

counterHandleMap.put(counter, p);

 

And remove all references to it here, making it eligible for GC:

 

HANDLEByReference href = counterHandleMap.remove(counter);

// null if handle wasn't present

if (href != null) {

    success = PerfDataUtil.removeCounter(href);

}

 

What I would LIKE to do is something like this when creating it:

 

Memory m = new Memory(Native.POINTER_SIZE);

HANDLEByReference p = new HANDLEByReference(m);

 

And when I’m done with it:

 

HANDLEByReference href = counterHandleMap.remove(counter);

// null if handle wasn't present

if (href != null) {

    success = PerfDataUtil.removeCounter(href);

    href.close();

}

 

This is not directly possible because HANDLEByReference extends com.sun.jna.ptr.ByReference and the only constructor takes a size argument, doing this:

 

protected ByReference(int dataSize) {

    setPointer(new Memory(dataSize));

}

 

One possibility with the existing code is to retrieve the pointer, and cast it to Memory to call close(), e.g.,

 

HANDLEByReference href = counterHandleMap.remove(counter);

// null if handle wasn't present

if (href != null) {

    success = PerfDataUtil.removeCounter(href);

    Pointer p = href.getPointer();

    if (p != null && p instanceof com.sun.jna.Memory) {

        (Memory) p.close();

    }

}

 

While this would work, casting always makes me uncomfortable. I would prefer to do one of the following:

1. Make ByReference Closeable; pull the “new Memory(dataSize)” out of setPointer(); and set an instance field to it that is closed in a close() method.   This introduces complexity on what to do when someone calls setPointer(); do we release the memory then (to match current behavior) or still hold on to it until close() is called (or it’s GC’d/cleaned)?
2. Add a constructor to ByReference that takes an existing allocation as a Memory or Pointer argument.  Memory avoids casting but prevents use of an existing native allocation for which we only have a Pointer. Pointer requires the same casting behavior as above. And in all cases, multiply this decision: do we implement pointer constructors in all the existing subclasses, and if so how do we handle PointerByReference that already has a pointer constructor with a different interpretation?  Or do we just add it to a few that we need?

 

I’m leaning toward 1 above, but can’t decide when to release the memory, and think 2 is a lot more flexible but also requires a lot more changes.  Or I can just do the casting check which I’ll probably end up doing if I can’t decide on anything else.  Or I might just use the try-with-resources with the Structure, and leave the HANDLEByReference as they are and trust the Cleaner to do its job.

    --

    You received this message because you are subscribed to the Google Groups "Java Native Access" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

    To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/2ea959cf5b727fa61a54c0d2c5c60ca8c3edc18b.camel%40doppel-helix.eu.