Is TOKEN_USER losing its memory allocation?
Daniel Widdis
Matthias Bläsing
Am Mittwoch, den 10.03.2021, 09:42 -0800 schrieb Daniel Widdis:
> [Strange behavior of Advapi32Util.getTokenAccount]
> The superclass (Structure) only tracks the pointer to that Memory
> object, assuming it's been allocated on the native side. But after
> calling the constructor here, Memory is available for GC and release
> of its native allocation.
>
> I believe the correct fix here would be to store a separate private
> Memory object in the class to maintain the reference.
>
> I'm unclear why this problem would only (or at least primarily)
> present in the case of users SYSTEM or Administrator, but I suppose
> they might have a much larger SID memory allocation requirement than
> other users, and thus be more likely to be recycled.
>
> Can anyone confirm my observations here?
The issue roots deeper: JNA IMHO makes the wrong assumption, that you
could handle native memory with a GC. If I remember correctly project
panama (the potential JNI successor) uses a scoped approach, but is
still shuffeling the API.
The simplest fix would be to ensure, that the memory stays referenced.
With JDK 9+ this should work:
public static Account getTokenAccount(HANDLE hToken) {
// ...
try {
return getAccountBySid(user.User.Sid);
} finally {
Reference.reachabilityFence(user.getPointer());
}
}
I would also expect, that this should work:
public static Account getTokenAccount(HANDLE hToken) {
// ...
try {
return getAccountBySid(user.User.Sid);
} finally {
user.getPointer().getByte(0);
}
}
Greetings
Matthias
Daniel B. Widdis
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/5d9d48bfb6c3f80bcb0787a49f70d2cad24f2ec6.camel%40doppel-helix.eu.
Matthias Bläsing
I don't think this will work. Here is what I think is happening:
- Memory is allocated in https://github.com/java-native-access/jna/blob/master/contrib/platform/src/com/sun/jna/platform/win32/Advapi32Util.java#L538
It does not matter whether the memory is allocated with new Memory
or allocateMemory. The latter only clears the Memory prior to
handing it out.
- I assume, that the SID returned in the structure is located inside
the Memory object
- the SID is used two times
- once without error: https://github.com/java-native-access/jna/blob/5.7.0/contrib/platform/src/com/sun/jna/platform/win32/Advapi32Util.java#L268
- again with error: https://github.com/java-native-access/jna/blob/5.7.0/contrib/platform/src/com/sun/jna/platform/win32/Advapi32Util.java#L283
The only difference between the two calls is, that allocations happen in between:
https://github.com/java-native-access/jna/blob/5.7.0/contrib/platform/src/com/sun/jna/platform/win32/Advapi32Util.java#L280-L281
The object allocated in the first step is already out of reach, so it can be GCed.
We are back to the requirement, that a strong reference to Memory/the original TOKEN_USER needs to be retained.
Greetings
Matthias
Tres Finocchiaro
Matthias Bläsing
Am Donnerstag, den 11.03.2021, 13:57 -0500 schrieb Tres Finocchiaro:
> As a stop-gap, would it make sense to use reflection to favor the
> Java 9 technique, if available?
here - it would be great if it could be tested:
https://github.com/matthiasblaesing/jna/commit/8f4685bf37abf29630621a7eaae023de753e65c5
Greetings
Matthias
Daniel B. Widdis
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/6e4c70fa4038f3e543766085b7755acb95a1c820.camel%40doppel-helix.eu.
Daniel B. Widdis
Matthias Bläsing
memory _immediately_ eligable for GC.
The problem is, that our original Memory is tied to the TOKEN_USER, but
only PSID is retained. That PSID has no reference to the original
memory.
Greetings
Matthias
Matthias Bläsing
> So a new Memory object is created, but only its Pointer is passed to
> the superclass (Structure) constructor
public class Memory extends Pointer {
...
}
So the _Memory_ object is passed and stays referenced. If that would
not happen, we would be even more sad. The problem is in this case,
that there is a additional indirection:
There is a PSID.ByReference in there. From the POV of JNA this is a
pointer and that pointer is read and a structure is wrapped around it.
That code can be found in Pointer#getVaue (line 367). The pointer of
the structure is found via:
Pointer#getPointer(offset)
that constructs a raw pointer like this:
return Native.getPointer(peer + offset);
so that does not retain a reference to the original memory object.
As an idea:
if Memory would override getPointer (and all methods of Pointer, that
return a Pointer objects), it could work like this:
- get the target pointer via the original method
- check if the new pointer points into the Memory object itself
- if the target of the pointer is inside the memory object, discard
the calculated value and replace it with a call to Memory#share
The Pointer/Memory object returnd by Memory#share holds a hard
reference to the Memory object it was created from and thus will
prevent GC of the original memory object.
I thought about creating a ManualMemory object, that is only freed,
then an explicit close method is called, but that won't work, as the
Struture object is hard tied to the GC based Memory object.
Greetings
Matthias
Daniel B. Widdis
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/5a122cc7cfe692a49aba9e6749b8a6dda55b3b49.camel%40doppel-helix.eu.
Durchholz, Joachim
> Early on with no pressure on the heap, this isn't a problem as GC is delayed, but when heap limited, GC is more aggressive and I would guess tries to free larger allocations first.
Don’t even guess, switch on GC tracing and look at the traces.
For any OpenJDK-based JVM since 1.5, you have a copying collector.
Copying collectors never touch unreachable objects, they just ignore them and copy the reachable ones.
Now OpenJDK got a ton of detail optimization since 1.5 (and it wasn’t half bad even in 1.5), and I recall having seen that there’s some special treatment for “humongous” blocks; my assumption that unreachable large blocks don’t get any special treatment may be wrong as well ;-)
Regards,
Jo
Sensitivity: C2 Internal
The content of this e-mail is intended only for the
confidential use of the person addressed.
If you are not the intended
recipient, please notify the sender and delete this e-mail immediately.
Thank
you.
Daniel Widdis
Given I’m a developer busy with a day job and supporting open source in my spare time, not having the equipment to reproduce this myself, and relying on the goodwill of users to do enough logging for me to try to identify the problem, this seems overkill.
But if you’d like to try to reproduce the problem and trace it to confirm my guess, I’d love the input!
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/DB6PR0102MB2725A15B219F794B867CD308B96F9%40DB6PR0102MB2725.eurprd01.prod.exchangelabs.com.
Durchholz, Joachim
Heh. Sorry. That was just a comment from the peanut gallery.
Same situation for me :-)
All I wanted to say that you shouldn’t base decisions on assumptions what GC is doing, that’s risky.
I’m not saying it’s important for the topic at hand :-)
Sensitivity: C2 Internal
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/A8E4E9B1-E37F-43D9-9206-FC305D2CB2B5%40gmail.com.
Daniel Widdis
All good.
I’m only really guessing at the symptoms. But tracing through the code and identifying where references are lost are more deterministic ways of fixing it, which is what I tried to do, and Matthias did much better than I did!
I do appreciate the advice and wouldn’t be surprised if I end up using it for some other problem in the future.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/DB6PR0102MB27250826799C5022770047AAB96F9%40DB6PR0102MB2725.eurprd01.prod.exchangelabs.com.
Daniel Widdis
OK, I'm still confused here.
Our fix(es) are based on the idea that the variable "user" has gone out of scope in this line because we’re passing on a reference not attached to “user”:
return getAccountBySid(user.User.Sid);
Initially I thought this was associated with the method "ending" with the return call, like tail recursion. However, as I've read more, Java does not do any optimization for tail recursion and every reference I see indicates scope lasts until the trailing curly brace. I wrote this test class which seems to confirm the variable is still in scope throughout the return call:
public class Test {
static boolean bar = false;
static Foo foo = null;
static class Foo {
public int baz = 42;
public Foo() {
bar = true;
}
@Override
protected void finalize() {
bar = false;
}
}
static boolean scopeTest() {
foo = new Foo();
System.out.println("Foo is " + foo + ", bar is " + bar);
return tailCall(foo.baz);
}
private static boolean tailCall(int baz) {
// has the scopeTest() method ended yet?
System.gc();
System.gc();
Util.sleep(2000);
System.gc();
System.gc();
System.out.println("Foo is " + foo + ", bar is " + bar);
return bar;
}
public static void main(String[] args) {
scopeTest();
}
}
So despite our “fixes” apparently working, I still don’t understand exactly why Memory is losing the allocation in the return statement.
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/a2fee5e0b535128e3649e74e6c94ea121b85a031.camel%40doppel-helix.eu.
Daniel Widdis
Uh, ignore that. By making foo static it’s always, forever, in scope. Doh!
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/a2fee5e0b535128e3649e74e6c94ea121b85a031.camel%40doppel-helix.eu.
Daniel Widdis
OK, same question, with a better example. I don’t see Foo going out of scope, which should trigger the finalize to switch bar back to false.
Output:
After instantiation, bar is true
After finalize, bar is false
Before call, bar is true
During call bar is true
After call bar is true
Code:
public class Test {
static boolean bar = false;
static class Foo {
public int baz = 42;
public Foo() {
bar = true;
}
@Override
protected void finalize() {
bar = false;
}
}
static boolean scopeTest() {
Foo foo = new Foo();
System.out.println("After instantiation, bar is " + bar);
foo = null;
System.gc();
System.gc();
Util.sleep(2000);
System.out.println("After finalize, bar is " + bar);
foo = new Foo();
System.out.println("Before call, bar is " + bar);
return tailCall(foo.baz);
}
private static boolean tailCall(int baz) {
// has the scopeTest() method ended yet?
System.gc();
System.gc();
Util.sleep(2000);
System.gc();
System.gc();
System.out.println("During call bar is " + bar);
return bar;
}
public static void main(String[] args) {
System.out.println("After call bar is " + scopeTest());
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/a2fee5e0b535128e3649e74e6c94ea121b85a031.camel%40doppel-helix.eu.
Neil C Smith
> Our fix(es) are based on the idea that the variable "user" has gone out of scope ... every reference I see indicates scope lasts until the trailing curly brace
Don't assume scope has any bearing on garbage collection though. A
Java object can be garbage collected before its constructor has even
completed. The discussions that led to the introduction of
Reference.reachabilityFence are an interesting rabbit warren to
explore.
Best wishes,
Neil
--
Neil C Smith
Codelerity Ltd.
www.codelerity.com
Codelerity Ltd. is a company registered in England and Wales
Registered company number : 12063669
Registered office address : Office 4 219 Kensington High Street,
Kensington, London, England, W8 6BD
Matthias Bläsing
OK, I'm still confused here.
Our fix(es) are based on the idea that the variable "user" has gone out of scope in this line because we’re passing on a reference not attached to “user”:
return getAccountBySid(user.User.Sid);
Initially I thought this was associated with the method "ending" with the return call, like tail recursion. However, as I've read more, Java does not do any optimization for tail recursion and every reference I see indicates scope lasts until the trailing curly brace. I wrote this test class which seems to confirm the variable is still in scope throughout the return call:
Daniel Widdis
I knew the fix was a good thing in general, just not why in this particular case!
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
Daniel Widdis
https://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.6.1-200
This bug/fix has been very educational for me!
On 3/14/21, 11:24 AM, "Neil C Smith" <jna-...@googlegroups.com on behalf of ne...@codelerity.com> wrote:
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
Durchholz, Joachim
Going out of scope does not necessarily trigger finalization.
Actually, finalization may never run at all. It’s purely advisory; if you want any guarantees, you need other mechanisms, and even these aren’t guaranteed to be run at any particular point in time.
If you wish to guarantee that a specific action is run, you have to call it from normal code.
If you wish to run it when going out of scope, your only (and actually quite good) option is try-with-resources.
Regards,
Jo
Sensitivity: C2 Internal
From: jna-...@googlegroups.com <jna-...@googlegroups.com> On Behalf Of Daniel Widdis
Sent: Sonntag, 14. März 2021 18:50
To: jna-...@googlegroups.com
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/5C7CF744-355D-4021-886D-FEE952356989%40gmail.com.