struct get different result with same code and process same data

39 views
Skip to first unread message

星野はじめ

unread,
May 16, 2024, 6:13:19 AMMay 16
to Java Native Access
run my code in linux host, I can get true result
but run in docker container the result is wrong

C code:

struct Buf
{
char *buf;
size_t buf_size;
size_t buf_used_size;
};

extern "C" {
Buf *bufs(Record *ptr) {
unsigned int *count = 0;
return ptr->Cols(count);
}
}

java code:

@FieldOrder({"buf", "buf_size", "buf_used_size"})
public class Buf extends Structure {

public Buf() {
super(ALIGN_NONE);
}

public Pointer buf;
public long buf_size = 0L;
public long buf_used_size = 0L;

public static class ByReference extends Buf implements Structure.ByReference {
}

public static class ByValue extends Buf implements Structure.ByValue {
}
}

public interface TestLibrary extends Library {

RecordColStruct.ByReference bufs(Pointer ptr);
}

public void test() {
var bufPtr = testLib.bufs(ptr);
bufPtr.read();
var array = (RecordColStruct[])bufPtr.toArray(count);

for (int i = 0; i < count; i++) {
var col = array[i];
var size = (int) col.buf_used_size;
var size2 = (int) col.buf_size;
logger.info("index: {}, size:{}, size2:{}",i,size,size2);
}
} 

run with same code and same data, get different result

host result:
index: 0, size:5, size2:5
index: 1, size:5, size2:5
index: 2, size:5, size2:5
index: 3, size:5, size2:5
index: 4, size:4, size2:4
index: 5, size:5, size2:5
index: 6, size:0, size2:0
index: 7, size:0, size2:0
index: 8, size:16, size2:16
index: 9, size:5, size2:5
index: 10, size:5, size2:5
index: 11, size:5, size2:5
index: 12, size:218, size2:218
index: 13, size:46, size2:46
index: 14, size:10, size2:10
index: 15, size:8, size2:8

docker result:
index: 0, size:5, size2:5
index: 1, size:5, size2:96542240
index: 2, size:96542368, size2:0
index: 3, size:0, size2:5
index: 4, size:5, size2:5
index: 5, size:4, size2:96542624
index: 6, size:96542752, size2:0
index: 7, size:0, size2:5
index: 8, size:0, size2:0
index: 9, size:0, size2:0
index: 10, size:96600720, size2:0
index: 11, size:0, size2:16
index: 12, size:5, size2:5
index: 13, size:5, size2:96600960
index: 14, size:96601072, size2:0
index: 15, size:0, size2:5

Matthias Bläsing

unread,
May 16, 2024, 1:37:27 PMMay 16
to jna-...@googlegroups.com
Hi,

the most important aspect: not for all architectures the equation sizeof(long) == sizeof(size_t) holds true. size_t is large enough to hold the size of a a theoretical object.
I.e. on 32bit platforms size_t is most probably 32bit and on 64bit platforms 64bit.

As long as you are on 64bit using java long for a native size_t will work, if your docker container is a 32bit environment, you have you explanation.

Greetings

Matthias
--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/e83b50dc-e183-493e-88ab-b3dbade95662n%40googlegroups.com.

Matthias Bläsing

unread,
May 16, 2024, 1:40:07 PMMay 16
to jna-...@googlegroups.com
Hi,

additional info:

  • sizeof(size_t) is available at runtime in JNA through: com.sun.jna.Native.SIZE_T_SIZE
  • size_t is bound in jna-platform: com.sun.jna.platform.unix.LibCAPI.size_t

HTH

Matthias
Message has been deleted
Message has been deleted

星野はじめ

unread,
May 16, 2024, 10:21:11 PMMay 16
to Java Native Access
docker run on the same host, cpu is the same, both 64-bit

星野はじめ

unread,
May 16, 2024, 10:36:51 PMMay 16
to Java Native Access
host cpu:
➜ ~ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7K62 48-Core Processor
Stepping: 0
CPU MHz: 2595.122
BogoMIPS: 5190.24
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat

container cpu:
➜ ~ docker exec -it obcdc4 bash
bash-4.2# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7K62 48-Core Processor
Stepping: 0
CPU MHz: 2595.122
BogoMIPS: 5190.24
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat
bash-4.2#

image inspect:
➜ ~ docker image inspect devops.datapipeline.cc/datapipeline/dp-dev/snapshot/obcdc4:test
[
{
"Id": "sha256:32d2aef8b80192ef4112ffa4fbe13778792ae0e28a92137ee47093710a3bc410",
"RepoTags": [
],
"RepoDigests": [],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-05-16T10:32:38.876837921+08:00",
"Container": "",
"ContainerConfig": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": null,
"Cmd": null,
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"5000/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/opt/jdk/bin:/opt/maven/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LD_LIBRARY_PATH=/home/admin/oceanbase/lib",
"HOME=/data",
"JAVA_HOME=/opt/jdk",
"TERM=dumb"
],
"Cmd": [
"top",
"-b",
"-d",
"60"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/data",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"io.buildah.version": "1.32.2",
"org.label-schema.build-date": "20201113",
"org.label-schema.license": "GPLv2",
"org.label-schema.name": "CentOS Base Image",
"org.label-schema.schema-version": "1.0",
"org.label-schema.vendor": "CentOS",
"org.opencontainers.image.created": "2020-11-13 00:00:00+00:00",
"org.opencontainers.image.licenses": "GPL-2.0-only",
"org.opencontainers.image.title": "CentOS Base Image",
"org.opencontainers.image.vendor": "CentOS",
"pkgscan.task.pid": "8c138ad2f65111ee830feee6083cdae2"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 3617603107,
"VirtualSize": 3617603107,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/4fpk5kgus79r3shh5y9htcawt/diff:/var/lib/docker/overlay2/iftw27nybdqs6v4vc53buor00/diff:/var/lib/docker/overlay2/8phc9bmcslva1y2w1uc572frz/diff:/var/lib/docker/overlay2/z6380qkeryukva7qxu5putcow/diff:/var/lib/docker/overlay2/jnrd8mfuu80beeiswa49xl5p8/diff:/var/lib/docker/overlay2/xmsommjgv27awkjhkibdvknjx/diff:/var/lib/docker/overlay2/mmxwp9g7odutlff2mslp33xxf/diff:/var/lib/docker/overlay2/5v293hs5dv3k6zpgc2ks3b21q/diff:/var/lib/docker/overlay2/q6k5cw0vwrjztvi165or9yiie/diff:/var/lib/docker/overlay2/0ukrwooohjnpdb2h77juotfyg/diff:/var/lib/docker/overlay2/5660xc69iwx6hhfe3c8io4cty/diff:/var/lib/docker/overlay2/k2hnjlw4ll6clpe14kav9diql/diff:/var/lib/docker/overlay2/06768c2651ae5836ce979bedf59312ef29c3be4d1f1caeb3b4ca783190342959/diff:/var/lib/docker/overlay2/bb82314ac1f4380a3921824d9fb6eb3005960832b630efe8a03a05d0da751ee3/diff:/var/lib/docker/overlay2/0d743e7f27860a35482ff60683240f8fff320f390247d2a09864a1c4f66f7b33/diff",
"MergedDir": "/var/lib/docker/overlay2/iekahsohe5v113fl7bnt9pmfc/merged",
"UpperDir": "/var/lib/docker/overlay2/iekahsohe5v113fl7bnt9pmfc/diff",
"WorkDir": "/var/lib/docker/overlay2/iekahsohe5v113fl7bnt9pmfc/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02",
"sha256:e99430acc719185257eaa41632dc1c971754e347cf10e31944101d62ffec5026",
"sha256:2bbfb6856bf04c3b2de6886e1c2580ede94fc698ca3736afad2e61135ae0e3b5",
"sha256:e782521e651517504e309f731a55a2a56305569a819892ce7398d6d0025f491c",
"sha256:9bec9e8cb74bbdd1d6472ffa24b52021822742973f2cebe7b9137ccef2c8c9bd",
"sha256:8f3817bfcc5a5fe434fe38eabc63840742aa50a4e49b0d8286edee95ef029392",
"sha256:820e95db26b947712a0cd6afd4f3c8345564c4a334c3dec6db359ddb392b6fee",
"sha256:4d6485ae21787a89ddb15cf33a6b3fcdb37537d8fa25d1660305ae7064f2ded3",
"sha256:3b2351c33c1cc968b5ea6645d878d08ac540edf12d73ab558a5046e3ed03d7dd",
"sha256:41ed5e8ee6a785668d8e8e4307e0c31bb72bbec64d41e96a98551590fa65f775",
"sha256:44f64419e2661aa97d60fbf7e8091b32f16f663e0b9f8356917a014e4755a9ed",
"sha256:f6d357aa2acd98f1268138587f86a0e6709f2dd3716049f48d7e2c401f3d0175",
"sha256:a88815fd7a6964194b791b57840dc7dbc81cfbdc50ec35171963e5c8fa88707a",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
"sha256:30cec146736ad31d3b175822bbb3e2643399309f1e9f49f87067bc24a325e7d3",
"sha256:ffbabe3c0a2ea82877d3befb8e4b275489ab7e76fa36212275790b2d943c2735"
]
},
"Metadata": {
"LastTagTime": "2024-05-16T10:32:41.617155294+08:00"
}
}
]

Architecture is amd64

星野はじめ

unread,
May 16, 2024, 10:46:30 PMMay 16
to Java Native Access

I used LibCAPI.size_t get the same result

Daniel B. Widdis

unread,
May 17, 2024, 12:38:02 AMMay 17
to jna-...@googlegroups.com
Looking at the values they seem to follow a pattern. You can see the repeating 5's, 4's, and 0's in the container output but spread out a bit more.

The first 4 in host data is in index 4, in the docker data it appears in index 5.
The first 0 in host data is in index 6, in the docker data there are lots of stray 0's but a steady stream appears in index 8
The first 16 in host data is in index 8, in the container data it appears at index 11.
The first 218 in host data is in index 12, we never see it in the container data meaning it's past index 15.

The docker data also has lots of random large values which could be unwritten memory, or pointer values found where not expected.

So generally it looks like the data in the docker memory fills out more space.  Hard to specifically align (pun intended) the values but it seems close to a 4:3 (16:12) ratio.

It's possible there is different alignment / padding.

Can you do a sizeof(Buf) in both environments to see if the structure sizes differ?  Are there any configurations for Docker on how it aligns bytes?



--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/4f388a07-3647-476a-b559-562d5a286efcn%40googlegroups.com.


--
Dan Widdis

星野はじめ

unread,
May 17, 2024, 1:48:06 AMMay 17
to Java Native Access
I test size with this code, both size is 24. I don't know the memory align configuration of docker. 

 @Structure.FieldOrder({"buf", "buf_size", "buf_used_size"})
  public static class Buf extends Structure {


    public Buf() {
      super(ALIGN_NONE);
    }

    public Pointer buf;
    public LibCAPI.size_t buf_size = new LibCAPI.size_t();
    public LibCAPI.size_t buf_used_size = new LibCAPI.size_t();

    public static class ByReference extends com.datapipeline.obcdc.obcdc4.lib.RecordColStruct implements Structure.ByReference {
    }

    public static class ByValue extends com.datapipeline.obcdc.obcdc4.lib.RecordColStruct implements Structure.ByValue {
    }
  }

  @Test
  public void testSize() {
    var buf = new Buf();
    System.out.println("sizeof buf: " + buf.size());
  }

星野はじめ

unread,
May 17, 2024, 1:58:52 AMMay 17
to Java Native Access
I only wrote some data above, and below is the complete test data

host:

index: 0, size:5, size2:5
index: 1, size:4, size2:4

index: 2, size:5, size2:5
index: 3, size:5, size2:5
index: 4, size:5, size2:5
index: 5, size:5, size2:5
index: 6, size:0, size2:0
index: 7, size:0, size2:0
index: 8, size:16, size2:16
index: 9, size:5, size2:5
index: 10, size:5, size2:5
index: 11, size:5, size2:5
index: 12, size:44, size2:44
index: 13, size:89, size2:89

index: 14, size:10, size2:10
index: 15, size:8, size2:8
index: 16, size:4, size2:4
index: 17, size:25, size2:25
index: 18, size:23, size2:23
index: 19, size:207, size2:207
index: 20, size:693, size2:693
index: 21, size:331, size2:331
index: 22, size:684, size2:684
index: 23, size:48, size2:48
index: 24, size:272, size2:272
index: 25, size:46, size2:46
index: 26, size:494, size2:494
index: 27, size:0, size2:0
index: 28, size:0, size2:0
index: 29, size:5, size2:5
index: 30, size:2, size2:2
index: 31, size:70, size2:70

docker:

index: 0, size:5, size2:5
index: 1, size:4, size2:-1553917648
index: 2, size:-1553917520, size2:0

index: 3, size:0, size2:5
index: 4, size:5, size2:5
index: 5, size:5, size2:-1553917264
index: 6, size:-1553917136, size2:0

index: 7, size:0, size2:5
index: 8, size:0, size2:0
index: 9, size:0, size2:0
index: 10, size:-1553916832, size2:0

index: 11, size:0, size2:16
index: 12, size:5, size2:5
index: 13, size:5, size2:-1553916592
index: 14, size:-1553916480, size2:0

index: 15, size:0, size2:5
index: 16, size:44, size2:44
index: 17, size:89, size2:-1553923515
index: 18, size:-1553916176, size2:0
index: 19, size:0, size2:10
index: 20, size:8, size2:8
index: 21, size:4, size2:-1553915936
index: 22, size:-1553858192, size2:0
index: 23, size:0, size2:25
index: 24, size:23, size2:23
index: 25, size:207, size2:-1553923401
index: 26, size:-1553923190, size2:0
index: 27, size:0, size2:693
index: 28, size:331, size2:331
index: 29, size:684, size2:-1553922158
index: 30, size:-1553921474, size2:0
index: 31, size:0, size2:48

Daniel B. Widdis

unread,
May 17, 2024, 3:37:00 AMMay 17
to jna-...@googlegroups.com
Still looking like an alignment thing.
  • The pattern of large numbers is consistent, indicating gaps of undefined data.
  • The comparison shows the same number at a different offset: "size:207" appears on host at index 19, in docker it's index 25.  This is pretty much a 4:3 (24:18) ratio of space.
Given a size of 24, it seems reasonable to think it's aligned to 16-byte boundaries, and on docker possibly includes 8 padding bytes between elements.

It's probably not that simple. Or maybe it is. Have you tried a different alignment than ALIGN_NONE?



To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/9c70a9d7-b853-4df0-88ad-92627e119b22n%40googlegroups.com.


--
Dan Widdis

星野はじめ

unread,
May 17, 2024, 11:16:13 AMMay 17
to Java Native Access
docker run on the same host, cpu is the same, both 64-bit

星野はじめ

unread,
May 17, 2024, 11:16:17 AMMay 17
to Java Native Access
docker run on the same host,  the cpu is same, both 64-bit

在2024年5月17日星期五 UTC+8 01:37:27<mbla...@doppel-helix.eu> 写道：

Matthias Bläsing

unread,
May 17, 2024, 3:00:06 PMMay 17
to jna-...@googlegroups.com
Am Donnerstag, dem 16.05.2024 um 19:18 -0700 schrieb 星野はじめ:
docker run on the same host, cpu is the same, both 64-bit

The CPU is not relevant here. amd64 CPUs can run 32bit and 64bit code. If you run a 32bit userland, you will still have the wrong size of size_t.

BUT I just read this:

public Buf() {
super(ALIGN_NONE);
}

I see no pragmas in your headers, if there is none, this alignment request is wrong. Don't specify alignment if you don't know what is the right one!

To make progress please create a minimal sampe and provide:

a) the project with the java code
b) the headers you base your code on
c) if possible the library itself

Greetings

Matthias
Reply all
Reply to author
Forward
0 new messages