Mapping for Structure containing an array of variable sized elements on a continuous memory

[Cnico]

Hello, 

In order to communicate with a legacy application, I have to send a message with the following C definition : 

typedef struct PARAM_TYPE {
    int  cFlag ;
    int  cDataSiz ; // Size of the data
    char *bData         ; // Data with a max size of 64b
} ParamInvoke, *LPParamInvoke;


typedef struct INVOKE_TYPE {

 int  iVal1 ; // Value 1
 char cValue2 ; // Value 2
 char sValue3[16] ; // Value 3
 int  nbParams ; // Number of effective params in the array
 ParamInvoke params[10]                   ; // Array of params whose max size is 10
} Invoke, *LPInvoke;

I could not map this structure with documented JNA samples and the Structure class because at runtime it JNA refuses to write to memory because of the variable size of the ParamInvoke content.

I managed to make it work with a mapping of params which are Structure.ByReference but my legacy application is parsing the memory directly and expects it as continuous with data and not with pointers of data.

My try with Structure.ByValue implementation does not work because of the variable size of the bData.

So I finally found a solution with this implementation : 

public static class ParamInvoke  {
 //Not a JNA structure because of the dynamic size of the data.
 public int cFlag = 0;
 public int cDataSiz = 0;
 public byte[] bData = new byte[1];

}

public static class Invoke extends DynaStructure {
  //Dynamic Structure
 
 public int iVal1 = 0;
 public byte cValue2 = 0;
 public byte[] sValue3 = new byte[16];
 public int nbParams = 0;
 public ParamInvoke[] params= new ParamInvoke[10];


 protected void allocateMemory() {
   memory = new Memory(size);
   long offset = 0;
   memory.setInt(offset, iVal1);
   offset+=Integer.BYTES;
   memory.setByte(offset, cValue2);
   offset+=Integer.BYTES;
   memory.write(offset, sValue3, 0, 16);
   offset+=16;
   memory.setInt(offset, nbParams);
   offset+=Integer.BYTES;
 
   for (int i = 0; i < nbParams ; i++) {
     ParamInvoke paramInvoke = params[i];
     memory.setInt(offset, paramInvoke.cFlag);
     offset+=Integer.BYTES;
     memory.setInt(offset, paramInvoke.cDataSiz);
     offset+=Integer.BYTES;
     memory.write(offset, paramInvoke.bData , 0, paramInvoke.cDataSiz);
     offset+=paramInvoke.cDataSiz;
   }
 
 }
 
 protected void computeSize() {
   this.size = 4 + 4 + 16 + 4;
   for (int i = 0; i < nbParam ; i++) {
     ParamInvoke paramInvoke = params[i];
     this.size += 4 + 4 + paramInvoke.cDataSiz;
   }
 }
}

/**

 * Dynamic JNA Structure for user defined content.

 * Current Structure does not support Array of variable sized elements on a continuous memory.

 * This class address this use case and generally every user manually handled mapping of java objects.

 *

 */

public static abstract class DynaStructure {
 
 //Memory to send.
 protected Memory memory;
 //Size of the memory zone.
 protected int size = -1;


 /**
 * Computes the size of the memory to use and the sets the result in the size attribute.
 */
 protected abstract void computeSize();
 
 /**
 * Allocation of the memory content using the memory protected attribute.
 * Example of implementation :
 * protected void allocateMemory() {
 *    memory = new Memory(size);
 *    long offset = 0;
 *    memory.setInt(offset, val1);
 *    offset+=Integer.BYTES;
 *    ...
 *
 */
 protected abstract void allocateMemory();
 
 private void ensureAllocated() {
   if (memory == null || size == -1) {
     computeSize();
     allocateMemory();
   }
 }
 
 /**
 * @return the pointer to the current memory allocated
 */
 public Pointer getPointer() {
   ensureAllocated();
   return memory.getPointer(0);
 }
 
 /**
 * @return the size of the allocated memory.
 */
 public int size() {
   ensureAllocated();
   return this.size;
 }

 /** Other methods here for toString largely copy pasted from Structure */
 
}

What do you think of my use case ?

Is there a better way of using JNA's Structure to map my C definition ?

Do you think this class DynaStructure should be added to JNA for such use cases ?

Regards,

[Timothy Wall]

You don't need anything so complex.  Just initialize the byte array once you know how big it is, then allow the autoallocation to do the rest.

--


You received this message because you are subscribed to the Google Groups "Java Native Access" group.


To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.


For more options, visit https://groups.google.com/d/optout.

[Cnico]

Do you have an example ?

[Timothy Wall]

There are two situations to account for.

a) JNA allocates the struct

b) Native code allocates the struct and you only have a pointer to it

From the JavaDoc overview:

Variable-sized structures

Structures with variable size, or with primitive array elements, for example:

// Original C code
typedef struct _Header {
  int flags;
  int buf_length;
  char buffer[1];
} Header;

require a constructor which establishes the required size for the structure and initializes things appropriately. For example:

// Equivalent JNA mapping
class Header extends Structure {
  public int flags;
  public int buf_length;
  public byte[] buffer;
  public Header(int bufferSize) {
    buffer = new byte[bufferSize];
    buf_length = buffer.length;
    allocateMemory();
  }
}

It's generally recommended to initialize the `buffer` field, e.g. (javadoc needs updating to reflect this):

public byte[] buffer = new byte[1];

The constructor for the native-allocated case would look liked this:

public Header(Pointer p) {
    // Read the desired size from whatever offset it's at
    // Alternatively you might use "readField()", but that
    // might get complicated
    this.buf_length = p.readInt(SIZE_OFFSET);
    buffer = new byte[this.buf_length];
    allocateMemory();
  }

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+unsubscribe@googlegroups.com.

[Cnico]

I am in the first situation : my java code, so JNA, allocates the struct.

Here is my try :

public static class ParamInvoke extends Structure {

  public ParamInvoke() {
  }

  public ParamInvoke(int cFlag , byte[] data) {
         this.cFlag = cFlag ;
         this.cDataSiz = data.length;
         this.bData = data;
         allocateMemory();

  }

 public int cFlag = 0;
 public int cDataSiz = 0;
 public byte[] bData = new byte[1];

 protected List<String> getFieldOrder() {
   return Arrays.asList(new String[] { "cFlag", "cDataSiz", "bData" });
 }

}

public static class Invoke extends Structure {

  public SagaInvoke() {

  }

 public int iVal1 = 0;
 public byte cValue2 = 0;
 public byte[] sValue3 = new byte[16];
 public int nbParams = 0;
 public ParamInvoke[] params= new ParamInvoke[10];

  protected List<String> getFieldOrder() {
            return Arrays.asList(new String[] { "iVal1", "cValue2", "sValue3", "nbParams", "params" });
        }
}





The empty constructor of ParamInvoke is required by JNA because otherwise, I get an exception when initializing : SagaInvoke :
java.lang.IllegalArgumentException: Invalid Structure field in class test.MainPOCInterop$Invoke, field name 'params' (class test.MainPOCInterop$ParamInvoke): Can't instantiate class test.MainPOCInterop$ParamInvoke
    at com.sun.jna.Structure.validateField(Structure.java:1109)
    at com.sun.jna.Structure.validateField(Structure.java:1101)
    at com.sun.jna.Structure.validateFields(Structure.java:1119)
    at com.sun.jna.Structure.<init>(Structure.java:179)
    at com.sun.jna.Structure.<init>(Structure.java:172)
    at com.sun.jna.Structure.<init>(Structure.java:159)
    at com.sun.jna.Structure.<init>(Structure.java:151)
    at test.MainPOCInterop$Invoke.<init>(MainPOCInterop.java:301)
    at test.MainPOCInterop.messageRun(MainPOCInterop.java:447)
    at test.MainPOCInterop$5.widgetSelected(MainPOCInterop.java:158)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at test.MainPOCInterop.main(MainPOCInterop.java:90)
Caused by: java.lang.IllegalArgumentException: Can't instantiate class test.MainPOCInterop$ParamInvoke
    at com.sun.jna.Structure.newInstance(Structure.java:1781)
    at com.sun.jna.Structure.newInstance(Structure.java:1759)
    at com.sun.jna.Structure.size(Structure.java:1030)
    at com.sun.jna.Native.getNativeSize(Native.java:1172)
    at com.sun.jna.Structure.getNativeSize(Structure.java:2072)
    at com.sun.jna.Structure.getNativeSize(Structure.java:2062)
    at com.sun.jna.Structure.validateField(Structure.java:1105)
    ... 16 common frames omitted
Caused by: java.lang.InstantiationException: test.MainPOCInterop$ParamInvoke
    at java.lang.Class.newInstance(Unknown Source)
    at com.sun.jna.Structure.newInstance(Structure.java:1773)
    ... 22 common frames omitted
Caused by: java.lang.NoSuchMethodException: test.MainPOCInterop$ParamInvoke.<init>()
    at java.lang.Class.getConstructor0(Unknown Source)
    ... 24 common frames omitted


With the empty constructor, my problem is that at runtime, the data sent to the wire is not correct.
The first ParamInvoke element of the array is for example of size 20 and it writes correctly to the output buffer.
But the second ParamInvoke parameter is written in memory at wrong offset, overriding my first parameter instead of being placed just next to it.
The override happens exactly at the place of bData's array size, i.e : 4 + 4 + 4 bytes are correct but next bytes of my first ParamInvoke are replaced by bytes of the second...

I suspect it the need of the default constructor that is problem, but I am not sure.

Note : I use jna 4.2.2

To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

[Timothy Wall]

You're not storing an array, you're storing a series of differently-sized items contiguously in memory.

If your structures are of different sizes, even though they're contiguous memory, it's not an array.  You shouldn't pretend it is one.  You certainly can't access it in native code as an array, e.g.

    saga.params[2].bData = malloc(100); // try to compile THAT, my friend

You can't do that in native code, you can't do it with JNA mapping, either.

Usually when dealing with memory buffers of different sizes, a pointer to the item is used, rather than adding a bunch of overhead calculating offsets into a generic buffer (which is essentially what you're doing).

To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+unsubscribe@googlegroups.com.

[Timothy Wall]

What you should do here instead is treat the entire block of params as a buffer (`Memory` will work), and then provide a method on the structure to access any given param at a given offset into that buffer, e.g.

    class MyStructure {

        public Param getParam(int n) {

            // Recursive implementation, in production use a loop or cache the data

            int params_offset = xxx; // offset of the first param within the structure

            Pointer buffer = getPointer(params_offset);

            if (n == 0) {

               return new Param(buffer.share());

            }

            Param param = getParam(n-1);

            Pointer p = param.getPointer().share(param.cDataSize);

            return new Param(p);

        }

    }

[Cnico]

I am not a C expert and your are certainly right it is not an array. To be honnest, the legacy C code that uses this structure is made of pointers manipulations.

Thank you very much for you answers and the mapping you propose,