DLL interface 15!:20 and 15!:21

[Thomas McGuire]

I have been playing around trying to make a curl call with libcurl in J. This particular rabbit hole came about because I was just going to implement a small openai api I found on GitHub. The C code utilizes libcurl to make the access to chatGPT. I started to think that I would like to control the libcurl code in J. Now there is a major trap in the libcurl code as it relies on a callback function to capture the HTTP response data. I haven’t tackled that part as yet. 

Many times when I need access to a DLL I have been writing routines to hand me pointers in the original code and using that to give J greater access. As I was working on this, the thought occured to me that there must be some direct access from J to the variables in DLL. After all the function signatures just reference the name of the function that is in the DLL and everything gets found. 

Reviewing the 15!: foreigns I came across 15!:21 (get procedure address) and that needs 15!:20 (get library handle) to be run first to get a numeric handle for the DLL. So the usage is as follows: 

hlibopenai =: (15!:20) jpath '~/cppdev/openai-c/build/libopenai.dylib'

hlibopenai

15340937760

hlibopenai (15!:21) 'write_callback' NB. procedure in lib

4841714632

hlibopenai (15!:21) 'print_slist' NB. procedure in lib

4841714464

hlibopenai (15!:21) 'chunk' NB. variable in lib

4841734160

So the interesting thing is that even though 15!:21 is called "get procedure address", it seems to get the address of any named variable or function in the C DLL (assuming you know the correct name). Which would make sense because C doesn’t make much of a distinction between the address of a variable or function. 

I gather the use of this function was intended to be able to pass C callbacks into J and since it works on variables you don’t need to write C interface routines of the following format: 

void * chunkptr() {

  struct memory *chunkptr;

  chunkptr = &chunk;

  return chunkptr;

}

I have the following interface for the above function in my DLL interface in J:

chunkptr =: (LIBOPENAI,' chunkptr *') & cd

chunkptr_OpenAI_ ''

┌──────────┐

│4841734160│

└──────────┘

So this confirms that 15!:21 is passing the correct address.

As I said the above is more for documentation purposess since an ArrayPortal on 15!:20 only brings up 3 references none of which are an actual implementation. In the future this forum post might be helpful.

Now for the documentation and usage questions:

The documentation in the foreigns page for 15! indicates the y argument for both 15!:20 and 15!:21 is supposed to be a boxed string. Yet it accepts a single string. Is this just documenting possible future functionality?

Usually addresses are boxed to be used in calls. While the library handle isn’t necessarily a typical address, should it be boxed when it is used as an argument into 15!:21 just to be consistent?

Should I modify the documentation (https://code.jsoftware.com/wiki/Vocabulary/Foreigns#m15) and take out the word “Boxed” in the parameter entry for table row on both 15!:20, 15!:21 and add in that the y parameter for 15!:21 can be a procedure name or a variable name?

Tom McGuire

[Eric Iverson]

Thomas,

re: DLL callback

Have you looked at the DLL labs? In the System category there is a lab called: Shared Library... - callback. This lab might be relevant to what you are trying to  do.

To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@jsoftware.com.

[Henry Rich]

Yes, please update NuVoc as you see fit.  Here it what JE does:

15!:20

1. audit for byte string (NOT boxed or other char type)
2. if nonempty string:
 2a. if Windows, convert string to wide chars & open with LoadLibraryW, getting handle
 2b. if wasm, error
 2c. otherwise open with dlopen(string,RTLD_LAZY), getting handle
3. otherwise (empty string)
 3a. If Windows, call GetModuleHandle(NULL), returning handle for current module
 3b. if wasm, error
 3c. otherwise return dlopen(0,RTLD_LAZY)

15!:21

1. audit y for nonnull byte string
2. audit x for atomic integer
3a. On Windows, return GetProcAddress result
3b. on wasm, error
3c. otherwise, return dlsym(x,y)

Henry Rich

To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@jsoftware.com.

[bill lam]

Did you mean 15!:20/21 work as expected but documentation needs improvement?

[Thomas McGuire]

I meant there is scant documentation and the documentation that is there didn’t slow me down in playing with the functions because I neglected to box the string in the first place and was surprised that it worked. So part of the posting is just documenting what I tried and how it worked since the foreign’s page does not have a lot of room for examples.

However, for 15!:20 I could potentially see that it may work on a boxed list of strings and return a boxed list of handles. So I was also asking is the intended functionality as it is operating currently or was the documentation pointing to some future functionality that has not been implemented yet. I will go in and change the documentation on the foreigns page but if there may be future stuff I would add in that it may change in the distant future.

Tom McGuire

[Thomas McGuire]

Funny you should ask because I was going to post a separate thread on the subject. So here is a short snippet of C code that uses the libcurl functions:

    curl_easy_setopt(curl, CURLOPT_URL, url_base);

    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);

    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, json);

    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);

    curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void*)&chunk);

    CURLcode res = curl_easy_perform(curl);

So in implementing part of the FFI for libcurl in J the curl_easy_setopt function interface is this:

curl_setopt=: (LIBCURL,' curl_easy_setopt n * i *') & cd

and the similar snippet of code in J is as follows:

hlibopenai =: (15!:20) LIBOPENAI

mycbp =: hlibopenai (15!:21) 'write_callback'

chunkp =: hlibopenai (15!:21) 'chunk'

headers =: <0

headers =: {. curl_slist_append (headers);'Content-Type: application/json'

NB. headers =: {. curl_slist_append (headers);'Authorization: Bearer sk-123'

curl_setopt C;CURLOPT_URL;url

curl_setopt C;CURLOPT_HTTPHEADER;<headers

NB. curl_setopt C;CURLOPT_POSTEFIELDS;json

curl_setopt C;CURLOPT_WRITEFUNCTION;mycbp

curl_setopt C;CURLOPT_WRITEDATA;<chunkp

where C is a pointer returned by the library from “curl_init" that save up all these options before you would call “curl_perform”

The problem is that I get a domain error when running:

curl_setopt C;CURLOPT_WRITEFUNCTION;mycbp

I have tried the boxed form and placed <mycbp at the end, I have also tried it with a J callback cp4 =: cdcb 4. Both addresses are integer types in J.

Both cause a Domain Error. the cder’’ comes back with 6 2 indicating it has a problem with my callback address.

Not sure why it’s not accepting it. Do I need to pack the address with ‘ic’ before I use it as a parameter?

I will probably code around this and see if I can get access to the ‘chunk’ data from C. It is implemented in a way that I may be able to ignore the C ’struct’ and treat it as a ‘memr’ of a string. 

But if anyone sees something wrong in the way I am implementing this I appreciate the help. 

Tom McGuire

[Henry Rich]

We have no plans for changes to 15!:20-21.  I wouldn't allude to any
possible future changes in NuVoc.  The reader should assume that any
part of J is subject to future improvement.

The 15!:x features were documented in the Dictionary only up to 15!:5. 
Much of NuVoc's treatment of them is based on my experimentation before
I had access to the JE source.  Don't treat it as gospel.

Henry Rich

[bill lam]

Try

curl_easy_setopt n * x *

[Thomas McGuire]

So the x works in place of i for the constant as suggested by Bill Lam. However, from the cder‘’ being 6 2 the error message means it has a problem with the third parameter to the curl_setopt call (and still does despite the change). 

So this made me redouble my efforts on the call sequence and the problem stems from how the “link" verb (;) works. 

1;(<3);1

┌─┬───┬─┐

│1│┌─┐│1│

│ ││3││ │

│ │└─┘│ │

└─┴───┴─┘

1;3;<1

┌─┬─┬─┐

│1│3│1│

└─┴─┴─┘

1;<3;<<1

┌─┬───────┐

│1│┌─┬───┐│

│ ││3│┌─┐││

│ ││ ││1│││

│ ││ │└─┘││

│ │└─┴───┘│

└─┴───────┘

nuvoc indicates that link: x ; y creates a boxed list out of x and y with the caveat that if y is already boxed it is not boxed again. This means you get the functionality of the first 2 examples above. 

I always think of link as just boxing items together and looked at the apparent concatenation of the last item (if it’s boxed) as an annoying implementation detail of the verb. However thinking about it is actually a necessity of the implementation since J evaluates from right to left. If link (;) did box the right operand you would create boxing of the type of the 3rd example since after the right most 2 items are “link”ed they would appear as boxed to the next “link” verb on the left. You would end up with a nested box structure that would look something like the 3rd example above.

What does this “link” tutorial have to do with my problem with the call to curl_setopt?

Bill Lam’s suggestion made me redouble my efforts in thinking about the parameters being sent in and I realized that the header structure I had created was being accepted (see the call sequence below). It is just an address as is my callback and as is the chunk (which was also having a problem if I commented out the callback option statement). However it turns out that the pointer address returned by the single linked list routine from the C libcurl library is different than the address returned by 15!:21.

Call sequence from my code:

headers =: <0

headers =: {. curl_slist_append (headers);'Content-Type: application/json'

NB. headers =: {. curl_slist_append (headers);'Authorization: Bearer sk-123'

curl_setopt C;CURLOPT_URL;url

curl_setopt C;CURLOPT_HTTPHEADER;<headers

NB. curl_setopt C;CURLOPT_POSTEFIELDS;json

curl_setopt C;CURLOPT_WRITEFUNCTION;<mycbp

curl_setopt C;CURLOPT_WRITEDATA;<chunkp

Differences between headers vs mycbp and chunkp:

headers__mchat

┌───────────────┐

│105553163181072│

└───────────────┘

mycbp__mchat

4621022152

chunkp__mchat

4621041680

This all comes down to appropriate boxing of arguments and the functioning of the “link” (;) verb. Because headers is already returned as boxed from its creation in curl_slist_append you just need a single box operator to pass it in correctly:

parameters from HTTPHEADER setopt:

C__mchat;CURLOPT_HTTPHEADER__mchat;<headers__mchat

┌────────────┬─────┬─────────────────┐

│┌──────────┐│10023│┌───────────────┐│

││4689265152││ ││105553163181072││

│└──────────┘│ │└───────────────┘│

└────────────┴─────┴─────────────────┘

Parameters from the callback pointer call to setopt:

C__mchat;CURLOPT_WRITEFUNCTION__mchat;<mycbp__mchat

┌────────────┬─────┬──────────┐

│┌──────────┐│20011│4621022152│

││4689265152││ │ │

│└──────────┘│ │ │

└────────────┴─────┴──────────┘

so the callback address needs to be double boxed to get the appropriate boxing for the parameters.

C__mchat;CURLOPT_WRITEFUNCTION__mchat;<<mycbp__mchat

┌────────────┬─────┬────────────┐

│┌──────────┐│20011│┌──────────┐│

││4689265152││ ││4621022152││

│└──────────┘│ │└──────────┘│

└────────────┴─────┴────────────┘

The double boxing indeed get these parameters accepted without error. 

Thanks everyone for all the help and sorry my poor mental constructs around J caused me to ask you about a simple problem I should have known about.

Tom McGuire

[bill lam]

My usual practice is to use x instead of * in the protocol, and then no need to worry about boxed arguments.

[Henry Rich]

It sounds like you have solved your problem.

(x ; y) works well for unboxed y but there is that troublesome exception for boxed y.  I think of the last in a series of ; as being one of

   ;    ,&<    (,<)

If nothing is boxed, I use a;b;c;d.  If the last might be boxed, I use a;b;c,&<d.  If something other than the last might be boxed - say I had (a;b;c);d - I use (a;b;c)(,<)d.

These are all implemented as primitives in JE.

Henry Rich