C api .gw.asyncexec

[hoffmanroni]

Hello, I have a client asking about getting a callback using c api from the .gw.asyncexec func.  I have only really used this in q myself by passing in q function names which works for me. Wondering if you can help?

Is it possible to get a callback here in .gw.asyncexecjpts using c api by passing in the name of a callback function?

Also he is asking about closue over the calling context to route the response back to the correct user.

Thanks very much,

[Matthew Moore]

Hi hoffmanroni,

The officially supported APIs for Java, C# and C will allow asynchronous calls.

When using the .gw.asyncexec, the easiest way to use this is to block from the client on the handle. This is what's called deferred synchronous. The client waits for a result but as the query is sent asynchronously, this frees the gateway up to decide when to process the request. As your client is concerned about routing the result back to the correct user, when blocking on the handle in this way, the gateway will route the result back down the socket it came from..gw.asyncexecjpt can be used to get a callback by defining a callback function with a lambda or by using a callback function already defined in the gateway process. It is worth noting that jpt at the end refers to join, postback and timeout respectively. These are the additional parameters required when using .gw.asyncexecjpt.Example of this from a client q process:

h:hopen `::34007:admin:admin //this connects to a localhost gateway process in my TorQ stack
(neg h)(`.gw.asyncexecjpt;"5#select time, sym, price, size from trade where sym = `MSFT";`hdb`rdb;raze;`handleresults;0D00:00:30); myresult:h(::)
myresult[2]

time                          sym  price size
---------------------------------------------
2019.08.20D00:00:00.439626000 MSFT 17.31 92
2019.08.20D00:00:00.439626000 MSFT 17.32 43
2019.08.20D00:00:01.927037000 MSFT 17.38 39
2019.08.20D00:00:02.925681000 MSFT 17.26 90

Breaking down what's going on here:

(neg h)(`.gw.asyncexecjpt; //neg h for async calls

"5#select time, sym, price, size from trade where sym = `MSFT"; //this is the query the gateway will process
//it is also worth noting that accessing the hdb with no date in the where clause is generally not advised but my hdb only has 2 dates so this query does not cause any issues.

`hdb`rdb; //the servers you want to query

raze; //the join function for combining the results of more than 1 server

`handleresults; //this is a basic postback function I have defined in the gateway process (shown below)
handleresults:{-1(string .z.z)," got results"; -3!x; show y} 
//this returns a list of 3 items, 1st is the time results are received, the 2nd is the query and the 3rd is the result returned which in this example is a table

0D00:00:30) //this is the time allowed before timeout. In this case 30 seconds.

myresult:h(::) //this blocks on the handle and assigns the results to a variable called myresult

As the result/table is the 3rd item in my handleresults postback you can access it by indexing:
myresult[2]
You should also ensure that async calls are allowed from your gateway. Check that .gw.synccallsallowed is set to 0b otherwise you will get  a 'asynchronous calls are not allowed error.

Now utilising this with an example with C API on x64 Linux:

//file name: gwapi,c
#include"k.h" //this allows the use of K objects for kdb, ensure this is in the same directory as this file
#include"time.h" //this used by the printT function and is included in the standard c library
#include<stdio.h>
#include <stdlib.h>

void printTime(J t) { //this function presents kdb times in c
    time_t timval= t / 1000000000;
    struct tm *timeInfo= localtime(&timval);

    printf("%dD%02d:%02d:%02d.%09lld ", 
        timeInfo->tm_yday,
        timeInfo->tm_hour,
        timeInfo->tm_min,
        timeInfo->tm_sec,
        t % 1000000000);
}
int main(){
        I i;
        K response, table, columnNames, columnValues;
        int c=khpu("localhost",34007, "admin:admin");                   //connect to my gateway process in TorQ
        k(-c,".gw.asyncexecjpt[\"5#select time, sym, price, size from trade where sym = `MSFT\";`hdb`rdb;raze;`handleresults;0D00:00:30]",(K)0);
        //negative c for async call
        //same .gw.asyncexecjpt query as demonstrated in q example

        response= k(c,(S)0); //flush - this blocks on the handle awaiting a response from the gateway
        printf(" Callback Function: %s\n",kK(response)[0]->s); //this displays the callback function name which was called
        table= kK(response)[2]->k; //gets the table from the response
        columnNames= kK(table)[0]; //gets the column headings
        columnValues= kK(table)[1]; //gets the column values

        printf(" %s %s %s %s\n", kS(columnNames)[0], kS(columnNames)[1], kS(columnNames)[2], kS(columnNames)[3]);
        printf(" ---------------------------------------------\n");
        //this sets up the column headers for the table

        for(i=0; i < kK(columnValues)[0]->n; i++)  {
        //this for loop prints the values for each column, row by row
                printTime(kJ(kK(columnValues)[0])[i]);
                printf("%s ", kS(kK(columnValues)[1])[i]);
                printf("%lf ", kF(kK(columnValues)[2])[i]);
                printf("%lld \n", kJ(kK(columnValues)[3])[i]);
        }
}

Then compile and run:

gcc gwapi.c c.o -DKXVER=3 -lpthread -o gwapi  
//-DKXVER=3 is an environment variable required for kdb 3.0+
// c.o and k.h can both be obtained from https://github.com/KxSystems/kdb/
// l64/c.o and c/c/k.h
 ./gwapi

Result:

 Callback Function: handleresults
 time sym price size
 ---------------------------------------------
231D01:00:00.439626000 MSFT 17.310000 184683593820
231D01:00:00.439626000 MSFT 17.320000 386547056679
231D01:00:01.927037000 MSFT 17.380000 408021893216
231D01:00:02.925681000 MSFT 17.260000 103079215154

[hoffmanroni]

Thanks for this Mathew,  what I was wondering mostly about was is it possible to have a callback function, like you have handleresults, on the c++ client instead of the gateway?  So its non blocking on the client as well.  

I can do this in q myself with something like

.gw.asyncexecjpts[(func;args);handle;raze;client_callback;timeout;0b]

So here I have a q client gets data from the gateway async, and is also non blocking in the client so I get a callback.  I'm just not as familiar with c api, if its possible or not.

Thanks!

[Matthew Moore]

Hi hoffmanroni,

The best way to avoid blocking on the client is to create listener threads which await the query, freeing you up on another thread to continue working. Please see the attached expanded example (gwapi_multithread.c).

In this example, I setup quote and trade listener threads so that I can await multiple queries. I then count to 5 in the main thread as a simple example of doing additional work while awaiting results from the gateway.After this is done, I then check if each of my threads has finished and retrieved the data before I begin to process it. In this example, I just display both tables once I have received the data.

Result:

gcc -g gwapi_multithread.c c.o -DKXVER=3 -lpthread -o gwapi_multithread
./gwapi_multithread

 1
 2
 3
 4
 5
trade data ready!
quote data ready!

 Callback Function: handleresults
 time sym price size
 ---------------------------------------------

237D01:00:00.481001000 MSFT 9.850000 403726925912
237D01:00:00.481001000 MSFT 9.870000 390842023948
237D01:00:00.698669000 MSFT 9.860000 184683593748
237D01:00:00.698669000 MSFT 9.850000 73014444068
237D01:00:03.099537000 MSFT 9.800000 240518168630
239D10:19:23.275544000 MSFT 28.990000 0
239D10:19:23.476643000 MSFT 29.040000 6520833
239D10:19:23.476643000 MSFT 29.040000 2
239D10:19:24.476083000 MSFT 28.910000 140002513128320
239D10:19:27.275603000 MSFT 28.920000 140002513128384
 time sym bid ask
 ---------------------------------------------
237D01:00:00.899803000 MSFT 9.610000 10.530000
237D01:00:00.899803000 MSFT 8.970000 10.010000
237D01:00:00.899803000 MSFT 9.490000 10.100000
237D01:00:00.899803000 MSFT 9.670000 10.590000
237D01:00:00.899803000 MSFT 9.450000 9.850000
239D10:19:23.875490000 MSFT 28.670000 29.730000
239D10:19:23.875490000 MSFT 28.100000 29.750000
239D10:19:23.875490000 MSFT 28.340000 29.580000
239D10:19:24.275372000 MSFT 28.230000 29.140000
239D10:19:24.275372000 MSFT 28.110000 29.580000

[hoffmanroni]

Thanks very much for this Mathew, makes sense this helps a lot.