Interacting with C++

[John Smith]

Hi,

I'm currently working in C++ and wish to export data to tables in kdb, real time.

I believe this is possible, and as a start I must include a k.h header file. Can someone tell me which is the appropriate file, and where to get it, as I think some example files on the web are outdated (possibly)?

I'm using the 32 bit version of kdb.

best,

J. 

[mark....@aquaq.co.uk]

Hi John,

You should just use the C interface to interact with kdb+ using C++. The header file is located in the svn repository: http://code.kx.com/svn/kx/kdb+/c/c/k.h.

If you are building a shared library, you will need to be aware of the name mangling that the C++ compiler will perform when linking. To avoid this
you will need to give C linkage to any functions that you will dynamically load using extern "C".

The svn repository also contains object files that you might need if you are building an executable or a binary that will not be loaded into kdb+:

http://code.kx.com/svn/kx/kdb+/l32/    (for linux 32 bit)
http://code.kx.com/svn/kx/kdb+/w32/  (for windows 32 bit)
http://code.kx.com/svn/kx/kdb+/m32/ (for mac 32 bit)

You can find more information in these links:
http://code.kx.com/wiki/Cookbook/InterfacingWithC
http://code.kx.com/wiki/Cookbook/ExtendingWithC (for shared libraries)

Thanks
---
Mark Rooney
Financial Software Developer
AquaQ Analytics

[John Smith]

Hi Mark,

thanks for your help on this. When I include this header file and compile, line 8 returns an error. The problem seems to rest with #ifndef KXVER, resulting in all lines from 9 onwards commented out in the header file. What is this KXVER? I actually tried a different header file see: code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h?rev=1442&peg=1442

This compiles, though I'm unable to open a connection using khpu. This may not be related though. So not sure if this is the correct header to use. Any advice?

-regards J

[mark....@aquaq.co.uk]

Are you defining KXVER? The easiest way to do this is to add a define before you include the header in your source code. You can also do it by passing a flag to your
compiler but the syntax for this differs based on compiler & platform used (e.g. for gcc on Linux you would pass -DKXVER=3).

e.g

#define KXVER 3
#include "k.h"

[mark....@aquaq.co.uk]

I forgot to add that the KXVER check is there to handle a change to the k struct that was not backwards compatible. To quote the Kx Wiki and the error message
in the k.h header file:

"Note that the k struct changed with the release of v3.0, and if you are compiling using the c library (c.o/c.dll) stamped on or after 2012.06.25 you should ensure you use the correct k struct by defining KXVER accordingly."

and

"Set KXVER=3 for kdb+3.0 or standalone c-api after 2011-04-20. Otherwise set KXVER=2"

[John Smith]

This works. Thanks! Ie putting the #define KXVER=3 in main.

With regard to my other prob, when trying to run int c=khpu("localhost",5001,"password");

This fails since it doesn't like the types, "warning: deprecated conversion from string constant to 'S {aka char*}' [-Wwrite-strings] .cannot convert ' complex int' to 'I {aka int}' for argument 2 to 'I khpu(S, I, S)'.

Should I expect this to compile ok?

Regards John

[mark....@aquaq.co.uk]

You will want to make sure you are using  #define KXVER 3 and not #define KXVER=3 before including your header file (the syntax is #define token value, no equals is required). Those
warnings about strings shouldn't be a problem (and shouldn't stop compilation unless you have a flag like gcc's -Werror enabled). If you want to get rid of the warnings, you can explicitly cast the values
to non-const ones.

e.g.

int c = khpu((S)"localhost", 1234,(S)"username:password");

Thanks

[John Smith]

Sorry my bad. The error at compile is coming from: undefined reference to 'khpu'

Are you sure we only need the k.h file. There's a time stored example online that shows compiling from Linux command window, where some libraries are also included. So maybe i need to include some libraries no? Also, I see examples on the tomestored using C not C++. You think C++ should be ok. Ie the issue doesn't rest here. Thanks for you help on this.

-John

[mark....@aquaq.co.uk]

Yep, using C++ should be fine. If you are creating a standalone executable, you will need to link against the c.o file. If you are using
Linux you can find it here: http://code.kx.com/svn/kx/kdb+/l32/

The c.o object provides all the definitions of the functions in the header file. You will also need to link against pthreads to resolve the missing
symbols for sem_post, pthread_rwlock_init etc.

For example, if I have a file called main.cpp that contains:

#include <iostream>

#define KXVER 3
#include "k.h"

int main(int argc, char* argv[])
{
    int c = khpu((S)"localhost", 5000, (S)"username:pass");
    std::cout << "c: " << c << std::endl;
    return 0;
}

I can compile it with the following line on Linux:

g++ main.cpp c.o -lpthread

Thanks

[John Smith]

Hi Mark, success! The 64 bit version of the c.o file I had to use. I appreciate your help with all this. Great job.

Now for the hard bit...to do what I need to do.
I will likely ask more questions later. Thanks.
-John

[John Smith]

Hi Mark,

After successfully connecting I've been able to play around with examples creating K objects. However I'm unaware of how to send variables defined in C++ to the kdb server. The examples seem to focus on the other way around eg:

For some table tab created on q, I can access this in C++, by:
K tab1=k(handle,"tab",(K)0);

However, say I have an int defined in C++, how do I retrieve this int variable in kdb.?

Thanks,
John.

[mark....@aquaq.co.uk]

Hi,

You can send C++ data to kdb+ by wrapping it up as a K object and passing it as a parameter to a function.

e.g

// Send a single integer to a function called myfunc in kdb. The ki function
// converts the integer into the corresponding K object representation that
// is used by kdb+. There are other functions for the other kdb+ types that
// can be found on the wiki page.
K result = k(handle, "myfunc", ki(10), (K) 0);

// You can send multiple arguments by just listing them out and finishing the
// list with a terminating (K) 0. The example below sends a float and an integer
// to the function myfuncb.
K result = k(handle, "myfuncb", kf(3.14), ki(17), (K) 0);

You would then define the following two functions in kdb to handle the data:

myfunc:{[x] ... }
myfuncb:{[x;y] ...}

You can see more realistic examples of this in the commented out part of the following file (calling the insert function with data records):
http://code.kx.com/svn/kx/kdb+/c/c/c.c

Thanks

[John Smith]

Hi Mark,

Thank you, This is working great. Again much help appreciated.

However, just when I thought I was home and dry.
I've been testing this out on a simple project, and it works fine. However when I try to do the exact same on my C++ application which includes additional header files, build now returns an error. It does not like the inclusion of one of my headers , error:

"Include/k.h:99:11: error expected identifier before 'return'."
#define R return

"Include/boost/type traits/detail/is men fun pointer impl.hpp:38:17: note: in expansion of macro 'R'"
template <class R,class T>

Include/k.h:99:11: error: expected '>' before 'return'"
#define R return



Thanks
-John

[mark....@aquaq.co.uk]

The #define R return that is in the k.h file is causing the R in your template parameter list to be replaced with return (the pre-processor
just blindly replaces all instances of R). To fix it, just undefine R after including the k.h header file:

e.g

#define KXVER 3
#include "k.h"

#undef R

[John Smith]

Fantastic! It works. I owe you!

-j

[Kim Tang]

Yes, this is correct.

 

k.h has more these kinds of macros and it is only a matter of time that you would have another clashes. Especially when using with boost library together.

 

I would suggest you to put ur own identifier before each macros defined in k.h.

 

Here is an example:

 

http://code.kx.com/wsvn/code/contrib/kuentang/c%2B%2B/kdb/k.h

 

 

Kim

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[John Smith]

Hi Mark/Kim,

Now that I have this up and running, my goal is to collect some data in C++ and transfer it to kdb in the form of a dictionary. This seems like the most sensible method.

Then in kdb I can insert the data onto a table. The goal is to have this real-time.

I have been able to do the simpler task of sending across atoms with type float, however generalising to a dictionary is more difficult.

On top of that I cannot create K string objects as I cannot get the ss function to work.

Here is example code of my attempt which is not correct. Hopefully it's not too far away, and you could point out the errors (please):

//Define empty table in kdb

tbl:([] col1:();col2:();col3:())

//A dictionary K object is made up of key object and value object

//Define key

K key=ktn(KS,2);    //only need 3 values

string col1="string1";  //created string variable col1

string col2="string2";

string col3="string3";

kS(key)[0]=ss(col1);  //intern the string using ss, of course not correct

kS(key)[1]=ss(col2);

kS(key)[2]=ss(col3);

//Define value

K val=ktn(0,2);

kK(val)[0]=kf(price1);  //price1 and price2 are floats

kK(val)[1]=kf(price2);

kK(val)[2]=ks(sym);       //not sure what to do here again for a string intern first? how?

//Create the dictionary

K result=xD(xkey,val);

K result1=k(handle,"func",result,(K)0);     //I opened a handle to kdb in earlier post

//where func is defined in kdb below:

func:{[d] `tbl insert(d[`col1];d[`col2];d[`col3])}

Note: I'm not experienced in C++ as you can tell. I want to get the data across from C++ to kdb where I am more comfortable.

I was initially not going to create a dictionary, just pass each value across separately where each takes a parameter is func.

However, if I have a lot of values then there'll be too many parameters, so a dictionary seems like the sensible thing to use.

Though, as I mentioned i also had problems with strings, and I think the examples online are always char types, so how to do it for strings?

again, any help appreciated.

thanks,

John

[Charles Skelton]

Note you must call khpu before generating any K data, and the very first call to khpu must not be concurrent to other khpu calls. To initialise memory without making a connection, use khp("",-1);

In your example
K val=ktn(0,2);

should be
K val=ktn(0,3);

same for key length.

K result=xD(xkey,val);

should be key not xkey.

You can create a char vector with kp("some string here"). Also kpn("hello world",5);

ks("hello world") automatically interns the string.

--

[John Smith]

Hi Charles et al,

Thanks for your corrections. I have two problems it seems at this point.

1. ks("hello world") works, and this produces a symbol type in kdb. However the below for example won't work:

string sval="hello";
ks(sval); //doesn't like this

ks(ss(sval)); // nor does it like this.

I need to do this because my sval (string) will continually change.

2. Given I cannot get the string type to work I'll revert to a char example which does seem to, to point out second problem, though difference here we pass the address of char variables?

int handle=khpu((S)"localhost",5002,(S)"password");

K key = ktn(KS,2);
char buf1='a';
char buf2='b';
char* buff1=&buf1;
char* buff2=&buf2;
kS(key)[0]=ss(buff1);
KS(key)[1]=ss(buff2);

K val=ktn(0,2);
kK(val)[0]=kf(24);
kK(val)[1]=kf(34);

K dict=xD(key,val);
K result=k(handle,"func",dict,(K)0);

kclose(handle);

// where in kdb: func={[d] z::d};

The problem is, the output for z which is almost fine gives jibberish (Unicode replacement character) for the key values, though the val values are fine. The type of z is 99h so it is ok that way.

Any ideas appreciated.

Thanks,

John.

[John Smith]

[Charles Skelton]

see
http://www.cplusplus.com/reference/string/string/c_str/

char buf1='a';

is not a null terminated string. So for that you would use sn(&buf1,1) to create a symbol, or kpn(&buf1,1) to create a char vector.

[mark....@aquaq.co.uk]

If you just want to insert all the fields into a table, it's perhaps easier to use a list rather than a dictionary. That way you can just pass a table name and a table row
to a function and then call insert in kdb+.

To take an example from the Kx Wiki:

/ Build a table row as a list of K objects
K row = knk(3, ks((S)"ibm"), kf (93.5), ki(300));

/ Call the function with the table name and the row data.
/ The upd function will append the row onto the trade table
k(-kdbSocketHandle, "func", ks((S)"trade"), row, (K)0);

Then in kdb+ you would have your function that could be defined as:

func:{[t;x] t insert x; }
or
func:insert

[John Smith]

I meant to say thank you Mark and Charles. For a variable in C++ of type string, call it var1. To access this as a symbol in kdb I needed to do ks(strdup(var1.c_str())).

Furthermore, when I used this to create a dictionary key value eg (13th value):

kK(val)[12]=ks(strdup(var1.c_str());

Did not like the assignment from ks. To get around this problem,I used what Mark mentioned on how to create lists. So I created a one element list.:

kK(val)[12]=knk(1,ks(strdup(var1.c_str())));
and it worked. Maybe not the best thing to do, but it worked.

[John Smith]

Hi,

Coming back to the question of transferring data from C++ to kdb real time.

The process I have of doing this: firstly the feed handler receives the data, parses it and publishes to a port using zmq.

I listen to the data in my C++ application and for each message update from the port I create a K dictionary of the variables I want (approx 13 variables), then I send this dictionary across to kdb as follows:

K result = k(handle, "myfunc", dict, (K) 0);

where handle takes me to a kdb session on the localhost and specific port. The function myfunc takes the dictionary, dict extracts a couple of longs and converts them to datetimes and then inserts all the values to the end of a pre-defined table.

The problem I have is, this process is too slow to consume the data that is being published to the port, from the zmq. This means my application is missing lots of packets. Actually it's missing 70% of packets it is that slow. A run I did, earlier for 5 mins, the zmq published approx. 500,000 messages and my kdb application only recorded roughly 150000. which equates to only 500 messages per second on average. When I remove the above line from the C++ application, and print the count of messages received to screen, there is no loss of packets.

Therefore the bottleneck must occur in the line above, i.e. sending the dictionary of values to kdb from C++, across the handle. 

I would like to know, is sending a (small) dictionary across to kdb slow? and therefore is this a pretty inefficient way of transferring the data realtime.

An alternative way of way of doing it then may be to append the dictionaries together in C++ (i.e. clumping the data) and then after every 5 seconds (say) send it across to kdb, therefore vastly reducing the number of times the I call the above line?

Another problem is, if kdb crashes, then when I reopen, I will wish to replay all the data into the table. So, actually the current method is, have another application receive the data from the port and write to a md file (binary) and have my application read and decode the md file, then sending to kdb as mentioned before. This means if kdb crashes, I can restart the application and all data is replayed from the md file and written to kdb. Problem is, reading and decoding the binary adds an extra level of slowness. 

Maybe my method here is generally bad. Another solution I thought of, was to have a separate application write the 13 variables of each message update to a csv file and every so often have kdb read the csv file in using a read0 or something. But im not keen on something like this because it seems more restrictive, plus I don't know how a realtime verion of this would work. i.e. I can't keep reading in a csv every so often, as the csv obviously will grow very large by the end of the day.

The goal is,listen to the port, write the data using C++ to a file, then have c++ read this file constantly and send to kdb any updates. At end of the day, save the table to disk, and repeat for the next day. I don't care about a few seconds or more of latency doing this.

Someone I spoke to who's familiar with mongoDB said threading would help here, I guess multiple threads doing the transfer of data to kdb. Any thoughts?

I wouldn't have thought capturing 1000 messages per sec on avg and sending to kdb would be a problem?

best,

John.

 

[Charles Skelton]

are you sending async (negative handle) or sync (positive handle) ? Feeds should always send async.

In general, try to bulk up data in fewer msgs - it's more efficient, less cpu load.

Ensure your socket buffer sizes (an OS setting) on your machines is sufficient for your payload and for any backlog if the receiver is temporarily busy with something else.

>Another problem is, if kdb crashes,...

That should be an exceptional case; ticker plants should never crash and they allow for recovery of downstream components through re-subscription.

Usually feedhandlers push into the tickerplant which does the logging, and the tp handles pub/sub for subscribers. The tp can also bulk up data too and publish later. It sounds like you may be reinventing the wheel. See

http://code.kx.com/wiki/Startingkdbplus/tick

--