updating a kbd table from C (shared lib)

[Tiago Rodrigues]

Hi,

  I'm extending some functionality of Kdb with a shared library which subscribes to quote feeds from within kdb (instead of using an external feed handler).

  This means that the event processing is being done in real time from inside the shared library, and I'd like to update an existing table in kbd each time a new quote arrives. Is this even possible without callbacks or IPC?

Thanks,

Tiago Rodrigues

[Tiago Rodrigues]

Hi,

  just found the answer to my question on kov's post 'Question about callbacks from C-lib to q via C-function sd1()'.

  I can use the following, with handle 0, for in process requests to kdb:

     K result=k(0,"0N!",ki(42),(K)0);

Best regards,

Tiago Rodrigues

[Tiago Rodrigues]

Well, it seems I still have a problem after all. I can run queries in q using the k(0, ...), but since I'm calling from a slave thread I only have read-only access to kdb variables.

Any hints on how can I manipulate a kdb variable from inside a thread in the shared lib?

Many thanks,

Tiago Rodrigues

[Charles Skelton]

There are a few key points to adhere to:

use k(0,..) from the kdb+ main thread only.
use sd1 to register a callback from the kdb+ main thread.

free objects in the thread in which they were allocated.

do not share objects between threads.
to pass objects between threads, use serialization.

internalize strings (via ss or ks) in the main thread only, or call setm(1) to allow internalization from other threads.

Some may choose to ignore the above claiming that it seems to work ok anyway ;-) But they may be setting themselves up for tracking down nasty race conditions with random crashes sometime later.

Hope this helps,

Charlie

[Felix Lungu]

On 2015.05.26, at 15:54, Charles Skelton <cha...@kx.com> wrote:

 or call setm(1) ???

what is this function doing?

[Charles Skelton]

I setm(I f) is part of the sharedlib c-api; it can be called to activate locks around the interning of symbols in threads other than the kdb+ main thread. It should only be called when you can guarantee that no threads could be inside sn at the time of the call. e.g. call it from the kdb+ main thread on startup. It returns the previous setting.

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[Tiago Rodrigues]

Thanks, the callback works nicely. I was making things harder than they needed to be.

Tiago

[Jay Han]

Charles: 

- if k(0,...) and q function it calls do not write, can't they be run safely on non-main threads?

- does m9() inside thread take care of reference counts?

--

[Jay Han]

what is the meaning of the argument for setm(I f)?

...

setm(1);

ks("asymbolinternedforever");

setm(0); //???

...

[Charles Skelton]

>- if k(0,...) and q function it calls do not write, can't they be run safely on non-main threads?

If the q function does not read any global variables, it may be ok.

- does m9() inside thread take care of reference counts?

No, it just frees whatever was allocated for that thread.

[Charles Skelton]

> what is the meaning of the argument for setm(I f)?

The argument f is to activate locks around sym interning. If you need to intern from threads other than main, you should set this just once, from the main thread, and never reset it.

[Charles Skelton]

>>- if k(0,...) and q function it calls do not write, can't they be run safely on non-main threads?

>If the q function does not read any global variables, it may be ok.

Actually, not even then. There is still potential for ref count anomalies in the vm, which could lead to a crash after some time.

kdb+ itself handles these issues differently during peach/multithreaded input mode, but this is not available to the capi directly.

[Tiago Rodrigues]

I was wondering if there was a reference manual for the sharedlib API (beyond whats on the integrating/extending with C on the wiki). That would be really helpful to better understand the internals and avoid some head-banging on the wall :)

You received this message because you are subscribed to a topic in the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/personal-kdbplus/xRtTODPLFt8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to personal-kdbpl...@googlegroups.com.

[Charles Skelton]

there are some additional docs floating around

e.g.
http://code.kx.com/wsvn/code/contrib/aquaqanalytics/InterfacingKDBtoC/AquaQ%20Analytics%20Transferring%20Data%20between%20kdb%2B%20and%20C%20V1.0.pdf

The internals of kdb+ are really quite complicated and subject to change; we try to hide that with the simple c-api.

[Jay Han]

when I read setm() above I thought thread-safety for ks()/ss().

if read-only access from non-main thread is not safe, the little descriptions for setm() and m9() are confusing... documentation does *not* say something like

you can launch threads with callbacks to main.

call setm() in the main once (and only once) before any threads use ks()/ss().

call m9() inside threads before exiting.

ok I am probably misreading mails and docs and second guessing too much!

[Jay Han]

another vote for reference doc. (lately i've been learning about a few new functions not appearing in wiki pages.)

something I've been wondering about: c api doesn't seem to include a dicionary lookup support.

[Tiago Rodrigues]

Hi Charles,

  I really appreciate your help so far and hope I'm not abusing of your good will, but I still have some questions.

 For instance, this works on the main thread, but results on a kdb crash when in a thread:

char *CALLBACK = NULL;

K callback() {

  K a=kp("C");

  k(0,CALLBACK,a,(K)0);

  R (K)0; }

 

Z K1(test) {

  CALLBACK=x->s;

  sd1(0,callback);

  R (K)0; }

The callback function in Q:

upd:{[x] update state:x from `.mbt.c}

Table definition:

c:([name:0#`] s:0#0N; state:0#`; server:0#`)    

Also, there seems to be a 1-2 second delay on the update from the callback. Does it have some kind of deferred execution? 

Here is an example:

q).mbt.c

name| s state server

----| --------------

qt  | 6 N         

q).mbt.test[`upd]

q).mbt.c       <------  About 1 second after .mbt.test call

name| s state server

----| --------------

qt  | 6 N           

q).mbt.c       <------  About 3 seconds after .mbt.test call

name| s state server

----| --------------

qt  | 6 C       

This also means that if I call r0(a) after the call to k(0,...) it causes a sigsegv (I assume its freeing 'a' before finishing the update?) 

q).mbt.c

name| s state server

----| --------------

qt  | 6 N           

q).mbt.test[`upd]

q).mbt.c

name| s state server

----| --------------

qt  | 6 N           

q).mbt.c

name| s state server

----| --------------

qt  | 6 .           

q)

rlwrap: warning: q crashed, killed by SIGSEGV.

Am I completely misunderstanding the correct way to do this?

Tiago

[Charles Skelton]

the first arg to sd1 should be a file descriptor that the main kdb+ loop can monitor for readable activity via the system select call. For example, see the section using eventfds at http://code.kx.com/wiki/Cookbook/InterfacingWithC

I don't see the sense in using stdin (fd 0) for the sd1 call sd1(0,...; I think that can only confuse kdb+ since it is already monitoring stdin.

For a system which is not under load, the latency should just be how long the select call takes to return - so a few uS. The callback is not deferred.

For these kinds of issues, it is really helpful to have a snippet of code that will fully compile and yet reproduces the issue.

Also please include (most of) the kdb+ startup banner as that has relevant details for the OS, kdb+ version/release etc.

[andrew...@aquaq.co.uk]

Thanks Charlie

We've updated that document recently and added some source code : https://github.com/AquaQAnalytics/kdb-c-interface

We've been building some feed handlers in C for clients recently and created a feed handler tutorial document which is probably also relevant. It walks through how to build an example feed handler in c as both a shared object loaded into the q process and a standalone executable (https://github.com/AquaQAnalytics/kdb-feedhandler-tutorial)

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsub...@googlegroups.com.

To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsub...@googlegroups.com.

To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsub...@googlegroups.com.

To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

-- 

You received this message because you are subscribed to a topic in the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/personal-kdbplus/xRtTODPLFt8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to personal-kdbplus+unsub...@googlegroups.com.

To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[Tiago Rodrigues]

Charlie, you are right, using the stdin was causing the delay (it waited for some input on q console before triggering the callback) which as you said didn't make sense. The example feed handler Andrew linked to was quite helpful in understanding the way to follow and I've managed to get it working.

Andrew, if you won't mind me asking, is there any special reason for your choice of using sockets instead of a pipe to communicate to the main process or is it just a matter of personal preference?

[mark....@aquaq.co.uk]

Hi,

It's probably optimal in terms of performance to use pipes on a Linux machine, but we used sockets instead of pipes for a few reasons:

  - Trying to keep the code simple: One of the goals was that the code should compile and run on both Linux and Windows, so we made use of the selectable socketpair available here: https://github.com/ncm/selectable-socketpair.
This made it easier to abstract over the different operating systems and is reliable well tested code. Keeping the code simple also allows people to focus on the interaction with the kdb+ api to see how it works rather than wading through
platform specific implementation details or having to bother with the differences between sockets vs pipes (e.g send/recv and read/write calls).

  - Sockets are not that much slower than pipes on Linux: If you look at the implementation of the socket pair in the library above, you will see that it uses the AC_UNIX/AC_LOCAL socket family on Linux.
This is intended for communicating between processes/threads on the same machine and cuts down on a fair bit of the overhead you would get when using AC_INET sockets. On Windows I believe that the
performance gap between named pipes and sockets is bigger, but you can't just blindly replace the socket with a pipe because of the issue below.

  - It's tricky to get pipes to work nicely with some functions on Windows: the implementations of pipes on Linux and Windows are very different, and in particular trying to use the select() function on a pipe fails.
This can make the sd1() callback hang if passed a file descriptor that belongs to a pipe. I think that the Windows implementation of select() will only work on the handles generated via WinSock (but I'm not 100%
sure on this).

I hope this answers your question!

Thanks

[Tiago Rodrigues]

Hi Mark,

  I expect a pipe to be about 10% faster than a UNIX socket on OSX and Linux . This shouldn't be much of a problem since other parts of the code will probably be on the critical path well before the performance delta becomes an issue, Also, your note on windows select() failing on a pipe justifies the option for sockets on portability alone.

  Just a note. If you'll be having multiple threads triggering the same callback function, a pipe will guarantee that the writes are atomic (if less than PIPE_BUF). The selectable-socketpair is using a STREAM socket, which is not guaranteed to be atomic and I assume could be problematic in this scenario. One could probably change it to use a DATAGRAM socket to get over this (though I haven't tested). You probably knew this already, but might be useful for someone using the feedhandler tutorial as a template. :)

  Again, thank you all for your answers. They have been most helpful.

Tiago