Best practices setting up KDB+ server architecture?

[Charles Lewis]

Good day,

We are testing the feasibility of using KDB in the energy space and while it appears to work extremely well,  we are in unchartered territory on the architecture side. 

We've been testing KDB functionality on our local machines and have recently set up a test server to push functions to in order to process large data sets. We've spun up KDB+ in a Docker container and can successfully connect via an IP address. Our next steps are trying load .q function library (some are quite lengthy) from a local dev client to this server and have run out of ideas on how this may be done without putting the files on the server itself and loading from there. 

Much of what we've found through online resources has been introductory and assumes running two instances on the same machine. https://code.kx.com/q4m3/11_IO/#116-interprocess-communication.

I have two questions:

1. Since we are accessing the server via IP address, is it possible to load data/files from a client machine or how to move functions from a local dev environment to this server.  
2. Are there suggested resources (literature or best practices) on how best to set up client/server development environments? 

FWIW - We are using VS Code/VS Studio as our IDE for both q and other technologies

Regards,

Jason Lewis

[Charles Lewis]

UPDATE - Spent some time working on this over the weekend and it looks like we have things working now.  We sorted out loading lengthy functions via raze read0 to build out the strings to pass to the server.   

[Himanshu Gupta]

Hi,

If you are dealing with real-time data, you will need a feed handler, ticker plant, real-time db and a historical db.

• Feed handler will be responsible for getting the data and pushing it to ticker plant.
• The ticker plant structures the data in q/kdb+ tables and routes the updates to downstream processes that have subscribed to it. 
• A real-time database is one of those downstream processes which will store these updates in memory. 
• Then, usually at the end of the day, you will push the data to disk into a historical database and flush it from memory.

You can use IPC to directly communicate with the processes or you can use an event broker such as Solace PubSub+ in the middle for a loosely coupled architecture. Your processes will distribute data over the broker. More info on how to do that here.

If you are dealing with historical data only where you have flat files that need to be loaded into kdb+ hourly or daily then Kx has a nice white paper on that. 

Based on how you plan on querying the data, you will need to properly partition your tables when storing them in memory and on disk. Here is a good white paper on data management in kdb.

As for resources,

• q for mortals is the best book for learning q/kdb+. 
• Once you know the basics, start reading Kx white papers which discuss more practical usecases. 
• I have a blog where I write about q/kdb+ 
• YouTube channel
• Intro to kdb+ videos by Jeffry Borror

Hope that helps!

- Himanshu Gupta

[Charles Lewis]

Many thanks Himanshu.

Actually, I've been pouring over the resources you provided for the last two months including subscribing to your own YouTube videos as well as your site, Enlist(q) site so thank you. Great videos by the way so thank you for sharing. We have been performing a feasibility study on using kdb and q as a calc engine in the energy space where we generate voluminous time series data based on various datasets. 

By now, I've we have written a fair amount of code thanks to resources such on this community, its been so helpful and friendly and patient with my questions. 

[Caolan rafferty]

Hi Jason,

You can pass function definitions across IPC like so:

q)func:{x+y}

q)h:hopen 5555

q){h({x set y};x;value x)}each value "\\f";

Thanks,

Caolan

-- 
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/a4bb30c4-d13b-4e90-a119-74308eee2bebn%40googlegroups.com.