Hi,
If you are dealing with real-time data, you will need a feed handler, ticker plant, real-time db and a historical db.
- Feed handler will be responsible for getting the data and pushing it to ticker plant.
- The ticker plant structures the data in q/kdb+ tables and routes the updates to downstream processes that have subscribed to it.
- A real-time database is one of those downstream processes which will store these updates in memory.
- Then, usually at the end of the day, you will push the data to disk into a historical database and flush it from memory.
You can use IPC to directly communicate with the processes or you can use an event broker such as
Solace PubSub+ in the middle for a loosely coupled architecture. Your processes will distribute data over the broker. More info on how to do that
here.
If you are dealing with historical data only where you have flat files that need to be loaded into kdb+ hourly or daily then Kx has a nice
white paper on that.
Based on how you plan on querying the data, you will need to properly partition your tables when storing them in memory and on disk. Here is a good white paper on
data management in kdb.
As for resources,
Hope that helps!
- Himanshu Gupta