Question about storing large amounts of data per second

53 views
Skip to first unread message

Xiaomu Wing

unread,
Apr 6, 2021, 8:52:04 AMApr 6
to Kdb+ Personal Developers
There are 1024 real-time data per second, how to save al these 1024 floats into a partition table per second? The insert string is out of length......

Thanks ALL!

TerryLynch

unread,
Apr 6, 2021, 11:58:28 AMApr 6
to Kdb+ Personal Developers
Not quite sure I understand the problem: are you saying that you have 1024 live/realtime messages per second and you want to write them every second to a partitioned table on disk? Or that you want to insert them into an in-memory table?

Can you give examples of what the records looks like?

András Dőtsch

unread,
Apr 6, 2021, 12:25:29 PMApr 6
to personal...@googlegroups.com
You have quite a few options to connect to kdb.

Andras

On Tue, Apr 6, 2021 at 2:52 PM Xiaomu Wing <xiaom...@gmail.com> wrote:
There are 1024 real-time data per second, how to save al these 1024 floats into a partition table per second? The insert string is out of length......

Thanks ALL!

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/0e7b1a63-1d6f-4bb7-98fe-7d69acef28d4n%40googlegroups.com.

Xiaomu Wing

unread,
Apr 11, 2021, 2:45:16 PMApr 11
to Kdb+ Personal Developers
not into in-mem table but a partitioned table, because of the real-time data is sensitive and have to flush them to disk immediately.  
Thank u for response!

Xiaomu Wing

unread,
Apr 11, 2021, 2:45:22 PMApr 11
to Kdb+ Personal Developers
Thank u for response , I'm not quite sure but you mean send parameters by bytes stream ? or what.....

András Dőtsch

unread,
Apr 11, 2021, 5:55:28 PMApr 11
to personal...@googlegroups.com
I mean sending data through the q ipc protocol, yes.
I guess you used http get and you run into some limit of the url size? You can also split up the message btw, but I recommend using the ipc protocol.

András

Reply all
Reply to author
Forward
0 new messages