Question about storing large amounts of data per second

[Xiaomu Wing]

There are 1024 real-time data per second, how to save al these 1024 floats into a partition table per second? The insert string is out of length......

Thanks ALL!

[TerryLynch]

Not quite sure I understand the problem: are you saying that you have 1024 live/realtime messages per second and you want to write them every second to a partitioned table on disk? Or that you want to insert them into an in-memory table?

Can you give examples of what the records looks like?

[András Dőtsch]

You have quite a few options to connect to kdb.

Check out https://code.kx.com/q/interfaces/#kdb-as-server

Andras

On Tue, Apr 6, 2021 at 2:52 PM Xiaomu Wing <xiaom...@gmail.com> wrote:

There are 1024 real-time data per second, how to save al these 1024 floats into a partition table per second? The insert string is out of length......

Thanks ALL!

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/0e7b1a63-1d6f-4bb7-98fe-7d69acef28d4n%40googlegroups.com.

[Xiaomu Wing]

not into in-mem table but a partitioned table, because of the real-time data is sensitive and have to flush them to disk immediately.  

Thank u for response!

[Xiaomu Wing]

Thank u for response , I'm not quite sure but you mean send parameters by bytes stream ? or what.....

[András Dőtsch]

I mean sending data through the q ipc protocol, yes.

I guess you used http get and you run into some limit of the url size? You can also split up the message btw, but I recommend using the ipc protocol.

András

To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/f3eba4ec-8b3c-4bbe-8cea-192ee35dcd42n%40googlegroups.com.