Fluctuation in Streaming

[kadir.k...@alaricsecurities.com]

Hello,

I set a tickerplant and a chained-tickerplant following the official best practices. Pretty straight-forward; a simulate a feed handler to push data into the tickerplant, which forwards the data to the chained-tickerplant and finally I have an application subscribing to this chained-tickerplant receiving data over TCP using the C# interface.

The problem is that I see fluctuations in the message count like;

 | 3096/3222 (# Msgs per sec./Total)

 | 2499/21743 (# Msgs per sec./Total)

 | 1666/23409 (# Msgs per sec./Total)

 | 1414/24823 (# Msgs per sec./Total)

 | 947/25770 (# Msgs per sec./Total)

 | 0/25770 (# Msgs per sec./Total)

 | 1559/41387 (# Msgs per sec./Total)

 | 1933/43320 (# Msgs per sec./Total)

 | 2138/45458 (# Msgs per sec./Total)

 | 1783/47241 (# Msgs per sec./Total)

 | 1573/48814 (# Msgs per sec./Total)

 | 892/49706 (# Msgs per sec./Total)

 | 0/49706 (# Msgs per sec./Total)

 | 1/49707 (# Msgs per sec./Total)

 | 3484/53191 (# Msgs per sec./Total)

 | 1340/54531 (# Msgs per sec./Total)

 | 1195/55726 (# Msgs per sec./Total)

 | 1816/57542 (# Msgs per sec./Total)

 | 2011/67516 (# Msgs per sec./Total)

 | 0/72030 (# Msgs per sec./Total)

As seen here, this goes for ever; data flows, then freezes for a second or two, then continues to flow. No data loss, just a delay repeating intermittently.

I checked the documentation, white-papers, and Google without any luck. Is there a chance that the socket connection from the KDB+ side is queuing up on purpose?

[kadir.k...@alaricsecurities.com]

Up! It's still there and I'm desperate to find an answer, thank you.

[Aaron]

How often the feed handler publishes to your tickerplant ? How often does the tickerplant publish to your ctp ? And .. how often does your ctp push data to your subscriber ?

On Mar 9, 2021, at 20:25, kadir.k...@alaricsecurities.com <kadir.k...@alaricsecurities.com> wrote:

Up! It's still there and I'm desperate to find an answer, thank you.

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/03b6663e-e737-4764-82d2-62fbf4971136n%40googlegroups.com.

[kadir.k...@alaricsecurities.com]

Hello Aaron,

FH => TP (0 ms - There is no explicit timer or batching, it calls .u.upd on TP as soon as the data is available from the simulator)

TP => CTP (0 ms - There is no explicit timer or batching, it calls .u.pub as soon as the FH pushes data through .u.upd)

CTP => Subscribers (250 ms - Data is batch sent every 250 ms without any conflation)

Any ideas are welcomed, thank you, Aaron.

Regards.

[TerryLynch]

There's probably too many variables and questions for anyone to know what's up here, but some lines of inquiry:

1. Where are you observing the message counts? Are you observing the message counts in a kdb subscriber? Is this subscriber an RDB (aka is it storing all of the messages in memory)? If so, then it's possible that this RDB is consuming more and more memory and will eventually have to request more memory from the OS once it hits a threshold (heap size in kdb increases as a step function). The blips you're observing might be caused by this as your blips seem to occur at a pretty regular interval. 

2. Or are you observing the message counts in the C# subscriber? If so, which C# interface are you using? Is it the most recent version? Are you doing any particular logic on the incoming messages?

3. Are your processes pegged to specific CPU cores? Are they sharing CPU cores with other processes on the machine? There could be some contention there. 

4. Is your tickerplant logging to disk? Is this local disk or mounted volume? How performant is the disk? Slow logging can affect tickerplant performance. 

To answer your question "Is there a chance that the socket connection from the KDB+ side is queuing up on purpose?" - kdb will not be holding back messages if it's not being impeded either by some resource contention, a slow consumer, non-performant hardware or some other custom logic in the tickerplant stack. 

Your best bet for profiling tickerplant latency is to timestamp the data before and after every hop:

a. Timestamp when feed sent

b. Timestamp when TP received, timestamp before TP sent

c. Timestamp when CTP received, timestamp before CTP sent

d. Timestamp when subscriber received

e. You could also set up some monitoring of the outgoing queues in each tickerplant (.z.W)

Using all of that info you should be able to narrow down where any delay/queueing is. 

Terry

[Alvi Kabir]

If I had to guess, the C# subscriber is off due to deserialization+processing cost. 25K messages/sec sounds like a high rate at which your C# app would have to have deserialize and process data. My recommendation would be to condense the data being sent downstream from the CTP

[kadir.k...@alaricsecurities.com]

Thank you very much for all the ideas and help. We are testing as you suggest and will update you. It's more likely a CPU core contention.