strange behaviour in a virtual environment

[Erik Svensson]

Howdy guys!

My journey into mystification with virtual environments continues.

We have a number of market data feeds from a number of markets. Each market feed is running on its own virtual host.
I’m keeping an eye of gc logs and such.
One thing I noticed after a while is that kernel time seems to grow over time, eventually exceeding user time. If you reboot the machine it goes back to ‘normal’ and then starts over again.
I have never seen this behaviour with physical hw.

Does this ring a bell with anyone? Right now I’m thinking about rebooting the virtual hosts every weekend but that is an unsatisfactory solution.

cheers
Erik

[Matt Fleming]

Hi,

Have you tried using the usual kernel performance profling tools to see where the system is spending its time?

Matt

Erik

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[singh.janmejay]

Try perf record a little while after restart (give it enough warmup room after restart), and then again once sys% starts to match usr%. Offender should show up if your put them side by side.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Peter Booth]

When you say "virtual environments" what do you mean? 

With "containers" or Solaris zones you have a very lightweight virtualized runtime environment where each container uses teh same OS kernel, with separation being really just a chroot combined with a cgroup. Performance tuning is no different than a native environment.

With a hypervisor setup (Xen, Virtual Box, etc) you have the coarse-grained Xen scheduler taking the place of the OS scheduler. This is great mainstream technology. Electronic trading is not a mainstream use case. It shouldn't be a surprise that a mainstream platform doesn't fit a low latency use case. 

My two cents.

Peter

[Erik Svensson]

(sorry for late answer, been on vacation)

When you say "virtual environments" what do you mean? 

With "containers" or Solaris zones you have a very lightweight virtualized runtime environment where each container uses teh same OS kernel, with separation being really just a chroot combined with a cgroup. Performance tuning is no different than a native environment.

With a hypervisor setup (Xen, Virtual Box, etc) you have the coarse-grained Xen scheduler taking the place of the OS scheduler. This is great mainstream technology. Electronic trading is not a mainstream use case. It shouldn't be a surprise that a mainstream platform doesn't fit a low latency use case. 

It’s a hypervisor setup and I agree with you totally, I just have no say in the matter. I was not consulted before we moved to virtual and I have even less influence now.

/Erik

My two cents.

Peter

 

On Tuesday, March 28, 2017 at 8:46:03 AM UTC-4, Erik Svensson wrote:

Howdy guys!

My journey into mystification with virtual environments continues.

We have a number of market data feeds from a number of markets. Each market feed is running on its own virtual host.
I’m keeping an eye of gc logs and such.
One thing I noticed after a while is that kernel time seems to grow over time, eventually exceeding user time. If you reboot the machine it goes back to ‘normal’ and then starts over again.
I have never seen this behaviour with physical hw.

Does this ring a bell with anyone? Right now I’m thinking about rebooting the virtual hosts every weekend but that is an unsatisfactory solution.

cheers
Erik

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.