Request for Comment: Live Migration Stall Detection and Lower Downtimes

[Aseef Imran]

Hi everyone,

Live Migration stall detection is now in v1.9 and was added by VEP 248.

Originally, the goal of stall detector was that when the default QEMU target downtime of 300ms cannot be achieved, find the next best downtime value that does work in order to avoid relying on timeouts (which could take several dozen minutes). However, this does mean that the stall detector pathway only triggers when a migration fails to meet the 300ms downtime which for many migrations is quite generous.

How It Works

As a reminder, how stall detector worked was that it monitored the "bytes remaining" of a migration over time, and through some heuristic it detected when the migration had stalled. However, VEP 248 did more than just detect stall: it used this historic data of "bytes remaining" to also figure out what an ideal time to switchover to stop-and-copy (or post-copy) would be by triggering switch-over at a local minima of remaining bytes.

The Question

What if all migrations begin with a target downtime of 1ms (unattainable)? This guarentees that migrations have to go through the stall detector. In practice, what this would mean is that all migrations would switch-over at the lowest possible downtime for that particular workload by optimizing for the local minima.

There is No Free Lunch

The trade-off here is increased migration time. The added time here is due to (a) simply the additional time required to migrate enough bytes to get downtime lower even if it isn't stalled [since in a live migration the last few bytes take longer to transfer due to concurrent dirtying], (b) additional time needed to collect enough data for the algorithm to get a sense of what a good migration time looks like, (c) and waiting for a previously known "good value" of remaining bytes to reoccur.

Not Everyone Needs This

There are use cases for trying to drive down downtime as low as possible like cloud gaming services. But for many other users, 300ms of downtime is more than good enough and not worth the added cost. For this reason, maybe it makes sense to add a new API option for whether you want to enable this "mode" like in the KubeVirt CR/MigrationPolicy CRD:

```

spec: configuration: migrations: minimizeDowntime: true

```

Comments anyone? Suggestions?

I know we would want to avoid further bloating an already complicated migration API. So with that in mind, if anyone has any alternatives, thoughts, or comments in mind, please let me know!

[Fabian Deutsch]

Hi Aseef,

What is the downside of sending all migrations through the stall detector?

And: Will the "odnwtime window" be configurable? Even if just with an annotation.

- fabian

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/bdd54fe1-d09f-497d-88c1-bdc4eed8c61en%40googlegroups.com.

[Aseef Imran]

Hi Fabian,

Thanks for your message!

>> What is the downside of sending all migrations through the stall detector?

The trade-off is increased total migration time. The sources of this increase are listed above in the original thread message under "There is No Free Lunch".

>> And: Will the "odnwtime window" be configurable? Even if just with an annotation.

Yes, the downtime window/progress timeout window will be configurable. This window controls the trade-off between achieving better downtimes using more data, and total migration time.

[Felix Enrique Llorente Pastora]

On Friday, July 24, 2026 at 2:49:04 PM UTC+2 Aseef Imran wrote:

Hi Fabian,

Thanks for your message!

>> What is the downside of sending all migrations through the stall detector?

The trade-off is increased total migration time. The sources of this increase are listed above in the original thread message under "There is No Free Lunch".

>> And: Will the "odnwtime window" be configurable? Even if just with an annotation.

Yes, the downtime window/progress timeout window will be configurable. This window controls the trade-off between achieving better downtimes using more data, and total migration time.

Sure we have check it out, but at openstack they are doing this kind of things from long time ago, maybe kubevirt should mimic it ? 

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L871