nomad and qemu live host migration

[pixel fairy]

How does nomad handle live qemu host migration? has anyone here tried this?

[Alex Dadgar]

What do you mean by live migration? Do you mean snapshotting the state of the VM and migrating that? Nomad does not have support for this.

Thanks,

Alex

On Fri, May 6, 2016 at 2:23 AM, pixel fairy <pixel...@gmail.com> wrote:

How does nomad handle live qemu host migration? has anyone here tried this?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/95ee814d-4100-47fd-bb13-6dcdba5248a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pixel fairy]

i mean this, http://www.linux-kvm.org/page/Migration

during the migration there are 2 instances of that vm running. what im asking is if nomad can tell its just a migration, or if theres a way to tell nomad not to kill one of the instances.

[Diptanu Choudhury]

Nomad doesn't support migration of VMs at all currently. However we have talked about doing this, but never got prioritized because no one has asked for it until now.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/2d9f52ce-6996-4fcd-9151-02956f2550c5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[pixel fairy]

The idea was to run qemu using the exec task driver, because the qemu one is too limited for our use. were you thinking of expanding the qemu driver, or finding a solution that would work with exec?

libvirt has "transient domains" that could be used too. they automatically become undefined when they shutdown, but their resources (disk images etc) do persist. while migrating, the copy on the destination is said to be "invisible" until its complete, see live vs non-live http://wiki.libvirt.org/page/VM_lifecycle#Migration . if nomad could use virsh to see whats running, this could make that seamless.

one feature id really like is to be able to vacate a hardware node for maintenance.

[Alex Dadgar]

Interesting!

Unfortunately there is a lot more low hanging fruit to tackle before we would look into live migration of workloads. Out of curiosity, what were you missing in the qemu driver? We would love PRs to improve its features!

Thanks,

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/fb7d5d14-8ce4-499a-bc0a-a5894054cc4e%40googlegroups.com.

[pixel fairy]

I was thinking of waiting till you had a plugin interface, but since your asking, here goes

1. storage other than a file. in my case rbd(ceph), but im sure others would want iscsi, glusterfs etc
2. multiple nic support. sometimes we want a box that can talk to multiple internal networks
3. resource allocation. cpus, and memory. ballooning.
   
4. other storage, for example a cdrom.
5. display, monitor and serial line sockets. aside from console, another use case is logging over the virtual serial line. the display socket is for the spice/vnc server. monitor is for qemu commands. then you can use socat to attach that to a network port or something like screen or minicom if you want to connect to the console

most other things could be done by sending commands to the monitor once its running. libvirt gives you a lot of those things out of the box which is why i mentioned it.

[Alex Dadgar]

Hey,

storage other than a file. in my case rbd(ceph), but im sure others would want iscsi, glisters etc

This is something we would like to implement in a generic way so that all drivers including Qemu can benefit.

multiple nic support. sometimes we want a box that can talk to multiple internal networks

This will probably be coming in 0.4 or 0.4.1 (for qemu) but it is not as simple as an enhanced driver. Requires work on both the scheduler, client and drivers

resource allocation. cpus, and memory. ballooning. 

Could you explain this one?

other storage, for example a cdrom.

Can you explain the use case of this? When using a cluster scheduler you shouldn't really be baking in assumptions about the machine it will be running on? I may not be understanding the ask here as well,

display, monitor and serial line sockets. aside from console, another use case is logging over the virtual serial line. the display socket is for the spice/vnc server. monitor is for qemu commands. then you can use socat to attach that to a network port or something like screen or minicom if you want to connect to the console

This sounds like a pretty cool use case. If it is something you are interested in contributing we would love to see and RFC and would be happy to review a PR.

Thanks,

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/9a3343aa-d740-4982-8506-1671271c28c5%40googlegroups.com.

[work.a...@gmail.com]

Does this https://github.com/hashicorp/nomad/issues/1588 provide you the ability to pass few options you want to use for your use case ? I am sure, you would like to see the control coming through the driver and keeping the scheduler/driver unaware of what few important command line options mean, can be troublesome. However, you may be able to pass various other command line options if you wish for the qemu executable through the QEMU driver.