IP assignment

[Alexander Gallego]

Hi Guys, 

Trying to understand how to map k8s concepts and this networking-guide into an example I can test

RE: https://github.com/kubevirt/kubevirt/blob/master/docs/libvirt-pod-networking.md

I want to - somehow: either dynamically or statically - pass 2 IP's to the VM that kubevirt launches, binded to specific vNICs. 

Say the host creates a bunch of virtual interfaces apriori. I want to assign these interfaces to the VM and assign specific IP's to these interfaces.

I didn't see necessarily a place to pass this information to libvirt from the manifests / examples.

If it's not supported, where would one add this capability to kubevirt proper?

Alternatively, these can be assigned from k8s and one can just map them transparently - but not sure how yet*

Any pointers are hugely appreciated! 

Thanks!

 

[Stuart Gott]

Hi Alexander,

Currently KubeVirt assigns a single NIC to a VM and attaches that the the Kubernetes pod network. This way Virtual Machines are able to interact with native k8s services.

Adding other IPs is not yet supported as we're still working on what the API should look like. It's unfortunately more complicated than just adding NICs -- for instance, we don't yet have a way to model what network they should be attached to.

You might be interested in looking at https://github.com/kubevirt/kubernetes-device-plugins to get a sense of how we're thinking of approaching that feature.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/23f30f09-2ecd-498e-ab82-e004cc13dff2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Gallego]

On Tue, May 15, 2018 at 3:29 PM, Stuart Gott <sg...@redhat.com> wrote:

Hi Alexander,

Currently KubeVirt assigns a single NIC to a VM and attaches that the the Kubernetes pod network. This way Virtual Machines are able to interact with native k8s services.

Adding other IPs is not yet supported as we're still working on what the API should look like. It's unfortunately more complicated than just adding NICs -- for instance, we don't yet have a way to model what network they should be attached to.

You might be interested in looking at https://github.com/kubevirt/kubernetes-device-plugins to get a sense of how we're thinking of approaching that feature.

Got it, so if I follow (correct me if I'm wrong) you are making resources of these devices (https://asciinema.org/a/165047)

and then attaching them to the VM via k8s resources?

resources:
	limits:
	bridge.network.kubevirt.io/mybr0: 1

Makes sense, I will checkout the code to see if I can understand it a bit better.

Thank stuart

.alex

[Itamar Heim]

On 05/15/2018 03:43 PM, Alexander Gallego wrote:
>
>
> On Tue, May 15, 2018 at 3:29 PM, Stuart Gott <sg...@redhat.com
> <mailto:sg...@redhat.com>> wrote:
>
> Hi Alexander,
>
> Currently KubeVirt assigns a single NIC to a VM and attaches that
> the the Kubernetes pod network. This way Virtual Machines are able
> to interact with native k8s services.
>
> Adding other IPs is not yet supported as we're still working on what
> the API should look like. It's unfortunately more complicated than
> just adding NICs -- for instance, we don't yet have a way to model
> what network they should be attached to.
>
> You might be interested in looking at
> https://github.com/kubevirt/kubernetes-device-plugins
> <https://github.com/kubevirt/kubernetes-device-plugins> to get a
> sense of how we're thinking of approaching that feature.
>
>
>
> Got it, so if I follow (correct me if I'm wrong) you are making
> resources of these devices (https://asciinema.org/a/165047)
>
> and then attaching them to the VM via k8s resources?
>
> resources:
> limits:
> bridge.network.kubevirt.io/mybr0

> <http://bridge.network.kubevirt.io/mybr0>: 1

>
>
> Makes sense, I will checkout the code to see if I can understand it a
> bit better.
>
> Thank stuart
>
>
> .alex
>

the just published prototype of kubetron may be an interesting starting
point to consider as well:
https://twitter.com/YanivKaul/status/996475591066112000

Thanks,
Itamar

> it, send an email to kubevirt-dev...@googlegroups.com
> <mailto:kubevirt-dev...@googlegroups.com>.

> To post to this group, send email to
> kubevi...@googlegroups.com

> <mailto:kubevi...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubevirt-dev/23f30f09-2ecd-498e-ab82-e004cc13dff2%40googlegroups.com

> <https://groups.google.com/d/msgid/kubevirt-dev/23f30f09-2ecd-498e-ab82-e004cc13dff2%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to kubevirt-dev...@googlegroups.com
> <mailto:kubevirt-dev...@googlegroups.com>.

> To post to this group, send email to kubevi...@googlegroups.com

> <mailto:kubevi...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubevirt-dev/CAL%2BiW29Qz5XS31Zr7qf69nJEJqQdM-sce%2B5Sx6VoWN5AdJkpWw%40mail.gmail.com
> <https://groups.google.com/d/msgid/kubevirt-dev/CAL%2BiW29Qz5XS31Zr7qf69nJEJqQdM-sce%2B5Sx6VoWN5AdJkpWw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Alexander Gallego]

On Tuesday, May 15, 2018 at 7:46:50 PM UTC-4, Itamar Heim wrote:

On 05/15/2018 03:43 PM, Alexander Gallego wrote:
>
>
> On Tue, May 15, 2018 at 3:29 PM, Stuart Gott <sg...@redhat.com
> <mailto:sg...@redhat.com>> wrote:
>
>     Hi Alexander,
>
>     Currently KubeVirt assigns a single NIC to a VM and attaches that
>     the the Kubernetes pod network. This way Virtual Machines are able
>     to interact with native k8s services.
>
>     Adding other IPs is not yet supported as we're still working on what
>     the API should look like. It's unfortunately more complicated than
>     just adding NICs -- for instance, we don't yet have a way to model
>     what network they should be attached to.
>
>     You might be interested in looking at
>     https://github.com/kubevirt/kubernetes-device-plugins
>     <https://github.com/kubevirt/kubernetes-device-plugins> to get a
>     sense of how we're thinking of approaching that feature.
>

Hi Stuart, 

OK, i grok the device-as-k8s-resource idea. I downloaded the nvidia one and played around w/ it. 

Say I have a list of virtual nics / devices. and I have a device-plugin that does the accounting as a deamon set in k8s.

Also, assume I correctly pass in the name:

```

resource:

    limits:

         com.x.y.x/nic_type_one: 1

         com.x.y.x/nic_type_two:  1

```

 

How do I bind them / mount them  / pass them to libvirt through kubevirt?

Pointers at the code is OK too, just trying to understand what level of effort is required to get a prototype for me working. 

Thanks!

.alex

 

>         <mailto:kubevirt-dev+unsub...@googlegroups.com>.

>         To post to this group, send email to
>         kubevi...@googlegroups.com
>         <mailto:kubevi...@googlegroups.com>.
>         To view this discussion on the web visit
>         https://groups.google.com/d/msgid/kubevirt-dev/23f30f09-2ecd-498e-ab82-e004cc13dff2%40googlegroups.com
>         <https://groups.google.com/d/msgid/kubevirt-dev/23f30f09-2ecd-498e-ab82-e004cc13dff2%40googlegroups.com?utm_medium=email&utm_source=footer>.
>         For more options, visit https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubevirt-dev...@googlegroups.com

> <mailto:kubevirt-dev+unsub...@googlegroups.com>.

[Fabian Deutsch]

Hey Alex,

cool that you already found the path towards using the device plugins :)

I'm CCing Petr who is working on this, and  can help with the consumption of those.

Sadly the glue to get the additional NICs inside the pod into the VM is not yet started. Currently we are designing it, and I hope to output a proposal by the end of today to outline how the whole multi network thingy could look.

Networking is a complex matter and we try to place nice on the Kube front but also enable the use-cases traditional networking has.

- fabian

[phor...@redhat.com]

Hi Alexander,

as Fabian said, we are not yet able to connect VMs to secondary networks.

Please bear in mind that the Bridge plugin is a PoC and it is not really tested, you might find it unstable. This plugin allows you to expose fixed list of bridges available on Nodes to Pods that request connection to them. Configuration of the bridge and its attachment to external network is not covered by the plugin.

We are working now on OVN multi-network plugin "Kubetron" that Itamar mentioned. We just released first PoC version, but more patches are coming to make it ready for contributors. I'm going to open Issues with missing features soon, feel free to suggest more. We did not decide yet how the final version would look like, but it should allow you to connect your Pods to logical switches created on OVN. That means that whole configuration of networks (DHCP, static IPs, routing, connection to physical host interfaces) are up to OVN administrator, it will be completely in your hands.

Hope this helps.

Best regards,

Petr

[Alexander Gallego]

On Friday, May 18, 2018 at 8:21:13 AM UTC-4, phor...@redhat.com wrote:

Hi Alexander,

as Fabian said, we are not yet able to connect VMs to secondary networks.

Please bear in mind that the Bridge plugin is a PoC and it is not really tested, you might find it unstable. This plugin allows you to expose fixed list of bridges available on Nodes to Pods that request connection to them. Configuration of the bridge and its attachment to external network is not covered by the plugin.

We are working now on OVN multi-network plugin "Kubetron" that Itamar mentioned. We just released first PoC version, but more patches are coming to make it ready for contributors. I'm going to open Issues with missing features soon, feel free to suggest more. We did not decide yet how the final version would look like, but it should allow you to connect your Pods to logical switches created on OVN. That means that whole configuration of networks (DHCP, static IPs, routing, connection to physical host interfaces) are up to OVN administrator, it will be completely in your hands.

Thanks Petr!

 

I might have to write my own device-plugin regardless - Maybe I can get your thoughts. 

I'm trying to port some hardware to use vms managed by k8s/kubevirt.  At the moment, the hardware is already partially provisioned. 

I have say 100 virtual devices. each VM should get two. One for internal and one for external. For example:

"type1": {
         "target": "fg-013ec379-9e",
         "ip": "a.a.a.a",
         "mask": "255.255.255.0",
         "source": "vlan319",
         "mac": "52:54:00:07:f8:21",
         "gateway": "a.a.a.1"
       },
       "type2": {
         "source": "vlan124",
         "mac": "52:54:00:06:d7:4a",
         "mask": "255.0.0.0",
         "target": "bg-013ec379-9e",
         "ip": "b.b.b.b",

      "gateway": "b.b.b.1"
       }

This list is basically provided apriori  - even before k8s is installed, so as far as I understand it, i just need to figure out how to pass it to @Fabian's upcoming design =)

However! - I am really curious to see if you any any design doc/ wiki / etc for the open v switch (https://github.com/openvswitch/ovn-kubernetes) integration you mentioned, because maybe I can twists arms and find a way to use that instead while providing the same functionality. 

Thanks!!!

Thanks Fabian!

Awesome! Looking forward to reading it.  

[phor...@redhat.com]

That's interesting :)

So you have precreated 100 interfaces that are connected to a bridge (are they virtual interfaces)?

If you are going to write your own device plugin, take a look at our DP manager framework https://github.com/kubevirt/device-plugin-manager, it should make it easier for you.

For network DP there is a problem obtaining netns of the target Pod. In the Bridge plugin I used a hack that monitors devices attached to Docker containers. In Kubetron I used different hack (I feel this one is a bit safer), where I read DP checkpoint file. I'll be happy to help you with this.

From the top of my head:
Your DP will expose two resources "internal" and "external". In the KubeVirt network spec you would require one interface from each resource and then pass them to VM.
The DP would need to recognize which interface on your host belongs to which type, once Allocate call arrives requesting an interface, you would pass it to the Pod netns.
However, the solution might differ based on interface type.

ovn-kubernetes is made to provide primary Kubernetes network. Someone correct me if I'm wrong, but it is not likely that it will support multiple interface in near future. Anyway, I'd love to see it supporting such case :)

[Fabian Deutsch]

Spolier - You will likely be able to use VirtualMachinePresets to provide this list up front: http://www.kubevirt.io/user-guide/#/workloads/virtual-machines/presets

- fabian

 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/2e518a8f-fd47-44ae-b23e-17b518bc859a%40googlegroups.com.

[Alexander Gallego]

@fabian 

It would be nice (a requirement for me) to be able to pass metadata to libvirt w.r.t networking i.e.: 

      {%- if network.backend.queues is defined %}
      <driver queues="{{ network.backend.queues }}"/>
      {%- endif %}

 

for virtual nic multi-queues. 

- fabian

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.