Kubevirt VM networking question

[Garrett Tompkins]

Hi,

Not sure if this is the correct place to put this but I had a question regarding how networking works in kubevirt. I've read the manual a number of times and either my answer isn't there or I read it without knowing it.

My question is, is it possible to make a kubevirt VM reachable outside the cluster with a static IP assigned within the VM? For example, when using the test vm example provided by kubevirt, the IP address defaults to 10.0.2.2. However, on any cluster node, that ip address is unreachable. The VM can ping nodes and machines outside the cluster, but no machine can ping the 10.0.2.2 address. I can reach and ssh into the vm using the node IP address 172.30.X.X but for what I am working with, I need the VM to be reachable on the static IP inside the VM. Is this possible?

-Garrett Tompkins

[Miguel Duarte de Mora Barroso]

I think you're referring to a VM with the masquerade interface type. 

To reach VMs with those interface types, you **must** reach them over the IP address of the pod - we've tried to capture information in [0]. but seems our documentation could be more explicit.

The reported IPs will be seen if you `kubectl get vmi <vmi-name> `, as you can see in the short example below:

```

$ kubectl apply -f examples/vmi-masquerade.yaml
virtualmachineinstance.kubevirt.io/vmi-masquerade created
$ kubectl get vmi                                                    
NAME             AGE     PHASE     IP               NODENAME
vmi-masquerade   4m10s   Running   10.244.196.147   node01

```

For you to have a VM reachable with static IP, I think the best option is to use bridge interface type, and then use cloud-init to configure the static IP into your VM, as explained in this issue's comment [1].

[0] - https://kubevirt.io/user-guide/#/creation/interfaces-and-networks?id=masquerade

[1] - https://github.com/kubevirt/kubevirt/issues/4564#issuecomment-739802347

 

-Garrett Tompkins

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/86b90543-945c-44b1-b441-96068927db50n%40googlegroups.com.

[dvo...@redhat.com]

if you need the internal IP within the VM guest to represent the IP address other endpoints within the cluster can use to contact the VM, then use bridge mode instead of masquerade. 

 

-Garrett Tompkins

[Petr Horacek]

čt 4. 2. 2021 v 14:59 odesílatel dvo...@redhat.com <dvo...@redhat.com> napsal:

And if you want this IP to be accessible from outside of the cluster, I'm afraid the easiest option would be to use a secondary network through Multus, connect the VM to your enterprise network through a linux/OVS bridge and handle the IP management yourself. Then you would however lose all the benefits of Kubernetes networking (Services).

 

 

-Garrett Tompkins

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/dcf19c03-3e1c-4876-b5f8-b94a70b6dd8dn%40googlegroups.com.

[Shawn Lu]

Maybe you can try kube-eip
Kube-eip implement a eipbinding operator to represent a elastic ip bind to a kubevirt vmi,

and it will watch the change of vmi pod ip, if change it will call eip agent to update eip rules

on hyper,
For the eip binding, I insert two chans in iptables nat table, before kube-service to inject the

ingress and egress traffic, and use ipset to match the traffic outside k8s cluster.

And it's easy to deploy, hope it works good for you

[ekambaram varathan]

Hi Shawn Lu,

We were trying to setup kube-eip but getting error while deploying

kube-eip-agent    eip-agent-fvzgq                                0/1     CrashLoopBackOff   6 (40s ago)      6m12s
kube-eip-agent    eip-agent-vd95f                                0/1     CrashLoopBackOff   7 (3m45s ago)    14m

Error:-

kubectl logs eip-agent-vd95f --all-containers -n kube-eip-agent
1
time="2023-12-06 10:36:40" level=info msg="register ipset manager finished [5a6d69f2]"
time="2023-12-06 10:36:40" level=debug msg="eip route table exist, do nothing [66f60c7b]"
time="2023-12-06 10:36:40" level=error msg="setup policy route: Link not found [5a6d69f2]"
Link not found

[Shawn Lu]

It's necessary to config the configmap deploy the eip-agent, you can check it in kube-eip/config/agent/eip_agent.yaml For the error "Link not found", you should set eip_gw_dev according to your environment, which is the interface that access the public network,

and make sure each node with the same interface name. If not you can add a linux bridge such as br-pub and add the physical interface

to the linux bridge on each hyper.

Here is all config option and help info:
svc_net: 192.168.223.0/24 # K8s service ip cidr pod_net: 10.244.0.0/16 # K8s pod ip cidr eip_net: 192.168.18.0/24 # The public network cidr eip_gw_ip: 192.168.18.1 # The public network gateway eip_gw_dev: enp2s0 # The interface on each hyper, that access public netwrok interface(If interface name not same, add a linux bridge with the same name(such as br-pub) and add interface to linux bridge) log_level: debug

[ekambaram varathan]

Hi Shawn Lu,

It looks kube-eip plugin will work on cloud based k8s + kubevirt environment, actually we are looking "how do we connect with host machine local network outside cluster network" based on the on-premises(physical servers) k8s + kubevirt setup and using kube-eip.

Kindly