Ganeti node cannot reach the master IP

[Timothy Scoppetta]

Hello Folks,

I've got what seems to be a kind of strange error here and I'm at a
loss to figure out why. I'm trying to sell my group on using Ganeti to
manage our VM infrastructure and any help you guys could provide in
trying to figure this out would be amazing.

So, as you can see below when I try to verify my cluster it tells me
that vmhost-1 and vmhost-2 can't reach the master IP.

---------------------------------------------------------------------------------------------------------------------------------------------------------
root@vmhost-0:/# gnt-cluster verify
Submitted jobs 307, 308
Waiting for job 307 ...
Thu May 17 18:12:00 2012 * Verifying cluster config
Thu May 17 18:12:00 2012 * Verifying cluster certificate files
Thu May 17 18:12:00 2012 * Verifying hypervisor parameters
Thu May 17 18:12:00 2012 * Verifying all nodes belong to an existing
group
Waiting for job 308 ...
Thu May 17 18:12:01 2012 * Verifying group 'default'
Thu May 17 18:12:01 2012 * Gathering data (3 nodes)
Thu May 17 18:12:11 2012 * Gathering disk information (3 nodes)
Thu May 17 18:12:12 2012 * Verifying configuration file consistency
Thu May 17 18:12:12 2012 * Verifying node status
Thu May 17 18:12:12 2012 - ERROR: node vmhost-1.berkeley.edu: cannot
reach the master IP
Thu May 17 18:12:12 2012 - ERROR: node vmhost-2.berkeley.edu: cannot
reach the master IP
Thu May 17 18:12:12 2012 * Verifying instance status
Thu May 17 18:12:12 2012 * Verifying orphan volumes
Thu May 17 18:12:12 2012 * Verifying N+1 Memory redundancy
Thu May 17 18:12:12 2012 * Other Notes
Thu May 17 18:12:12 2012 * Hooks Results
---------------------------------------------------------------------------------------------------------------------------------------------------------

This seems strange to me because on one of those nodes:

---------------------------------------------------------------------------------------------------------------------------------------------------------
root@vmhost-2:/# gnt-cluster getmaster
vmhost-0.berkeley.edu
root@vmhost-2:/# ping vmhost-0.berkeley.edu
PING vmhost-0.berkeley.edu (128.32.244.132) 56(84) bytes of data.
64 bytes from vmhost-0.berkeley.edu (128.32.244.132): icmp_req=1
ttl=64 time=0.149 ms
64 bytes from vmhost-0.berkeley.edu (128.32.244.132): icmp_req=2
ttl=64 time=0.115 ms
---------------------------------------------------------------------------------------------------------------------------------------------------------

I do notice, however, that I get no output for gnt-cluster masterping.

Any ideas?

[Iustin Pop]

That command is (really) badly named. If you check it's output:

gnt-cluster master-ping --help
Usage
=====
gnt-cluster master-ping

Checks if the master is alive

So it does a different thing than what you expected (I think).

> Any ideas?

Could be a number of things, not enough data yet.

The way ganeti tests reachability is via a TCP connection to the target
address on the noded port (defaults to 1811). It could be that in your
environment this doesn't work (filtered?) whereas plain ping works.

Can you telnet to vmhost-0-berkeley.edu:1811 from various machines and
check what happens?

Alternatively, it could be that you have a network where the master IP
is in a different netblock (still reachable, but different) and that has
some corner-cases too.

thanks,
iustin

[Timothy Scoppetta]

Iustin,

Thanks for the quick reply. I'm very hopeful that I can use this whole
thing to virtualize our infrastructure and use it as a proof of
concept to prevent us having to go down other more expensive more
commercial roads.

It did turn out that my Master IP was placed on a separate network
entirely (but with proper routing, etc.) due to 'network politics.'
I've gone ahead and gotten an IP for the master reserved on the same
subnet, but I'm curious as to how I update the master IP for the
cluster. I updated it in /etc/hosts the master and all nodes (even
though it should resolve via DNS) and restarted ganeti everywhere it's
running, but I still get the same errors. I also noticed that br0:0
(the virtual device for the ganeti master) doesn't have the right IP
address. I could change it manually, but I feel that's not the
*proper* way to go about this. I did try 'changing' the interface
using gnt-cluster modify --master-netdev br0, but it had no real
effect and the IP remained the same.

Is there a way in the cluster's configuration that I can manually set
it?

As far as telnet, I get this from everywhere I try it:
root@vmhost-1:~# telnet
telnet> open vmhost-0.berkeley.edu:1811
telnet: could not resolve vmhost-0.berkeley.edu:1811/telnet: Name or
service not known

Thanks,
Tim

[Guido Trotter]

Hi Timothy,

Try gnt-cluster rename <same-name>

If the ip resolved at that name is changed, that will update the ip in Ganeti.

Thanks,

guido

[Iustin Pop]

On Fri, May 18, 2012 at 10:29:07AM -0700, Timothy Scoppetta wrote:
> Iustin,
>
> Thanks for the quick reply. I'm very hopeful that I can use this whole
> thing to virtualize our infrastructure and use it as a proof of
> concept to prevent us having to go down other more expensive more
> commercial roads.

Ack. I hope you find Ganeti good for your, or at least you will file
some nice bug reports :)

> It did turn out that my Master IP was placed on a separate network
> entirely (but with proper routing, etc.) due to 'network politics.'

Note that with Ganeti 2.6 (which we hope to release in the somewhat near
future), this will be supported too; but right now (with 2.5 and
earlier) it doesn't work well.

> I've gone ahead and gotten an IP for the master reserved on the same
> subnet, but I'm curious as to how I update the master IP for the
> cluster. I updated it in /etc/hosts the master and all nodes (even
> though it should resolve via DNS) and restarted ganeti everywhere it's
> running, but I still get the same errors. I also noticed that br0:0
> (the virtual device for the ganeti master) doesn't have the right IP
> address. I could change it manually, but I feel that's not the
> *proper* way to go about this. I did try 'changing' the interface
> using gnt-cluster modify --master-netdev br0, but it had no real
> effect and the IP remained the same.
>
> Is there a way in the cluster's configuration that I can manually set
> it?

I have to apologise; this is trivial to do but probably hard to find!

Just run:

gnt-cluster rename clustername

And (in your case) it will see that the IP address for clustername has
changed and will do the right thing. Of course, you can use this to
rename to a different name too (not just changed IP for the same name).

> As far as telnet, I get this from everywhere I try it:
> root@vmhost-1:~# telnet
> telnet> open vmhost-0.berkeley.edu:1811
> telnet: could not resolve vmhost-0.berkeley.edu:1811/telnet: Name or
> service not known

Sorry to give you false hints. In telnet syntax, this should be "telnet
host port"; I used the "host:port" syntax due to use with other tools…

Let us know how it goes!

iustin

[Timothy Scoppetta]

Success!

After renaming the cluster and having the IP update it verifies just fine. 

-------------------------------------------------------------------------------------

root@vmhost-0:~# gnt-cluster verify

Submitted jobs 395, 396

Waiting for job 395 ...

Fri May 18 10:55:28 2012 * Verifying cluster config

Fri May 18 10:55:28 2012 * Verifying cluster certificate files

Fri May 18 10:55:28 2012 * Verifying hypervisor parameters

Fri May 18 10:55:28 2012 * Verifying all nodes belong to an existing group

Waiting for job 396 ...

Fri May 18 10:55:28 2012 * Verifying group 'default'

Fri May 18 10:55:28 2012 * Gathering data (3 nodes)

Fri May 18 10:55:28 2012 * Gathering disk information (3 nodes)

Fri May 18 10:55:29 2012 * Verifying configuration file consistency

Fri May 18 10:55:29 2012 * Verifying node status

Fri May 18 10:55:29 2012 * Verifying instance status

Fri May 18 10:55:29 2012 * Verifying orphan volumes

Fri May 18 10:55:29 2012 * Verifying N+1 Memory redundancy

Fri May 18 10:55:29 2012 * Other Notes

Fri May 18 10:55:29 2012 * Hooks Results

-------------------------------------------------------------------------------------

Now I'm off to test failover and start reading the developers docs so that I and my army of interns can contribute some code this summer. If you have any areas or directions you'd particularly appreciate help on, development wise, feel free to let me know :)

Thanks again for your help, Ganeti is amazing!

Tim

--

Timothy Scoppetta
UC Berkeley

E: t...@eecs.berkeley.edu
P: 845-459-3002

[Lance Albertson]

If you get the point where you're wanting a web interface, make sure you check out Ganeti Web Manager [1]. Its a Django based project that we're working on at the OSL that gives a lot of nice features on top of Ganeti. If you run into any issues feel free to ping us in #ganeti-webmgr or on our mailing list. We're hoping to have a 0.9 release out by mid-June and a bug-fix release of 0.8.1 in the next day or so.

Cheers-

[1] https://code.osuosl.org/projects/ganeti-webmgr

On Fri, May 18, 2012 at 11:16 AM, Timothy Scoppetta <t...@eecs.berkeley.edu> wrote:

Now I'm off to test failover and start reading the developers docs so that I and my army of interns can contribute some code this summer. If you have any areas or directions you'd particularly appreciate help on, development wise, feel free to let me know :)

Thanks again for your help, Ganeti is amazing!

--
Lance Albertson

Associate Director of Operations

Oregon State University | Open Source Lab 

[Timothy Scoppetta]

Lance,

I had actually taken a look at GWM and deployed it in the django-dev sense to give it a try and it seems great. I didn't actually get the VNC portion working (which would be the most attractive portion for the less technical folks using this system) but I'm sure I'll get it going at some point. That's actually one of my main goals after getting Ganeti itself publicly configured.

I did have one question, and since I've got you here...

My initial goal of this implementation is to serve the needs of infrastructure here, your standard network services (DNS, DHCP, WDS, et al.) but I would like eventually to 'resell' virtualized instances to students for use on projects and what have you. The issue is that we need to track resource usage in a per user and per instance basis. Does your manager allow for that at all? Or is that something custom we'd have to cook up (and then contribute back, if you're open to such things)?

-Tim

[Lance Albertson]

On Fri, May 18, 2012 at 11:39 AM, Timothy Scoppetta <t...@eecs.berkeley.edu> wrote:

I had actually taken a look at GWM and deployed it in the django-dev sense to give it a try and it seems great. I didn't actually get the VNC portion working (which would be the most attractive portion for the less technical folks using this system) but I'm sure I'll get it going at some point. That's actually one of my main goals after getting Ganeti itself publicly configured.

That's been a pain point for us and can be tricky to get setup initially. We're hoping to get that simplified and fixed in future releases. If you have any specific issues you know where to find us and we can get it going for you.

 

I did have one question, and since I've got you here...

My initial goal of this implementation is to serve the needs of infrastructure here, your standard network services (DNS, DHCP, WDS, et al.) but I would like eventually to 'resell' virtualized instances to students for use on projects and what have you. The issue is that we need to track resource usage in a per user and per instance basis. Does your manager allow for that at all? Or is that something custom we'd have to cook up (and then contribute back, if you're open to such things)?

Our quota system in GWM should be able to handle that type of stuff fairly nicely. We have a quota and permission system built into GWM which will enable you to achieve that.

We also have a Google Summer of Code student working on integrating VM statistics [1] into GWM this summer so hopefully he can get that done for us.

Cheers-

[1] https://code.osuosl.org/issues/3615

[Timothy Scoppetta]

That's great to hear. I haven't really looked into it too thoroughly but I got some errors that I could find in the documentation and decided that I was putting the cart before the horse and my time would be better spent getting Ganeti up and running properly. That being said GWM is flashy and cool and I can't wait to get it deployed. It'll be the cherry on top of the virtualization scheme I'm hoping to pull off.

If there's any development needs you have (as I mentioned earlier about Ganeti) and you're looking for code contributions, let me know. I'm more than happy to give back to the projects I use and I have a few interns this summer that need the practice. Plus, it's fun to review their code :)

-Tim

[Lance Albertson]

On Fri, May 18, 2012 at 12:06 PM, Timothy Scoppetta <t...@eecs.berkeley.edu> wrote:

That's great to hear. I haven't really looked into it too thoroughly but I got some errors that I could find in the documentation and decided that I was putting the cart before the horse and my time would be better spent getting Ganeti up and running properly. That being said GWM is flashy and cool and I can't wait to get it deployed. It'll be the cherry on top of the virtualization scheme I'm hoping to pull off.

Glad you're looking forward to using it! Unfortunately the last few releases have had more bugs than we like in them but we're working towards improving that.

 

If there's any development needs you have (as I mentioned earlier about Ganeti) and you're looking for code contributions, let me know. I'm more than happy to give back to the projects I use and I have a few interns this summer that need the practice. Plus, it's fun to review their code :)

Outstanding! We're going to have a meeting soon to discuss some major refactoring we want to do moving forward which should simplify and reduce a lot of the bugs we're having. Once we have that squared away we can certain reach out to the community to ask for some help on some tasks.

And of course, we're always open to new features and suggestions!

[Immortal]

Hi .

I also get the same error. Well scenario is

192.168.2.180   public IP *.*.*.180
192.168.2.181   public IP *.*.*.181
192.168.2.182   public IP *.*.*.182

Well I have assigned the first two private IPs to the two nodes. They are able to ping each other. But when I intialize the cluster, every thing ran fine until I got the following error

ro...@node1.x.x# gnt-cluster verify

ERROR: node node2.x.x: cannot reach the master IP

While checking the ipconfig br-lan:0

I get to see the public IP *.*.*.182 instead of the private IP 192.168.2.182.  and there isn't ipconfig br-lan:0 on node2.

Please advice if am doing something wrong.

[Klaus Aehlig]

In order for someone to give advice, you probably have to share more
details on how you initialised the cluster.

In either case there are a few things to keep in mind.

- Ganeti assumes that the node names resolve to the
primary (from Ganeti's point of view) IP address of
the nodes and that the nodes can reach each other
via that IP.

- Ganeti assumes that the cluster name resolves to
the master IP address of that cluster. Note that the
master IP is an additional IP different from all the
node IPs.

Regards,
Klaus

--
Klaus Aehlig
Google Germany GmbH, Dienerstr. 12, 80331 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Graham Law, Christine Elizabeth Flores

[candlerb]

On Wednesday, 8 July 2015 04:54:33 UTC+1, Immortal wrote:

While checking the ipconfig br-lan:0

I get to see the public IP *.*.*.182 instead of the private IP 192.168.2.182.  and there isn't ipconfig br-lan:0 on node2.

I would guess that either in your /etc/hosts file or in the DNS you have mapped the cluster name to the public IP, not the management IP.

If you use:

    gnt-cluster init --master-netdev=br-lan .... foo.bar.baz

then foo.bar.baz should resolve to the management IP address you want on br-lan.  Then ganeti will create the alias br-lan:0 on the master node with that address.

The cluster itself doesn't need any public IP, and indeed the nodes themselves don't need public IPs if you are using private management IPs.

ganeti only requires that each node have one management IP (with all nodes on the same subnet), plus one cluster address on the same subnet. And if you have decided to use a secondary replication network then each node will need an interface on that network too.

HTH,

Brian.

[Tshering Dhendup]

Thank you all.

Well ! From my perception, nodes should have a FQDN which is DNS resolvable. Is this true? In my case, I have Private IP mapped to public ( in the router) and public IPs are DNS resolvable.

Or can we have private IPs in the /etc/hosts file ( without mapping) to public IP ?? Will ganeti work this way?

Just my doubt please.

On Wed, Jul 8, 2015 at 5:11 PM, candlerb <b.ca...@pobox.com> wrote:

This message is eligible for Automatic Cleanup! (b.ca...@pobox.com) Add cleanup rule | More info

On Wednesday, 8 July 2015 04:54:33 UTC+1, Immortal wrote:

While checking the ipconfig br-lan:0

I get to see the public IP *.*.*.182 instead of the private IP 192.168.2.182.  and there isn't ipconfig br-lan:0 on node2.

I would guess that either in your /etc/hosts file or in the DNS you have mapped the cluster name to the public IP, not the management IP.

If you use:

    gnt-cluster init --master-netdev=br-lan .... foo.bar.baz

then foo.bar.baz should resolve to the management IP address you want on br-lan.  Then ganeti will create the alias br-lan:0 on the master node with that address.

The cluster itself doesn't need any public IP, and indeed the nodes themselves don't need public IPs if you are using private management IPs.

ganeti only requires that each node have one management IP (with all nodes on the same subnet), plus one cluster address on the same subnet. And if you have decided to use a secondary replication network then each node will need an interface on that network too.

HTH,

Brian.

--

Tshering DHENDUP
IT ENGINEER.
BHUTAN TELECOM LTD
MOB # +975-17141707
WEB www.bt.bt

[candlerb]

The nodes and the cluster each need an FQDN which is resolvable within the cluster to the *actual* IP address that is configured on their interfaces and that they can use to talk to each other - not some NAT address on the other side of a firewall.

Overriding it in /etc/hosts will work. However it can be confusing if the external world resolves the same names to something else, and over time you will need to keep an increasing number of /etc/hosts files in sync.

Personally I have no problem putting private IP addresses in the DNS. If you prefer, you can use a separate subdomain for internal addresses, e.g. "node1.int.example.com" can be the private address and "node1.example.com" can be the public address. As long as each ganeti node knows that its own FQDN is "nodeX.int.example.com" (in /etc/hostname) then you should be OK.

Belt-and-braces, you can also put these private names in /etc/hosts on every ganeti node (use "gnt-cluster copyfile /etc/hosts" to distribute from the master to the slaves). Then you are also protected from DNS outages.

Regards,

Brian.

 

[Tshering Dhendup]

Thanks a lot Brian. Much understood.

Regards

On Thu, Jul 9, 2015 at 5:22 PM, candlerb <b.ca...@pobox.com> wrote:

This message is eligible for Automatic Cleanup! (b.ca...@pobox.com) Add cleanup rule | More info

Brian.