gnt-instance console Problem

[Alan Simon]

Hello -

I have just set up a Ganeti 2.6.2 cluster of two i386 servers and created four DRBDed instances, two on each node.  I am unable to connect to the consoles of the instances on the master node, but can connect to the consoles of the instances on the secondary node.

I have:

1) added the following lines to /etc/default/grub on each of the nodes (followed by a update-grub):

    GRUB_CMDLINE_LINUX="console=tty0 console=hvc0"
    GRUB_TERMINAL="console serial"
    GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1"
    GRUB_CMDLINE_XEN="com1=9600,8n1 console=com1,vga dom0_mem=1024M nosmp"

2) added the following line to /etc/inittab on each node:

    hvc0:2345:respawn:/sbin/getty 38400 hvc0

and

3) modified the cluster root_path parameter to:

    root_path: /dev/xvda1

I rebooted the nodes after these changes.

Why am I unable to connect to the consoles on the master node?

Thanks in advance for any help you can give me.

Alan

-- 
Alan C. Simon
Director of Data Services
HSLC
3600 Market Street, Suite 550
Philadelphia, PA  19104

[Thomas Thrainer]

Hi Alan,

Could you please give a bit more information?

 * Which distribution do you use? Which Xen version?

 * What's the output of `gnt-cluster info`, `gnt-instance info --all` and `gnt-node info`?

 * Which commands are you issuing? What is the output?

 * After a command fails, could you please attach the relevant portions of /var/log/ganeti/master-daemon.log and /var/log/ganeti/node-daemon.log? Ideally, you attach all output from the end of those files based on the timestamp, and obviously of all the affected nodes (probably only from the master node in your case).

Thanks,

Thomas

--

Thomas Thrainer | Software Engineer | thom...@google.com | 

Google Germany GmbH

Dienerstr. 12

80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[Alan Simon]

Thomas -

I am running Xen 4.0.1 under Debian Squeeze.

root@utility1:/# gnt-cluster info
Cluster name: utilitycluster.int.hslc.org
Cluster UUID: cb702497-d67e-4edf-94b7-f8cfdee54fad
Creation time: 2013-06-10 12:42:22
Modification time: 2013-06-10 13:50:55
Master node: utility1.int.hslc.org
Architecture (this node): 32bit (i686)
Tags: (none)
Default hypervisor: xen-pvm
Enabled hypervisors: xen-pvm
Hypervisor parameters:
  - xen-pvm:
      bootloader_args:
      bootloader_path:
      initrd_path: /boot/initrd-2.6-xenU
      kernel_args: ro
      kernel_path: /boot/vmlinuz-2.6-xenU
      migration_port: 8002
      root_path: /dev/xvda1
      use_bootloader: False
OS specific hypervisor parameters:
Cluster parameters:
  - candidate pool size: 10
  - master netdev: xen-br0
  - lvm volume group: xenvg
  - file storage path: /srv/ganeti/file-storage
  - maintenance of node health: False
  - uid pool:
Default instance parameters:
  - default:
      auto_balance: True
      memory: 128
      vcpus: 1
Default nic parameters:
  - default:
      link: xen-br0
      mode: bridged

root@utility1:/# gnt-instance info --all
Instance name: nagios.int.hslc.org
UUID: 83153af4-e8a7-4e56-9769-fc40229be76e
Serial number: 4
Creation time: 2013-06-10 15:12:19
Modification time: 2013-06-12 07:26:43
State: configured to be up, actual state is up
  Nodes:
    - primary: utility1.int.hslc.org
    - secondaries: utility2.int.hslc.org
  Operating system: debootstrap+default
  Allocated network port: None
  Hypervisor: xen-pvm
    - bootloader_args: default ()
    - bootloader_path: default ()
    - initrd_path: default (/boot/initrd-2.6-xenU)
    - kernel_args: default (ro)
    - kernel_path: default (/boot/vmlinuz-2.6-xenU)
    - root_path: default (/dev/xvda1)
    - use_bootloader: default (False)
  Hardware:
    - VCPUs: 1
    - memory: 2048MiB
    - NICs:
      - nic/0: MAC: aa:00:00:c8:e4:83, IP: 10.10.20.36, mode: bridged, link: xen-br0
  Disks:
    - disk/0: drbd8, size 8.0G
      access mode:  rw
      nodeA:        utility1.int.hslc.org, minor=1
      nodeB:        utility2.int.hslc.org, minor=1
      port:         11004
      auth key:     88d61ca031e41d56ca843a305f85c546a0ff511e
      on primary:   /dev/drbd1 (147:1) in sync, status ok
      on secondary: /dev/drbd1 (147:1) in sync, status ok
      child devices:
        - child 0: lvm, size 8.0G
          logical_id:   xenvg/70502cb0-17a9-4d52-af9e-329df2887061.disk0_data
          on primary:   /dev/xenvg/70502cb0-17a9-4d52-af9e-329df2887061.disk0_data (254:2)
          on secondary: /dev/xenvg/70502cb0-17a9-4d52-af9e-329df2887061.disk0_data (254:2)
        - child 1: lvm, size 128M
          logical_id:   xenvg/70502cb0-17a9-4d52-af9e-329df2887061.disk0_meta
          on primary:   /dev/xenvg/70502cb0-17a9-4d52-af9e-329df2887061.disk0_meta (254:3)
          on secondary: /dev/xenvg/70502cb0-17a9-4d52-af9e-329df2887061.disk0_meta (254:3)
Instance name: ns2new.dmz.hslc.org
UUID: 79f217f7-61cd-49f1-8bf8-2a5646a64f3a
Serial number: 2
Creation time: 2013-06-10 17:17:04
Modification time: 2013-06-10 17:22:48
State: configured to be up, actual state is up
  Nodes:
    - primary: utility2.int.hslc.org
    - secondaries: utility1.int.hslc.org
  Operating system: debootstrap+default
  Allocated network port: None
  Hypervisor: xen-pvm
    - bootloader_args: default ()
    - bootloader_path: default ()
    - initrd_path: default (/boot/initrd-2.6-xenU)
    - kernel_args: default (ro)
    - kernel_path: default (/boot/vmlinuz-2.6-xenU)
    - root_path: default (/dev/xvda1)
    - use_bootloader: default (False)
  Hardware:
    - VCPUs: 1
    - memory: 2048MiB
    - NICs:
      - nic/0: MAC: aa:00:00:b5:c1:6e, IP: 10.10.10.42, mode: bridged, link: xen-br0
  Disks:
    - disk/0: drbd8, size 8.0G
      access mode:  rw
      nodeA:        utility2.int.hslc.org, minor=3
      nodeB:        utility1.int.hslc.org, minor=3
      port:         11007
      auth key:     2773bd6cc4408222dbe0cf917c623dbbe50a7bf7
      on primary:   /dev/drbd3 (147:3) in sync, status ok
      on secondary: /dev/drbd3 (147:3) in sync, status ok
      child devices:
        - child 0: lvm, size 8.0G
          logical_id:   xenvg/87919150-53b5-48e8-8f39-e838e47f1dfd.disk0_data
          on primary:   /dev/xenvg/87919150-53b5-48e8-8f39-e838e47f1dfd.disk0_data (254:6)
          on secondary: /dev/xenvg/87919150-53b5-48e8-8f39-e838e47f1dfd.disk0_data (254:6)
        - child 1: lvm, size 128M
          logical_id:   xenvg/87919150-53b5-48e8-8f39-e838e47f1dfd.disk0_meta
          on primary:   /dev/xenvg/87919150-53b5-48e8-8f39-e838e47f1dfd.disk0_meta (254:7)
          on secondary: /dev/xenvg/87919150-53b5-48e8-8f39-e838e47f1dfd.disk0_meta (254:7)
Instance name: ns1new.dmz.hslc.org
UUID: 4038ee88-f9c5-402f-a647-7c452d8ee0ea
Serial number: 2
Creation time: 2013-06-10 15:05:43
Modification time: 2013-06-10 15:11:39
State: configured to be up, actual state is up
  Nodes:
    - primary: utility1.int.hslc.org
    - secondaries: utility2.int.hslc.org
  Operating system: debootstrap+default
  Allocated network port: None
  Hypervisor: xen-pvm
    - bootloader_args: default ()
    - bootloader_path: default ()
    - initrd_path: default (/boot/initrd-2.6-xenU)
    - kernel_args: default (ro)
    - kernel_path: default (/boot/vmlinuz-2.6-xenU)
    - root_path: default (/dev/xvda1)
    - use_bootloader: default (False)
  Hardware:
    - VCPUs: 1
    - memory: 2048MiB
    - NICs:
      - nic/0: MAC: aa:00:00:64:7d:79, IP: 10.10.10.41, mode: bridged, link: xen-br0
  Disks:
    - disk/0: drbd8, size 8.0G
      access mode:  rw
      nodeA:        utility1.int.hslc.org, minor=0
      nodeB:        utility2.int.hslc.org, minor=0
      port:         11003
      auth key:     80fe213407c56caada25c060ba4de2b30501d1c8
      on primary:   /dev/drbd0 (147:0) in sync, status ok
      on secondary: /dev/drbd0 (147:0) in sync, status ok
      child devices:
        - child 0: lvm, size 8.0G
          logical_id:   xenvg/6c8db4d3-5bad-46f8-a965-6add61f597ea.disk0_data
          on primary:   /dev/xenvg/6c8db4d3-5bad-46f8-a965-6add61f597ea.disk0_data (254:0)
          on secondary: /dev/xenvg/6c8db4d3-5bad-46f8-a965-6add61f597ea.disk0_data (254:0)
        - child 1: lvm, size 128M
          logical_id:   xenvg/6c8db4d3-5bad-46f8-a965-6add61f597ea.disk0_meta
          on primary:   /dev/xenvg/6c8db4d3-5bad-46f8-a965-6add61f597ea.disk0_meta (254:1)
          on secondary: /dev/xenvg/6c8db4d3-5bad-46f8-a965-6add61f597ea.disk0_meta (254:1)
Instance name: ehour.int.hslc.org
UUID: 681dc417-2dc9-4a1e-8c52-cd71a50135be
Serial number: 4
Creation time: 2013-06-10 16:58:07
Modification time: 2013-06-12 07:26:56
State: configured to be up, actual state is up
  Nodes:
    - primary: utility2.int.hslc.org
    - secondaries: utility1.int.hslc.org
  Operating system: debootstrap+default
  Allocated network port: None
  Hypervisor: xen-pvm
    - bootloader_args: default ()
    - bootloader_path: default ()
    - initrd_path: default (/boot/initrd-2.6-xenU)
    - kernel_args: default (ro)
    - kernel_path: default (/boot/vmlinuz-2.6-xenU)
    - root_path: default (/dev/xvda1)
    - use_bootloader: default (False)
  Hardware:
    - VCPUs: 1
    - memory: 2048MiB
    - NICs:
      - nic/0: MAC: aa:00:00:b3:62:bc, IP: 10.10.20.37, mode: bridged, link: xen-br0
  Disks:
    - disk/0: drbd8, size 8.0G
      access mode:  rw
      nodeA:        utility2.int.hslc.org, minor=2
      nodeB:        utility1.int.hslc.org, minor=2
      port:         11005
      auth key:     735f2df6aab09bd90cdd73766fd36fafa6c91822
      on primary:   /dev/drbd2 (147:2) in sync, status ok
      on secondary: /dev/drbd2 (147:2) in sync, status ok
      child devices:
        - child 0: lvm, size 8.0G
          logical_id:   xenvg/46a8e910-2917-4a0b-a471-17682fd08e0a.disk0_data
          on primary:   /dev/xenvg/46a8e910-2917-4a0b-a471-17682fd08e0a.disk0_data (254:4)
          on secondary: /dev/xenvg/46a8e910-2917-4a0b-a471-17682fd08e0a.disk0_data (254:4)
        - child 1: lvm, size 128M
          logical_id:   xenvg/46a8e910-2917-4a0b-a471-17682fd08e0a.disk0_meta
          on primary:   /dev/xenvg/46a8e910-2917-4a0b-a471-17682fd08e0a.disk0_meta (254:5)
          on secondary: /dev/xenvg/46a8e910-2917-4a0b-a471-17682fd08e0a.disk0_meta (254:5)

root@utility1:/# gnt-node info
Node name: utility1.int.hslc.org
  primary ip: 10.10.20.26
  secondary ip: 10.10.20.26
  master candidate: True
  drained: False
  offline: False
  primary for instances:
    - nagios.int.hslc.org
    - ns1new.dmz.hslc.org
  secondary for instances:
    - ehour.int.hslc.org
    - ns2new.dmz.hslc.org
Node name: utility2.int.hslc.org
  primary ip: 10.10.20.27
  secondary ip: 10.10.20.27
  master candidate: True
  drained: False
  offline: False
  primary for instances:
    - ehour.int.hslc.org
    - ns2new.dmz.hslc.org
  secondary for instances:
    - nagios.int.hslc.org
    - ns1new.dmz.hslc.org


Log entries after issuing the commands:

gnt-instance console nagios
gnt-instance console ns1new

master-daemon.log
--------------------------
2013-06-12 07:28:46,358: ganeti-masterd pid=32304/ClientReq1 INFO Received new j
ob
2013-06-12 07:28:46,583: ganeti-masterd pid=32304/JobQueue1 INFO Processing job
1081
2013-06-12 07:28:46,584: ganeti-masterd pid=32304/JobQueue1 INFO Op 1/1: Startin
g opcode INSTANCE_CONSOLE(nagios)
2013-06-12 07:28:46,587: ganeti-masterd pid=32304/ClientReq1 INFO Received job p
oll request for 1081
2013-06-12 07:28:46,717: ganeti-masterd pid=32304/ClientReq1 INFO Received job p
oll request for 1081
2013-06-12 07:28:46,723: ganeti-masterd pid=32304/ClientReq1 INFO Received job p
oll request for 1081
2013-06-12 07:28:47,305: ganeti-masterd pid=32304/JobQueue1 INFO Op 1/1: Success
fully finished opcode INSTANCE_CONSOLE(nagios)
2013-06-12 07:28:47,418: ganeti-masterd pid=32304/JobQueue1 INFO Finished job 10
81, status = success
2013-06-12 07:28:47,420: ganeti-masterd pid=32304/ClientReq1 INFO Received job q
uery request for 1081
2013-06-12 07:28:51,956: ganeti-masterd pid=32304/MainThread INFO Accepted conne
ction from pid=32393, uid=0, gid=0
2013-06-12 07:28:51,958: ganeti-masterd pid=32304/ClientReq2 INFO Received new j
ob
2013-06-12 07:28:52,182: ganeti-masterd pid=32304/JobQueue2 INFO Processing job
1082
2013-06-12 07:28:52,183: ganeti-masterd pid=32304/JobQueue2 INFO Op 1/1: Startin
g opcode INSTANCE_CONSOLE(ns1new)
2013-06-12 07:28:52,186: ganeti-masterd pid=32304/ClientReq2 INFO Received job p
oll request for 1082
2013-06-12 07:28:52,300: ganeti-masterd pid=32304/ClientReq2 INFO Received job p
oll request for 1082
2013-06-12 07:28:52,893: ganeti-masterd pid=32304/JobQueue2 INFO Op 1/1: Success
fully finished opcode INSTANCE_CONSOLE(ns1new)
2013-06-12 07:28:53,005: ganeti-masterd pid=32304/JobQueue2 INFO Finished job 10
82, status = success
2013-06-12 07:28:53,007: ganeti-masterd pid=32304/ClientReq2 INFO Received job q
uery request for 1082

node-daemon.log
-------------------------
2013-06-12 07:28:47,186: ganeti-noded pid=32382 INFO 10.10.20.26:44741 PUT /inst
ance_list HTTP/1.0 200
2013-06-12 07:28:52,772: ganeti-noded pid=32394 INFO 10.10.20.26:44748 PUT /inst
ance_list HTTP/1.0 200

Plet let me know if you see anything unusual.  Thank you.

Alan

[Thomas Thrainer]

Hi,

This all seems correct for me. Have you tried to get a console by using Xen directly (`xm console "nagios.int.hslc.org"` and `xm console "ns1new.dmz.hslc.org"' on utility1)?

Could you also try going through the ganeti wrapper (`/usr/lib/ganeti/tools/xen-console-wrapper xm nagios.int.hslc.org')?

Thanks,

Thomas

 

[Alan Simon]

Thomas -

xm console ... works, but gnt-instance console ...  still does not.  There is no '/usr/lib/ganeti/tools/xen-console-wrapper' on my ganeti host, so I cannot try that.

Alan


On Tuesday, June 11, 2013 9:21:16 PM UTC-4, Alan Simon wrote:

[Thomas Thrainer]

Hi,

The xen-console-wrapper might have been installed in another location depending on your distro/configure settings (mine is in /usr/local/lib/ganeti/tools/).

If `xm console ...` works, that's really weird. Ganeti does nothing else than ssh-ing into the machine and issuing the same command. Is there something which might prevent ssh to work on your master node? Does the master node hostname properly resolve, as well as the cluster name? Could you try to open the console via ssh (`ssh us...@master-node.com xm console ...`)?

Thanks,

Thomas

[Alan Simon]

Thomas -

It works!  I set up Ganeti on an existing server that had SSH root login disabled.  Re-enabling SSH root login solved the problem.  Thank you for your SSH suggestion that triggered my memory.

Alan
 

On Tuesday, June 11, 2013 9:21:16 PM UTC-4, Alan Simon wrote: