Transfer instances between clusters.

154 views

Skip to first unread message

panagiotis panagiotopoulos

unread,

Jul 27, 2014, 12:07:54 PM7/27/14

to gan...@googlegroups.com

Dear all,

I am using the following:

hspace (ganeti-htools) version v2.6.2

gnt-cluster version

Software version: 2.10.2

Internode protocol: 2100000

Configuration format: 2100000

OS api version: 20

Export interface: 0

VCS version: (ganeti) version v2.10.2

on Debian 3.2.60-1+deb7u1

I am trying to export and import an instance to move it to another cluster.

Running gnt-backup export -n node1.example.com instance1.vm.example.com everything seems to work fine.

Sun Jul 27 18:22:01 2014 Shutting down instance instance1.vm.example.com

Sun Jul 27 18:22:01 2014 Activating disks for instance1.vm.example.com

Sun Jul 27 18:22:01 2014 Creating a snapshot of disk/0 on node node1.example.com

Sun Jul 27 18:22:02 2014 Exporting snapshot/0 from node1.example.com to node1.example.com

Sun Jul 27 18:22:05 2014 snapshot/0 is now listening, starting export

Sun Jul 27 18:22:08 2014 snapshot/0 is receiving data on node1.example.com

Sun Jul 27 18:22:08 2014 snapshot/0 is sending data on node1.example.com

Sun Jul 27 18:22:13 2014 snapshot/0 sent 24M, 3.8 MiB/s

Sun Jul 27 18:23:14 2014 snapshot/0 sent 183M, 2.6 MiB/s

Sun Jul 27 18:24:15 2014 snapshot/0 sent 607M, 7.6 MiB/s

Sun Jul 27 18:25:16 2014 snapshot/0 sent 1.3G, 12.4 MiB/s

Sun Jul 27 18:26:17 2014 snapshot/0 sent 2.0G, 12.1 MiB/s

Sun Jul 27 18:27:19 2014 snapshot/0 sent 2.8G, 12.1 MiB/s

Sun Jul 27 18:28:20 2014 snapshot/0 sent 3.5G, 12.2 MiB/s

Sun Jul 27 18:29:21 2014 snapshot/0 sent 4.2G, 12.1 MiB/s

Sun Jul 27 18:30:22 2014 snapshot/0 sent 4.9G, 12.1 MiB/s

Sun Jul 27 18:31:23 2014 snapshot/0 sent 5.6G, 12.1 MiB/s

Sun Jul 27 18:32:24 2014 snapshot/0 sent 6.3G, 10.5 MiB/s

Sun Jul 27 18:33:25 2014 snapshot/0 sent 7.0G, 11.0 MiB/s

Sun Jul 27 18:34:26 2014 snapshot/0 sent 7.7G, 11.5 MiB/s

Sun Jul 27 18:35:27 2014 snapshot/0 sent 8.5G, 14.1 MiB/s

Sun Jul 27 18:36:28 2014 snapshot/0 sent 9.3G, 13.8 MiB/s

Sun Jul 27 18:37:29 2014 snapshot/0 sent 10.2G, 13.2 MiB/s

Sun Jul 27 18:38:32 2014 snapshot/0 sent 11.0G, 14.1 MiB/s

Sun Jul 27 18:39:33 2014 snapshot/0 sent 11.6G, 10.6 MiB/s

Sun Jul 27 18:40:19 2014 snapshot/0 finished receiving data

Sun Jul 27 18:40:20 2014 snapshot/0 finished sending data

Sun Jul 27 18:40:20 2014 Removing snapshot of disk/0 on node node1.example.com

Sun Jul 27 18:40:21 2014 Finalizing export on node1.example.com

Sun Jul 27 18:40:21 2014 Deactivating disks for instance1.vm.example.com

Sun Jul 27 18:40:21 2014 Removing old exports for instance instance1.vm.example.com

When i am trying to import it at node2.example.com the following errors occurs:

gnt-backup import -t plain -n node2.example.com --no-name-check --no-ip-check --src-node=node2.example.com --src-dir=/var/lib/ganeti/export/instance1.vm.example.com instance1.vm.example.com

Sun Jul 27 18:52:52 2014 * disk 0, size 50.0G

Sun Jul 27 18:52:52 2014 * creating instance disks...

Sun Jul 27 18:52:53 2014 adding instance instance1.vm.example.com to cluster config

Sun Jul 27 18:52:54 2014 - INFO: Waiting for instance instance1.vm.example.com to sync disks

Sun Jul 27 18:52:54 2014 - INFO: Instance instance1.vm.example.com disks are in sync

Sun Jul 27 18:52:54 2014 * running the instance OS import scripts...

Sun Jul 27 18:52:54 2014 Exporting disk/0 from node2.example.com to node2.example.com

Sun Jul 27 18:52:58 2014 disk/0 is now listening, starting export

Sun Jul 27 18:53:00 2014 - WARNING: import 'import-disk0-2014-07-27_18_52_54-liWJOt' on node2.example.com failed: Exited with status 1

Sun Jul 27 18:53:00 2014 disk/0 failed to receive data: Exited with status 1 (recent output: ssh: Could not resolve hostname f1e8461d-d8a2-430c-b857-c65316f7af6c: Name or service not known\ncp: cannot stat `:/srv/ganeti/export/instance1.vm.example.com/import_env.sh': No such file or directory\n/srv/ganeti/os/linux-image/import: line 75: [: -gt: unary operator expected\n/srv/ganeti/os/linux-image/import: line 80: /512: syntax error: operand expected (error token is "/512")\ndd: dd: writing `standard output': Broken pipe\ndd: 0 bytes (0 B) copied, 8.3423e-05 s, 0.0 kB/s\nsocat: E write(1, 0x7f0bfbb4e010, 16384): Broken pipe)

Sun Jul 27 18:53:00 2014 - WARNING: Aborting export 'export-disk0-2014-07-27_18_52_59-gXFqcp' on f1e8461d-d8a2-430c-b857-c65316f7af6c

Sun Jul 27 18:53:00 2014 - WARNING: export 'export-disk0-2014-07-27_18_52_59-gXFqcp' on node2.example.com failed: Exited with status 1

Sun Jul 27 18:53:00 2014 disk/0 failed to send data: Exited with status 1 (recent output: socat: E SSL_write(): Broken pipe\ndd: dd: writing `standard output': Broken pipe)

Sun Jul 27 18:53:00 2014 - WARNING: Some disks for instance instance1.vm.example.com on node node2.example.com were not imported successfully

Sun Jul 27 18:53:00 2014 Running rename script for instance1.vm.example.com

I also try

/usr/share/ganeti/2.10/move-instance --dest-ca-file=/user/ca.pem --src-ca-file=/var/lib/ganeti/rapi.pem --src-username=user --src-password-file=/user/pass --os-parameters os-image=image+wheezy --dest-instance-name=instance1.vm.example.com node1.example.com node2.example.com instance1.vm.example.com

2014-07-27 18:21:36,000: Move1 Caught unhandled exception

Traceback (most recent call last):

File "/usr/share/ganeti/2.10/move-instance", line 359, in HandleErrors

fn(*args)

File "/usr/share/ganeti/2.10/move-instance", line 615, in __init__

expinfo["x509_key_name"], mrt.dest_impinfo)

File "/usr/share/ganeti/2.10/move-instance", line 687, in _ExportInstance

(fin_resu, dresults) = poll_job_fn(cl, job_id)[0]

File "/usr/share/ganeti/2.10/move-instance", line 419, in PollJob

remote_import_fn))

File "/usr/share/ganeti/2.10/ganeti/rapi/client_utils.py", line 98, in PollJob

return cli.GenericPollJob(job_id, RapiJobPollCb(rapi_client), reporter)

File "/usr/share/ganeti/2.10/ganeti/cli.py", line 2087, in GenericPollJob

errors.MaybeRaise(msg)

File "/usr/share/ganeti/2.10/ganeti/errors.py", line 501, in MaybeRaise

raise errcls(*args)

OpExecError: Export failed, errors in disk export: disk(s) 0

2014-07-27 18:21:37,842: DestForMove1 Aborting despite job 123169 still running

Any ideas?

Thank you.

Phil Regnauld

unread,

Jul 27, 2014, 12:20:55 PM7/27/14

to gan...@googlegroups.com

panagiotis panagiotopoulos (ppanagiwtis) writes:
>
> I also try
>
> /usr/share/ganeti/2.10/move-instance --dest-ca-file=/user/ca.pem
> --src-ca-file=/var/lib/ganeti/rapi.pem --src-username=user
> --src-password-file=/user/pass --os-parameters os-image=image+wheezy
> --dest-instance-name=instance1.vm.example.com node1.example.com node2.example.com instance1.vm.example.com
> 2014-07-27 18:21:36,000: Move1 Caught unhandled exception

Hi,

What version of Ganeti are you running ? On which OS ?

> Traceback (most recent call last):
> File "/usr/share/ganeti/2.10/move-instance", line 359, in HandleErrors
> fn(*args)
> File "/usr/share/ganeti/2.10/move-instance", line 615, in __init__
> expinfo["x509_key_name"], mrt.dest_impinfo)
> File "/usr/share/ganeti/2.10/move-instance", line 687, in _ExportInstance
> (fin_resu, dresults) = poll_job_fn(cl, job_id)[0]
> File "/usr/share/ganeti/2.10/move-instance", line 419, in PollJob
> remote_import_fn))
> File "/usr/share/ganeti/2.10/ganeti/rapi/client_utils.py", line 98, in
> PollJob
> return cli.GenericPollJob(job_id, RapiJobPollCb(rapi_client), reporter)
> File "/usr/share/ganeti/2.10/ganeti/cli.py", line 2087, in GenericPollJob
> errors.MaybeRaise(msg)
> File "/usr/share/ganeti/2.10/ganeti/errors.py", line 501, in MaybeRaise
> raise errcls(*args)
> OpExecError: Export failed, errors in disk export: disk(s) 0

Hm, this is odd.

> 2014-07-27 18:21:37,842: DestForMove1 Aborting despite job 123169 still
> running

It took me a while to get move-instance to work - in particular I had
to make sure that:

- a cluster secret was set
- the PEM file(s) from the target cluster was available on the source
cluster
- the user existed on both sides (and had the right password as per the
pass file
- NTP was enabled and time was correct
- OS definitions had to be present on both sides

Also, gawk and curl had to be installed. But even then, I haven't seen
issues similar to what you describe..

What do the logs say ?

Cheers,
Phil

Message has been deleted

panagiotis panagiotopoulos

unread,

Jul 27, 2014, 12:48:01 PM7/27/14

to gan...@googlegroups.com

Both nodes have the same ganeti version 2.10.2 and running Debian 3.2.60-1+deb7u1

masterd logs from node1 (the node that sends the instance) :

2014-07-27 19:36:11,979: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26963, uid=114, gid=120

2014-07-27 19:36:11,980: ganeti-masterd pid=28749/ClientReq12 INFO Received instance query request for ['instance1.vm.example.com']

2014-07-27 19:36:12,119: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26965, uid=114, gid=120

2014-07-27 19:36:12,120: ganeti-masterd pid=28749/ClientReq16 INFO Receiving new job

2014-07-27 19:36:12,385: ganeti-masterd pid=28749/ClientReq16 INFO New job with id 264673, summary: INSTANCE_QUERY_DATA

2014-07-27 19:36:12,431: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26967, uid=114, gid=120

2014-07-27 19:36:12,431: ganeti-masterd pid=28749/ClientReq9 INFO Received job poll request for 264673

2014-07-27 19:36:12,479: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26969, uid=114, gid=120

2014-07-27 19:36:12,479: ganeti-masterd pid=28749/ClientReq15 INFO Received job poll request for 264673

2014-07-27 19:36:12,513: ganeti-masterd pid=28749/Jq8/Job264673 INFO Op 1/1: opcode INSTANCE_QUERY_DATA waiting for locks

2014-07-27 19:36:12,740: ganeti-masterd pid=28749/Jq8/Job264673 INFO Finished job 264673, status = success

2014-07-27 19:36:12,767: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26973, uid=114, gid=120

2014-07-27 19:36:12,767: ganeti-masterd pid=28749/ClientReq7 INFO Received job poll request for 264673

2014-07-27 19:36:12,871: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26976, uid=114, gid=120

2014-07-27 19:36:12,872: ganeti-masterd pid=28749/ClientReq4 INFO Receiving new job

2014-07-27 19:36:13,100: ganeti-masterd pid=28749/ClientReq4 INFO New job with id 264674, summary: BACKUP_PREPARE(instance1.vm.example.com)

2014-07-27 19:36:13,147: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26978, uid=114, gid=120

2014-07-27 19:36:13,147: ganeti-masterd pid=28749/ClientReq10 INFO Received job poll request for 264674

2014-07-27 19:36:13,195: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26980, uid=114, gid=120

2014-07-27 19:36:13,195: ganeti-masterd pid=28749/ClientReq11 INFO Received job poll request for 264674

2014-07-27 19:36:13,211: ganeti-masterd pid=28749/Jq13/Job264674 INFO Op 1/1: opcode BACKUP_PREPARE(instance1.vm.example.com) waiting for locks

2014-07-27 19:36:13,623: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26987, uid=114, gid=120

2014-07-27 19:36:13,623: ganeti-masterd pid=28749/ClientReq13 INFO Received job poll request for 264674

2014-07-27 19:36:13,671: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26988, uid=114, gid=120

2014-07-27 19:36:13,671: ganeti-masterd pid=28749/ClientReq2 INFO Received job poll request for 264674

2014-07-27 19:36:13,766: ganeti-masterd pid=28749/Jq13/Job264674 INFO Finished job 264674, status = success

2014-07-27 19:36:23,817: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26991, uid=114, gid=120

2014-07-27 19:36:23,818: ganeti-masterd pid=28749/ClientReq14 INFO Receiving new job

2014-07-27 19:36:24,070: ganeti-masterd pid=28749/ClientReq14 INFO New job with id 264675, summary: BACKUP_EXPORT(instance1.vm.example.com)

2014-07-27 19:36:24,119: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26993, uid=114, gid=120

2014-07-27 19:36:24,119: ganeti-masterd pid=28749/ClientReq6 INFO Received job poll request for 264675

2014-07-27 19:36:24,167: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=26995, uid=114, gid=120

2014-07-27 19:36:24,167: ganeti-masterd pid=28749/ClientReq8 INFO Received job poll request for 264675

2014-07-27 19:36:24,191: ganeti-masterd pid=28749/Jq4/Job264675 INFO Op 1/1: opcode BACKUP_EXPORT(instance1.vm.example.com) waiting for locks

2014-07-27 19:36:24,587: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27014, uid=114, gid=120

2014-07-27 19:36:24,588: ganeti-masterd pid=28749/ClientReq1 INFO Received job poll request for 264675

2014-07-27 19:36:24,635: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27016, uid=114, gid=120

2014-07-27 19:36:24,635: ganeti-masterd pid=28749/ClientReq5 INFO Received job poll request for 264675

2014-07-27 19:36:34,803: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27186, uid=114, gid=120

2014-07-27 19:36:34,804: ganeti-masterd pid=28749/ClientReq3 INFO Received job poll request for 264675

2014-07-27 19:36:35,915: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27192, uid=114, gid=120

2014-07-27 19:36:35,916: ganeti-masterd pid=28749/ClientReq12 INFO Received job poll request for 264675

2014-07-27 19:36:36,777: ganeti-masterd pid=28749/Jq4/Job264675/BACKUP_EXPORT INFO Started export 'export-disk0-2014-07-27_19_36_36-h2cNLh' on node1.example.com

2014-07-27 19:36:37,111: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27292, uid=114, gid=120

2014-07-27 19:36:37,112: ganeti-masterd pid=28749/ClientReq16 INFO Received job poll request for 264675

2014-07-27 19:36:47,271: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27385, uid=114, gid=120

2014-07-27 19:36:47,272: ganeti-masterd pid=28749/ClientReq9 INFO Received job poll request for 264675

2014-07-27 19:36:57,435: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27399, uid=114, gid=120

2014-07-27 19:36:57,436: ganeti-masterd pid=28749/ClientReq15 INFO Received job poll request for 264675

2014-07-27 19:37:07,595: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27414, uid=114, gid=120

2014-07-27 19:37:07,596: ganeti-masterd pid=28749/ClientReq7 INFO Received job poll request for 264675

2014-07-27 19:37:17,755: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27432, uid=114, gid=120

2014-07-27 19:37:17,756: ganeti-masterd pid=28749/ClientReq4 INFO Received job poll request for 264675

2014-07-27 19:37:27,919: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27594, uid=114, gid=120

2014-07-27 19:37:27,920: ganeti-masterd pid=28749/ClientReq10 INFO Received job poll request for 264675

2014-07-27 19:37:38,143: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27609, uid=114, gid=120

2014-07-27 19:37:38,144: ganeti-masterd pid=28749/ClientReq11 INFO Received job poll request for 264675

2014-07-27 19:37:48,307: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27624, uid=114, gid=120

2014-07-27 19:37:48,308: ganeti-masterd pid=28749/ClientReq13 INFO Received job poll request for 264675

2014-07-27 19:37:58,531: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27640, uid=114, gid=120

2014-07-27 19:37:58,532: ganeti-masterd pid=28749/ClientReq2 INFO Received job poll request for 264675

2014-07-27 19:38:08,691: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27658, uid=114, gid=120

2014-07-27 19:38:08,692: ganeti-masterd pid=28749/ClientReq14 INFO Received job poll request for 264675

2014-07-27 19:38:18,851: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27673, uid=114, gid=120

2014-07-27 19:38:18,852: ganeti-masterd pid=28749/ClientReq6 INFO Received job poll request for 264675

2014-07-27 19:38:29,011: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27835, uid=114, gid=120

2014-07-27 19:38:29,012: ganeti-masterd pid=28749/ClientReq8 INFO Received job poll request for 264675

2014-07-27 19:38:39,175: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27857, uid=114, gid=120

2014-07-27 19:38:39,176: ganeti-masterd pid=28749/ClientReq1 INFO Received job poll request for 264675

2014-07-27 19:38:48,545: ganeti-masterd pid=28749/Jq4/Job264675/BACKUP_EXPORT WARNING export 'export-disk0-2014-07-27_19_36_36-h2cNLh' on node1.example.com failed: Exited with status 1

2014-07-27 19:38:49,039: ganeti-masterd pid=28749/Jq4/Job264675/BACKUP_EXPORT INFO Finalizing export 'export-disk0-2014-07-27_19_36_36-h2cNLh' on 3dd18c66-bfcf-4022-be3e-2e86384e884c

2014-07-27 19:38:49,087: ganeti-masterd pid=28749/MainThread INFO Accepted connection from pid=27921, uid=114, gid=120

2014-07-27 19:38:49,088: ganeti-masterd pid=28749/ClientReq5 INFO Received job poll request for 264675

2014-07-27 19:38:49,122: ganeti-masterd pid=28749/Jq4/Job264675 ERROR Op 1/1: Caught exception in BACKUP_EXPORT(instance1.vm.example.com)

Traceback (most recent call last):

File "/usr/share/ganeti/2.10/ganeti/jqueue.py", line 1121, in _ExecOpCodeUnlocked

timeout=timeout)

File "/usr/share/ganeti/2.10/ganeti/jqueue.py", line 1432, in _WrapExecOpCode

return execop_fn(op, *args, **kwargs)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 534, in ExecOpCode

calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 460, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 408, in _LockAndExecLU

result = self._ExecLU(lu)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 375, in _ExecLU

result = _ProcessResult(submit_mj_fn, lu.op, lu.Exec(self.Log))

File "/usr/share/ganeti/2.10/ganeti/cmdlib/backup.py", line 461, in Exec

utils.CommaJoin(failures))

OpExecError: Export failed, errors in disk export: disk(s) 0

2014-07-27 19:38:49,259: ganeti-masterd pid=28749/Jq4/Job264675 INFO Finished job 264675, status = error

masterd logs from node2: (the node that receives the instance):

2014-07-27 19:36:14,810: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11008, uid=114, gid=120

2014-07-27 19:36:14,810: ganeti-masterd pid=23214/ClientReq1 INFO Receiving new job

2014-07-27 19:36:15,114: ganeti-masterd pid=23214/ClientReq1 INFO New job with id 123226, summary: INSTANCE_CREATE(instance1.vm.example.com)

2014-07-27 19:36:15,159: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11009, uid=114, gid=120

2014-07-27 19:36:15,160: ganeti-masterd pid=23214/ClientReq10 INFO Received job poll request for 123226

2014-07-27 19:36:15,207: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11010, uid=114, gid=120

2014-07-27 19:36:15,208: ganeti-masterd pid=23214/ClientReq13 INFO Received job poll request for 123226

2014-07-27 19:36:15,289: ganeti-masterd pid=23214/Jq22/Job123226 INFO Op 1/1: opcode INSTANCE_CREATE(instance1.vm.example.com) waiting for locks

2014-07-27 19:36:15,475: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11013, uid=114, gid=120

2014-07-27 19:36:15,476: ganeti-masterd pid=23214/ClientReq3 INFO Received job poll request for 123226

2014-07-27 19:36:15,807: ganeti-masterd pid=23214/Jq22/Job123226/I_CREATE INFO Selected nodes for instance instance1.vm.example.com via iallocator hail: node2.example.com

2014-07-27 19:36:15,999: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11022, uid=114, gid=120

2014-07-27 19:36:16,000: ganeti-masterd pid=23214/ClientReq4 INFO Received job poll request for 123226

2014-07-27 19:36:16,635: ganeti-masterd pid=23214/Jq22/Job123226/I_CREATE INFO Creating disk 0 for instance 'instance1.vm.example.com'

2014-07-27 19:36:16,796: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11026, uid=114, gid=120

2014-07-27 19:36:16,796: ganeti-masterd pid=23214/ClientReq15 INFO Received job poll request for 123226

2014-07-27 19:36:17,300: ganeti-masterd pid=23214/Jq22/Job123226/I_CREATE INFO Instance 'instance1.vm.example.com' DISK_LAYOUT: {'8a1305cd-a197-4151-97ef-004c48b84790': ['xenvg/d978f213-4145-41f7-b871-ff94113619ea.disk0']}

2014-07-27 19:36:17,455: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11029, uid=114, gid=120

2014-07-27 19:36:17,456: ganeti-masterd pid=23214/ClientReq16 INFO Received job poll request for 123226

2014-07-27 19:36:18,838: ganeti-masterd pid=23214/Jq22/Job123226/I_CREATE INFO Waiting for instance instance1.vm.example.com to sync disks

2014-07-27 19:36:19,003: ganeti-masterd pid=23214/Jq22/Job123226/I_CREATE INFO Instance instance1.vm.example.com disks are in sync

2014-07-27 19:36:19,023: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11033, uid=114, gid=120

2014-07-27 19:36:19,024: ganeti-masterd pid=23214/ClientReq11 INFO Received job poll request for 123226

2014-07-27 19:36:19,219: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11036, uid=114, gid=120

2014-07-27 19:36:19,220: ganeti-masterd pid=23214/ClientReq2 INFO Received job poll request for 123226

2014-07-27 19:36:19,478: ganeti-masterd pid=23214/Jq22/Job123226/I_CREATE INFO Started import 'import-disk0-2014-07-27_19_36_19-uN7C0q' on node2.example.com

2014-07-27 19:36:22,816: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11039, uid=114, gid=120

2014-07-27 19:36:22,816: ganeti-masterd pid=23214/ClientReq8 INFO Received job poll request for 123226

2014-07-27 19:36:32,975: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11174, uid=114, gid=120

2014-07-27 19:36:32,976: ganeti-masterd pid=23214/ClientReq5 INFO Received job poll request for 123226

2014-07-27 19:36:43,131: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11181, uid=114, gid=120

2014-07-27 19:36:43,132: ganeti-masterd pid=23214/ClientReq14 INFO Received job poll request for 123226

2014-07-27 19:36:53,292: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11577, uid=114, gid=120

2014-07-27 19:36:53,292: ganeti-masterd pid=23214/ClientReq12 INFO Received job poll request for 123226

2014-07-27 19:37:03,456: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11625, uid=114, gid=120

2014-07-27 19:37:03,456: ganeti-masterd pid=23214/ClientReq9 INFO Received job poll request for 123226

2014-07-27 19:37:13,616: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11629, uid=114, gid=120

2014-07-27 19:37:13,616: ganeti-masterd pid=23214/ClientReq7 INFO Received job poll request for 123226

2014-07-27 19:37:23,780: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11638, uid=114, gid=120

2014-07-27 19:37:23,780: ganeti-masterd pid=23214/ClientReq6 INFO Received job poll request for 123226

2014-07-27 19:37:33,940: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11779, uid=114, gid=120

2014-07-27 19:37:33,940: ganeti-masterd pid=23214/ClientReq1 INFO Received job poll request for 123226

2014-07-27 19:37:44,104: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11782, uid=114, gid=120

2014-07-27 19:37:44,104: ganeti-masterd pid=23214/ClientReq10 INFO Received job poll request for 123226

2014-07-27 19:37:54,264: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11788, uid=114, gid=120

2014-07-27 19:37:54,264: ganeti-masterd pid=23214/ClientReq13 INFO Received job poll request for 123226

2014-07-27 19:38:04,423: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11838, uid=114, gid=120

2014-07-27 19:38:04,424: ganeti-masterd pid=23214/ClientReq3 INFO Received job poll request for 123226

2014-07-27 19:38:14,583: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11841, uid=114, gid=120

2014-07-27 19:38:14,584: ganeti-masterd pid=23214/ClientReq4 INFO Received job poll request for 123226

2014-07-27 19:38:24,744: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11849, uid=114, gid=120

2014-07-27 19:38:24,744: ganeti-masterd pid=23214/ClientReq15 INFO Received job poll request for 123226

2014-07-27 19:38:34,907: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11987, uid=114, gid=120

2014-07-27 19:38:34,908: ganeti-masterd pid=23214/ClientReq16 INFO Received job poll request for 123226

2014-07-27 19:38:45,060: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=11992, uid=114, gid=120

2014-07-27 19:38:45,060: ganeti-masterd pid=23214/ClientReq11 INFO Received job poll request for 123226

2014-07-27 19:40:01,226: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=12271, uid=0, gid=0

2014-07-27 19:40:01,342: ganeti-masterd pid=23214/ClientReq2 INFO Received config values query request for ['master_node']

2014-07-27 19:40:01,344: ganeti-masterd pid=23214/ClientReq8 INFO Received job autoarchive request for age 21600, timeout 29

2014-07-27 19:40:01,344: ganeti-masterd pid=23214/ClientReq8 INFO Archiving jobs with age more than 21600 seconds

2014-07-27 19:40:01,351: ganeti-masterd pid=23214/ClientReq5 INFO Received group query request for []

2014-07-27 19:40:01,491: ganeti-masterd pid=23214/MainThread INFO Accepted connection from pid=12289, uid=0, gid=0

2014-07-27 19:40:01,492: ganeti-masterd pid=23214/ClientReq14 INFO Received config values query request for ['master_node']

2014-07-27 19:40:01,493: ganeti-masterd pid=23214/ClientReq12 INFO Receiving new job

2014-07-27 19:40:01,814: ganeti-masterd pid=23214/ClientReq12 INFO New job with id 123227, summary: QUERY(instance), QUERY(node)

2014-07-27 19:40:01,816: ganeti-masterd pid=23214/ClientReq9 INFO Received job poll request for 123227

2014-07-27 19:40:01,818: ganeti-masterd pid=23214/ClientReq7 INFO Received job poll request for 123227

2014-07-27 19:40:01,972: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:02,130: ganeti-masterd pid=23214/ClientReq6 INFO Received job poll request for 123227

2014-07-27 19:40:02,925: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:04,008: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:05,108: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:06,310: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:07,723: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:09,336: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:11,347: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:13,923: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:17,420: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:22,234: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:30,066: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:31,242: ganeti-masterd pid=23214/ClientReq1 INFO Received job poll request for 123227

2014-07-27 19:40:43,377: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:40:59,002: ganeti-masterd pid=23214/Jq7/Job123227 INFO Op 1/2: opcode QUERY(instance) waiting for locks

2014-07-27 19:41:00,378: ganeti-masterd pid=23214/ClientReq10 INFO Received job poll request for 123227

candlerb

unread,

Jul 28, 2014, 5:36:06 AM7/28/14

to gan...@googlegroups.com

(1) gnt-backup export/import are supposed to be used for moving images *within* a cluster.

If you want to move between clusters, you'll have to manually copy /srv/ganeti/export/XXX/* from one cluster to the other. (Unless it's an NFS share which is visible to both clusters, of course)

(2) There is a separate tool for moving instances between clusters, as Phil mentioned:

http://docs.ganeti.org/ganeti/2.10/html/move-instance.html

It's a PITA to set up, but would be useful if you are doing lots of inter-cluster moves.

(3) The biggest unknown here is what OS definition you are using. Try this:

gnt-instance list -o name,os

What does the "os" column show for the instance(s) you are moving?

The format in which a VM is moved is very much dependent on the OS choice, because it uses the export and import scripts contained within the OS definition.

http://docs.ganeti.org/ganeti/2.10/man/ganeti-os-interface.html

It is therefore very important (a) that you have identical OS definitions on both clusters, and (b) you understand what your particular OS definition does.

For example: ganeti-instance-debootstrap's export takes a filesystem dump(8) of the root partition, and the import script creates a new partition and does a restore(8). But that won't install grub, so you may find your system is not bootable.

This means in general, exports and imports *don't work* out-of-the-box.

For the best chance of this working, I think you select an OS definition which does a simple block-by-block export.

You can do this using ganeti-instance-image with NOMOUNT=yes; or you can use ganeti-os-noop but you need to use this version which has not yet made it into the main tree:

https://github.com/candlerb/ganeti-os-noop/tree/master/ganeti/os/noop

Just create directory /usr/share/ganeti/os/noop/ and copy those 5 files there.

snf-image also does the right thing, i.e. simple block image export.

For this to work you need to change the OS definition which your instance uses (easy enough: e.g. gnt-instance modify -o noop <instance-name>)

I'm afraid this part of ganeti is probably the biggest can of worms, and the most surprising to new users.

Regards,

Brian.

panagiotis panagiotopoulos

unread,

Jul 28, 2014, 2:48:01 PM7/28/14

to gan...@googlegroups.com

Thank you very much for your reply.

Both cluster have the same OS definitions. With the same variants and the same hooks. I tried to disable swap partition from variants conf file but nothing changes. I also try to use the noop OS definition but it gives me the following error:

2014-07-28 21:42:48,276: Move1 Caught unhandled exception

Traceback (most recent call last):

File "/usr/share/ganeti/2.10/move-instance", line 359, in HandleErrors

fn(*args)

File "/usr/share/ganeti/2.10/move-instance", line 615, in __init__

expinfo["x509_key_name"], mrt.dest_impinfo)

File "/usr/share/ganeti/2.10/move-instance", line 687, in _ExportInstance

(fin_resu, dresults) = poll_job_fn(cl, job_id)[0]

File "/usr/share/ganeti/2.10/move-instance", line 419, in PollJob

remote_import_fn))

File "/usr/share/ganeti/2.10/ganeti/rapi/client_utils.py", line 98, in PollJob

return cli.GenericPollJob(job_id, RapiJobPollCb(rapi_client), reporter)

File "/usr/share/ganeti/2.10/ganeti/cli.py", line 2087, in GenericPollJob

errors.MaybeRaise(msg)

File "/usr/share/ganeti/2.10/ganeti/errors.py", line 501, in MaybeRaise

raise errcls(*args)

OpExecError: Export failed, errors in disk export: disk(s) 0

and from master log file

2014-07-28 21:42:47,737: ganeti-masterd pid=16936/Jq11/Job265442 ERROR Op 1/1: Caught exception in BACKUP_EXPORT(instance1.example.com)

Traceback (most recent call last):

File "/usr/share/ganeti/2.10/ganeti/jqueue.py", line 1121, in _ExecOpCodeUnlocked

timeout=timeout)

File "/usr/share/ganeti/2.10/ganeti/jqueue.py", line 1432, in _WrapExecOpCode

return execop_fn(op, *args, **kwargs)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 534, in ExecOpCode

calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 460, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 469, in _LockAndExecLU

result = self._LockAndExecLU(lu, level + 1, calc_timeout)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 408, in _LockAndExecLU

result = self._ExecLU(lu)

File "/usr/share/ganeti/2.10/ganeti/mcpu.py", line 375, in _ExecLU

result = _ProcessResult(submit_mj_fn, lu.op, lu.Exec(self.Log))

File "/usr/share/ganeti/2.10/ganeti/cmdlib/backup.py", line 461, in Exec

utils.CommaJoin(failures))

OpExecError: Export failed, errors in disk export: disk(s) 0

All instances that I am trying to transfer are debian images created by a custom image with 3 partitions (boot, swap, and root, in that order.) The creation process works just fine with or without using swap partition in the variant file. (SWAP="yes"

SWAP_SIZE="4096" or SWAP="no")

candlerb

unread,

Jul 28, 2014, 5:20:19 PM7/28/14

to gan...@googlegroups.com

On Monday, 28 July 2014 19:48:01 UTC+1, panagiotis panagiotopoulos wrote:

All instances that I am trying to transfer are debian images created by a custom image with 3 partitions (boot, swap, and root, in that order.) The creation process works just fine with or without using swap partition in the variant file. (SWAP="yes"
SWAP_SIZE="4096" or SWAP="no")

But you still haven't said what os definition you used, so I'm afraid I'm still working blind.

(The only one I know which supports boot+swap+root out-of-the-box is ganeti-instance-image)

As for ganeti-os-noop: I've tested my version for export and import (but not for move-instance). The one which "apt-get" gives you (from the grnet repo) will fail, because the export and import scripts just exit without doing anything.

I'd strongly recommend you get export and import working within a cluster, before attempting move-instance.

And beware: move-instance will unconditionally delete the source instance whether or not the move was successful.

Hrvoje Ribicic

unread,

Jul 29, 2014, 2:24:56 PM7/29/14

to gan...@googlegroups.com

On Mon, Jul 28, 2014 at 11:20 PM, candlerb <b.ca...@pobox.com> wrote:

On Monday, 28 July 2014 19:48:01 UTC+1, panagiotis panagiotopoulos wrote:

All instances that I am trying to transfer are debian images created by a custom image with 3 partitions (boot, swap, and root, in that order.) The creation process works just fine with or without using swap partition in the variant file. (SWAP="yes"

SWAP_SIZE="4096" or SWAP="no")

But you still haven't said what os definition you used, so I'm afraid I'm still working blind.

(The only one I know which supports boot+swap+root out-of-the-box is ganeti-instance-image)

As for ganeti-os-noop: I've tested my version for export and import (but not for move-instance). The one which "apt-get" gives you (from the grnet repo) will fail, because the export and import scripts just exit without doing anything.

I'd strongly recommend you get export and import working within a cluster, before attempting move-instance.

+1 for this. If you cannot export and import an instance within the source/destination cluster, move-instance certainly will not work. That said, I will look at the original issue, it looks like a problem assembling the command line used for the import-export daemon.

And beware: move-instance will unconditionally delete the source instance whether or not the move was successful.

Looking at the move-instance code, the only gap where this can happen is in the case that the export succeeded, but the import did not.

I will experiment with the tool, and have a fix ready if this can happen, as this is extremely unacceptable behavior.

Phil Regnauld

unread,

Jul 29, 2014, 3:42:40 PM7/29/14

to 'Hrvoje Ribicic' via ganeti

'Hrvoje Ribicic' via ganeti (ganeti) writes:
> >
> > And beware: move-instance will unconditionally delete the source instance
> > whether or not the move was successful.
> >
>
> Looking at the move-instance code, the only gap where this can happen is in
> the case that the export succeeded, but the import did not.
> I will experiment with the tool, and have a fix ready if this can happen,
> as this is extremely unacceptable behavior.

I found this out while testing - it was quite surprising, especially
after an OS-image definition failed to export all disks.

Cheers,
Phil

panagiotis panagiotopoulos

unread,

Aug 3, 2014, 3:41:33 PM8/3/14

to gan...@googlegroups.com

gnt-instance list -o name,os instance1

Instance OS

instance1 image+wheezy

cat variants/wheezy.conf

SWAP="yes"

SWAP_SIZE="4096"

FILESYSTEM="ext4"

IMAGE_NAME="debian-7.3"

IMAGE_TYPE="dump"

IMAGE_DIR="/var/cache/ganeti-instance-image"

ARCH="x86_64"

CUSTOMIZE_DIR="/etc/ganeti/instance-image/hooks"

IMAGE_DEBUG=1

PARTITION_STYLE="none"

The above file (wheezy,conf) is the same at all nodes at both clusters.

I tried to export the instance then remove it and import it back. Everything work like a charm.

Then I tried to export the instance, copy the folder (/var/lib/ganeti/export/insance1) from one cluster to the other and import it at the second cluster. Also worked great.

Finally I tried to use move-instance.

gnt-job list

267324 success INSTANCE_QUERY_DATA

267325 success BACKUP_PREPARE(instance1)

267326 error BACKUP_EXPORT(instance1)

master-daemon.log

2014-08-03 22:32:30,104: ganeti-masterd pid=20477/ClientReq4 INFO New job with id 267324, summary: INSTANCE_QUERY_DATA

2014-08-03 22:32:30,106: ganeti-masterd pid=20477/ClientReq14 INFO Picking up new job from queue

2014-08-03 22:32:30,106: ganeti-masterd pid=20477/ClientReq14 INFO Restarting job 267324

2014-08-03 22:32:30,154: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26410, uid=114, gid=120

2014-08-03 22:32:30,154: ganeti-masterd pid=20477/ClientReq1 INFO Received job poll request for 267324

2014-08-03 22:32:30,202: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26412, uid=114, gid=120

2014-08-03 22:32:30,202: ganeti-masterd pid=20477/ClientReq8 INFO Received job poll request for 267324

2014-08-03 22:32:30,377: ganeti-masterd pid=20477/Jq14/Job267324 INFO Op 1/1: opcode INSTANCE_QUERY_DATA waiting for locks

2014-08-03 22:32:30,625: ganeti-masterd pid=20477/Jq14/Job267324 INFO Finished job 267324, status = success

2014-08-03 22:32:30,696: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=20592, uid=113, gid=118

2014-08-03 22:32:30,742: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26422, uid=114, gid=120

2014-08-03 22:32:30,742: ganeti-masterd pid=20477/ClientReq6 INFO Received job poll request for 267324

2014-08-03 22:32:30,842: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26426, uid=114, gid=120

2014-08-03 22:32:30,843: ganeti-masterd pid=20477/ClientReq7 INFO Receiving new job

2014-08-03 22:32:31,081: ganeti-masterd pid=20477/ClientReq7 INFO New job with id 267325, summary: BACKUP_PREPARE(instance1)

2014-08-03 22:32:31,082: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=20592, uid=113, gid=118

2014-08-03 22:32:31,082: ganeti-masterd pid=20477/ClientReq5 INFO Picking up new job from queue

2014-08-03 22:32:31,083: ganeti-masterd pid=20477/ClientReq5 INFO Restarting job 267325

2014-08-03 22:32:31,134: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26428, uid=114, gid=120

2014-08-03 22:32:31,134: ganeti-masterd pid=20477/ClientReq9 INFO Received job poll request for 267325

2014-08-03 22:32:31,182: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26430, uid=114, gid=120

2014-08-03 22:32:31,182: ganeti-masterd pid=20477/ClientReq12 INFO Received job poll request for 267325

2014-08-03 22:32:31,320: ganeti-masterd pid=20477/Jq5/Job267325 INFO Op 1/1: opcode BACKUP_PREPARE(instance1) waiting for locks

2014-08-03 22:32:31,494: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26434, uid=114, gid=120

2014-08-03 22:32:31,495: ganeti-masterd pid=20477/ClientReq10 INFO Received job poll request for 267325

2014-08-03 22:32:31,542: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26436, uid=114, gid=120

2014-08-03 22:32:31,543: ganeti-masterd pid=20477/ClientReq11 INFO Received job poll request for 267325

2014-08-03 22:32:31,777: ganeti-masterd pid=20477/Jq5/Job267325 INFO Finished job 267325, status = success

2014-08-03 22:32:31,827: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=20592, uid=113, gid=118

2014-08-03 22:32:45,252: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26472, uid=114, gid=120

2014-08-03 22:32:45,252: ganeti-masterd pid=20477/ClientReq13 INFO Receiving new job

2014-08-03 22:32:46,517: ganeti-masterd pid=20477/ClientReq13 INFO New job with id 267326, summary: BACKUP_EXPORT(instance1)

2014-08-03 22:32:46,517: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=20592, uid=113, gid=118

2014-08-03 22:32:46,518: ganeti-masterd pid=20477/ClientReq16 INFO Picking up new job from queue

2014-08-03 22:32:46,519: ganeti-masterd pid=20477/ClientReq16 INFO Restarting job 267326

2014-08-03 22:32:46,570: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26625, uid=114, gid=120

2014-08-03 22:32:46,571: ganeti-masterd pid=20477/ClientReq3 INFO Received job poll request for 267326

2014-08-03 22:32:46,622: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26631, uid=114, gid=120

2014-08-03 22:32:46,622: ganeti-masterd pid=20477/ClientReq2 INFO Received job poll request for 267326

2014-08-03 22:32:46,778: ganeti-masterd pid=20477/Jq19/Job267326 INFO Op 1/1: opcode BACKUP_EXPORT(instance1) waiting for locks

2014-08-03 22:32:46,938: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26635, uid=114, gid=120

2014-08-03 22:32:46,939: ganeti-masterd pid=20477/ClientReq15 INFO Received job poll request for 267326

2014-08-03 22:32:46,986: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26638, uid=114, gid=120

2014-08-03 22:32:46,986: ganeti-masterd pid=20477/ClientReq4 INFO Received job poll request for 267326

2014-08-03 22:32:47,266: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26642, uid=114, gid=120

2014-08-03 22:32:47,266: ganeti-masterd pid=20477/ClientReq1 INFO Received job poll request for 267326

2014-08-03 22:32:48,333: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=20592, uid=113, gid=118

2014-08-03 22:32:48,334: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=20592, uid=113, gid=118

2014-08-03 22:32:52,786: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26648, uid=114, gid=120

2014-08-03 22:32:52,786: ganeti-masterd pid=20477/ClientReq14 INFO Received job poll request for 267326

2014-08-03 22:32:53,471: ganeti-masterd pid=20477/Jq19/Job267326/BACKUP_EXPORT INFO Started export 'export-disk0-2014-08-03_22_32_53-J7115G' on node1.example.com

2014-08-03 22:32:53,546: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26652, uid=114, gid=120

2014-08-03 22:32:53,546: ganeti-masterd pid=20477/ClientReq8 INFO Received job poll request for 267326

2014-08-03 22:33:00,046: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26658, uid=114, gid=120

2014-08-03 22:33:00,046: ganeti-masterd pid=20477/ClientReq6 INFO Received job poll request for 267326

2014-08-03 22:33:10,210: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=26664, uid=114, gid=120

2014-08-03 22:33:10,211: ganeti-masterd pid=20477/ClientReq7 INFO Received job poll request for 267326

2014-08-03 22:33:20,733: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27065, uid=114, gid=120

2014-08-03 22:33:20,734: ganeti-masterd pid=20477/ClientReq9 INFO Received job poll request for 267326

2014-08-03 22:33:30,886: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27099, uid=114, gid=120

2014-08-03 22:33:30,886: ganeti-masterd pid=20477/ClientReq5 INFO Received job poll request for 267326

2014-08-03 22:33:41,050: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27107, uid=114, gid=120

2014-08-03 22:33:41,051: ganeti-masterd pid=20477/ClientReq12 INFO Received job poll request for 267326

2014-08-03 22:33:51,222: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27273, uid=114, gid=120

2014-08-03 22:33:51,223: ganeti-masterd pid=20477/ClientReq10 INFO Received job poll request for 267326

2014-08-03 22:34:00,790: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27279, uid=114, gid=120

2014-08-03 22:34:00,790: ganeti-masterd pid=20477/ClientReq11 INFO Received job poll request for 267326

2014-08-03 22:34:10,950: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27286, uid=114, gid=120

2014-08-03 22:34:10,951: ganeti-masterd pid=20477/ClientReq13 INFO Received job poll request for 267326

2014-08-03 22:34:12,517: ganeti-masterd pid=20477/Jq19/Job267326/BACKUP_EXPORT WARNING export 'export-disk0-2014-08-03_22_32_53-J7115G' on node1.example.com failed: Exited with status 1

2014-08-03 22:34:12,718: ganeti-masterd pid=20477/MainThread INFO Accepted connection from pid=27289, uid=114, gid=120

2014-08-03 22:34:12,718: ganeti-masterd pid=20477/ClientReq3 INFO Received job poll request for 267326

2014-08-03 22:34:13,292: ganeti-masterd pid=20477/Jq19/Job267326/BACKUP_EXPORT INFO Finalizing export 'export-disk0-2014-08-03_22_32_53-J7115G' on efdea1f1-ceac-4e8a-a722-f7053c700bfd

2014-08-03 22:34:13,374: ganeti-masterd pid=20477/Jq19/Job267326 ERROR Op 1/1: Caught exception in BACKUP_EXPORT(instance1)

Traceback (most recent call last):

File "/usr/share/ganeti/2.11/ganeti/jqueue.py", line 1130, in _ExecOpCodeUnlocked