Adding tags on instance creation

[Simon Deziel]

I've been trying to add a tag on instance creation since I upgrade my
cluster to 2.5.0 rc5 but I can't get it working. This seems to hang
forever waiting to acquire all the locks in the "instance-add-post.d"
hooks. The hook is essentially this command :

gnt-instance add-tags $GANETI_INSTANCE_NAME kvmuser:$
{GANETI_INSTANCE_HV_security_domain}

Some more info :

# gnt-job list | tail -3
1496 running INSTANCE_CREATE(test02.inet.example.com)
1497 success TAGS_SEARCH(^kvmuser:kvm4001$)
1498 waiting TAGS_SET

# gnt-debug locks
Name Mode
Owner Pending
BGL/BGL shared Jq8/Job1498/
TAGS_SET,Jq25/Job1496/I_CREATE -
BGL/[lockset] -
- -
instances/[lockset] -
- -
instances/test01.inet.example.com -
- -
instances/test02.inet.example.com exclusive Jq25/Job1496/
I_CREATE exclusive:Jq8/Job1498/TAGS_SET
instances/test04.inet.example.com -
- -
instances/test04.vpn.example.com -
- -
nodegroups/[lockset] -
- -
nodegroups/d031ad03-3232-44fe-b58d-f0985ba0a2b1 -
- -
nodes/[lockset] -
- -
nodes/node1.hv.example.com -
- -
nodes/node2.hv.example.com -
- -
nodes/node3.hv.example.com -
- -
nodes/node4.hv.example.com -
- -

# gnt-job watch 1498
Output from job 1498 follows
----------------------------
Job 1498 is trying to acquire all necessary locks


Can someone point me to what I'm doing wrong ?

Thanks in advance,
Simon

[Iustin Pop]

On Thu, Mar 01, 2012 at 01:02:05PM -0800, Simon Deziel wrote:
> I've been trying to add a tag on instance creation since I upgrade my
> cluster to 2.5.0 rc5 but I can't get it working. This seems to hang
> forever waiting to acquire all the locks in the "instance-add-post.d"
> hooks. The hook is essentially this command :
>
> gnt-instance add-tags $GANETI_INSTANCE_NAME kvmuser:$
> {GANETI_INSTANCE_HV_security_domain}
>
> Some more info :
>
> # gnt-job list | tail -3
> 1496 running INSTANCE_CREATE(test02.inet.example.com)
> 1497 success TAGS_SEARCH(^kvmuser:kvm4001$)
> 1498 waiting TAGS_SET
>

> Can someone point me to what I'm doing wrong ?

The hooks are executed while the instance is still locked, so they won't
be able to acquire the lock, thus leading to a deadlock.

You should add a "--submit" to the job, so that it's launched
asynchronously. Yes, the instance will be created first without tags,
and then immediately the tags will be added (so there will be a slight
interval when the instance will not have the tags).

Alternatively, you could add tags directly at the instance creation:

gnt-instance add --tags … [other options] $instance

hope this helps,
iustin

[Simon Deziel]

On 12-03-01 06:12 PM, Iustin Pop wrote:
> On Thu, Mar 01, 2012 at 01:02:05PM -0800, Simon Deziel wrote:
>> I've been trying to add a tag on instance creation since I upgrade my
>> cluster to 2.5.0 rc5 but I can't get it working. This seems to hang
>> forever waiting to acquire all the locks in the "instance-add-post.d"
>> hooks. The hook is essentially this command :
>>
>> gnt-instance add-tags $GANETI_INSTANCE_NAME kvmuser:$
>> {GANETI_INSTANCE_HV_security_domain}
>>
>> Some more info :
>>
>> # gnt-job list | tail -3
>> 1496 running INSTANCE_CREATE(test02.inet.example.com)
>> 1497 success TAGS_SEARCH(^kvmuser:kvm4001$)
>> 1498 waiting TAGS_SET
>>
>> Can someone point me to what I'm doing wrong ?
>
> The hooks are executed while the instance is still locked, so they won't
> be able to acquire the lock, thus leading to a deadlock.
>
> You should add a "--submit" to the job, so that it's launched
> asynchronously. Yes, the instance will be created first without tags,
> and then immediately the tags will be added (so there will be a slight
> interval when the instance will not have the tags).

I like the idea but gnt-instance add-tags does not support "--submit".

> Alternatively, you could add tags directly at the instance creation:
>
> gnt-instance add --tags … [other options] $instance

Much better idea ! No need for asynchronous add-tags then.

> hope this helps,

Yes, many thanks.

Regards,
Simon

[Iustin Pop]

On Thu, Mar 01, 2012 at 06:28:07PM -0500, Simon Deziel wrote:
> On 12-03-01 06:12 PM, Iustin Pop wrote:
> > On Thu, Mar 01, 2012 at 01:02:05PM -0800, Simon Deziel wrote:
> >> I've been trying to add a tag on instance creation since I upgrade my
> >> cluster to 2.5.0 rc5 but I can't get it working. This seems to hang
> >> forever waiting to acquire all the locks in the "instance-add-post.d"
> >> hooks. The hook is essentially this command :
> >>
> >> gnt-instance add-tags $GANETI_INSTANCE_NAME kvmuser:$
> >> {GANETI_INSTANCE_HV_security_domain}
> >>
> >> Some more info :
> >>
> >> # gnt-job list | tail -3
> >> 1496 running INSTANCE_CREATE(test02.inet.example.com)
> >> 1497 success TAGS_SEARCH(^kvmuser:kvm4001$)
> >> 1498 waiting TAGS_SET
> >>
> >> Can someone point me to what I'm doing wrong ?
> >
> > The hooks are executed while the instance is still locked, so they won't
> > be able to acquire the lock, thus leading to a deadlock.
> >
> > You should add a "--submit" to the job, so that it's launched
> > asynchronously. Yes, the instance will be created first without tags,
> > and then immediately the tags will be added (so there will be a slight
> > interval when the instance will not have the tags).
>
> I like the idea but gnt-instance add-tags does not support "--submit".

Oh, uh, well, that's called a "bug" then :), thanks for catching it.
Filled issue 220.

iustin