username/password combo for custom provider (couchbase)

[Jakov Sosic]

Hi,

I'm trying to program custom type for managing Couchbase buckets.

Problem is that every RW operation to Couchbase cluster (creating,
deleting, resizing bucket) requires admin privileges (username/password).


Do you have any suggestion how should I pass the username/password to
provider? If I make it a param, then every resource of type
'couchbucket' will need to have 'username =>' and 'password =>' params,
which will (probably) be the same across the board.

Now, to circumvent that repetition, I'm thinking of maybe setting
something like:

Couchbucket {
username => $::couchbase::admin_user,
password => $::couchbase::admin_password,
}

in the '::couchbase' class. Offcourse there will be similar settings for
hostname and port of the couchnode, and probably something like purge =>
true.


But still, if someone wants to use the type without including the base
class, this is obviously not a viable option, because he will have to
address the username/password in every couchbucket resource definition.

So, what I am interested in is your opinion about how to proceed with
this one.



--
Jakov Sosic
www.srce.unizg.hr

[Nan Liu]

Crosspost to puppet-dev to get better feedback.

On Sat, Jul 13, 2013 at 7:54 AM, Jakov Sosic <jso...@srce.hr> wrote:

I'm trying to program custom type for managing Couchbase buckets.

Problem is that every RW operation to Couchbase cluster (creating,
deleting, resizing bucket) requires admin privileges (username/password).


Do you have any suggestion how should I pass the username/password to
provider? If I make it a param, then every resource of type
'couchbucket' will need to have 'username =>' and 'password =>' params,
which will (probably) be the same across the board.

There's two options, one to store the username/password on server like mysql resource (~/my.cnf), or in catalog. I had discussions where some users prefer the former, but I'm in the later camp.

Now, to circumvent that repetition, I'm thinking of maybe setting
something like:

Couchbucket {
  username => $::couchbase::admin_user,
  password => $::couchbase::admin_password,
}

in the '::couchbase' class. Offcourse there will be similar settings for
hostname and port of the couchnode, and probably something like purge =>
true.

This works, just slightly cumbersome. 

But still, if someone wants to use the type without including the base
class, this is obviously not a viable option, because he will have to
address the username/password in every couchbucket resource definition.

So, what I am interested in is your opinion about how to proceed with
this one.

In vmware modules, we use a transport resource to specify connectivity.

https://github.com/vmware/vmware-vcsa/blob/master/manifests/init.pp#L44-L51

Right now, transport is a resource and metaparameter, but I think there's some value perhaps to turn it back to a regular parameter so we can specify defaults.

Puppet::Type.newtype(:couchbucket) do
...
  newparam(:transport) do
    defaultto 'couch'
  end
end

This should allow a one time specification of username password for all resources:

transport { 'couch'

  username => ...

  password => ...,

}

couchbucket { 'a':

  #transport => 'couch' implied

}

You can find examples in the vmware module how we search the catalog to find the transport data to initialize and reuse connection.

I've intended to convert transport to a stand alone module and support the following usage:

1. simple store for username/password (for your exact usage).

2. transport ssh.

3. transport rest.

4. transport soap.

Maybe that would be of interest? I have some other ideas about how to take advantage of the catalog as data, instead of just resource, but that's probably going to deviate from this topic. 

Thanks,

Nan

[Jakov Sosic]

On 07/13/2013 06:43 PM, Nan Liu wrote:
> There's two options, one to store the username/password on server like
> mysql resource (~/my.cnf), or in catalog. I had discussions where some
> users prefer the former, but I'm in the later camp.

First approach is OK, if the client program supports dot-conf file with
supplied password. But 'couchbase-cli' doesn't. So I'm stuck with the
former anyway.

> This works, just slightly cumbersome.

Actually I've just found that I have trouble getting it to work :(

This code is generating errors:

Couchbucket {
admin_user => 'Administrator',
admin_password => 'secret',
}

couchbucket { 'testbucket':
ensure => present,
}


Error: Could not set 'present' on ensure: can't convert nil into String
at 75:/etc/puppet/modules/system/manifests/autoload/nodes/server.pp

Error: Could not set 'present' on ensure: can't convert nil into String
at 75:/etc/puppet/modules/system/manifests/autoload/nodes/server.pp
...

And this code works as expected:

couchbucket { 'testbucket':
ensure => present,
admin_user => 'Administrator',
admin_password => 'secret',
}


So I'm buffled why is that...

> In vmware modules, we use a transport resource to specify connectivity.
>
> https://github.com/vmware/vmware-vcsa/blob/master/manifests/init.pp#L44-L51
>
> Right now, transport is a resource and metaparameter, but I think
> there's some value perhaps to turn it back to a regular parameter so we
> can specify defaults.
>
> Puppet::Type.newtype(:couchbucket) do
> ...
> newparam(:transport) do
> defaultto 'couch'
> end
> end
>
> This should allow a one time specification of username password for all
> resources:
>
> transport { 'couch'
> username => ...
> password => ...,
> }
>
> couchbucket { 'a':
> #transport => 'couch' implied
> }

This seems nice.

> You can find examples in the vmware module how we search the catalog to
> find the transport data to initialize and reuse connection.
>
> I've intended to convert transport to a stand alone module and support
> the following usage:
>
> 1. simple store for username/password (for your exact usage).
> 2. transport ssh.
> 3. transport rest.
> 4. transport soap.
>
> Maybe that would be of interest? I have some other ideas about how to
> take advantage of the catalog as data, instead of just resource, but
> that's probably going to deviate from this topic.

This would be great! It would certainly avoid collisions... Eg. two
different modules (couchbase and vmware for example) both bringing
'transport' resource with them...

It would be a good idea to merge transport into stdlib.

[Jakov Sosic]

On 07/13/2013 06:43 PM, Nan Liu wrote:

> Puppet::Type.newtype(:couchbucket) do
> ...
> newparam(:transport) do
> defaultto 'couch'
> end
> end
>
> This should allow a one time specification of username password for all
> resources:
>
> transport { 'couch'
> username => ...
> password => ...,
> }
>
> couchbucket { 'a':
> #transport => 'couch' implied
> }
>

OK, I will basically do something like this, but I will implement my own
transport-alike resource until you guys standardize this :)

[Jakov Sosic]

On 07/15/2013 01:54 PM, Jakov Sosic wrote:

> OK, I will basically do something like this, but I will implement my own
> transport-alike resource until you guys standardize this :)

OK, I'm having a big problem ... How can I get the values from the other
resource?

For example:

couchconnection { 'default':
username => 'admin',
password => 'admin',
}

couchbucket { 'test':
ramsize => '64',
connection => Couchconnection['default'],
}


So, how can I get the value of username in my couchbucket 'create method'?

I would like to have something like:

def create
username = @resource[:connection][:username]
end


but that obviously doesn't work :-/


I've tried something along these lines, inspired by vmware-vconsole module:


@couchconnection ||=
Puppet::Puppetlabs::Couchconnection.retrieve(:resource_ref =>
resource[:connection], :catalog => resource.catalog)


but zero succes...

And ideas?

[Jakov Sosic]

On 07/15/2013 07:55 PM, Jakov Sosic wrote:

> For example:
>
> couchconnection { 'default':
> username => 'admin',
> password => 'admin',
> }
>
> couchbucket { 'test':
> ramsize => '64',
> connection => Couchconnection['default'],
> }

Damn, even my autorequire isn't working properly...

couchbucket { 'test_cli':
ensure => present,
ramsize => '64M',
connection => Couchconnection['something'],
}



And this is part of couchbucket type:

newproperty(:connection) do
desc 'Connection parameters for administrating Couchbase.'
defaultto 'couch'
end

autorequire(:couchconnection) do
self[:connection]
end

[Jakov Sosic]

On 07/15/2013 08:37 PM, Jakov Sosic wrote:

> And this is part of couchbucket type:
>
> newproperty(:connection) do
> desc 'Connection parameters for administrating Couchbase.'
> defaultto 'couch'
> end
>
> autorequire(:couchconnection) do
> self[:connection]
> end

Hmm, autorequire doesn't work in my cobbler(repo|system|profile|distro)
providers either, and I'm kinda certain it did work before... Is there a
bug related to autorequire, or is my code somehow broken?

[Nan Liu]

On Mon, Jul 15, 2013 at 4:54 AM, Jakov Sosic <jso...@srce.hr> wrote:

OK, I will basically do something like this, but I will implement my own
transport-alike resource until you guys standardize this :)

Disclaimer, I don't work at Puppet Labs anymore, and this will be a personal project as time permits. 

This would be great! It would certainly avoid collisions... Eg. two
different modules (couchbase and vmware for example) both bringing
'transport' resource with them...

 

Yep, that's the goal. I've seen interest managing database with puppet device, and a common module could provide a standard pattern for these type of usages.

 

It would be a good idea to merge transport into stdlib.

It's probably too experimental at this point for me to send PR against stdlib.

> OK, I will basically do something like this, but I will implement my own
> transport-alike resource until you guys standardize this :)

OK, I'm having a big problem ... How can I get the values from the other
resource?

For example:

couchconnection { 'default':
  username => 'admin',
  password => 'admin',
}

couchbucket { 'test':
  ramsize    => '64',
  connection => Couchconnection['default'],
}


So, how can I get the value of username in my couchbucket 'create method'?

I would like to have something like:

  def create
    username = @resource[:connection][:username]
  end


but that obviously doesn't work :-/

The one mistake in my original email was it was searching by resource by name instead of reference, so perhaps it's as simple as fixing:

defaultto 'Transport[couch]' # or maybe defaultto "Transport['couch']"

If that's not the issue, then I'm not sure. I'll try to put together the transport module when I get some spare time.

Thanks,

Nan

[Jakov Sosic]

On 07/16/2013 02:56 AM, Nan Liu wrote:

> If that's not the issue, then I'm not sure. I'll try to put together
> the transport module when I get some spare time.

Reference is not a problem, but fetching parameters from that resource
is... As I can see in vmware modules, there is a
PuppetX::PuppetLabs::Transport module with retrieve method, and I'm not
sure if I'm supposed to write something along that lines for my
'transport-alike' resource also?

[Jakov Sosic]

On 07/16/2013 02:56 AM, Nan Liu wrote:

> Disclaimer, I don't work at Puppet Labs anymore, and this will be a
> personal project as time permits.

Also, best of luck in the future! And thank you for your contribution so
far. You've been really helpful (at least to me) in the process of
learning custom types and how are they supposed to work...

[Nan Liu]

This was long overdue, but finally had a opportunity to extend native puppet resource to manage remote resource via transport. The specific implementation uses ssh:

  transport { 'esx':
    username => $username,
    password => $password,
    server => $server,
  }

  service { 'SSH':
    ensure => 'running',
    provider => 'ssh',
    transport => Transport['esx'],
  }

This was written to run against an ESXi server to update some sshd config:
https://github.com/vmware/vmware-vmware_lib

Thanks,

Nan