Re: [Puppet Users] Running all apt source and update operations first, best practice.

[Nan Liu]

On Thu, May 9, 2013 at 11:52 AM, James Kyle <li...@jameskyle.org> wrote:

I've run into several incidences where a module attempts to install a package before the apt::source is added or an update is run. Result is a bunch apt errors and explosions.

Basically what should be done is all the apt::sources are added and and an update run _before_ any packages are installed to ensure you're pulling from the repos you want. 

I've gone through several iterations in my attempt to achieve that behavior. The one that works best so far is stages and wrapper classes. Here's a terse example of what it looks like:

class myorg::common {

  include stdlib

  Apt::Source {stage => "setup"}  

  apt::source { 'puppetlabs':

    location   => 'http://apt.puppetlabs.com',

    repos      => 'main',

    key        => '4BD6EC30',

    key_server => 'pgp.mit.edu',

  }

  Exec['apt_update'] -> Package<|  title != 'ubuntu-cloud-keyring' |>

}

node 'foo.bar.com' {

  include stdlib

  

  class {'myorg::common': stage => "setup"}

}

One thing that bothers me is you have to declare the stage for myorg::common in every node that uses it. And as the name implies, that's every node.

Is there a way to get rid of that duplication? I've thought of node inheritance, but the docs seem to strongly steer you away from that pattern.

Doesn't the relationship do the right thing without stages? Does this work?

class myorg::common (

  $staging  = 'setup',

) { ...

Nan

[jcbollinger]

It would depend on whether and how stages are used elsewhere in the manifest set.  With stages it's often "in for a penny, in for a pound."  Generally speaking, though, it should be possible to do this sort of thing (or anything, actually) without stages.

 

Does this work?

class myorg::common (

  $staging  = 'setup',

) { ...

Nan

Provided that all of the relevant Apt sources are managed via Apt:::Source resources, and that no virtual Apt::Source or Package resources are declared without elsewhere being realized, this declaration should work:

Apt::Source<| |> -> Exec['apt_update'] -> Package<| |>

That declaration is not itself affected by or dependent on stages, but it is possible for the referenced resources to be assigned to stages in such a way that the chain closes one or more dependency cycles.  Such cycles would reflect an error in the stage assignments, however, for the chain expression succinctly and precisely describes the requested resource application order; as long as it does so, it defines the correct orders.  It would be sufficient to make that declaration once, in some central place, but not harmful to make it multiple times.

There are all manner of tweaks and accommodations that could be applied to handle special cases, such as when there are virtual Apt::Source or Package resources must remain unrealized, but the chain above captures the essential essence of the required declaration.


John

[joe]

This can be achieved without stages if you put the relationship inside site.pp outside any class scope.

[James Kyle]

This is what I get with the above, slightly adapted to take care of an edge case:

Apt::Source<| |> -> Exec['apt_update'] -> Package<| title != 'ubuntu-cloud-keyring' |>

The ubuntu-cloud-keyring is a prerequisite for adding the ubuntu cloud archive (for openstack debs). Without this exception, it leads to a cycle since Sources -> packages otherwise.

Dependency Cycle.

(Anchor[apt::source::puppetlabs] => Apt::Source[puppetlabs] => Exec[apt_update] => Class[Apt::Update] => Anchor[apt::source::puppetlabs])

Now, the problem I'm trying to solve is I constantly see these package errors:

0 upgraded, 17 newly installed, 0 to remove and 12 not upgraded.

Need to get 5504 kB/6061 kB of archives.

After this operation, 20.8 MB of additional disk space will be used.

WARNING: The following packages cannot be authenticated!

  libvirt0 libvirt-bin

E: There are problems and -y was used without --force-yes

I'm sure this has to do with getting updates to run after adding sources because if I hop on the host and do this:

/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libvirt-bin

It reproduces the error. But if I then run 'apt-get update' and rerun it, no errors.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[jcbollinger]

On Friday, May 10, 2013 11:13:03 AM UTC-5, James Kyle wrote:

This is what I get with the above, slightly adapted to take care of an edge case:

Apt::Source<| |> -> Exec['apt_update'] -> Package<| title != 'ubuntu-cloud-keyring' |>

The ubuntu-cloud-keyring is a prerequisite for adding the ubuntu cloud archive (for openstack debs). Without this exception, it leads to a cycle since Sources -> packages otherwise.

Dependency Cycle.

(Anchor[apt::source::puppetlabs] => Apt::Source[puppetlabs] => Exec[apt_update] => Class[Apt::Update] => Anchor[apt::source::puppetlabs])

That last link, Class[Apt::Update] => Anchor[apt::source::puppetlabs], looks very suspicious.  It suggests that somewhere among your manifests and modules and stages, you are declaring that class apt::update should be applied before whichever class declares Anchor[apt::source::puppetlabs] (maybe a class of the same name?).  That seems like it would be erroneous -- surely the idea is to get all the apt sources correct first, and only then run Exec[apt_update].  Right?

That's just the kind of thing I was talking about when I said [paraphrasing] that the chain was fundamentally right, but it could close a dependency cycle if other relationships were wrong.


John

[James Kyle]

That's what I'm seeing as well, but no idea where it's coming from (tips on how to run something like that down welcome).

I only have a very few modules I've written myself and like 3 nodes. I've removed all dependencies declared there except the one above. And I've grepped the modules directory for all mentions of puuppetlabs source.

Also, the Apt::Source['puppetlabs'] is a source I declare in my module and I don't see it declared in any other modules.

Wait, this is at the bottom of the modules/apt/manifests/source.pp:

  # Need anchor to provide containment for dependencies.

  anchor { "apt::source::${name}":

    require => Class['apt::update'],

  }

Could that do it? That's the Class[apt::update] -> Anchor[apt::source::puppetlabs] -> Exec[apt_update] cycle right?

John

[David Schmitt]

On 10.05.2013 18:13, James Kyle wrote:
> I'm sure this has to do with getting updates to run after adding sources
> because if I hop on the host and do this:
>
> /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install
> libvirt-bin
>
> It reproduces the error. But if I then run 'apt-get update' and rerun
> it, no errors.

Yes, apt-get update refreshes the Release files and their signatures
which are used to verify the downloaded packages.


Regards, David

[jcbollinger]

On Friday, May 10, 2013 1:13:32 PM UTC-5, James Kyle wrote:

Wait, this is at the bottom of the modules/apt/manifests/source.pp:

  # Need anchor to provide containment for dependencies.

  anchor { "apt::source::${name}":

    require => Class['apt::update'],

  }

Could that do it? That's the Class[apt::update] -> Anchor[apt::source::puppetlabs] -> Exec[apt_update] cycle right?

That could and would do it.  And it looks wrong, in that it appears to be saying that Apt sources should be applied after running "apt-get update".

Nan: github fingers you as the person responsible for (the current version of) that anchor in puppetlabs-apt.  Do I misunderstand its meaning or purpose?  Do you agree that it is incorrect?  (Else, why not?)  If it is indeed incorrect, then what would you say it should be?


John

[jcbollinger]

Do note, by the way, that the idea appears to be that the Apt module automatically handles for you what you are trying to do manually: that is, to ensure that "apt-get update" is automatically run, once, after all Apt sources are managed, so that your packages can safely express relationships directly with the appropriate Apt::Source resources.

As I said, the implementation doesn't look right to me, but it's still Monday morning, so maybe I'm not seeing straight.

John