puppetdb, foreman, one particular node has a problem

[Steve Wray]

I have about 75 nodes now and recently added 6 of them. They are all running Debian squeeze and all have exactly the same Puppet packages. One is giving an error when I try to enroll it:

Failed to import facts: PG::UndefinedColumn: ERROR:  column "environment" of relation "hosts" does not exist

LINE 1: ...d", "created_at", "disk", "domain_id", "enabled", "environme...

: INSERT INTO "hosts" ("architecture_id", "build", "certname", "comment", "compute_resource_id", "created_at", "disk", "domain_id", "enabled", "environment", "environment_id", "hostgroup_id", "image_file", "image_id", "installed_at", "ip", "last_compile", "last_freshcheck", "last_report", "location_id", "mac", "managed", "medium_id", "model_id", "name", "operatingsystem_id", "organization_id", "owner_id", "owner_type", "ptable_id", "puppet_ca_proxy_id", "puppet_proxy_id","puppet_status", "root_pass", "serial", "source_file_id", "subnet_id", "type", "updated_at", "use_image", "uuid") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41) RETURNING "id"

  Rendered text template (0.0ms)

theres nothing special about this node and other enrollments I've done today have been fine. Exactly the same environments, same packages, same everything. Whats different?

[Steve Wray]

It turned out that there was an update to the foreman package which completely broke Puppets ability to enroll new nodes.

Call me old fashioned, I've been a Debian sysadmin for over 10 years, but on a 'stable' system an apt-get upgrade is not supposed to break things. Thats one of the things Debian is famous for. Just so happens my server is running Ubuntu server 'long term support' version and I expected something similar there.

I've spent some time now ripping Foreman out. Its too unreliable and I don't think the benefits outweigh the costs. Puppet without foreman is just fine.

Bye bye, foreman, I'll hardly miss you.

[Ohad Levy]

On Wed, Oct 16, 2013 at 11:07 AM, Steve Wray <steve...@gmail.com> wrote:

It turned out that there was an update to the foreman package which completely broke Puppets ability to enroll new nodes.

Call me old fashioned, I've been a Debian sysadmin for over 10 years, but on a 'stable' system an apt-get upgrade is not supposed to break things. Thats one of the things Debian is famous for. Just so happens my server is running Ubuntu server 'long term support' version and I expected something similar there.

Would you mind sharing information about which version did you upgrade from <-> to ?

we take releases very seriously (usually 4 RC with full packages) before rolling out an upgrade, I'm sorry it didn't work for you this time.

Ohad

[Andreas Ntaflos]

On 2013-10-16 10:07, Steve Wray wrote:
> It turned out that there was an update to the foreman package which
> completely broke Puppets ability to enroll new nodes.
>
> Call me old fashioned, I've been a Debian sysadmin for over 10 years,
> but on a 'stable' system an apt-get upgrade is not supposed to break
> things. Thats one of the things Debian is famous for. Just so happens my
> server is running Ubuntu server 'long term support' version and I
> expected something similar there.
>
> I've spent some time now ripping Foreman out. Its too unreliable and I

> don't think the benefits outweigh the costs. Puppet *without* foreman is

> just fine.
>
> Bye bye, foreman, I'll hardly miss you.

You are complaining on the wrong mailing list. If you have feedback for
the Foreman developers please head over to
https://groups.google.com/forum/#!forum/foreman-users and bring the
topic up there.

Andreas

[Steve Wray]

Yes, I know. At first I didn't anticipate that it was a foreman issue. 

To be honest I've found foremans usefulness marginal at best and its performance hit on the Puppet master server quite significant, so I'm not inclined to pursue it further.

[Steve Wray]

Sure, I'm using this repository

deb http://deb.theforeman.org/ precise stable

I upgraded from 

foreman-postgresql 1.2.3+debian1

foreman 1.2.3+debian1

foreman-proxy 1.2.1+ubuntu1

foreman-installer 1.2.1-debian1

to

foreman-postgresql 1.3.0-1

foreman 1.3.0-1

foreman-proxy 1.3.0-1

foreman-installer 1.3.0-1

it looks as if the upgrade didn't make the required changes to the database, or something like that. I dropped the db and recreated it and the error about the missing column went away but still couldn't enroll new nodes. 

I urgently needed to make this work so I just ripped out foreman. I didn't have time to research the issue further or try to fix foreman.

[Greg Sutcliffe]

On 16 October 2013 11:48, Steve Wray <steve...@gmail.com> wrote:

Sure, I'm using this repository

deb http://deb.theforeman.org/ precise stable

it looks as if the upgrade didn't make the required changes to the database, or something like that. I dropped the db and recreated it and the error about the missing column went away but still couldn't enroll new nodes.

Firstly, as the deb package maintainer for Foreman, I'm sorry you had issues. Thanks also for the bug report - we've just tracked down and fixed an issue which specifically affects sqlite3 upgrades (I'm not sure if you are on sqlite3, but your symptoms fit, I think). 1.3.0-2 is now in the repo with the fix.

With regards to upgrades not breaking things, I agree. We spent some time discussing the state of the Debian Foreman repo a few weeks ago[1] and one outcome of that was that we would create a repo for each major version (eg http://deb.theforeman.org/ precise 1.2) and that 'stable' would track the latest release. This was done, and as 1.3 was released on Monday, 'stable' now points to 1.3. The 1.2 repo allows people who wish to be conservative to lock themselves to a particular release.

An unavoidable side-effect of this is that since we only had "stable" before, everyone is currently using it. I accept we could have done a better job of publicising the existence of the 1.2 repo before 1.3 came out though. I've just sent a mail to our users and announce lists to make people more aware of the new repos.

Once again, apologies for your troubles, and thanks for the feedback.

Greg

[1]https://groups.google.com/d/msg/foreman-dev/76A7LmuPzLM/O8ob4aEX3VAJ

[Steve Wray]

Your response is encouraging, thanks.

I wasn't using sqlite, I was using postgresql. I have about 100 nodes (and growing) sqlite quickly became unusable.

One of the things I tried was aptitude install with =version but this didn't work, apparently it couldn't find the old versions in the repo. I started downloading the .debs to install the old versions manually but realised it would be quicker to just uninstall foreman, we had just commissioned a bunch of servers, and more arriving all the time, so I really had to just get a move on.

Honestly, if it hadn't been a Debian-based system, say CentOS, I'd have been a LOT more cautious about the 8upgrade but I've got so used to things just not breaking in stable (last time anything like that bit me was when someone decided to apply a 'security' patch to sudo).

When I have some breathing room I'll do some more tests with the newer version of foreman.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/Dfgwz8bPYvY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Greg Sutcliffe]

On 16 October 2013 14:39, Steve Wray <steve...@gmail.com> wrote:

Your response is encouraging, thanks.

I wasn't using sqlite, I was using postgresql. I have about 100 nodes (and growing) sqlite quickly became unusable.

Ah ok. There are changes to the ENC script (which also registers new Hosts in the Foreman interface) in 1.3, but due to a variety of reasons, the ENC (/etc/puppet/node.rb) isn't packaged. So potentially, that needed an update to fix your host registration problem.

One of the things I tried was aptitude install with =version but this didn't work, apparently it couldn't find the old versions in the repo. I started downloading the .debs to install the old versions manually but realised it would be quicker to just uninstall foreman, we had just commissioned a bunch of servers, and more arriving all the time, so I really had to just get a move on.

Understandable, I've been in those firefighting situations. It's interesting that this didn't work for you though - I was using that syntax this very morning to test the sqlite3 error fixes :) 

Honestly, if it hadn't been a Debian-based system, say CentOS, I'd have been a LOT more cautious about the 8upgrade but I've got so used to things just not breaking in stable (last time anything like that bit me was when someone decided to apply a 'security' patch to sudo).

When I have some breathing room I'll do some more tests with the newer version of foreman.

Looking forward to your feedback. See you on foreman-users :)