Jira (PUP-4651) systemd service provider stars/enables in the wrong order

9 views
Skip to first unread message

Wolfram Strauss (JIRA)

unread,
May 26, 2015, 11:03:18 AM5/26/15
to puppe...@googlegroups.com
Wolfram Strauss created an issue
 
Puppet / Bug PUP-4651
systemd service provider stars/enables in the wrong order
Issue Type: Bug Bug
Affects Versions: PUP 3.7.5, PUP 3.6.2
Assignee: Kylo Ginsberg
Components: Client
Created: 2015/05/26 8:02 AM
Environment:

Oracle Linux 7.1
rsyslog-8.9.0-1.el7.x86_64 (from http://rpms.adiscon.com/v8-stable/epel-7/x86_64)

Priority: Normal Normal
Reporter: Wolfram Strauss

The systemd provider for the service resource type seems to always start a service before enabling it. This behavior does not work for the rsyslog service. Enabling the rsyslog service must happen before starting it because enabling it creates symlinks for syslog.service to rsyslog.service. Those are necessary for the syslog.socket unit to work properly which is again required by rsyslog.service. Thus starting rsyslog.service is not possible, before it has been enabled.

How to reproduce:

1. Prepare environment
$ systemctl stop rsyslog.service
$ systemctl stop syslog.socket
$ systemctl disable rsyslog.service
$ systemctl is-active rsyslog.service
unknown
$ systemctl is-enabled rsyslog.service
disabled

2. Try to activate rsyslog through puppet (will fail)
$ puppet apply --debug --execute 'service

{ "rsyslog.service": ensure => 'running', enable => true, provider => 'systemd' }

'

3. Activate rsyslog manually (how puppet should do it)
$ systemctl enable rsyslog.service
$ systemctl start rsyslog.service

The relevant log messages from step 2. are:

Debug: Executing '/bin/systemctl is-active rsyslog.service'
Debug: Executing '/bin/systemctl is-enabled rsyslog.service'
Debug: Executing '/bin/systemctl start rsyslog.service'
Error: Could not start Service[rsyslog.service]: Execution of '/bin/systemctl start rsyslog.service' returned 1: A dependency job for rsyslog.service failed. See 'journalctl -xn' for details.
Error: /Stage[main]/Main/Service[rsyslog.service]/ensure: change from stopped to running failed: Could not start Service[rsyslog.service]: Execution of '/bin/systemctl start rsyslog.service' returned 1: A dependency job for rsyslog.service failed. See 'journalctl -xn' for details.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
Atlassian logo

Wolfram Strauss (JIRA)

unread,
May 26, 2015, 11:04:21 AM5/26/15
to puppe...@googlegroups.com

Wolfram Strauss (JIRA)

unread,
May 26, 2015, 11:04:24 AM5/26/15
to puppe...@googlegroups.com

Wolfram Strauss (JIRA)

unread,
May 26, 2015, 11:05:17 AM5/26/15
to puppe...@googlegroups.com

Wolfram Strauss (JIRA)

unread,
May 26, 2015, 11:06:18 AM5/26/15
to puppe...@googlegroups.com
Wolfram Strauss updated an issue
systemd service provider starts/enables in the wrong order
Change By: Wolfram Strauss
Summary: systemd service provider  stars  starts /enables in the wrong order

Wolfram Strauss (JIRA)

unread,
May 26, 2015, 11:07:24 AM5/26/15
to puppe...@googlegroups.com
Wolfram Strauss updated an issue
The systemd provider for the service resource type seems to always start a service before enabling it. This behavior does not work for the rsyslog service. Enabling the rsyslog service must happen before starting it because enabling it creates symlinks for syslog.service to rsyslog.service. Those are necessary for the syslog.socket unit to work properly which is again required by rsyslog.service. Thus starting rsyslog.service is not possible, before it has been enabled.

How to reproduce:

1. Prepare environment
$ systemctl stop rsyslog.service
$ systemctl stop syslog.socket
$ systemctl disable rsyslog.service
$ systemctl is-active rsyslog.service
unknown
$ systemctl is-enabled rsyslog.service
disabled

2. Try to activate rsyslog through puppet (will fail)
$  puppet apply --debug --execute 'service { "rsyslog.service": ensure => 'running', enable => true, provider => 'systemd' }'

3. Activate rsyslog manually (how puppet should do it)
{{ $ systemctl enable rsyslog.service
$ systemctl start rsyslog.service
}}

The relevant log messages from step 2. are:

Debug: Executing '/bin/systemctl is-active rsyslog.service'
Debug: Executing '/bin/systemctl is-enabled rsyslog.service'
Debug: Executing '/bin/systemctl start rsyslog.service'
Error: Could not start Service[rsyslog.service]: Execution of '/bin/systemctl start rsyslog.service' returned 1: A dependency job for rsyslog.service failed. See 'journalctl -xn' for details.
Error: /Stage[main]/Main/Service[rsyslog.service]/ensure: change from stopped to running failed: Could not start Service[rsyslog.service]: Execution of '/bin/systemctl start rsyslog.service' returned 1: A dependency job for rsyslog.service failed. See 'journalctl -xn' for details.



Kylo Ginsberg (JIRA)

unread,
May 26, 2015, 11:23:16 AM5/26/15
to puppe...@googlegroups.com
Kylo Ginsberg commented on Bug PUP-4651
 
Re: systemd service provider starts/enables in the wrong order

This may be the same as PUP-4605 (which spawned off from

PUP-1253 ). In PUP-4605, we thought the issue was specific to 'masked' services, but from this report it sounds like it may be a more general problem.

Kylo Ginsberg (JIRA)

unread,
May 26, 2015, 11:23:18 AM5/26/15
to puppe...@googlegroups.com
Kylo Ginsberg updated an issue
 
Change By: Kylo Ginsberg
Scrum Team: Client Platform

Kylo Ginsberg (JIRA)

unread,
May 26, 2015, 11:23:21 AM5/26/15
to puppe...@googlegroups.com
Kylo Ginsberg assigned an issue to Unassigned
Change By: Kylo Ginsberg
Assignee: Kylo Ginsberg

Russell Maclean (JIRA)

unread,
Feb 24, 2016, 10:03:02 PM2/24/16
to puppe...@googlegroups.com
Russell Maclean commented on Bug PUP-4651
 
Re: systemd service provider starts/enables in the wrong order

This is still unresolved in 3.8.6 after several hours head scratching its clear that the provider still enables in the wrong order, Causing resource failure due to no generated .service file. Its been fixed in 4.x but left in 3.x for some reason.

==> default: Error: //pm0.*****************/Puppet: Could not start Service[puppetmaster_unicorn]: Execution of '/bin/systemctl start puppetmaster_unicorn' returned 6: Failed to start puppetmaster_unicorn.service: Unit puppetmaster_unicorn.service failed to load: No such file or directory.
==> default: Error: //pm0.*********************//Stage[main]/Rm_profiles::Puppetmaster/Service[puppetmaster_unicorn]/ensure: change from stopped to running failed: Could not start Service[puppetmaster_unicorn]: Execution of '/bin/systemctl start puppetmaster_unicorn' returned 6: Failed to start puppetmaster_unicorn.service: Unit puppetmaster_unicorn.service failed to load: No such file or directory.

Running: systemctl enable puppetmaster_unicorn generates the service file and the next puppet run succeeds.

It is entirely reproducible although sometimes it appears to enable first and does create the service file.

Its been fixed in 4.x https://tickets.puppetlabs.com/browse/PUP-4605.

This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
Atlassian logo

Jon McKenzie (JIRA)

unread,
Oct 5, 2016, 4:30:03 PM10/5/16
to puppe...@googlegroups.com
Jon McKenzie commented on Bug PUP-4651

I believe this issue is hitting us as well on Puppet 3.x (we use 3.8.7 currently) on EL7 (CentOS or RHEL). From the --debug logs on puppet agent, we can see that systemctl start <service> runs prior to any systemctl enable <service>, which fails because the unit file has not yet been symlinked into the appropriate place (this particular service is actually a SysV init-type service, so the unit file is generated on the fly when it's enabled). If we manually run systemctl enable <service> and then re-run Puppet, everything works as expected.

Just to provide some additional context for this, the main problem is that with SysVinit (e.g. from CentOS 6), the service command can start a service even if it has not yet been enabled with chkconfig. In SystemD (in CentOS 7), this is NOT true. You cannot start a SysV init-style service in SystemD prior to that service being enabled. So the ordering of the service start/enable doesnt matter in CentOS 6, but it DOES matter in CentOS 7.

This is similar to the issue author's problem, which also generates unit files on the fly when the service is enabled.

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Taras Kostyuk (JIRA)

unread,
Apr 24, 2017, 12:13:03 PM4/24/17
to puppe...@googlegroups.com
Taras Kostyuk updated an issue
 
Change By: Taras Kostyuk
Affects Version/s: PUP 4.10.0

Taras Kostyuk (JIRA)

unread,
Apr 24, 2017, 12:14:03 PM4/24/17
to puppe...@googlegroups.com
Taras Kostyuk commented on Bug PUP-4651
 
Re: systemd service provider starts/enables in the wrong order

OS: CentOS 7.3
SELinux: enforcing
Puppet version: 4.10.0

service

{ "any-service": enable => true, ensure => running, provider => systemd, }

As default selinux policy on Centos prevents enabling already started systemd service a service should be enabled before starting.

Rob Lucke (JIRA)

unread,
May 15, 2017, 4:39:04 PM5/15/17
to puppe...@googlegroups.com
Rob Lucke updated an issue
 
Change By: Rob Lucke
Labels: help_wanted service systemd  triaged

Moses Mendoza (JIRA)

unread,
Aug 14, 2017, 6:46:04 PM8/14/17
to puppe...@googlegroups.com

Branan Riley (JIRA)

unread,
May 10, 2018, 6:20:03 PM5/10/18
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Labels: help_wanted linux service systemd triaged type_and_provider
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (Jira)

unread,
Jun 9, 2021, 9:36:03 PM6/9/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-4651
 
Re: systemd service provider starts/enables in the wrong order

This was fixed awhile ago. Following the OP instructions:

Debug: Executing: '/bin/systemctl is-active -- rsyslog.service'
Debug: Executing: '/bin/systemctl is-enabled -- rsyslog.service'
Debug: Executing: '/bin/systemctl show --property=NeedDaemonReload -- rsyslog.service'
Debug: Executing: '/bin/systemctl unmask -- rsyslog.service'
Debug: Executing: '/bin/systemctl start -- rsyslog.service'
Debug: Executing: '/bin/systemctl is-enabled -- rsyslog.service'
Debug: Executing: '/bin/systemctl unmask -- rsyslog.service'
Debug: Executing: '/bin/systemctl enable -- rsyslog.service'
Notice: /Stage[main]/Main/Service[rsyslog.service]/ensure: ensure changed 'stopped' to 'running'
...
# systemctl is-active rsyslog.service
active

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages