Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#832713: systemd: after "systemd (231-1) unstable" update systemd-jurnald.service fails to start

89 views
Skip to first unread message

Michael Biebl

unread,
Jul 28, 2016, 7:10:02 AM7/28/16
to
control: tags -1 + moreinfo
Am 28.07.2016 um 12:08 schrieb Rick Thomas:
> Main PID: 477 (code=exited, status=228/SECCOMP)
...
> Kernel: Linux 4.6.0-1-marvell

That looks like you are using a custom kernel. Is the problem
reproducible with the default Debian Linux kernel?


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

signature.asc

Rick Thomas

unread,
Jul 28, 2016, 4:10:03 PM7/28/16
to
No. This is a stock kernel. It’s available from both testing and unstable repos. The machine itself is a SheevaPlug. Nothing custom about it.

What makes you think it’s custom?

Rick

Michael Biebl

unread,
Jul 28, 2016, 4:30:03 PM7/28/16
to
Am 28.07.2016 um 22:01 schrieb Rick Thomas:
> No. This is a stock kernel. It’s available from both testing and unstable repos. The machine itself is a SheevaPlug. Nothing custom about it.
>
> What makes you think it’s custom?

This was more of a question then a statement.
I wanted to rule out, that the kernel was missing any relevant features.

I assume the previous version worked properly and the addition of

SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount
@obsolete @raw-io

is causing the problem? If you comment out that line from
systemd-journald.service, does it start properly?

SystemCallFilter uses libseccomp, maybe libseccomp on armel is not fully
functional and we need to reassign this.
signature.asc

Michael Biebl

unread,
Jul 28, 2016, 5:10:03 PM7/28/16
to
Am 28.07.2016 um 22:50 schrieb Rick Thomas:
> In the interest of having a working system, I reverted that machine to systemd version 230-7. Unsurprisingly, the problem went away.
>
> I’ll try re-installing 231-1 and commenting that line. I’ll probably have a chance tonight. I’ll report when I have something.
>
> It may be worth noticing that other things failed as well when 231-1 was in. I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note are “Failed to start Raise network interfaces” and “Failed to start Login Service.”
>
> Are there other places where I should remove a “SystemCallFilter” ?
>

Various units were locked down like e.g. in
https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736

If the SystemCallFilter= is what causes journald to fail, it's likely it
also affects those other services.

Michael
signature.asc

Rick Thomas

unread,
Jul 28, 2016, 8:30:04 PM7/28/16
to
I upgraded one of my test systems (an armhf Cubox-i4Pro) to Sid.
After rebooting, I saw the same symptoms as on the SheevaPlug.

I commented the line in question in
/lib/systemd/system/systemd-logind.service
and
/lib/systemd/system/systemd-journald.service

There was no corresponding line in
/lib/systemd/system/networking.service
so I did nothing there.

I then did ‘update-initramfs -u’.

Then I rebooted. Symptoms still persist, so that doesn’t seem to be it.

I still have
root@cube:~# systemctl status systemd-journald.service
* systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Thu 2016-07-28 17:05:29 PDT; 8min ago
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 608 (code=exited, status=228/SECCOMP)

Any other thoughts?

Rick

Rick Thomas

unread,
Jul 29, 2016, 5:40:03 PM7/29/16
to
Hmmm… Curiouser and curiouser!

I upgraded a VM (amd64) to latest Sid (with systemd version 231-1). The problem is *not* present there.

The problem may be specific to arm hardware? I’ll try it on a PowerPC G4 (Apple Mac PPC) machine later today.

Rick

Michael Biebl

unread,
Jul 29, 2016, 6:10:02 PM7/29/16
to
Am 29.07.2016 um 23:29 schrieb Rick Thomas:
> Hmmm… Curiouser and curiouser!
>
> I upgraded a VM (amd64) to latest Sid (with systemd version 231-1). The problem is *not* present there.
>
> The problem may be specific to arm hardware? I’ll try it on a PowerPC G4 (Apple Mac PPC) machine later today.

I think this might be arch specific, yes.
See my reply earlier:

> SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount
> @obsolete @raw-io
>
> is causing the problem? If you comment out that line from
> systemd-journald.service, does it start properly?
>
> SystemCallFilter uses libseccomp, maybe libseccomp on armel is not fully
> functional and we need to reassign this.




signature.asc

Rick Thomas

unread,
Jul 30, 2016, 4:50:03 AM7/30/16
to
As I replied to that (did you see it? There may have been some problems with my email about that time),
commenting out the “SystemCallFilter” did not make the problem go away on either of the ARM systems.

Another interesting datapoint: I upgraded my PowerMac G4 powerpc machine to latest Sid (including the systemd version in question) and it does *not* have the problem. This is the same result I saw on the amd64 (virtual) machine.

So it seems to be arm specific…

Any thoughts?

Rick Thomas

unread,
Jul 31, 2016, 5:40:02 PM7/31/16
to
Thanks for you help so far, Michael!

Can I ask one last favor on this?

It seems that the bug (whatever it is) depends on things like kernel version and machine architecture.

So, can you suggest someone who can take this further?

Thanks!
Rick

Felipe Sateler

unread,
Aug 1, 2016, 5:50:03 PM8/1/16
to
On 28 July 2016 at 17:04, Michael Biebl <bi...@debian.org> wrote:
> Am 28.07.2016 um 22:50 schrieb Rick Thomas:
>> In the interest of having a working system, I reverted that machine to systemd version 230-7. Unsurprisingly, the problem went away.
>>
>> I’ll try re-installing 231-1 and commenting that line. I’ll probably have a chance tonight. I’ll report when I have something.
>>
>> It may be worth noticing that other things failed as well when 231-1 was in. I’m attaching a ‘grep -i fail -C20’ of the screen log. Of particular note are “Failed to start Raise network interfaces” and “Failed to start Login Service.”
>>
>> Are there other places where I should remove a “SystemCallFilter” ?
>>
>
> Various units were locked down like e.g. in
> https://github.com/systemd/systemd/commit/4e069746fe0de1f60bd1b75c113b0f40ffe86736
>
> If the SystemCallFilter= is what causes journald to fail, it's likely it
> also affects those other services.

Turns out seccomp is disabled in the arm* kernels:

% grep SECCOMP boot/config-4.6.0-1-marvell
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
# CONFIG_SECCOMP is not set

% grep SECCOMP boot/config-4.6.0-1-armmp
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
# CONFIG_SECCOMP is not set

So I think the kernel should enable SECCOMP.

However, I think systemd should also simply (warn and) ignore seccomp
calls if seccomp is not available in the current kernel.

--

Saludos,
Felipe Sateler

Rick Thomas

unread,
Aug 1, 2016, 6:40:03 PM8/1/16
to
Thanks, Filipe!

What do we have to do at this point to test this and then translate it into a patch?

Michael, do you have any suggestions?

Thanks!
Rick

Michael Biebl

unread,
Aug 3, 2016, 2:20:02 AM8/3/16
to
Am 01.08.2016 um 23:40 schrieb Felipe Sateler:
> So I think the kernel should enable SECCOMP.

I agree, unless SECCOMP on arm has some unwanted side-effects.
Felipe, can you file a bug report against the linux package accordingly?

> However, I think systemd should also simply (warn and) ignore seccomp
> calls if seccomp is not available in the current kernel.

I agree as well, Systemd should log an error and continue.
We need to file a bug report upstream for that? Any takers?

Assuming upstream actually decides to make SECCOMP (once enabled) a hard
dependency, we need to find a different solution, like a check in the
maintainer scripts. That said, we should first try to get that adressed
upstream.

Regards,
signature.asc

Rick Thomas

unread,
Aug 3, 2016, 2:40:02 AM8/3/16
to

On Aug 2, 2016, at 11:15 PM, Michael Biebl <bi...@debian.org> wrote:

> Am 01.08.2016 um 23:40 schrieb Felipe Sateler:
>> So I think the kernel should enable SECCOMP.
>
> I agree, unless SECCOMP on arm has some unwanted side-effects.
> Felipe, can you file a bug report against the linux package accordingly?

It may be that there are size considerations that preclude turning on such features for kernels on certain arm devices (armel comes to mind with SheevaPlug and OpenRD as examples).

I’m not sure of that — and definitely unsure of the details, but I guess the linux package maintainers will know for sure.

Rick

Felipe Sateler

unread,
Aug 3, 2016, 11:40:03 AM8/3/16
to
Control: forwarded -1 https://github.com/systemd/systemd/issues/3882
Control: severity -1 serious

On 3 August 2016 at 02:15, Michael Biebl <bi...@debian.org> wrote:
> Am 01.08.2016 um 23:40 schrieb Felipe Sateler:
>> So I think the kernel should enable SECCOMP.
>
> I agree, unless SECCOMP on arm has some unwanted side-effects.
> Felipe, can you file a bug report against the linux package accordingly?

Already reported

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833183

>
>> However, I think systemd should also simply (warn and) ignore seccomp
>> calls if seccomp is not available in the current kernel.
>
> I agree as well, Systemd should log an error and continue.
> We need to file a bug report upstream for that? Any takers?

I have forwarded this report and marked this bug as such.

> Assuming upstream actually decides to make SECCOMP (once enabled) a hard
> dependency, we need to find a different solution, like a check in the
> maintainer scripts. That said, we should first try to get that adressed
> upstream.

Yes, I'd like upstream's opinion on this first. However, I think this
bug should be bumped to RC to prevent migration to testing, as this
makes arm systems basically unusable. I have marked the bug as serios.
Feel free to downgrade if you disagree.


--

Saludos,
Felipe Sateler

Felipe Sateler

unread,
Aug 3, 2016, 7:10:02 PM8/3/16
to
On 3 August 2016 at 16:44, Rick Thomas <rbth...@pobox.com> wrote:
>
> On Aug 3, 2016, at 9:06 AM, Felipe Sateler <fsat...@debian.org> wrote:
>
>> On 1 August 2016 at 18:32, Rick Thomas <rbth...@pobox.com> wrote:
>>>
>>> Thanks, Filipe!
>>>
>>> What do we have to do at this point to test this and then translate it into a patch?
>>
>> OK, so I have a proof-of-concept patch. Rick, could you test it in your machine?
>
> I’m afraid I don’t have the facilities or the expertise to turn this into a loadable kernel .deb .
>
> If someone can provide an installable kernel .deb file for a Cuboxi4-Pro machine, I’ll he happy to test it.

This is a patch for systemd, not the kernel. I prepared a source
package for easier testing. Do the following:

apt-get install dpkg-dev devscripts
apt-get build-dep systemd
dget https://people.debian.org/~fsateler/systemd_231-2.dsc
cd systemd-231
dpkg-buildpackage

That should leave you with several .debs (after a long time). Install
libsystemd0 and systemd from those debs.

--

Saludos,
Felipe Sateler

Pete Batard

unread,
Aug 12, 2016, 12:30:04 PM8/12/16
to
Okay. I have now gone through a dpkg -i install of all the (non dbgsym)
.deb I see on your server, and also issued a reboot for good measure,
but I still see the same problem with journald being failed, along with
dependent services

---------------------------------------------------------------------
root@pi ~ # systemctl start systemd-journald.service
Job for systemd-journald.service failed because the control process
exited with error code.
See "systemctl status systemd-journald.service" and "journalctl -xe" for
details.
root@pi ~ # systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service;
static; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Fri 2016-08-12
17:17:12 IST; 5s ago
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Process: 1560 ExecStart=/lib/systemd/systemd-journald (code=exited,
status=228/SECCOMP)
Main PID: 1560 (code=exited, status=228/SECCOMP)
root@pi ~ # systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-journald.service loaded failed failed Journal Service
● systemd-logind.service loaded failed failed Login Service
● systemd-networkd.service loaded failed failed Network Service
● systemd-resolved.service loaded failed failed Network Name
Resolution
● systemd-journald-dev-log.socket loaded failed failed Journal Socket
(/dev/log)
● systemd-journald.socket loaded failed failed Journal Socket
● systemd-networkd.socket loaded failed failed Network Service
Netlink Socket

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.

7 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
---------------------------------------------------------------------

One thing I still notice on the download server is that there exists a
'systemd-journal-remote-dbgsym_231-2_armhf.deb' but no
'systemd-journal-remote_231-2_armhf.deb'. There's also no 'systemd-sysv'
despite being listed in the .dsc. Maybe those .deb's are still missing?

Regards,

/Pete

Felipe Sateler

unread,
Aug 28, 2016, 10:40:02 AM8/28/16
to
On 27 August 2016 at 20:20, Pete Batard <pe...@akeo.ie> wrote:
> ...and 231-4 broke the whole thing down again. :(

Could you try 231-5 from unstable?


--

Saludos,
Felipe Sateler

Pete Batard

unread,
Aug 28, 2016, 4:50:02 PM8/28/16
to
On 2016.08.28 15:37, Felipe Sateler wrote:
> Could you try 231-5 from unstable?

Oops, I mean 231-5 broke it. I'm seeing the issue back in 231-5 and not
231-4 as I mentioned earlier.

231-4 seemed to be okay, and it's only when I upgraded to 231-5 that the
problem reappeared.

Regards,

/Pete

Felipe Sateler

unread,
Aug 29, 2016, 11:40:03 AM8/29/16
to
On 29 August 2016 at 12:10, Pete Batard <pe...@akeo.ie> wrote:
> Please find my boot log. Note that I've tried commenting out
> 'MemoryDenyWriteExecute=yes' in the various .service config files, but it
> didn't help.
>
> If you need anything else, please let me know.

This log is not verbose. Could you boot with kernel argument
systemd.log_level=debug ?

Also, on the arm machine could you attach the contents of /proc/self/status ?

--
Saludos,
Felipe Sateler

Pete Batard

unread,
Aug 30, 2016, 12:40:02 PM8/30/16
to
Thanks Felipe.

I guess with the new detection process, the resurgence of the issue is
starting to make sense now.

For the record, I am using one of the latest official Raspberry Pi
kernels from https://github.com/raspberrypi/firmware (which I get
indirectly through the https://github.com/Hexxeh/rpi-firmware repo and
its rpi-update script at https://github.com/Hexxeh/rpi-update).

Unfortunately, these kernels do not provide a config.gz in /proc, so I
can't tell you precisely what compilation options were used in case it
matters. But I'll be happy to run more tests on my machine as needed.

Regards,

/Pete

Felipe Sateler

unread,
Aug 30, 2016, 12:40:03 PM8/30/16
to
On 30 August 2016 at 13:26, Pete Batard <pe...@akeo.ie> wrote:
> Thanks Felipe.
>
> I guess with the new detection process, the resurgence of the issue is
> starting to make sense now.
>
> For the record, I am using one of the latest official Raspberry Pi kernels
> from https://github.com/raspberrypi/firmware (which I get indirectly through
> the https://github.com/Hexxeh/rpi-firmware repo and its rpi-update script at
> https://github.com/Hexxeh/rpi-update).

Ah, I thought you were using the debian kernels.

>
> Unfortunately, these kernels do not provide a config.gz in /proc, so I can't
> tell you precisely what compilation options were used in case it matters.
> But I'll be happy to run more tests on my machine as needed.

I managed to get the config from a jessie rpi by loading the 'configs'
module (sudo modprobe configs). After that the config is found on
/proc/config.gz


--

Saludos,
Felipe Sateler

Pete Batard

unread,
Aug 30, 2016, 7:30:03 PM8/30/16
to
Okay.

I have logged issue #651
(https://github.com/raspberrypi/firmware/issues/651) with the rpi team
so that they try to sort their SECCOMP configuration in future kernels.

Regards,

/Pete
0 new messages