/var/log excessive filesystem usage

[Tai...@gmx.com]

my /var/log was using a massive 5GB on an install that is only 6 months old

Why is this? did someone jack up the log files? I simply deleted it as I
am quite busy but I imagine this is responsible for most of my SSD wear.

[Tai...@gmx.com]

Update: deleting the contents of /var/log, /tmp and /var/tmp caused my
system to be unbootable which is silly as these are not meant to be
permanent locations

I received errors about qmemmman not being able to write a file, to
which I had to revert the changes and re-create it's directory to render
the system bootable again.

[Alex]

That's very strange - not the fact that qmemman does not work if you
remove its log directory, but the size on disk.

I've had this R3.2 installation since october 2016, and my /tmp has 4KB
worth of data, /var/tmp 20KB and the biggest is /var/log with 1.8GB.

But inside /var/log the biggest directory is journald/, that takes 99%
of the space, while qubes/ takes only 3.2 MB - the second biggest
directory being xen/ at 8.3MB.

To check directory size I used "du", with a line like this:
/var/log# du --max=1 -h

Please check settings in /etc/systemd/journald.conf to make sure
journald only logs what you need (and, in my case, does not discard what
it thinks I don't need).

--
Alex

[David Hobach]

Yes the default settings in 3.2 were quite ridiculous - also made me get
a few GB over time, journalctl wouldn't even load at some point...
Didn't check that in 4.0 yet...

[haaber]

Could you me more specific which entry in the default settings are
responsible for this space-consuming logging ? Thanks, Bernhard

[Alex]

It's not much a question of "responsible for space-consuming logging"
but about "make sure journald logs what you need" so you don't miss
important things and are not submerged by useless logs.

By default, in Qubes, the default settings for journald are kept; this
means that:
* log files are persisted across reboots only if /var/log/journal exists
* log files are compressed
* each log file keeps ~1 month of logs
* total /var/log/journal size cannot be more than 10% (of filesystem
total space) or more than 4G

I'd recommend to explicitly set:
* Storage=persistent (this will take care of creating /var/log/journal
and setting the correct permissions)
* MaxFileSec=1week
* SystemMaxUse=1GB
* RuntimeMaxUse=1GB
* MaxRetentionSec=1month

To ensure that:
* even if /var/log/journal does not exist, it gets created and files
are persisted: this can be useful in debugging Qubes failed boots
* should a lot of logs occur, no more than 1GB of data is kept on disk
(for privacy purposes)
* normally no more than 1 month of data is kept (for privacy purposes)

More info on
https://www.freedesktop.org/software/systemd/man/journald.conf.html

--
Alex

[Tai...@gmx.com]

Thanks, I don't normally use systemd on my other computers

Another reason to hate systemd.
Systemd linux takes 1min+ to boot vs 15 seconds on devuan's plane jane SysVinit (redhat only created systemd to run a hostile takeover of the linux community, hell they probably bribed people to make it happen why else would so many distros start using such a crappy thing right at the same time)

[Chris Laprise]

On 11/10/2017 05:57 PM, Tai...@gmx.com wrote:
>
> On 09/26/2017 03:56 AM, Alex wrote:
>

>> On 09/26/2017 09:44 AM,Tai...@gmx.com wrote:
>>> Update: deleting the contents of /var/log, /tmp and /var/tmp caused my
>>> system to be unbootable which is silly as these are not meant to be
>>> permanent locations
>>>
>>> I received errors about qmemmman not being able to write a file, to
>>> which I had to revert the changes and re-create it's directory to render
>>> the system bootable again.
>>>
>> That's very strange - not the fact that qmemman does not work if you
>> remove its log directory, but the size on disk.
>>
>> I've had this R3.2 installation since october 2016, and my /tmp has 4KB
>> worth of data, /var/tmp 20KB and the biggest is /var/log with 1.8GB.
>>
>> But inside /var/log the biggest directory is journald/, that takes 99%
>> of the space, while qubes/ takes only 3.2 MB - the second biggest
>> directory being xen/ at 8.3MB.
>>
>> To check directory size I used "du", with a line like this:
>> /var/log# du --max=1 -h
>>
>> Please check settings in /etc/systemd/journald.conf to make sure
>> journald only logs what you need (and, in my case, does not discard what
>> it thinks I don't need).
> Thanks, I don't normally use systemd on my other computers
>

You can also run 'journalctl' to prune the logs. That's what I've done
since Fedora doesn't come with a sensible default setting.

> Another reason to hate systemd.
> Systemd linux takes 1min+ to boot vs 15 seconds on devuan's plane jane
> SysVinit (redhat only created systemd to run a hostile takeover of the
> linux community

... you must be new to Linux. :) Redhat has long exercised undue
influence over Linux development. When "desktop Linux" was a trend over
a decade ago, they threw their weight around in that arena too.
Unfortunately the community is stupid enough to let an unabashed
server-only company determine the direction of desktop development.

OTOH, going back to init instead of fostering one of the alternatives
has exposed a regressive streak in the community. Sysvinit sucked eggs
for use cases involving power management (sleep/wake/etc), peripheral
hotplug, anything where the system had to enter different global states.
It probably still does suck and somehow I can't believe that devuan is
thoroughly testing for PC use cases (doubt they even recognize 'use
case' as a development concept); On my last survey, no one except
Canonical does this.

The tragedy here is that Ubuntu tried to address the issue reasonably in
their usual fashion (follow Apple's lead) and Redhat and their neckbeard
camp said "No". Over the years: No apt, No Mir, No upstart, No
addressing desktop security bug reports, No repo signing on Fedora
(can't compete with RHEL on update security!), No certification of
PCs... They'll wait 7-10 years until their boys get around to doing it
over. Redhat are the Knights Who Say NIH (Not Invented Here). Now
Canonical is taking their business and they are flailing about.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Tai...@gmx.com]

On 11/11/2017 06:58 AM, Chris Laprise wrote:

>> Systemd linux takes 1min+ to boot vs 15 seconds on devuan's plane
>> jane SysVinit (redhat only created systemd to run a hostile takeover
>> of the linux community
> ... you must be new to Linux. :) Redhat has long exercised undue
> influence over Linux development. When "desktop Linux" was a trend
> over a decade ago, they threw their weight around in that arena too.
> Unfortunately the community is stupid enough to let an unabashed
> server-only company determine the direction of desktop development.

I have been using it for a decade :<

>
> OTOH, going back to init instead of fostering one of the alternatives
> has exposed a regressive streak in the community. Sysvinit sucked eggs
> for use cases involving power management (sleep/wake/etc), peripheral
> hotplug, anything where the system had to enter different global
> states. It probably still does suck and somehow I can't believe that
> devuan is thoroughly testing for PC use cases (doubt they even
> recognize 'use case' as a development concept); On my last survey, no
> one except Canonical does this.

I dunno SVI works fine for me.
You can install a few other init systems in devuan too, it is your choice.
Gentoo is also a great option if you have a fast CPU and time on your
hands then you can have whatever init system you desire. (I wish it came
with a nice management GUI so it wasn't such a time sink)

> The tragedy here is that Ubuntu tried to address the issue reasonably
> in their usual fashion (follow Apple's lead) and Redhat and their
> neckbeard camp said "No". Over the years: No apt, No Mir, No upstart,
> No addressing desktop security bug reports, No repo signing on Fedora
> (can't compete with RHEL on update security!), No certification of
> PCs... They'll wait 7-10 years until their boys get around to doing it
> over. Redhat are the Knights Who Say NIH (Not Invented Here). Now
> Canonical is taking their business and they are flailing about.

Wait fedora doesn't sign their stuff? Damn that's terrible! So when you
dnf something there isn't any gpg verification of the files?

Closed - wontfix User: lpottering

[Chris Laprise]

On 11/11/2017 03:38 PM, Tai...@gmx.com wrote:
> Wait fedora doesn't sign their stuff? Damn that's terrible! So when
> you dnf something there isn't any gpg verification of the files?
>

Fedora signs packages individually. But nearly all (except Fedora) sign
the overall repository manifest as well. Lack of repo signatures allows
an attacker to selectively prevent individual updates from being
installed. On a typical non-Fedora distro, the attacker can only hold
back the entire repository (and they can't change the timestamp to make
it appear current).