move /var/log to a RAMDISK

[The Natural Philosopher]

Embedded Raspberry Pi Zero W. Minimal logs generated but I only use
logs for debugging and why wear the SD card?

My question is simple. If I make a 50MByte or so RAMDISK and mount it on
/var/log, will the logging daemons recreate all the files and
directories they need at boot time?


I am barely using any memory on a headless server - there is room for a
ramdisk

# free -m
total used free shared buff/cache
available
Mem: 429 61 131 3 237
310
Swap: 99 0 99

--
The theory of Communism may be summed up in one sentence: Abolish all
private property.

Karl Marx

[David W. Hodgins]

On Sat, 29 Jul 2023 13:34:45 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> Embedded Raspberry Pi Zero W. Minimal logs generated but I only use
> logs for debugging and why wear the SD card?

> My question is simple. If I make a 50MByte or so RAMDISK and mount it on
> /var/log, will the logging daemons recreate all the files and
> directories they need at boot time?

No. Files in /var are expected to survive reboot, so that's how the packages
are configured.

They are created during package installation. For example,
$ rpm -q -l postgresql13-server|grep /var/log
/var/log/postgres

Looking at the spec file for the rpm package, it has ...
%attr(700,postgres,postgres) %dir /var/log/postgres
which is used by rpm to set the ownership and permissions.

Regards, Dave Hodgins

[Robert Heller]

This is in fact the default behaviour for Armbian on my Banana Pi M64.

If you want, I can send you the scripts used. They create the RAMDISK, copy
the existing files on boot, then copy back during shutdown (so the log files
are not lost).

At Sat, 29 Jul 2023 18:34:45 +0100 The Natural Philosopher <t...@invalid.invalid> wrote:

>
> Embedded Raspberry Pi Zero W. Minimal logs generated but I only use
> logs for debugging and why wear the SD card?
>
> My question is simple. If I make a 50MByte or so RAMDISK and mount it on
> /var/log, will the logging daemons recreate all the files and
> directories they need at boot time?
>
>
> I am barely using any memory on a headless server - there is room for a
> ramdisk
>
> # free -m
> total used free shared buff/cache
> available
> Mem: 429 61 131 3 237
> 310
> Swap: 99 0 99
>

--

Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
hel...@deepsoft.com -- Webhosting Services

[The Natural Philosopher]

I feared that might be the case.
--
"Nature does not give up the winter because people dislike the cold."

― Confucius

[The Natural Philosopher]

On 29/07/2023 19:45, Robert Heller wrote:
> This is in fact the default behaviour for Armbian on my Banana Pi M64.
>
> If you want, I can send you the scripts used. They create the RAMDISK, copy
> the existing files on boot, then copy back during shutdown (so the log files
> are not lost).
>

I can write that ok myself.
However there would be no *controlled shutdown*. We are talking power cuts.
The actual application does not write to SD disk at all, except for
persistent configuration. Runtime files use a ram disk.

So the aim is to limit random writes that might happend when power is
pulled from under.
In use the only likely things would be systemd which logs everything
all the time, and apache.
Exim might send a mail a month.

Looking at the log files there is a huge amount of installation stuff
that simply isn't needed, and stuff that appears at boot time that
doesn't need to be preserved on power loss.

So I guess this is really a more general discussion about how to reduce
logging to a minimum, and reduce writing to persistent storage to a
minimum as well

I can set apache to log to a volatile storage area OK I think.
And probably Exim too - mount ramdisks on /var/log/apache2 and
/var/log/exim4

The problem is that the whole*nix logging system is based on how a good
minicomputer sysadmin would like things, not how an embedded linux
running off an SD card needs to be :-)

[Robert Heller]

At Sat, 29 Jul 2023 20:42:12 +0100 The Natural Philosopher <t...@invalid.invalid> wrote:

>
> On 29/07/2023 19:45, Robert Heller wrote:
> > This is in fact the default behaviour for Armbian on my Banana Pi M64.
> >
> > If you want, I can send you the scripts used. They create the RAMDISK, copy
> > the existing files on boot, then copy back during shutdown (so the log files
> > are not lost).
> >
> I can write that ok myself.
> However there would be no *controlled shutdown*. We are talking power cuts.
> The actual application does not write to SD disk at all, except for
> persistent configuration. Runtime files use a ram disk.
>
> So the aim is to limit random writes that might happend when power is
> pulled from under.
> In use the only likely things would be systemd which logs everything
> all the time, and apache.
> Exim might send a mail a month.
>
> Looking at the log files there is a huge amount of installation stuff
> that simply isn't needed, and stuff that appears at boot time that
> doesn't need to be preserved on power loss.
>
> So I guess this is really a more general discussion about how to reduce
> logging to a minimum, and reduce writing to persistent storage to a
> minimum as well

Embeded servers (eg OpenWRT) mount /var from a ramdisk at boot, and likely
initially populate it that file system from some "static" initial state (with
all of the basic directory tree elements) at boot time. And then on
reboot/power failure, all of that is just lost (discarded). (Most
OpenWRT-based servers have the option of remote logging, if one wants
persistent logs.)

DRBL (Diskless Remote Boot Linux), also does this. It mounts the root (/) and
/usr file systems read-only from the server and create RAMDISK file systems
for /etc /var and /root (~root) and populates these FSs from pre-generated
tarballs and rsyncs host specific content from a template tree at boot time
and then discards them on shutdown.

>
> I can set apache to log to a volatile storage area OK I think.
> And probably Exim too - mount ramdisks on /var/log/apache2 and
> /var/log/exim4
>
> The problem is that the whole*nix logging system is based on how a good
> minicomputer sysadmin would like things, not how an embedded linux
> running off an SD card needs to be :-)
>
>

--

[David W. Hodgins]

On Sat, 29 Jul 2023 14:54:25 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:

> On 29/07/2023 19:42, David W. Hodgins wrote:
>> On Sat, 29 Jul 2023 13:34:45 -0400, The Natural Philosopher
>> <t...@invalid.invalid> wrote:
>>> Embedded Raspberry Pi Zero W. Minimal logs generated but I only use
>>> logs for debugging and why wear the SD card?
>>
>>> My question is simple. If I make a 50MByte or so RAMDISK and mount it on
>>> /var/log, will the logging daemons recreate all the files and
>>> directories they need at boot time?
>>
>> No. Files in /var are expected to survive reboot, so that's how the
>> packages
>> are configured.
>>
>> They are created during package installation. For example,
>> $ rpm -q -l postgresql13-server|grep /var/log
>> /var/log/postgres
>>
>> Looking at the spec file for the rpm package, it has ...
>> %attr(700,postgres,postgres) %dir /var/log/postgres
>> which is used by rpm to set the ownership and permissions.

> I feared that might be the case.

It can be done using scripts that hook into systemd, but is not easy, and may
require changes to scripts with any package install that uses /var/log, and
also risks losing parts of the logs in the case of an unclean shutdown, which
is typically when you are going to want the logs.

Regards, Dave Hodgins

[David W. Hodgins]

On Sat, 29 Jul 2023 15:42:12 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> The problem is that the whole*nix logging system is based on how a good
> minicomputer sysadmin would like things, not how an embedded linux
> running off an SD card needs to be :-)

That's not a nix thing. Live iso images work fine with no permanent storage,
though most have an option to have persistent storage too.

While running a general purpose linux distribution installed on an sd card does
work, they are not tailored for use on embedded systems.

An embedded system should be designed to work like a live iso, but with essential
data stored on persistent storage, and with only the minimum needed applications,
and system tools.

Regards, Dave Hodgins

[Computer Nerd Kev]

In comp.os.linux.misc Robert Heller <hel...@deepsoft.com> wrote:
> At Sat, 29 Jul 2023 20:42:12 +0100 The Natural Philosopher <t...@invalid.invalid> wrote:
>> Looking at the log files there is a huge amount of installation stuff
>> that simply isn't needed, and stuff that appears at boot time that
>> doesn't need to be preserved on power loss.
>>
>> So I guess this is really a more general discussion about how to reduce
>> logging to a minimum, and reduce writing to persistent storage to a
>> minimum as well
>
> Embeded servers (eg OpenWRT) mount /var from a ramdisk at boot, and likely
> initially populate it that file system from some "static" initial state (with
> all of the basic directory tree elements) at boot time. And then on
> reboot/power failure, all of that is just lost (discarded). (Most
> OpenWRT-based servers have the option of remote logging, if one wants
> persistent logs.)
>
> DRBL (Diskless Remote Boot Linux), also does this. It mounts the root (/) and
> /usr file systems read-only from the server and create RAMDISK file systems
> for /etc /var and /root (~root) and populates these FSs from pre-generated
> tarballs and rsyncs host specific content from a template tree at boot time
> and then discards them on shutdown.

Yes a few distros are designed similar to that. The server computer
on my home LAN (which also runs off an SD card) is set to only save
/var/lib/hwclock between reboots. If you don't run a syslog daemon
then logs from programs that send them to syslog don't go anywhere.
Busybox comes with a syslogd implementation which can be started to
show all log info in the terminal with "busybox syslogd -n -O -".
So run that during development for debugging, then when it's all
working, go syslog-free.

On a system that isn't designed for temporary /var, if you just
recreate empty directories in /var (those listed by
"find /var -type d") after reboot, I expect all the programs that
are already installed will create fresh files inside those
directories by themselves.

--
__ __
#_ < |\| |< _#

[David W. Hodgins]

On Sat, 29 Jul 2023 20:22:13 -0400, Computer Nerd Kev <n...@telling.you.invalid> w

> On a system that isn't designed for temporary /var, if you just
> recreate empty directories in /var (those listed by
> "find /var -type d") after reboot, I expect all the programs that
> are already installed will create fresh files inside those
> directories by themselves.

The owner, group, and permissions must be set correctly too.

Regards, Dave Hodgins

[Ahem A Rivet's Shot]

On Sat, 29 Jul 2023 20:35:49 +0000
Robert Heller <hel...@deepsoft.com> wrote:

> Embeded servers (eg OpenWRT) mount /var from a ramdisk at boot, and
> likely initially populate it that file system from some "static" initial
> state (with all of the basic directory tree elements) at boot time.

Another common technique is to overlay a ramdisk on a read-only
file system so that the ram disk only holds the changes.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/
Host: Beautiful Theory meet Inconvenient Fact
Obit: Beautiful Theory died today of factual inconsistency

[The Natural Philosopher]

On 29/07/2023 21:35, Robert Heller wrote:
> Embeded servers (eg OpenWRT) mount /var from a ramdisk at boot, and likely
> initially populate it that file system from some "static" initial state (with
> all of the basic directory tree elements) at boot time. And then on
> reboot/power failure, all of that is just lost (discarded).

That is an thought. Dont rely on fstab, but run a script.
Really its only /var/log I care about. Rest of var can do as it pleases
- its so little used ro write to.


--
Climate Change: Socialism wearing a lab coat.

[The Natural Philosopher]

Well you would just copy /var/log to somewhere on persistent, delete all
the files and then do a cp - R on reboot. Except that the first thing
that happens on boot is the daemons fill up /var/log with boot messages :-(

I note since its Sunday, that the newest log files have been created by
the logrotate process and are all empty.

Ah well. The smallest SD card I could easily get was 16GB, and RasPios
and friends have only used 1.5GB.

It will probably take a long time to wear out.

--
“The fundamental cause of the trouble in the modern world today is that
the stupid are cocksure while the intelligent are full of doubt."

- Bertrand Russell

[Computer Nerd Kev]

In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
> On 30/07/2023 04:27, David W. Hodgins wrote:
>> On Sat, 29 Jul 2023 20:22:13 -0400, Computer Nerd Kev
>> <n...@telling.you.invalid> wrote:
>>> On a system that isn't designed for temporary /var, if you just
>>> recreate empty directories in /var (those listed by
>>> "find /var -type d") after reboot, I expect all the programs that
>>> are already installed will create fresh files inside those
>>> directories by themselves.
>>
>> The owner, group, and permissions must be set correctly too.
>

> Well you would just copy /var/log to somewhere on persistent, delete all
> the files and then do a cp - R on reboot. Except that the first thing
> that happens on boot is the daemons fill up /var/log with boot messages :-(

This is of course not really the first thing that happens on boot,
so if you can get in before the init system starts daemons then
that solves your problem. I wouldn't like to try figuring out how
to do that with something like Systemd, but then I avoid that sort
of thing anyway. Busybox has a simple init system, good for
embedded systems.

> Ah well. The smallest SD card I could easily get was 16GB, and RasPios
> and friends have only used 1.5GB.
>
> It will probably take a long time to wear out.

I was curious about this once too - does a 16GB SD card where only
1.5GB is used last longer than say a 2GB card? I couldn't find a
conclusive answer online, except that an SD card's unlikely to be
near as smart about wear levelling as an SSD is.

[Theo]

The directory structure may be created by the package manager on install,
but the files are created on the fly. In other words I might have
/var/log/foo/error.log - foo is a directory that needs to exist, but if I
delete error.log it'll be created next time there's something to go in
there. This is commonly used by 'logrotate', which moves old files and
compresses them, allowing new files to be created for future log entries.

So all you really need to do is reproduce the directory structure:

find /var/log -type d -exec mkdir -p "/tmp/{}" \;

which reproduces /var/log/foo/bar/ to /tmp/var/log/foo/bar/

Theo

[The Natural Philosopher]

Now that is interesting, and if I leave the old /var/log there, and
mount a ramdisk over it, provided that the ramdisk is *immediately*
populated and directories created, then all should work?

I guess I have to learn how to add a script to systemd :-(


--
“People believe certain stories because everyone important tells them,
and people tell those stories because everyone important believes them.
Indeed, when a conventional wisdom is at its fullest strength, one’s
agreement with that conventional wisdom becomes almost a litmus test of
one’s suitability to be taken seriously.”

Paul Krugman

[Rich]

In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
> On 30/07/2023 10:24, Theo wrote:
>> The directory structure may be created by the package manager on
>> install, but the files are created on the fly. In other words I
>> might have /var/log/foo/error.log - foo is a directory that needs to
>> exist, but if I delete error.log it'll be created next time there's
>> something to go in there. This is commonly used by 'logrotate',
>> which moves old files and compresses them, allowing new files to be
>> created for future log entries.
>>
>> So all you really need to do is reproduce the directory structure:
>>
>> find /var/log -type d -exec mkdir -p "/tmp/{}" \;
>>
>> which reproduces /var/log/foo/bar/ to /tmp/var/log/foo/bar/
>>
>> Theo
>
> Now that is interesting, and if I leave the old /var/log there, and
> mount a ramdisk over it, provided that the ramdisk is *immediately*
> populated and directories created, then all should work?

You /may/ need to trigger the logging deamon to redo the files. With
traditional syslog that would be send the log deamon's a sighub. With
systemd I have no idea how to do this as none of my linux systems
contain systemd.

> I guess I have to learn how to add a script to systemd :-(

True. Hopefully there is a way.

[Ahem A Rivet's Shot]

On Sun, 30 Jul 2023 15:45:54 +0100
The Natural Philosopher <t...@invalid.invalid> wrote:

> Now that is interesting, and if I leave the old /var/log there, and
> mount a ramdisk over it, provided that the ramdisk is *immediately*
> populated and directories created, then all should work?
>
> I guess I have to learn how to add a script to systemd :-(

You might like this approach it layers a ramdisc over an existing
filesystem.

https://unix.stackexchange.com/questions/339496/mounting-var-as-overlayfs

[John-Paul Stewart]

On 7/30/23 10:45, The Natural Philosopher wrote:
> On 30/07/2023 10:24, Theo wrote:
>> So all you really need to do is reproduce the directory structure:
>>
>> find /var/log -type d -exec mkdir -p "/tmp/{}" \;
>>
>> which reproduces /var/log/foo/bar/ to /tmp/var/log/foo/bar/
>>
>> Theo
> Now that is interesting, and if I leave the old /var/log there, and
> mount a ramdisk over it, provided that the ramdisk is *immediately*
> populated and directories created, then all should work?

Any process that has a log file open before you mount the ramdisk won't
automatically start using it. The process will still have the open file
handle to the original, disk-based file instead. So you'll either need
to be sure to mount the ramdisk sufficiently early in the boot sequence
to ensure that no daemons have opened their logfiles yet, or you'll need
to signal those daemons reopen their logs. Most do that in response to
SIGHUP. Logrotate often uses that mechanism after rotating logs, so its
configuration can help you figure out what needs to be done after
mounting the ramdisk if programs are still logging to disk.

[David W. Hodgins]

On Sun, 30 Jul 2023 10:45:54 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> Now that is interesting, and if I leave the old /var/log there, and
> mount a ramdisk over it, provided that the ramdisk is *immediately*
> populated and directories created, then all should work?
>
> I guess I have to learn how to add a script to systemd :-(

I would add an entry to fstab to handle the mount ...
mylog /var/log tmpfs defaults,user,size=1G 1 1

Copy the directory structure from /var/log somewhere else such as /my/log.

copy /usr/lib/systemd/system/systemd-remount-fs.service to /etc/systemd/system/,
(create the dir if it doesn't exist already), then modify the /etc service to
add a second ExecStart to run a script that copies the directory structure from
/my log to /var/log.

Run "dracut -f" and reboot.

If you later install something that creates a directory in /var/log, just create
a directory with that name, owner, group, and permissions in /my/log.

Make a backup on a second sd card first though, just in case! :-)

Regards, Dave Hodgins

[Carlos E.R.]

On 2023-07-30 09:58, Computer Nerd Kev wrote:
> In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
>> On 30/07/2023 04:27, David W. Hodgins wrote:
>>> On Sat, 29 Jul 2023 20:22:13 -0400, Computer Nerd Kev
>>> <n...@telling.you.invalid> wrote:
>>>> On a system that isn't designed for temporary /var, if you just
>>>> recreate empty directories in /var (those listed by
>>>> "find /var -type d") after reboot, I expect all the programs that
>>>> are already installed will create fresh files inside those
>>>> directories by themselves.
>>>
>>> The owner, group, and permissions must be set correctly too.
>>
>> Well you would just copy /var/log to somewhere on persistent, delete all
>> the files and then do a cp - R on reboot. Except that the first thing
>> that happens on boot is the daemons fill up /var/log with boot messages :-(
>
> This is of course not really the first thing that happens on boot,
> so if you can get in before the init system starts daemons then
> that solves your problem. I wouldn't like to try figuring out how
> to do that with something like Systemd, but then I avoid that sort
> of thing anyway.

On my desktop machine, using openSUSE Leap, the service
"bootmsg.service" initializes some the logs. Writes "var/log/boot.msg",
for instance.

Syslog and or journal are services, so you can create the files before
any of those two services run. For instance, I have my own
"boot-marker.service" which writes a timestamp to /var/log/messages:

[Unit]
Description=Write boot markers in /var/log/messages
Before=syslog.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=-/root/ThingsNeededForBoot/write-boot-marker start
ExecStop=-/root/ThingsNeededForBoot/write-boot-marker stop

[Install]
WantedBy=multi-user.target



A systemd system might not have syslog, but journal, and non permanent
(not on disk). It may use temporary files, though.




--
Cheers, Carlos.

[David W. Hodgins]

On Sun, 30 Jul 2023 16:39:26 -0400, Carlos E.R. <robin_...@es.invalid> wrote:
> A systemd system might not have syslog, but journal, and non permanent
> (not on disk). It may use temporary files, though.

With systemd, systemd-journald collects messages during early boot, in ram. The
in ram copy is flushed to persistent storage shortly after the / filesystem is
remounted rw over the ro / filesystem from the initrd.

That's why I recommend adding an extra ExecStart to systemd-remount-fs.service
to set up the directory structure in a tmpfs mount of /var/log right after
the / filesystem is ready to accept writes. That way it's done before the
journal or, if present, rsyslogd starts writing to /var/log.

Regards, Dave Hodgins

[23k.302]

On 7/29/23 2:42 PM, David W. Hodgins wrote:
> On 7/30/23 10:45, The Natural Philosopher wrote:
>> On 30/07/2023 10:24, Theo wrote:
>>> So all you really need to do is reproduce the directory structure:
>>>

>>> find/var/log -type d -exec mkdir -p "/tmp/{}" \;
>>>
>>> which reproduces/var/log/foo/bar/ to/tmp/var/log/foo/bar/
>>>
>>> Theo

>> Now that is interesting, and if I leave the old /var/log there, and
>> mount a ramdisk over it, provided that the ramdisk is*immediately*
>> populated and directories created, then all should work?

> Any process that has a log file open before you mount the ramdisk won't
> automatically start using it. The process will still have the open file
> handle to the original, disk-based file instead. So you'll either need
> to be sure to mount the ramdisk sufficiently early in the boot sequence
> to ensure that no daemons have opened their logfiles yet, or you'll need
> to signal those daemons reopen their logs. Most do that in response to
> SIGHUP. Logrotate often uses that mechanism after rotating logs, so its
> configuration can help you figure out what needs to be done after
> mounting the ramdisk if programs are still logging to disk.

Hmm ... MIGHT be able to make that work with minimal
pain using systemd. It has 'depends on','run after' and
such params. He could init the ramdisk REALLY early in a
quasi-structured/quasi-safe environment.

Not everyone likes systemd - but it DOES have its uses.

Another trick is to just make the ramdisk whenever and
then make /var/log into a symlink/mountpoint. Yes, there'd
be some early stuff still in the real /var/log, but maybe
that's not important for him.

However I'd rec ramdisks for where they're more useful.
I have an app that does a lot of image manipulation
before the results are shown via PHP/Apache on a Pi.
I don't want any of that to be reading/writing to the
SD card - too slow AND burns it out - but I did want
access to be "normal", like with any 'real' drive. A
small ramdisk was by far the best/easiest solution.

[David Taylor]

On 29/07/2023 18:34, The Natural Philosopher wrote:
> Embedded Raspberry Pi Zero W. Minimal logs generated but I only use
> logs for debugging and why wear the SD card?
>
> My question is simple. If I make a 50MByte or so RAMDISK and mount it on
> /var/log, will the logging daemons recreate all the files and
> directories they need at boot time?

Would this be of any interest?

https://github.com/openenergymonitor/emonpi/blob/master/rpi-ro


--
Cheers,
David
Web: https://www.satsignal.eu

[The Natural Philosopher]

On 30/07/2023 08:58, Computer Nerd Kev wrote:
>> Ah well. The smallest SD card I could easily get was 16GB, and RasPios
>> and friends have only used 1.5GB.
>>
>> It will probably take a long time to wear out.
> I was curious about this once too - does a 16GB SD card where only
> 1.5GB is used last longer than say a 2GB card? I couldn't find a
> conclusive answer online, except that an SD card's unlikely to be
> near as smart about wear levelling as an SSD is.

I think that SD cards (apart from a very few) have no sophisticated wear
levelling *at all*.

From what I could find out they never map *already written* sectors to
new ones so the old sectors get their allocation of writes, too.

My guess is that they would probably *at best* allocate new blocks to
writes till they ran out.

Then sequentially reuse all the already used but 'erased' blocks. That
sort of behaviour would work well for e.g. camera SD cards which get
full but no so often


So, the more blocks the merrier.

If they are *really* cheap they wont do anything at all and there will
be a 1:1 correlation between physical and logical sectors, in which case
it doesn't matter how big they are, they will wear out the directory and
sector allocation sectors without ever using the majority of the card.

I am not sure if e.g. linux can map out bad sectors in those areas. I
suspect not.

My other Pi zero has been on for about 3 years now, writing the log
files. and rotating them. It doesn't do a lot - its simple allows me to
access and play my recorded music and the radio to one of the home hi fis.

[The Natural Philosopher]

A cursory glance at systemd reveals a huge potential flaw in it, or at
least that is my understanding.

There is no order of execution of systemd scripts. They all get executed
in parallel. Instead there are dependencies, which I THINK are
encapsulated in the scripts as 'depends upon', not 'is depended on by'.
So a standalone systemd script wont do the job. Instead one would
probably have to find the mount script and add a patch to that.

>> I guess I have to learn how to add a script to systemd :-(
>
> True. Hopefully there is a way.

--
"Strange as it seems, no amount of learning can cure stupidity, and
higher education positively fortifies it."

- Stephen Vizinczey

[The Natural Philosopher]

On 30/07/2023 17:40, David W. Hodgins wrote:
> On Sun, 30 Jul 2023 10:45:54 -0400, The Natural Philosopher
> <t...@invalid.invalid> wrote:
>> Now that is interesting, and if I leave the old /var/log there,
>> and mount a ramdisk over it, provided that the ramdisk is
>> *immediately* populated and directories created, then all should
>> work?
>>
>> I guess I have to learn how to add a script to systemd :-(
>
> I would add an entry to fstab to handle the mount ... mylog /var/log
> tmpfs defaults,user,size=1G 1 1
>

Pi Zero only has 512M RAM!

> Copy the directory structure from /var/log somewhere else such as
> /my/log.
>
> copy /usr/lib/systemd/system/systemd-remount-fs.service to
> /etc/systemd/system/, (create the dir if it doesn't exist already),
> then modify the /etc service to add a second ExecStart to run a
> script that copies the directory structure from /my log to /var/log.
>

Ah. That is interesting. Patching the mount service *itself*. That
sounds like a plan, unless any updates wipe out the config...but who
bothers to update an embedded system that Just Works?

> Run "dracut -f" and reboot.
>

Can you explain why that would be necessary?

> If you later install something that creates a directory in /var/log,
> just create a directory with that name, owner, group, and
> permissions in /my/log.
>
> Make a backup on a second sd card first though, just in case! :-)
>
> Regards, Dave Hodgins

--
No Apple devices were knowingly used in the preparation of this post.

[The Natural Philosopher]

On 30/07/2023 22:13, David W. Hodgins wrote:
> On Sun, 30 Jul 2023 16:39:26 -0400, Carlos E.R.
> <robin_...@es.invalid> wrote:
>> A systemd system might not have syslog, but journal, and non
>> permanent (not on disk). It may use temporary files, though.
>
> With systemd, systemd-journald collects messages during early boot,
> in ram. The in ram copy is flushed to persistent storage shortly
> after the / filesystem is remounted rw over the ro / filesystem from
> the initrd.
>

Ahaha. First thing that systemd does that actually makes sense.

> That's why I recommend adding an extra ExecStart to
> systemd-remount-fs.service to set up the directory structure in a
> tmpfs mount of /var/log right after the / filesystem is ready to
> accept writes. That way it's done before the journal or, if present,
> rsyslogd starts writing to /var/log.
>

Yes.
The advantage of that is that no info is lost writing to the on-disk
/var/log - the boot stuff will all be flushed to the ramdisk once it is
available.

Well it will be several weeks before I am ready to do any practical
experimentation. I try to ask first, implement later. Soi as not to
waste time on finding out what others already know, by experiment

> Regards, Dave Hodgins

Many thinks. I think your solutions is the one to use. It surgically
splices in the minimum of configuration required to do the job. It OUGHT
to be an optional part of Raspios really...

--
Any fool can believe in principles - and most of them do!

[The Natural Philosopher]

I use them in this and my other pi-zero context as fundamentally simple
interprocess communicators. So in my Hifi Project, a daemon reads the
icecast info off radio streams and writes it to a file in a temporary
file system.

The web server then has access to that to let me know what misc is being
played, via ajax calls from the web servers radio interface.

As long as only one process is writing them, this is a very simple and
effective way to transfer data from one daemon to another.

In the current project which involves driving 4 mains relays using a
daemon that collects various bits of information in order to decide when
to switch them on, it is used to store the relay state, so again ajax
calls from the web server can determine not what *should be on, but what
is *actually* on.

I am sure the semaphores messages and pipes might have done the job, but
simply recreating a file ion a ramdisk every few seconds works fine.

[Richard Kettlewell]

The Natural Philosopher <t...@invalid.invalid> writes:
> There is no order of execution of systemd scripts. They all get
> executed in parallel. Instead there are dependencies, which I THINK
> are encapsulated in the scripts as 'depends upon', not 'is depended on
> by'. So a standalone systemd script wont do the job. Instead one
> would probably have to find the mount script and add a patch to that.

AFAICS all the dependency options have reverses, see the table in
systemd.unit(5).

There are both functional dependencies (e.g. Wants/WantedBy) and
ordering dependencies (e.g. After/Before). If you want to serialize
startup you’ll need both.

--
https://www.greenend.org.uk/rjk/

[Rich]

In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:

> I think that SD cards (apart from a very few) have no sophisticated
> wear levelling *at all*.

Given their race to the bottom price wise, this is a reasonably safe
assumption.

> If they are *really* cheap they wont do anything at all and there
> will be a 1:1 correlation between physical and logical sectors, in
> which case it doesn't matter how big they are, they will wear out the
> directory and sector allocation sectors without ever using the
> majority of the card.
>
> I am not sure if e.g. linux can map out bad sectors in those areas.
> I suspect not.

If the hardware does not handle bad sector replacement then you'd need
to run a Linux filesystem that does do so on top. For an SD card,
that would likely best be one of the 'flash' filesystems.
https://en.wikipedia.org/wiki/Flash_file_system#Linux_flash_filesystems

As to which might be best for an SD card that might have minimal to no
on card wear leveling I can not say.

[Robert Heller]

I've been using SanDisk Class 10 uSD cards in various PIs, with the machines
running continiously for years without errors. Some of these machines have
survived multiple power failures without problems. Several of these machines
I use as "build boxes". I have been using good old Ext4 as the FS.

>
>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
hel...@deepsoft.com -- Webhosting Services

[The Natural Philosopher]

On 31/07/2023 16:11, Robert Heller wrote:
> At Mon, 31 Jul 2023 13:25:28 -0000 (UTC) Rich <ri...@example.invalid> wrote:
>
>>
>> In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
>>> I think that SD cards (apart from a very few) have no sophisticated
>>> wear levelling *at all*.
>>
>> Given their race to the bottom price wise, this is a reasonably safe
>> assumption.
>>
>>> If they are *really* cheap they wont do anything at all and there
>>> will be a 1:1 correlation between physical and logical sectors, in
>>> which case it doesn't matter how big they are, they will wear out the
>>> directory and sector allocation sectors without ever using the
>>> majority of the card.
>>>
>>> I am not sure if e.g. linux can map out bad sectors in those areas.
>>> I suspect not.
>>
>> If the hardware does not handle bad sector replacement then you'd need
>> to run a Linux filesystem that does do so on top. For an SD card,
>> that would likely best be one of the 'flash' filesystems.
>> https://en.wikipedia.org/wiki/Flash_file_system#Linux_flash_filesystems
>>
>> As to which might be best for an SD card that might have minimal to no
>> on card wear leveling I can not say.
>
> I've been using SanDisk Class 10 uSD cards in various PIs, with the machines
> running continiously for years without errors. Some of these machines have
> survived multiple power failures without problems. Several of these machines
> I use as "build boxes". I have been using good old Ext4 as the FS.
>

Indeed. People DO have failures, but I can find no really clear data on
*why* they fail in a Pi situation

>>
>>
>>
>

--
Renewable energy: Expensive solutions that don't work to a problem that
doesn't exist instituted by self legalising protection rackets that
don't protect, masquerading as public servants who don't serve the public.

[Robert Heller]

Ultra "cheap" no-name brand uSD cards?

[druck]

On 29/07/2023 21:14, David W. Hodgins wrote:
> On Sat, 29 Jul 2023 14:54:25 -0400, The Natural Philosopher
> <t...@invalid.invalid> wrote:
>
>> On 29/07/2023 19:42, David W. Hodgins wrote:

>>> On Sat, 29 Jul 2023 13:34:45 -0400, The Natural Philosopher
>>> <t...@invalid.invalid> wrote:
>>>> Embedded Raspberry Pi Zero W. Minimal logs generated but I
>>>> only use logs for debugging and why wear the SD card?
>>>
>>>> My question is simple. If I make a 50MByte or so RAMDISK and
>>>> mount it on /var/log, will the logging daemons recreate all the
>>>> files and directories they need at boot time?

> It can be done using scripts that hook into systemd, but is not easy,
> and may require changes to scripts with any package install that
> uses /var/log, and also risks losing parts of the logs in the case of
> an unclean shutdown, which is typically when you are going to want
> the logs.

On Bullseye and later the largest amount of logging activity to disc is
bloody systemd's journald, which by default will grow /var/log/journal/*
to 2GB, taking up valuable space on SD cards and severely reducing their
write life.

Two things you can do; first as fucking Peottering always knows best,
there is no way of turning off journald and just using the god given
rsyslog, but you can stop journald logging to disc and just keep it in
RAM instead by adding the following to /etc/systemd/journald.conf

Storage=volatile
RuntimeMaxUse=32M
RuntimeMaxFileSize=32M
ForwardToConsole=no
ForwardToWall=no

Secondly stop systemd spamming /var/log/syslog with all sorts of crap
you have no interest in. Create a file called
/etc/rsyslog.d/drop.systemd.conf containing

# Drop messages from f***ing systemd
:programname,startswith,"systemd" stop

If you do that, you'll be left with far less logging activity to disk,
so you may not have to bother moving it to RAM. If you still want no
logging at all to disc, you can disable rsyslog entirely and just use
journald restricted RAM. But that way madness lies.

---druck

[mm0fmf]

On 29/07/2023 18:34, The Natural Philosopher wrote:
> Embedded Raspberry Pi Zero W. Minimal logs generated but  I only use
> logs for debugging and why wear the  SD card?
>
> My question is simple. If I make a 50MByte or so RAMDISK and mount it on
> /var/log, will the logging daemons recreate all the files and
> directories they need at boot time?
>
>

Works fine for me. Pi W running Raspbian (Debian 11 version)

Cron job reboots it every morning so I don't worry about ramdisk filling up.

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.0G 2.1G 4.7G 31% /
devtmpfs 183M 0 183M 0% /dev
tmpfs 215M 0 215M 0% /dev/shm
tmpfs 86M 2.7M 84M 4% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 100M 8.0K 100M 1% /tmp
tmpfs 100M 176K 100M 1% /var/log
tmpfs 30M 0 30M 0% /var/spool/mqueue
tmpfs 30M 0 30M 0% /var/tmp
/dev/mmcblk0p1 253M 51M 202M 20% /boot
tmpfs 43M 0 43M 0% /run/user/1001

Logging works and everything needed seems to be created on boot.

$ ls /var/log -la
total 180
drwxr-xr-x 3 root root 260 Aug 1 05:33 .
drwxr-xr-x 11 root root 4096 Jan 11 2021 ..
-rw-r----- 1 root adm 23843 Aug 1 08:06 auth.log
-rw-rw---- 1 root utmp 3072 Aug 1 08:03 btmp
-rw-r----- 1 root adm 21318 Aug 1 08:00 daemon.log
-rw-r----- 1 root adm 1032 Aug 1 05:33 debug
-rw------- 1 root root 2559 Aug 1 08:03 fail2ban.log
-rw-r----- 1 root adm 26940 Aug 1 05:33 kern.log
-rw-rw-r-- 1 root utmp 292584 Aug 1 08:00 lastlog
-rw-r----- 1 root adm 26239 Aug 1 05:33 messages
drwx------ 2 root root 40 Aug 1 05:33 private
-rw-r----- 1 root adm 50106 Aug 1 08:00 syslog
-rw-rw-r-- 1 root utmp 1920 Aug 1 08:00 wtmp

It's been running like this on the same SDCARD for 2+ years.

[The Natural Philosopher]

On 31/07/2023 21:27, Robert Heller wrote:
> At Mon, 31 Jul 2023 19:59:48 +0100 The Natural Philosopher <t...@invalid.invalid> wrote:
>
>> Indeed. People DO have failures, but I can find no really clear data on
>> *why* they fail in a Pi situation
>
> Ultra "cheap" no-name brand uSD cards?
>

Possibly, again, no direct evidence, sadly.

Its another of those details that Pi users probably need to know, but no
one else cares about.,

There are SD cards for data transfer - video capture - and sd cards for
multiple use like cameras.

no one sells one for a lot of micro writes...

--
“it should be clear by now to everyone that activist environmentalism
(or environmental activism) is becoming a general ideology about humans,
about their freedom, about the relationship between the individual and
the state, and about the manipulation of people under the guise of a
'noble' idea. It is not an honest pursuit of 'sustainable development,'
a matter of elementary environmental protection, or a search for
rational mechanisms designed to achieve a healthy environment. Yet
things do occur that make you shake your head and remind yourself that
you live neither in Joseph Stalin’s Communist era, nor in the Orwellian
utopia of 1984.”

Vaclav Klaus

[The Natural Philosopher]

On 31/07/2023 21:36, druck wrote:
> On 29/07/2023 21:14, David W. Hodgins wrote:
>> On Sat, 29 Jul 2023 14:54:25 -0400, The Natural Philosopher
>> <t...@invalid.invalid> wrote:
>>
>>> On 29/07/2023 19:42, David W. Hodgins wrote:
>>>> On Sat, 29 Jul 2023 13:34:45 -0400, The Natural Philosopher
>>>> <t...@invalid.invalid> wrote:
>>>>> Embedded Raspberry Pi Zero W. Minimal logs generated but  I
>>>>> only use logs for debugging and why wear the  SD card?
>>>>
>>>>> My question is simple. If I make a 50MByte or so RAMDISK and
>>>>> mount it on /var/log, will the logging daemons recreate all the
>>>>> files and directories they need at boot time?
>
>> It can be done using scripts that hook into systemd, but is not easy,
>>  and may require changes to scripts with any package install that
>> uses /var/log, and also risks losing parts of the logs in the case of
>> an unclean shutdown, which is typically when you are going to want
>> the logs.
>
> On Bullseye and later the largest amount of logging activity to disc is
> bloody systemd's journald, which by default will grow /var/log/journal/*
> to 2GB, taking up valuable space on SD cards and severely reducing their
> write life.
>
> Two things you can do; first as fucking Peottering always knows best,

I am already liking your world view. As I said 'designed to make life
easier for a minicomputer system admin'
Not really what we want in a Pi.

> there is no way of turning off journald and just using the god given
> rsyslog, but you can stop journald logging to disc and just keep it in
> RAM instead by adding the following to /etc/systemd/journald.conf
>
> Storage=volatile
> RuntimeMaxUse=32M
> RuntimeMaxFileSize=32M
> ForwardToConsole=no
> ForwardToWall=no
>

Ah. I had already limited its size.
Where in RAM does it keep it, or is that only known the the Great
Poettering, who has provided some buggy tool to access it?

In normal use I would have access and hopefully not error logs from
apache, but only minimal. And perhaps the odd error message being mailed
out via exim.


Mostly the machine will be, apart from a daemon waking up every few
seconds and seeing what's what, essentially idle

> Secondly stop systemd spamming /var/log/syslog with all sorts of crap
> you have no interest in. Create a file called
> /etc/rsyslog.d/drop.systemd.conf containing
>
> # Drop messages from f***ing systemd
> :programname,startswith,"systemd" stop
>

is ryslog still running? Oh, yes it is.

I think stage one is to create and mount something like /var/ramlog and
see how much standard logging can be moved there by patching
rsyslog.conf and the logrotate files.

apache2 can be moved there too. That's defined in /etc/apache2/envvars

The only hardwired one I can see so far is exim4, but I can cope with that.
Certainly should be able to push a huge amount off the SD

> If you do that, you'll be left with far less logging activity to disk,
> so you may not have to bother moving it to RAM. If you still want no
> logging at all to disc, you can disable rsyslog entirely and just use
> journald restricted RAM. But that way madness lies.
>

Logging is phenomenally useful for debugging a *development* project,
but there is absolutely no need for persistence

So a ram disk plus limiting what goes on it, is handy.
...a few moments later...
successfuly got systemd journal to cease to exist on disk, and moved
nearly everything to a ram log.
As the man falling past the 13th floor yelled: 'so far. so good!'

--
Religion is regarded by the common people as true, by the wise as
foolish, and by the rulers as useful.

(Seneca the Younger, 65 AD)

[Ahem A Rivet's Shot]

On Tue, 1 Aug 2023 11:03:45 +0100
The Natural Philosopher <t...@invalid.invalid> wrote:

> I think stage one is to create and mount something like /var/ramlog and
> see how much standard logging can be moved there by patching
> rsyslog.conf and the logrotate files.

Why don't you just use overlayfs to overlay a ramdisk on /var ?

[Dave]

On 01/08/2023 09:58, The Natural Philosopher wrote:
> Its another of those details that Pi users probably need to know, but no
> one else cares about.,
>
> There are SD cards for data transfer - video capture - and sd cards for
> multiple use like cameras.
>
> no one sells one for a lot of micro writes...

There are "Application Performance Class" cards now available which
focus on I/O operations per second instead of raw streaming read/write
speeds. They are identified as class A1 and A2. Sandisk seems to be the
main supplier.

One would hope that these cards would have wear levelling set up to
handle lots of small I/O operations but I suspect that's not part of the
spec...

[Ahem A Rivet's Shot]

On Tue, 1 Aug 2023 14:25:33 +0100
Dave <da...@cyw.uklinux.net> wrote:

> One would hope that these cards would have wear levelling set up to
> handle lots of small I/O operations but I suspect that's not part of the
> spec...

SD cards have had wear levelling for about a decade now, I doubt
there are many without it.

[John-Paul Stewart]

On 7/31/23 16:36, druck wrote:
>
> On Bullseye and later the largest amount of logging activity to disc is
> bloody systemd's journald, which by default will grow /var/log/journal/*
> to 2GB, taking up valuable space on SD cards and severely reducing their
> write life.
>
> Two things you can do; first as fucking Peottering always knows best,
> there is no way of turning off journald and just using the god given
> rsyslog, but you can stop journald logging to disc and just keep it in
> RAM instead by adding the following to /etc/systemd/journald.conf
>
> Storage=volatile
> RuntimeMaxUse=32M
> RuntimeMaxFileSize=32M
> ForwardToConsole=no
> ForwardToWall=no
>
> Secondly stop systemd spamming /var/log/syslog with all sorts of crap
> you have no interest in. Create a file called
> /etc/rsyslog.d/drop.systemd.conf containing
>
> # Drop messages from f***ing systemd
> :programname,startswith,"systemd" stop

Thanks for posting that!

I'm not the OP, but I still found that info immensely useful for a
totally different reason. And I'm probably not the only one who doesn't
want to waste a day sifting through the systemd and journalctl docs to
figure it out for myself.

[druck]

On 01/08/2023 11:03, The Natural Philosopher wrote:
> Where in RAM does it keep it, or is that only known the the Great
> Poettering, who has provided some buggy tool to access it?

It's stored in /run/log/journal (/run is tmpfs in RAM)

Despite the configuration specifying a maximum journal size of 32M and
even 512MB Pi's having a 64MB /run, if you do a df /run and see it at
100% it's down to bloody journald, and you have to use Peottering,s
buggy tool to manually clear out the crap.

journal --vacuum-files=1

> successfuly got systemd journal to cease to exist on disk, and moved
> nearly everything to a ram log.
> As the man falling past the 13th floor yelled: 'so far. so good!'

If you aiming for uptimes of 80+ days or more, then bung that command in
a weekly or monthly cron job, as things can get a bit temperamental if
/run is full.

---druck

[Pancho]

On 7/29/23 19:45, Robert Heller wrote:
> This is in fact the default behaviour for Armbian on my Banana Pi M64.
>

+1, :-) I was just thinking, I spent ages undoing this, in order to
find out why my system was crashing. But I'm running Armbian.

TNP, FFS, SD cards cost buttons. Pack of two 32GB for less than a
tenner. If you are worried about wear, clone a new one every year, or
whatever.

[druck]

On 01/08/2023 08:08, mm0fmf wrote:
> Cron job reboots it every morning so I don't worry about ramdisk filling
> up.
>

> It's been running like this on the same SDCARD for 2+ years.

Lucky. I had an older Pi which I had set to reboot every day, and I had
lots of occasions when it didn't come back up by itself. Often just
needed power cycling, occasionally filesystem corruption needed fixing.
Turned out it was a dodgy WiFi dongle, and after replacing that I was
able to abandon the daily reboots, and the reliability was rock solid
after that.

My suggestion is to look at the logging levels of various processes, and
ensure everything has a suitable log rotate configuration, so the size
of your logging directory is kept pretty static over time without having
to reboot. Particularly as booting does lots of logging, and once that
is rotated out, the size may go down slightly.

I do nightly differential backups across my fleet of Pi's and the first
thing I look for in the monitoring graphs is the size of data
transferred, any significant increase is invariably due to some problem
causing a log to grow massively. A quick intervention has often avoided
considerable downtime.

---druck

[mm0fmf]

On 01/08/2023 21:48, druck wrote:
> On 01/08/2023 08:08, mm0fmf wrote:
>> Cron job reboots it every morning so I don't worry about ramdisk
>> filling up.
>>
>> It's been running like this on the same SDCARD for 2+ years.
>

[snip]

> My suggestion is to look at the logging levels of various processes, and
> ensure everything has a suitable log rotate configuration, so the size
> of your logging directory is kept pretty static over time without having
> to reboot. Particularly as booting does lots of logging, and once that
> is rotated out, the size may go down slightly.
>

I think you are right. I really should set it up better than it is. It's
a classic "round tuit" problem as it works acceptably as is. I suppose
the problem with Pi's and the like is they feel disposable along with
their SD Cards and once you have something working the urge to do the
job properly wanes. And you can always just stick the backup SD Card in
and reboot!

[23k.304]

Agreed. If you CAN keep it simple, DO keep it simple.
Remember YOU might have to debug/enhance it a few
years LATER :-)

Things like Pi's are special cases - not blazing fast,
not a lot of RAM, run off SD cards/eMMC that have to be
protected from re-write fatigue. If the need for space
isn't excessive, and simplicity is worthwhile (almost
always) then a RAMdisk is often your best solution.

Because of the filesystem overhead, RAMdisks just MIGHT
not be fast enough ... then you have to go to "less simple"
approaches alas. To each app, its own.

> In the current project which involves driving 4 mains relays using a
> daemon that collects various bits of information in order to decide when
> to switch them on, it is used to store the relay state, so again ajax
> calls from the web server can determine not what *should be on, but what
> is *actually* on.

Cool stuff

> I am sure the semaphores messages and pipes might have done the job, but
> simply recreating  a file ion a ramdisk every few seconds works fine.

"Pipes" are a kind of cheat - really just "invisible" R/W files.
I used them kinda extensively on some forking servers I wrote a
couple of years ago in Python and 'C' to make data gathered by
new forks accessible to the mother process. They WORK just fine
and are commonly used and I *think* a tad faster than a RAMdisk,
but still ...

I'm fond of such 'servers' - even cobbled together a good
"pre-forked" one - supposedly the highest-capacity/speed -
for both TCP and UDP - but never had a good high-volume
reason to use them. The best was almost like a 'chat' app,
non-sync bi-directional where the queen server process could
initiate, even push, tasks and messages to the clients. Fun !

Hmm ... now apparently some possible servants of Xi here
are calling me a "troll" because I repeated a few things
(said by HS/NSA/CIA) they didn't WANT to hear about
CCP sabotage-ware infiltrating almost everything in
our important infrastructure/military systems and
'devices' (which are almost entirely Linux/Unix-based).
One day soon they ARE gonna get a big wake-up call ... but
meanwhile denial means they won't DO anything about it :-)

ANYway, /var/log CAN be moved to a RAMdisk if you want.
Not 100% sure WHY you'd want to, but it CAN. If a few
very early logs get 'lost' as you re-direct /var/log
then that MIGHT not be all so important. If you want
it all on RAMdisk then you don't CARE if it all
vanishes on reboot. I very rarely look in /var/log
anyhow so ....

[The Natural Philosopher]

Some info I gleaned suggested that [some of] the better cards do have
full wear levelling.

--
“It is not the truth of Marxism that explains the willingness of
intellectuals to believe it, but the power that it confers on
intellectuals, in their attempts to control the world. And since...it is
futile to reason someone out of a thing that he was not reasoned into,
we can conclude that Marxism owes its remarkable power to survive every
criticism to the fact that it is not a truth-directed but a
power-directed system of thought.”
Sir Roger Scruton

[The Natural Philosopher]

On 01/08/2023 14:46, Ahem A Rivet's Shot wrote:
> On Tue, 1 Aug 2023 14:25:33 +0100
> Dave <da...@cyw.uklinux.net> wrote:
>
>> One would hope that these cards would have wear levelling set up to
>> handle lots of small I/O operations but I suspect that's not part of the
>> spec...
>
> SD cards have had wear levelling for about a decade now, I doubt
> there are many without it.
>

I could find no clear evidence of that.

[The Natural Philosopher]

Yup. excellent advice.

Now my /var/log isn't being used at all

oot@heating-controller:/var/ramlog# du -sh /var/log
92K /var/log
root@heating-controller:/var/ramlog# ls -l
total 21224
-rw-r--r-- 1 root root 21280021 Aug 2 16:42 access.log
-rw-r----- 1 root adm 119049 Aug 2 16:39 auth.log
-rw-r----- 1 root adm 88082 Aug 2 16:39 daemon.log
-rw-r----- 1 root adm 3119 Aug 1 11:05 debug
-rw-r--r-- 1 root root 31171 Aug 2 15:39 error.log
-rw-r----- 1 root adm 31714 Aug 1 11:05 kern.log
-rw-r----- 1 root adm 30892 Aug 1 11:05 messages
-rw-r--r-- 1 root root 0 Aug 1 11:05 other_vhosts_access.log
-rw-r----- 1 root adm 133829 Aug 2 16:39 syslog

But ramlog is. My server instructs the browser to to do an ajax poll
every second

# ls -l /var/ramlog
total 21240
-rw-r--r-- 1 root root 21296720 Aug 2 16:45 access.log
-rw-r----- 1 root adm 119049 Aug 2 16:39 auth.log
-rw-r----- 1 root adm 88082 Aug 2 16:39 daemon.log
-rw-r----- 1 root adm 3119 Aug 1 11:05 debug
-rw-r--r-- 1 root root 31171 Aug 2 15:39 error.log
-rw-r----- 1 root adm 31714 Aug 1 11:05 kern.log
-rw-r----- 1 root adm 30892 Aug 1 11:05 messages
-rw-r--r-- 1 root root 0 Aug 1 11:05 other_vhosts_access.log
-rw-r----- 1 root adm 133829 Aug 2 16:39 syslog

Next step is to turn off such verbose logging from apache...ah
simply comment out the line dictating access logs in the sites-enabled
location and no more access..log.
its getting there.

--
It’s easier to fool people than to convince them that they have been fooled.
Mark Twain

[The Natural Philosopher]

On 01/08/2023 21:30, druck wrote:
> On 01/08/2023 11:03, The Natural Philosopher wrote:
>> Where in RAM does it keep it, or is that only known the the Great
>> Poettering, who has provided some buggy tool to access it?
>
> It's stored in /run/log/journal (/run is tmpfs in RAM)
>
> Despite the configuration specifying a maximum journal size of 32M and
> even 512MB Pi's having a 64MB /run, if you do a df /run and see it at
> 100% it's down to bloody journald, and you have to use Peottering,s
> buggy tool to manually clear out the crap.
>

Its looking OK.
root@heating-controller:/run/log/journal# ls -la
total 0
drwxr-sr-x+ 3 root systemd-journal 60 Aug 1 11:05 .
drwxr-xr-x 3 root root 60 Aug 1 11:05 ..
drwxr-s---+ 2 root systemd-journal 80 Aug 2 06:05
5804893eb1ba4979aa14b83e642e215c
root@heating-controller:/run/log/journal# du -sh *
6.3M 5804893eb1ba4979aa14b83e642e215c
root@heating-controller:/run/log/journal#

> journal --vacuum-files=1
>
>> successfuly got systemd journal to cease to exist on disk, and moved
>> nearly everything to a ram log.
>> As the man falling past the 13th floor yelled: 'so far. so good!'
>
> If you aiming for uptimes of 80+ days or more, then bung that command in
> a weekly or monthly cron job, as things can get a bit temperamental if
> /run is full.
>

I will bear that in mind
But power failures are relatively common here when the wind blows

> ---druck
>

--
"And if the blind lead the blind, both shall fall into the ditch".

Gospel of St. Mathew 15:14

[The Natural Philosopher]

This product may end up in places where the user isn't smart enough to
do that.

[The Natural Philosopher]

This pi will be running my central heating. It will be tucked away in a
boiler cupboard. The existing controller keeps crashing during power
cuts so I decided that with a $5000 heating bill, making a rock sold
controller to have different temperatures at different times might
easily save me $1000 a year (as well as allowing me to hit the
controller via the bedroom laptop and turn bathroom heaters on before I
get out of bed).

And by not continuously writing to the SD card with log files the chance
of a non booting unit is really rather small. In fact it is hardly
writing to the SD card AT ALL. now.


Its going slower than I hoped but i haven't met any major issues on the
controller - the Zero W.

The Pico Ws to measure temperature and oil tank levels may be more of a
challenge. But the existing Wireless oil tank gauge sucks - hasn't got
the range, but to my surprise, holding up the mobile next to the oil
tank showed two fairly weak, but potentially useable WiFi points in the
house.

WE shall see.Having a machine far more capable than a PDP/11 cost less
than £25 with the SD card, and less than £50 all in with power supply
relays and hopefully 3D printed box, is stunning for basic networked
process control

Looking forward to hacking the Picos in due course. Ive been working on
the simplest and cheapest electronics that will, if the Pi instructs it,
shut all power down to the Pi and draw at most a few microamps until a
capacitor charges up again, when it will restore power and reboot the
PICO..my existing oil tank monitor wakes up once an hour. For a second
or two. The battery lasts for years.

[The Natural Philosopher]

Oh, I learnt my code writing contract assembler for people who wanted it
well commented enough that the stupidest permie could fix it years later.
Or myself. Sometimes I look at stuff I wrote and marvel at ait.

I am not a computer scientist. Elegance isn't in my nature. I am a
software and other sorts of engineer, and making it work, and keep on
working at the lowets possible prices is burned into my soul

"An engineer is someone who can do for five bob what any damned fool can
do for a quid"

>   Things like Pi's are special cases - not blazing fast,
>   not a lot of RAM, run off SD cards/eMMC that have to be
>   protected from re-write fatigue. If the need for space
>   isn't excessive, and simplicity is worthwhile (almost
>   always) then a RAMdisk is often your best solution.
>

Ha! compared with a Z80 with 48k of usable RAM or a 6809, they ARE
blazingly fast. I cut my teeth writing C and assembler for those, stuff
that was burned into ROM.
There is so much on this zero ROM - 16GB! And I have ram coming out of
my ears.

>   Because of the filesystem overhead, RAMdisks just MIGHT
>   not be fast enough ... then you have to go to "less simple"
>   approaches alas. To each app, its own.
>

My access requrements are measured in seconds, not micro seconds.

>> In the current project which involves driving 4 mains relays using a
>> daemon that collects various bits of information in order to decide
>> when to switch them on, it is used to store the relay state, so again
>> ajax calls from the web server can determine not what *should be on,
>> but what is *actually* on.
>
>   Cool stuff
>
>> I am sure the semaphores messages and pipes might have done the job,
>> but simply recreating  a file ion a ramdisk every few seconds works fine.
>
>   "Pipes" are a kind of cheat - really just "invisible" R/W files.
>   I used them kinda extensively on some forking servers I wrote a
>   couple of years ago in Python and 'C' to make data gathered by
>   new forks accessible to the mother process. They WORK just fine
>   and are commonly used and I *think* a tad faster than a RAMdisk,
>   but still ...
>

Oh there surely are faster, but who is in a hurry? a central heating
controller changes states every few HOURS. and if the users browser view
of that is a second behind, frankly who gives a damn?

Like wise room temperatures and tank oil levels are not things that
change much in an hour...

>   I'm fond of such 'servers' - even cobbled together a good
>   "pre-forked" one - supposedly the highest-capacity/speed -
>   for both TCP and UDP - but never had a good high-volume
>   reason to use them. The best was almost like a 'chat' app,
>   non-sync bi-directional where the queen server process could
>   initiate, even push, tasks and messages to the clients. Fun !
>
>   Hmm ... now apparently some possible servants of Xi here
>   are calling me a "troll" because I repeated a few things
>   (said by HS/NSA/CIA) they didn't WANT to hear about
>   CCP sabotage-ware infiltrating almost everything in
>   our important infrastructure/military systems and
>   'devices' (which are almost entirely Linux/Unix-based).
>   One day soon they ARE gonna get a big wake-up call ... but
>   meanwhile denial means they won't DO anything about it  :-)
>

The problem is that no one knows what is really happening, and although
the political narratives that are spun for consumption by hoi polloi,
are obviously not the real truth, no one knows what is. Not even the
spinners.

I myself know for example, that there couldn't have been any weapons of
mass destruction *directly threatening British interests* in Iraq, and
that our prime minister lied about that. I spent a couple of years
designing missile parts and know in principle where the bleeding edge
is, and it sure ain't in Iraq.

I know that Covid injections do not put goverment control chips in your
brain, They don't need to. They have the mass mediafor that :-)

I know the moon landings were real, because there wasn't the technology
then to fake it.

I know that 911 was a simple case of structural failure in a fire,
because everything about it is consistent with my knowledge of
structural engneering.

I dont know who apart from lee harvey oswald, might have shot JFK , or
why. |Jury still out.

I know that 'renewable' energy is a greater disaster than any climate
change, and I know that climate change is largely natural with very
little man made components. Because I did the years of research. And the
problem is within my pay garde as an engineer with a good STEM background


I don't know who runs the world banks, but I an fairly sure its not
Jews, or Lizards.
I am however sure that they haven't a clue what they are doing.

>   ANYway, /var/log CAN be moved to a RAMdisk if you want.
>   Not 100% sure WHY you'd want to, but it CAN. If a few
>   very early logs get 'lost' as you re-direct /var/log
>   then that MIGHT not be all so important. If you want
>   it all on RAMdisk then you don't CARE if it all
>   vanishes on reboot. I very rarely look in /var/log
>   anyhow so ....

As I said, whoever you are, the less writing to an SD drive there is,
the less chance there is of file system corruption when the power goes out.

Its simply a natural habit to ensure than whatever an embedded system
does is the minimum necessary to get the job done with the maximum
reliability.

The SD card looks like it might be the weakest link. So I am trying to
reduce stress on it.


--
"Women actually are capable of being far more than the feminists will
let them."

[druck]

We are more likely to have power failures when the wind doesn't blow!

---druck

[druck]

On 01/08/2023 23:56, mm0fmf wrote:

> On 01/08/2023 21:48, druck wrote:
>> On 01/08/2023 08:08, mm0fmf wrote:
>> My suggestion is to look at the logging levels of various processes,
>> and ensure everything has a suitable log rotate configuration

> I think you are right. I really should set it up better than it is. It's
> a classic "round tuit" problem as it works acceptably as is.

Yes it works, right up to when it doesn't :-)

>  I suppose
> the problem with Pi's and the like is they feel disposable along with
> their SD Cards and once you have something working the urge to do the
> job properly wanes. And you can always just stick the backup SD Card in
> and reboot!

That's why I set up the nightly differential backups, which write the
changes to an image file of the SD card, so if the card fails I can just
write the image to a new SD card and be back up and running in minutes.

That's fine for my Pi's at home, but now I have them running services at
several other locations, which I may not be able to get to for weeks. So
I've set them up far more carefully to avoid using up the SD write life,
and have invested and trying out the Samsung and SanDisk high endurance
variants.

---druck

[The Natural Philosopher]

Ah, well the qualifier 'local' versus 'national' comes into play here,
and it's not exactly true either

high wind leads to far more fluctuation in generation as the wind can
stop wuite quickly, and if all the other generators are shut down,m
there ain't nuthin' left.

People are coming to realise that renewable energy is a far greater
threat to civilisation than climate change, man made or not, could ever
be...



--
Those who want slavery should have the grace to name it by its proper
name. They must face the full meaning of that which they are advocating
or condoning; the full, exact, specific meaning of collectivism, of its
logical implications, of the principles upon which it is based, and of
the ultimate consequences to which these principles will lead. They must
face it, then decide whether this is what they want or not.

Ayn Rand.

[The Natural Philosopher]

On 02/08/2023 21:32, druck wrote:
> That's why I set up the nightly differential backups, which write the
> changes to an image file of the SD card, so if the card fails I can just
> write the image to a new SD card and be back up and running in minutes.
>
> That's fine for my Pi's at home, but now I have them running services at
> several other locations, which I may not be able to get to for weeks. So
> I've set them up far more carefully to avoid using up the SD write life,
> and have invested and trying out the Samsung and SanDisk high endurance
> variants.

Interesting philosophy. accept failure and plan round it rather than
eliminating failure in the first place?
I will take an SD card image once the design is finalised, but fir now
the Piz are outside my nightly backup regime simply because they are not
changing, and such changes as there are are simply a mater of remounting
my server on a fresh pi and saying 'make;make install'


--
"An intellectual is a person knowledgeable in one field who speaks out
only in others...”

Tom Wolfe

[David W. Hodgins]

On Wed, 02 Aug 2023 22:47:44 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> Interesting philosophy. accept failure and plan round it rather than
> eliminating failure in the first place?

Keeping backups comes from experience. Hardware fails, even sd cards, and
people make mistakes when deleting things.

Regards, Dave Hodgins

[23k.304]

Came across an embedded app I wrote about six years
ago. It had its uses and then was obsolete. A big 'C'
program for an Arduino Mega... lots of sensors, interrupts,
data-polishing, what you'd expect. Even after such a
relatively short period I did indeed "marvel" at its
complexity and how it got through a bunch of issues -
and I *wrote* the damned thing in about a month !
How'd I *do* that ???

> I am not a computer scientist. Elegance isn't in my nature. I am a
> software and other sorts of engineer, and making it work, and keep on
> working at the lowets possible prices is burned into my soul
>
> "An engineer is someone who can do for five bob what any damned fool can
> do for a quid"

"What Works" is usually MUCH more important than "What
is theoretically correct". I do not plan/diagram
apps, I just GO for it. Gotta kinda hold the whole thing
in your head so you can see every interaction/dependency.
Always a buzz ! Gets all those old neurons working !

Yea, "computational theory" has its place - but in the
real world ...... shit, I don't even use most of those
"advanced" tricks in Python - too obscure and of little
advantage except to decrease transparency.

It's all 'C' under the hood. All the new "tricks"/"shortcuts/
"methods" are really the Same Old Shit, same (or more) number
of cpu cycles and bytes - just a lot less transparent. "Ease"
and "sophistication" are so often illusions and it's getting
worse and worse.

The "AI's" will soon take over, writing everything THEIR way,
according to THEIR logic. Won't be long after that mere humans
are incapable of understanding/verifying what they create.
It all becomes "magic".

>>    Things like Pi's are special cases - not blazing fast,
>>    not a lot of RAM, run off SD cards/eMMC that have to be
>>    protected from re-write fatigue. If the need for space
>>    isn't excessive, and simplicity is worthwhile (almost
>>    always) then a RAMdisk is often your best solution.
>>
> Ha! compared with a Z80 with 48k of usable RAM or a 6809, they ARE
> blazingly fast. I cut my teeth writing C and assembler for those, stuff
> that was burned into ROM.
> There is so much on this zero ROM - 16GB! And I have ram coming out of
> my ears.

Well, "blazingly fast", except by modern desktop/server measures :-)

PI's are kind of in the gap between microcontrollers and
"real" modern CPU boards. The power-consumption kinda
pisses me off though ... you can basically turn a PIC
or Arduino OFF until some important event happens, thus
making them battery/solar-friendly. NOT so with PI's.

PIs can be good for a lot of applications - but their
limits MUST be considered. They are NOT high-end Xeons,
more like 10 year old laptops.

>>    Because of the filesystem overhead, RAMdisks just MIGHT
>>    not be fast enough ... then you have to go to "less simple"
>>    approaches alas. To each app, its own.
>>
> My access requrements are measured in seconds, not micro seconds.

Makes it easy ! This is OFTEN the case.

>>> In the current project which involves driving 4 mains relays using a
>>> daemon that collects various bits of information in order to decide
>>> when to switch them on, it is used to store the relay state, so again
>>> ajax calls from the web server can determine not what *should be on,
>>> but what is *actually* on.
>>
>>    Cool stuff
>>
>>> I am sure the semaphores messages and pipes might have done the job,
>>> but simply recreating  a file ion a ramdisk every few seconds works
>>> fine.
>>
>>    "Pipes" are a kind of cheat - really just "invisible" R/W files.
>>    I used them kinda extensively on some forking servers I wrote a
>>    couple of years ago in Python and 'C' to make data gathered by
>>    new forks accessible to the mother process. They WORK just fine
>>    and are commonly used and I *think* a tad faster than a RAMdisk,
>>    but still ...
>>
> Oh there surely are faster, but who is in a hurry? a central heating
> controller changes states every few HOURS. and if the users browser view
> of that is a second behind, frankly who gives a damn?

Well, 'device controllers' run at their own pace but
things like comm servers need to run a lot faster and
be prepared for LOTS of clients at the same time. At
the time I was interested in all practical variants and
thus took some skeleton code I found and blew it out
into full real apps.

"Pre-Forked" really ARE "the best" - you set up maybe
ten or fifteen forks all running at the same time and
distribute work accordingly. If NEEDED you can dynamically
expand the number of forks. Think "Google".

Good for very high traffic (there
are graphs on this). The flip is that setting up and
then managing all those forks IS a pain in the ass.
It all depends on what you NEED. For MOST uses the
simplest, non-forking/non-threaded, server will,
well, serve.

> Like wise room temperatures and tank oil levels are not things that
> change much in an hour...
>
>
>>    I'm fond of such 'servers' - even cobbled together a good
>>    "pre-forked" one - supposedly the highest-capacity/speed -
>>    for both TCP and UDP - but never had a good high-volume
>>    reason to use them. The best was almost like a 'chat' app,
>>    non-sync bi-directional where the queen server process could
>>    initiate, even push, tasks and messages to the clients. Fun !
>>
>>    Hmm ... now apparently some possible servants of Xi here
>>    are calling me a "troll" because I repeated a few things
>>    (said by HS/NSA/CIA) they didn't WANT to hear about
>>    CCP sabotage-ware infiltrating almost everything in
>>    our important infrastructure/military systems and
>>    'devices' (which are almost entirely Linux/Unix-based).
>>    One day soon they ARE gonna get a big wake-up call ... but
>>    meanwhile denial means they won't DO anything about it  :-)
>>
> The problem is that no one knows what is really happening, and although
> the political narratives that are spun for consumption by hoi polloi,
> are obviously not  the real truth, no one knows what is. Not even the
> spinners.

The spinners generally know NOTHING. They don't program.
They merely repeat what they heard. However WHAT they've
been hearing from the Highest Levels now IS very worrisome.

My experience HERE is now Very Bad - the Linux People
just do NOT WANT TO HEAR that the CCP has been sabotaging
distros/drivers/etc. I remember about five years back when
a Mint distro was heavily sabotaged ... they went after the
code on the official distribution download page. One of
the guys in the office HAD just installed it - I had to
tell him to HOSE it. He wasn't very happy.

BUT ... when's the last time YOU went through, say, the
code for SSH or even actually used those checksum files
that come with distros ? We all just ASSUME everything
is fine, that SOMEBODY is watching every bit of it.

Alas there are now SO many bits of it that said assumptions
might NOT be true. Hell, even a printer driver can be
tweaked by a malicious agent. It's so low-level that
likely NOBODY checks often. And, according to our big
letter agencies, there are MANY secret CCP tweaks in
such stuff.

> I myself know for example, that there couldn't have been any weapons of
> mass destruction *directly threatening British interests* in Iraq, and
> that our prime minister lied about that. I spent a couple of years
> designing missile parts and know in principle where the bleeding edge
> is, and it sure ain't in Iraq.

The "justification" for the Iraq War was 99% bullshit.
The REAL reasons were purely political - Hussein was
sliding over to the Russian side. Can't HAVE that !
Too much oil involved.

There is "politics" and "REALPOLITIK" ... a world
of difference.

> I know that Covid injections do not put goverment control chips in your
> brain, They don't need to. They have the mass mediafor that :-)

I've had - and will have - every Covid vax they produce.
No microchips. I forget WHO came up with that paranoid
theory ...

> I know the moon landings were real, because there wasn't the technology
> then to fake it.

I lived 20 miles from Kennedy back in the day. The
launches shook the sky. My Dad WORKED there. They
weren't faking anything. It was all stuff just
*barely* within the current tech. Surprised there
weren't more disasters.

For fun, look up "rope memory" ... that's what the
Apollo computers ran on (even though much better
stuff HAD been invented since the Original Contract).

> I know that 911 was a simple case of structural failure in a fire,
> because everything about it is consistent with my knowledge of
> structural engneering.

Yep. However the popular vision of falling buildings
comes from movies - where the tall structures fall
over dramatically instead of collapsing in-place.
Sorry, but the structures can't support "falling over".
MAYBE the Empire State.

I was watching on 911 ... was able to predict the
Big Crush well ahead of time. You could SEE what
was happening, what was about to happen.

> I dont know who apart from lee harvey oswald, might have shot JFK , or
> why. |Jury still out.

I think he *did* it ... but WHY, and who was PAYING,
we'll NEVER find out. There are now SO many takes
on it that we'd never recognize the REAL one from
all the bullshit. This seems intentional. All "high
officials" from 1963 are DEAD. Nobody blabbed.

> I know that 'renewable' energy is a greater disaster than any climate
> change, and I know that climate change is largely natural with very
> little man made components. Because I did the years of research. And the
> problem is within my pay garde as an engineer with a good STEM background

"Renewable" is almost all hype/bullshit at this
point. It'll make a lot of $$$ for a few who
will kick it back to their favorite pols.

Now in 20-25 years 'renewable' CAN be much more
real. Better PVs, better batteries, it CAN be
very useful. But for TODAY, a disaster .....

> I don't know who runs the world banks, but I an fairly sure its not
> Jews, or Lizards.
> I am however sure that they haven't a clue what they are doing.

They mostly just "fake it" ... IMHO. Not the best
thing but the whole world financial thing is SO
fluid and weird that there may be no better approach.
Pure Reaction.

However the ILLUSION of sense and logic is paramount.
Economies these days are all illusion. None of the
actual numbers seem to work out. They've become a
'religion' of sorts. Heretics are suppressed.

As for the "Lizard People" ... well :-) I'll put
my trust in the Space Aliens instead !

>>    ANYway, /var/log CAN be moved to a RAMdisk if you want.
>>    Not 100% sure WHY you'd want to, but it CAN. If a few
>>    very early logs get 'lost' as you re-direct /var/log
>>    then that MIGHT not be all so important. If you want
>>    it all on RAMdisk then you don't CARE if it all
>>    vanishes on reboot. I very rarely look in /var/log
>>    anyhow so ....
>
> As I said, whoever you are, the less writing to an SD drive there is,
> the less chance there is of file system corruption when the power goes out.

Yep !!! Leave the SD/eMMC *out* of it all as much as possible !

> Its simply a natural habit to ensure than whatever an embedded system
> does is the minimum necessary to get the job done with the maximum
> reliability.

Yea, but do the "new guys" really GET that ??? I've run
into a LOT who don't seem to. They don't understand the
tech, just follow the glossy hype. The more "churning"
the more you should shift to SSDs, then magnetics, then
RAMdisks if possible.

> The SD card looks like it might be the weakest link. So I am trying to
> reduce stress on it.

In most PI/embedded any SD card *is* the weakest link, the
prime point for failure. Always buy the best (Samsung
"Endurance") but you STILL have to be smart.

I came across a PI-1b that I'd set up to do ONE simple
thing buried way in the background. It'd been doing its
thing for about 7 YEARS without fanfare. Samsung card
(pre-"Endurance"). Ancient version of Raspbian. Not a
security risk, not on the network. It simply Just Worked.
The main junk is done on a RAMdisk. I did put a newer
card in it, the latest Raspbian, and now it's still doing
its one thing. Should be good for a decade. I'll be
long-retired, maybe dead, by then. After that they
can spend $2500 for what I did for $79.95 ......

Oh well, maybe Too Much ... I'd prefer simpler replies,
but there were MANY issues raised here. I'm NOT gonna
stop posting to COLM even IF some fools think I'm
some kind of "troll". What I say generally NEEDS to
be said. If they Don't Want to hear it ... well .....

Linux/Unix are NOT invulnerable. This latest news
makes that all the more obvious. Plan/act accordingly.
The thing that makes your electricity stay on, that keeps
big chemical factory from exploding, mekes your bank
account work ... NOT as secure as you'd LIKE to think.
The CCP and friends have VAST resources to make that so
and Confrontation Time is counting down.

[Jan Panteltje]

On a sunny day (Wed, 2 Aug 2023 17:47:56 +0100) it happened The Natural
Philosopher <t...@invalid.invalid> wrote in <uae1bt$5slp$1...@dont-email.me>:

>
>
>>   ANYway, /var/log CAN be moved to a RAMdisk if you want.
>>   Not 100% sure WHY you'd want to, but it CAN. If a few
>>   very early logs get 'lost' as you re-direct /var/log
>>   then that MIGHT not be all so important. If you want
>>   it all on RAMdisk then you don't CARE if it all
>>   vanishes on reboot. I very rarely look in /var/log
>>   anyhow so ....
>
>As I said, whoever you are, the less writing to an SD drive there is,
>the less chance there is of file system corruption when the power goes out.

To avoid restarts on power failure use a UPS (I do) or just a simple battery circuit
was it not called 'battery hat' or something? with the Pi.

>Its simply a natural habit to ensure than whatever an embedded system
>does is the minimum necessary to get the job done with the maximum
>reliability.
>
>The SD card looks like it might be the weakest link. So I am trying to
>reduce stress on it.

For many things Pi is too much power, much can be done with a simple PIC 18F14K22
https://panteltje.nl/panteltje/pic/index.html
take scope_pic or freq_pic from that site.
*IF* you like to code in asm :-)
Only milli-amps curent consumption
https://panteltje.nl/panteltje/pic/gm_pic2/

So much bloat these days just to say 'hello world'
I designed a ram disk for the Z80, wrote a CP/M clone too
https://panteltje.nl/panteltje/z80/system14/diagrams/index.html

[Jan Panteltje]

On a sunny day (Wed, 02 Aug 2023 22:56:57 -0400) it happened "David W.
Hodgins" <dwho...@nomail.afraid.org> wrote in
<op.182yg7s...@hodgins.homeip.net>:

+1

[The Natural Philosopher]

On 03/08/2023 07:44, Jan Panteltje wrote:
> On a sunny day (Wed, 2 Aug 2023 17:47:56 +0100) it happened The Natural
> Philosopher <t...@invalid.invalid> wrote in <uae1bt$5slp$1...@dont-email.me>:
>
>>
>>
>>>   ANYway, /var/log CAN be moved to a RAMdisk if you want.
>>>   Not 100% sure WHY you'd want to, but it CAN. If a few
>>>   very early logs get 'lost' as you re-direct /var/log
>>>   then that MIGHT not be all so important. If you want
>>>   it all on RAMdisk then you don't CARE if it all
>>>   vanishes on reboot. I very rarely look in /var/log
>>>   anyhow so ....
>>
>> As I said, whoever you are, the less writing to an SD drive there is,
>> the less chance there is of file system corruption when the power goes out.
>
> To avoid restarts on power failure use a UPS (I do) or just a simple battery circuit
> was it not called 'battery hat' or something? with the Pi.
>
>

Restarts are not a problem really. the whole central heating system dies
when the power goes out anyway.

If I wanted to run through that I'd UPS the whole damned thing.

>
>> Its simply a natural habit to ensure than whatever an embedded system
>> does is the minimum necessary to get the job done with the maximum
>> reliability.
>>
>> The SD card looks like it might be the weakest link. So I am trying to
>> reduce stress on it.
>

> So much bloat these days just to say 'hello world'

no argument there. Was there even at the time of MSDOS. I once traced
through the code that ran when you hit a key on the keyboard. Thousands
of instructions

> I designed a ram disk for the Z80, wrote a CP/M clone too
> https://panteltje.nl/panteltje/z80/system14/diagrams/index.html
>
>

Neat stuff


--
"I guess a rattlesnake ain't risponsible fer bein' a rattlesnake, but ah
puts mah heel on um jess the same if'n I catches him around mah chillun".

[Pancho]

On 02/08/2023 17:09, The Natural Philosopher wrote:
> On 01/08/2023 21:47, Pancho wrote:
>> On 7/29/23 19:45, Robert Heller wrote:
>>> This is in fact the default behaviour for Armbian on my Banana Pi M64.
>>>
>>
>> +1, :-)  I was just thinking, I spent ages undoing this, in order to
>> find out why my system was crashing. But I'm running Armbian.
>>
>> TNP, FFS, SD cards cost buttons. Pack of two 32GB for less than a
>> tenner. If you are worried about wear, clone a new one every year, or
>> whatever.
>>
> This product may end up in places where the user isn't smart enough to
> do that.
>

OK, I remember looking at this back in December and coming to the
conclusion it was hopeless. Micro SD cards (probably) don't do active
wear levelling + I wasn't clear what IO actually killed them. I seem to
remember, someone suggesting it was the EXT4 journal file that was the
problem rather than logs.

[Charlie Gibbs]

On 2023-08-03, The Natural Philosopher <t...@invalid.invalid> wrote:

> Interesting philosophy. accept failure and plan round it rather than
> eliminating failure in the first place?

That's old hat for anyone who's had to work with M$ products.

--
/~\ Charlie Gibbs | You can't save the earth
\ / <cgi...@kltpzyxm.invalid> | unless you're willing to
X I'm really at ac.dekanfrus | make other people sacrifice.
/ \ if you read it the right way. | -- Dogbert the green consultant

[Allodoxaphobia]

On Thu, 03 Aug 2023 16:44:42 GMT, Charlie Gibbs wrote:
> On 2023-08-03, The Natural Philosopher <t...@invalid.invalid> wrote:
>
>> Interesting philosophy. accept failure and plan round it rather than
>> eliminating failure in the first place?
>
> That's old hat for anyone who's had to work with M$ products.

For many years scientists have tried to perfect fault tolerant systems.
Meanwhile, Microsoft profits by exploiting fault tolerant customers.

[Ahem A Rivet's Shot]

On Thu, 3 Aug 2023 03:47:44 +0100

The Natural Philosopher <t...@invalid.invalid> wrote:

> Interesting philosophy. accept failure and plan round it rather than
> eliminating failure in the first place?

I've seen that as a programming design philosophy. The idea is that
it should be possible to kill the program at any time without depending on
it going through some cleanup/finish code it can just stop without losing
anything. The startup code always goes through the crash recovery and
everything necessary to recover must be flushed to disc before
acknowledging an input.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/
Host: Beautiful Theory meet Inconvenient Fact
Obit: Beautiful Theory died today of factual inconsistency

[The Natural Philosopher]

That is a slightly terrifying thought...does anyone have any more
information?
Mm. It would seem that while journalling multiplies the number of
writes, it wont impact what amounts to a read-only or 'write very
rarely' filesystem.
Phew.
Obviously of you are wring huge amounts of data over and over again you
would probably attach an SSD via USB anyway..

--
"The great thing about Glasgow is that if there's a nuclear attack it'll
look exactly the same afterwards."

Billy Connolly

[The Natural Philosopher]

On 03/08/2023 17:44, Charlie Gibbs wrote:
> On 2023-08-03, The Natural Philosopher <t...@invalid.invalid> wrote:
>
>> Interesting philosophy. accept failure and plan round it rather than
>> eliminating failure in the first place?
>
> That's old hat for anyone who's had to work with M$ products.
>

Well that's why we eliminated M$ - and installed Linux, innit?

--
“The urge to save humanity is almost always only a false face for the
urge to rule it.”
– H. L. Mencken

[Rich]

In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
> On 03/08/2023 10:43, Pancho wrote:
>> OK, I remember looking at this back in December and coming to the
>> conclusion it was hopeless. Micro SD cards (probably) don't do
>> active wear levelling + I wasn't clear what IO actually killed them.
>> I seem to remember, someone suggesting it was the EXT4 journal file
>> that was the problem rather than logs.
>
> That is a slightly terrifying thought...does anyone have any more
> information?
> Mm. It would seem that while journalling multiplies the number of
> writes, it wont impact what amounts to a read-only or 'write very
> rarely' filesystem.

The problem with the ext4 journal and a SD card /without wear leveling/
is the journal is in a fixed location on the disk, so any write
anywhere that is journaled also becomes a write onto the journal area
of the disk. So the flash cells underlying the area of the journal see
far more writes than the rest of the cells, and wear out sooner.

Now, *with wear leveling* those journal writes that are logically to
the same blocks on the disk get spread all over the flash cells via the
wear leveling controller, mitigating the issue of a lot of writes
happening to the same blocks on the disk. The write amplification
issue is still present in any case.

But agreed, as you've reduced the number of writes to a bare minimum,
you likely won't need to worry about whether your sd card has, or does
not have, a wear leveling controller. Few to zero writes means little
to no wear.

[David W. Hodgins]

On Thu, 03 Aug 2023 14:57:22 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> That is a slightly terrifying thought...does anyone have any more
> information?
> Mm. It would seem that while journalling multiplies the number of
> writes, it wont impact what amounts to a read-only or 'write very
> rarely' filesystem.
> Phew.
> Obviously of you are wring huge amounts of data over and over again you
> would probably attach an SSD via USB anyway..

I'm using 11 GB of a 32 GB sd card using ext4 on an rpi 4b. A little slow to boot
but otherwise performance pretty good. It's running kde plasma, with no workarounds
to avoid extra writes, just disabling things I don't like in plasma like desktop
effects and most background services.

# free -m
total used free shared buff/cache available
Mem: 3831 1289 1141 49 1400 2305
Swap: 6696 0 6696

It's currently running konversation and guvcview for a camera, but I've used
firefox on it too. Firefox is slow, but not unusable.

I use it mainly for konversation so I can do things on my main desktop system
while discussing things in irc.

I have some spare sd cards just in case and keep regular backups on my main
system. I've been running it with the same sd card since Feb. 2021.

The sd cards are in packages labeled Lexar high-performance 633x microSDHC UHS-I.

I have no idea if that does wear leveling.

The rpi 4b has much better performance than I expected given that it's using
an sd card. I was going to try xfce4 with it, but after seeing it's performance
decided to try kde plasma.

Using something like f2fs would probably be better, but ext4 is working fine for
me. Knock on wood. :-)

Regards, Dave Hodgins

[Pancho]

I guessed that the worry was that the journal was a smallish single
file, used as a circular buffer (maybe 128MB). So, without wear
levelling, the same few blocks would be getting hit all the time.
Meaning the SD card dies well below any quoted TB Written endurance stat.

But.., I appear to have been wrong, some micro SD cards do, indeed, do
wear levelling, the Kingston Industrial for example. But it is pricey
£24 quid for £64 GB (32 GB even more expensive).

Due to this high cost of the industrial SD, for my rPi, I still arrive
at the same conclusion, yearly replacement of a cheap SD.

[Ahem A Rivet's Shot]

On Thu, 3 Aug 2023 23:28:06 +0100
Pancho <Pancho...@Proton.Me> wrote:

> I guessed that the worry was that the journal was a smallish single
> file, used as a circular buffer (maybe 128MB). So, without wear
> levelling, the same few blocks would be getting hit all the time.

Remember write amplification - flash storage uses very large blocks
1MB or more so small writes involve copying most of a block, writing a new
one with the changes and queing the old one to be wiped.

[The Natural Philosopher]

On 03/08/2023 20:50, Rich wrote:
> In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
>> On 03/08/2023 10:43, Pancho wrote:
>>> OK, I remember looking at this back in December and coming to the
>>> conclusion it was hopeless. Micro SD cards (probably) don't do
>>> active wear levelling + I wasn't clear what IO actually killed them.
>>> I seem to remember, someone suggesting it was the EXT4 journal file
>>> that was the problem rather than logs.
>>
>> That is a slightly terrifying thought...does anyone have any more
>> information?
>> Mm. It would seem that while journalling multiplies the number of
>> writes, it wont impact what amounts to a read-only or 'write very
>> rarely' filesystem.
>
> The problem with the ext4 journal and a SD card /without wear leveling/
> is the journal is in a fixed location on the disk, so any write
> anywhere that is journaled also becomes a write onto the journal area
> of the disk. So the flash cells underlying the area of the journal see
> far more writes than the rest of the cells, and wear out sooner.
>

Mmm. But that goes for directory entries which get time stamped, as well
I think any card with NO wear levelling - that is no mapping from
logical to physical sectors - is not going to work as a card at all,
because the OS will be dealing in much smaller sectors than the SD card,
and the card has to erase large blocks.

So there must be some processor shit going on to map logical to physical
blocks and it is then no cost saving to have shit software.
Unfortunately that means you need to have somewhere to store that
mapping, and THAT area will get a lot of writes. Dunno how they manage that.


I suspect the cheaper chips have substandard NVRAM

> Now, *with wear leveling* those journal writes that are logically to
> the same blocks on the disk get spread all over the flash cells via the
> wear leveling controller, mitigating the issue of a lot of writes
> happening to the same blocks on the disk. The write amplification
> issue is still present in any case.
>
> But agreed, as you've reduced the number of writes to a bare minimum,
> you likely won't need to worry about whether your sd card has, or does
> not have, a wear leveling controller. Few to zero writes means little
> to no wear.

Yes. When I thought about what was happening in the application, the
user makes occasional config changes, but apart from that, nothing needs
to change. If it goes wrong, unless it's me, logs are pointless. And I
am keeping RAM logs, which are worth it for live debugging.


--
"When one man dies it's a tragedy. When thousands die it's statistics."

Josef Stalin

[The Natural Philosopher]

I dunno how fast an SSD with a USB interface would be compared with a
SATA, but if I was using a pi for user level stuff, I'd want some kind
of SSD in there with the SD card only there to boot the thing.

My impressions is that a Pi is about as fast as a 486 used to be. or
maybe a bit more. Many people say the latest Pi is pentium 4 level or
thereabouts.


--
In theory, there is no difference between theory and practice.
In practice, there is.
-- Yogi Berra

[The Natural Philosopher]

On 03/08/2023 23:59, Ahem A Rivet's Shot wrote:
> On Thu, 3 Aug 2023 23:28:06 +0100
> Pancho <Pancho...@Proton.Me> wrote:
>
>> I guessed that the worry was that the journal was a smallish single
>> file, used as a circular buffer (maybe 128MB). So, without wear
>> levelling, the same few blocks would be getting hit all the time.
>
> Remember write amplification - flash storage uses very large blocks
> 1MB or more so small writes involve copying most of a block, writing a new
> one with the changes and queing the old one to be wiped.
>

Indeed.

Oddly enough my SSD drives seem to be lasting better than the spinning
rust they replaced. The wear levelling in THOSE really works.
So my tentative feeling at this point in time is that while /boot might
be on the SD card, if I were to use a pi for serious R/W everything else
would be an SSD mounted somehow at boot. I believe with later PIs you
can boot the whole thing from USB/SSD or rust.

For me that leads to a sort of tentative conclusions - if I want a busy
server, or user desktop, I'd pick a late model Pi and USB boot it. If I
want an embedded device, I'd pick an SD card and tune the OS not to use
it, if possible.

And for serious storage, the later OS supports SATA hats...




--
The urge to save humanity is almost always a false front for the urge to
rule.
– H. L. Mencken, American journalist, 1880-1956

[23k.304]

On 8/3/23 2:44 AM, Jan Panteltje wrote:
> On a sunny day (Wed, 2 Aug 2023 17:47:56 +0100) it happened The Natural
> Philosopher <t...@invalid.invalid> wrote in <uae1bt$5slp$1...@dont-email.me>:
>
>>
>>
>>>   ANYway, /var/log CAN be moved to a RAMdisk if you want.
>>>   Not 100% sure WHY you'd want to, but it CAN. If a few
>>>   very early logs get 'lost' as you re-direct /var/log
>>>   then that MIGHT not be all so important. If you want
>>>   it all on RAMdisk then you don't CARE if it all
>>>   vanishes on reboot. I very rarely look in /var/log
>>>   anyhow so ....
>>
>> As I said, whoever you are, the less writing to an SD drive there is,
>> the less chance there is of file system corruption when the power goes out.
>
> To avoid restarts on power failure use a UPS (I do) or just a simple battery circuit
> was it not called 'battery hat' or something? with the Pi.

I remember the "battery hat" - still sold :

https://www.waveshare.com/li-ion-battery-hat.htm

But MOST Pi's, because of the high power consumption,
still run off the mains - so a UPS is probably the
simplest option. The "battery hat" however might
serve to deal with very short interruptions.

However I don't think the most stress to SD cards
is on boot - but during regular USE ... the usual
data churning and loading system apps from the
card. If "ping" is used, well, where does it COME
from on a PI ? The SD card. Each application has
to be examined to see what routines are used and
put them into a RAMdisk or whatever. Remember,
even reading an SD card involves re-writing the
thing, that's how the tech works.

>> Its simply a natural habit to ensure than whatever an embedded system
>> does is the minimum necessary to get the job done with the maximum
>> reliability.
>>
>> The SD card looks like it might be the weakest link. So I am trying to
>> reduce stress on it.
>
> For many things Pi is too much power, much can be done with a simple PIC 18F14K22
> https://panteltje.nl/panteltje/pic/index.html
> take scope_pic or freq_pic from that site.
> *IF* you like to code in asm :-)
> Only milli-amps curent consumption
> https://panteltje.nl/panteltje/pic/gm_pic2/
>
> So much bloat these days just to say 'hello world'
> I designed a ram disk for the Z80, wrote a CP/M clone too
> https://panteltje.nl/panteltje/z80/system14/diagrams/index.html

Hey, I *remember* those days ! Bill Gates kinda rose
to fame because there used to be contests over how to
write basic functions in the very least number of
bytes/cycles. Consider how TINY his BASIC was at the
start, yet could DO so much. These days - ten or more
times the pork for barely more function and if it
doesn't have a giant slick GUI nobody wants to use it.

Yea, Bill became a total asshole, but at the beginning ...

I've always liked micro-controllers. Very little RAM/ROM
or even speed. The goal is to use every trick to squeeze
as much function from every single byte as possible. That
is an entirely different philosophy from what we commonly
see today.

Used to do ASM on micro-controllers, but over the past
decade the better 'C' compilers are actually smarter,
can do the same in even fewer bytes/cycles than most
human-writ code. Not by a HUGE margin, but, on tiny
devices, maybe enough.

[23k.304]

Yep, and FAMILIAR too. This was the kind of stuff
"developers" had to deal with back in the day, right
down to making the hardware. No "standard drivers"
or off-the-shelf plug-in cards back then worth shit.
If it was worth doing you had to do it ALL yourself
from the crappy chips on up.

Such INTERESTING days ! :-)

I'd say maybe 1830-1900 were equally interesting for
the old MECHANICAL developers. Ah to be on Edison's
team (even though he was an asshole) !!! :-)

Some new "interesting days", or "day", may be yet to
come - but it's in "AI" development. After that
there's nothing left (for humans). It'll be
robotopia, or the "Savage Reservation" ....

[Jan Panteltje]

On a sunny day (Fri, 4 Aug 2023 01:08:14 -0400) it happened "23k.304"
<23k...@bfxw9.net> wrote in <x4mcnaG1KMJQGlH5...@earthlink.com>:

I dunno..
I have an old Pi uname -a says Feb 2013.. so >10 years, on 24/7,
https://panteltje.nl/panteltje/xgpspc/index.html
I use it to monitor things like air traffic, shipping traffic,
weather data like air pressure, temperature etc, runs as server.
It ran on some old sdcard then when I moved house some years ago
I replaced that sdcard wih a Samsung one, think reason was I messed up something.
It does log weather all the time..
An other Pi as client logs everything else.
Some screenshots:
panteltje.nl/pub/xgpspc_functie_H.gif
panteltje.nl/pub/boats_and_planes.gif
panteltje.nl/pub/xgpspc_5_planes.gif

But never a problem with the old card and the Samsung SD cards.
This I write on a Pi4 8 GB with the Usenet news reader I wrote that I ported to it:
https://panteltje.nl/panteltje/newsflex/index.html
32 GB Samsung SDcard..
There is several GB data - Usenet posts - from the last 20 years or so and updated every time I go online and post or read here.
firefox or chromium browser used every day,
I do have a Toshiba 4 TB harddisk connected but most writing from firefox and downloads I do go to the SDcard,
then are backed up to that Toshiba.

raspberrypi: ~ # df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 30446232 23535028 5566020 81% /
...
...
/dev/mmcblk0p1 258095 50411 207685 20% /boot
tmpfs 808848 20 808828 1% /run/user/1000
/dev/sdb2 3844510712 2411345180 1237804868 67% /mnt/sda2
raspberrypi: ~ # uname -a
Linux raspberrypi 5.15.32-v7l+ #1538 SMP Thu Mar 31 19:39:41 BST 2022 armv7l GNU/Linux

>
>>> Its simply a natural habit to ensure than whatever an embedded system
>>> does is the minimum necessary to get the job done with the maximum
>>> reliability.
>>>
>>> The SD card looks like it might be the weakest link. So I am trying to
>>> reduce stress on it.

Well we will see....

It makes sense to backup your data on a regular basis and make a copy of the card every month? year?
Card hardware failures are rare, I suspect more failures are with the filesystem.
I have a 20 year? old 16 GB Duracell USB stick with Reiserfs that seems to always work, used every day.
Some ext4 I have had serious problems with.. Becoming unreadable.

[David W. Hodgins]

On Thu, 03 Aug 2023 22:29:31 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> I dunno how fast an SSD with a USB interface would be compared with a
> SATA, but if I was using a pi for user level stuff, I'd want some kind
> of SSD in there with the SD card only there to boot the thing.

sd cards are a lot cheaper. :-)

> My impressions is that a Pi is about as fast as a 486 used to be. or
> maybe a bit more. Many people say the latest Pi is pentium 4 level or
> thereabouts.

This is an rpi 4b, which is a quad core (1 thread per core) and 4GB ram.

lscpu shows 108.0 BogoMIPS, while my desktop system with an AMD FX(tm)-4170
Quad-Core Processor from 2012 shows 8428.66 BoboMIPS.
Despite the low number of BogoMIPS, I'm still impressed by it overall.

It's fine for running the few applications I use it for.

Regards, Dave Hodgins

[Ahem A Rivet's Shot]

On Fri, 4 Aug 2023 03:38:17 +0100

The Natural Philosopher <t...@invalid.invalid> wrote:

> For me that leads to a sort of tentative conclusions - if I want a busy
> server, or user desktop, I'd pick a late model Pi and USB boot it. If I
> want an embedded device, I'd pick an SD card and tune the OS not to use
> it, if possible.

That seems right to me too.

[23k.304]

On 8/3/23 5:08 AM, The Natural Philosopher wrote:

Heh, heh ... ALWAYS a lot more complicated than it SEEMS :-)

On the old IBM-PCs there were a bunch, a whole bunch, of
BIOS routines. These were pretty well optimized code
due to size limits of the time. Using ASM you could
exploit them nicely - get KB input, do X-Y output to
the monitor, like all the stuff you'd need for a
smart WYSIWYG text editor, all from these built-in
routines.

Still have the old IBM Technical Reference Manual.
All that stuff was meticulously documented.

[The Natural Philosopher]

I am not sure it actually does. If you turn off 'last read' with 'noatime'
Which you surely would if you cared enough

>   Used to do ASM on micro-controllers, but over the past
>   decade the better 'C' compilers are actually smarter,
>   can do the same in even fewer bytes/cycles than most
>   human-writ code. Not by a HUGE margin, but, on tiny
>   devices, maybe enough.

Oh yes. The days of trying to write C that didn't turn into bloat on an
8 bit 6809...are long gone

I occasionally looked at .a files on *86, and they wrote better
assembler than I could.

So much so that I gave up looking.


--
“There are two ways to be fooled. One is to believe what isn’t true; the
other is to refuse to believe what is true.”

—Soren Kierkegaard

[The Natural Philosopher]

On 04/08/2023 04:15, David W. Hodgins wrote:
> On Thu, 03 Aug 2023 22:29:31 -0400, The Natural Philosopher
> <t...@invalid.invalid> wrote:
>> I dunno how fast an SSD with a USB interface would be compared with a
>> SATA, but if I was using a pi for user level stuff, I'd want some kind
>> of SSD in there with the SD card only there to boot the thing.
>
> sd cards are a lot cheaper. :-)

But like a bicycle compared to a Ferrari, they are a lot slower, too
SSD simply flies.

>
>> My impressions is that a Pi is about as fast as a 486 used to be. or
>> maybe a bit more. Many people say the latest Pi is pentium 4 level or
>> thereabouts.
>
> This is an rpi 4b, which is a quad core (1 thread per core) and 4GB ram.
>
> lscpu shows 108.0 BogoMIPS, while my desktop system with an AMD FX(tm)-4170
> Quad-Core Processor from 2012 shows 8428.66 BoboMIPS.
> Despite the low number of BogoMIPS, I'm still impressed by it overall.
>
> It's fine for running the few applications I use it for.
>
> Regards, Dave Hodgins

Pi Zero W: BogoMIPS: 697.95
My desktop - refurbed HP: BogoMIPS: 5399.81

But it makes no sense - my old server which is nowher as fast show s
more bogomips


--
“Politics is the art of looking for trouble, finding it everywhere,
diagnosing it incorrectly and applying the wrong remedies.”
― Groucho Marx

[The Natural Philosopher]

On 04/08/2023 07:03, 23k.304 wrote:
> Still have the old IBM Technical Reference Manual.
>   All that stuff was meticulously documented.

I gave that away sometime in the late 80s.

--
“Progress is precisely that which rules and regulations did not foresee,”

– Ludwig von Mises

[The Natural Philosopher]

On 04/08/2023 09:51, The Natural Philosopher wrote:

> I am not sure it actually does. If you turn off 'last read' with 'noatime'
> Which you surely would if you cared enough

My Pi Zero W came with 'noatime' configured

root@heating-controller:~# more /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=b8c9fbb7-01 /boot vfat defaults 0 2
PARTUUID=b8c9fbb7-02 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
tmpfs /var/www/data/volatile tmpfs
nodev,nosuid,noexec,nodiratime,size=1M 0 0
tmpfs /var/ramlog tmpfs nodev,nosuid,noexec,nodiratime,size=25M 0 0

It seems Raspios has pretty decent defaults.

This is good because I will be reading files a * lot*. Probably 16 a second

PS I also managed to find out from the net how to set the PI Zeros USB
port to look like an ethernet adapter and service DHCP requests, so it
can be a web sever to a PC just by plugging it into the PC's USB port.
Great for setting up the wifi!
If anyone else wants to know, ask.

[Jan Panteltje]

On a sunny day (Fri, 4 Aug 2023 09:57:30 +0100) it happened The Natural
Philosopher <t...@invalid.invalid> wrote in <uaiehq$170d6$6...@dont-email.me>:

>On 04/08/2023 04:15, David W. Hodgins wrote:
>> On Thu, 03 Aug 2023 22:29:31 -0400, The Natural Philosopher
>> <t...@invalid.invalid> wrote:
>>> I dunno how fast an SSD with a USB interface would be compared with a
>>> SATA, but if I was using a pi for user level stuff, I'd want some kind
>>> of SSD in there with the SD card only there to boot the thing.
>>
>> sd cards are a lot cheaper. :-)
>
>But like a bicycle compared to a Ferrari, they are a lot slower, too
>SSD simply flies.
>
>>
>>> My impressions is that a Pi is about as fast as a 486 used to be. or
>>> maybe a bit more. Many people say the latest Pi is pentium 4 level or
>>> thereabouts.
>>
>> This is an rpi 4b, which is a quad core (1 thread per core) and 4GB ram.
>>
>> lscpu shows 108.0 BogoMIPS, while my desktop system with an AMD FX(tm)-4170
>> Quad-Core Processor from 2012 shows 8428.66 BoboMIPS.
>> Despite the low number of BogoMIPS, I'm still impressed by it overall.
>>
>> It's fine for running the few applications I use it for.
>>
>> Regards, Dave Hodgins
>
>Pi Zero W: BogoMIPS: 697.95
>My desktop - refurbed HP: BogoMIPS: 5399.81
>
>But it makes no sense - my old server which is nowher as fast show s
>more bogomips

My Pi4 8GB I am posting this from says:
raspberrypi: ~ # while [ 1 ] ; do cat /proc/cpuinfo | grep BogoMIPS ; sleep 1 ; done
BogoMIPS : 216.00
BogoMIPS : 324.00
BogoMIPS : 324.00
BogoMIPS : 324.00
BogoMIPS : 324.00
BogoMIPS : 144.00
BogoMIPS : 144.00
BogoMIPS : 144.00
BogoMIPS : 144.00
BogoMIPS : 180.00
BogoMIPS : 180.00
BogoMIPS : 180.00
BogoMIPS : 180.00
BogoMIPS : 198.00

raspberrypi: ~ # lscpu
Architecture: armv7l
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: ARM
Model: 3
Model name: Cortex-A72
Stepping: r0p3
CPU max MHz: 1800.0000
CPU min MHz: 600.0000
BogoMIPS: 126.00


??
And my core i5 laptop says:
4788.51
runs an old Slackware version

So much for BogoMIPS
:-)

[Martin Gregorie]

On Fri, 4 Aug 2023 03:38:17 +0100, The Natural Philosopher wrote:

> Oddly enough my SSD drives seem to be lasting better than the spinning
> rust they replaced. The wear levelling in THOSE really works.
> So my tentative feeling at this point in time is that while /boot might
> be on the SD card, if I were to use a pi for serious R/W everything else
> would be an SSD mounted somehow at boot. I believe with later PIs you
> can boot the whole thing from USB/SSD or rust.
>

IIRC it used to be well known that SD cards (class 10 cards?) that were
sold for use in video cameras etc. had:

(a) large storage blocks and
(b) no wear levelling

In consequence these cards are not generally useful for use as computer
filing systems, being designed to hold large (MB to GB) data files that
would only be written to sequentially and would usually not be edited in
situ, just deleted in order to make room for recording another video. In
consequence only file level operations (record a video, copy or delete a
complete file) worked at an acceptable speed.

--

Martin | martin at
Gregorie | gregorie dot org

[Chris Elvidge]

Slackware used to do BogoMIPS at startup (perhaps still does) but then
went on to say measuring BogoMIPS was a waste of time.


--

Chris Elvidge, England
I WILL NOT SELL LAND IN FLORIDA

[Chris Elvidge]

On 04/08/2023 03:38, The Natural Philosopher wrote:
> On 03/08/2023 23:59, Ahem A Rivet's Shot wrote:
>> On Thu, 3 Aug 2023 23:28:06 +0100
>> Pancho <Pancho...@Proton.Me> wrote:
>>
>>> I guessed that the worry was that the journal was a smallish single
>>> file, used as a circular buffer (maybe 128MB). So, without wear
>>> levelling, the same few blocks would be getting hit all the time.
>>
>> Remember write amplification - flash storage uses very large blocks
>> 1MB or more so small writes involve copying most of a block, writing a
>> new
>> one with the changes and queing the old one to be wiped.
>>
> Indeed.
>
> Oddly enough my SSD drives seem to be lasting better than the spinning
> rust they replaced. The wear levelling in THOSE really works.
> So my tentative feeling at this point in time is that while /boot might
> be on the SD card, if I were to use a pi for serious R/W everything else
> would be an SSD mounted somehow at boot.

That's what I do

My cmdline.txt:

console=tty1 root=PARTUUID=72020302-c396-2148-9d13-caf4f68cd18c
rootfstype=ext4 fsck.repair=yes rootwait quiet splash
plymouth.ignore-serial-consoles net.ifnames=0 biosdevname=0
usbhid.mousepoll=0 dwc_otg.lpm_enable=0

The PARTUUID is that of half a 120Gb SSD (/dev/sda2). /boot is on a
250Mb SD card.

It was (seemed) easier than trying to get USB boot to work (RPi3B).

> I believe with later PIs you
> can boot the whole thing from USB/SSD or rust.

> For me that leads to a sort of tentative conclusions - if I want a busy
> server, or user desktop, I'd pick a late model Pi and USB boot it. If I
> want an embedded device, I'd pick an SD card and tune the OS not to use
> it, if possible.
>
> And for serious storage, the later OS supports SATA hats...
>
>
>
>


--

[Ahem A Rivet's Shot]

On Fri, 4 Aug 2023 09:51:48 +0100
The Natural Philosopher <t...@invalid.invalid> wrote:

> I am not sure it actually does. If you turn off 'last read' with 'noatime'
> Which you surely would if you cared enough

Oh that brings back a memory from a long time back. On a visit to
Altos I saw a machine that one of the engineers there was playing with - it
was a XENIX machine with four eight inch floppy drives as storage - the
labels stuck over the drives read (/ /usr /etc and /tmp) - He'd had to turn
of atime because it was wearing out the tracks holding the inodes.

[David W. Hodgins]

On Fri, 04 Aug 2023 06:26:42 -0400, Jan Panteltje <al...@comet.invalid> wrote:
> So much for BogoMIPS

Agreed. BogoMIPS is useless for showing the cpu speed. It's much faster than
I expected, which allowed me to use it for much more than simple testing of
aarch64 installs, which is all I bought it for.

Regards, Dave Hodgins

[David W. Hodgins]

On Fri, 04 Aug 2023 04:57:30 -0400, The Natural Philosopher <t...@invalid.invalid> wrote:
> But like a bicycle compared to a Ferrari, they are a lot slower, too
> SSD simply flies.

Agreed. I use multiple ssd drives on my main system. The difference in speed
between a spinning rust drive and an ssd drive is well worth it. As I do
a lot of installs and other testing on that system, the speed is worth it.

I have as my main drive an OCZ-AGILITY4 240GB SSD drive that I've been using
since 2011.

I back it up regularly as I've been expecting it to fail anytime for years. :-)
smartctl shows ...
5 Reallocated_Sector_Ct 0x0000 100 100 000 Old_age Offline - 1
9 Power_On_Hours 0x0000 100 100 000 Old_age Offline - 89199
12 Power_Cycle_Count 0x0000 100 100 000 Old_age Offline - 489
232 Lifetime_Writes 0x0000 100 100 000 Old_age Offline - 93035004612

The Reallocated_Sector_Ct has been at 1 since shortly after I started using it.
The spinning rust drive it replaced only lasted about 5 years.

For the rpi 4b, I'd be looking at an external enclosure connected via usb, so
it wouldn't see the speed of a sata or pcie ssd, but would be a lot faster
than the sd card, but for what I use it for, not worth the price.

Regards, Dave Hodgins

[Computer Nerd Kev]

In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
> On 04/08/2023 04:15, David W. Hodgins wrote:
>> lscpu shows 108.0 BogoMIPS, while my desktop system with an AMD FX(tm)-4170
>> Quad-Core Processor from 2012 shows 8428.66 BoboMIPS.
>> Despite the low number of BogoMIPS, I'm still impressed by it overall.
>>
>> It's fine for running the few applications I use it for.
>

> Pi Zero W: BogoMIPS: 697.95
> My desktop - refurbed HP: BogoMIPS: 5399.81
>
> But it makes no sense - my old server which is nowher as fast show s
> more bogomips

BogoMIPS on ARM is known to show incorrect results. The CPU tricks
the routine that Linux uses to calculate it, so at least it's
meaningless compared to the number given on architectures like x86,
or actual Million Instructions Per Second. Somehow the Linux
developers haven't been able to fix this.

"In 2012, ARM contributed a new udelay implementation allowing the
system timer built into many ARMv7 CPUs to be used instead of a
busy-wait loop. This implementation was released in Version 3.6 of
the Linux kernel. Timer-based delays are more robust on systems that
use frequency scaling to dynamically adjust the processor's speed at
runtime, as loops_per_jiffies values may not necessarily scale
linearly. Also, since the timer frequency is known in advance, no
calibration is needed at boot time.

One side effect of this change is that the BogoMIPS value will reflect
the timer frequency, not the CPU's core frequency. Typically the
timer frequency is much lower than the processor's maximum frequency,
and some users may be surprised to see an unusually low BogoMIPS value
when comparing against systems that use traditional busy-wait loops."
https://en.wikipedia.org/wiki/BogoMips

The core frequency on different models isn't directly correlated to
the CPU's clock frequency on ARM (it gets multiplied), so it's pretty
much meaningless.

--
__ __
#_ < |\| |< _#

[Computer Nerd Kev]

In comp.os.linux.misc The Natural Philosopher <t...@invalid.invalid> wrote:
> My impressions is that a Pi is about as fast as a 486 used to be.

Perhaps it might seem as fast as a 486 running Linux from the mid
90s would be. In practice even the Pi Zero should be able to emulate
that 486 and run the same software at a usable speed. Linux and all
the software bolted onto it has been getting ever slower over the
years, but it can seem much faster on the Pi Zero when running more
minimal distros than RPi OS (OpenWrt, Tiny Core).

[23k.304]

On 8/4/23 4:58 AM, The Natural Philosopher wrote:
> On 04/08/2023 07:03, 23k.304 wrote:
>> Still have the old IBM Technical Reference Manual.
>>    All that stuff was meticulously documented.
>
> I gave that away sometime in the late 80s.

I consider it to be a "valuable artifact" :-)

At minimum, it showed how serious and deep they'd
go with the early PCs. That's when programming was
still painfully "real". Mostly you could not buy
what you needed - so you had to make it yourself.

And I *did* find many uses for that info.

[The Natural Philosopher]

That seems a logical best solution for an early Pi. The later ones will
do USB boot, I think.

(where on earth did you get a 250MB SD card: I cant find anything much
less than gigabytes these days)


--
For in reason, all government without the consent of the governed is the
very definition of slavery.

Jonathan Swift

[The Natural Philosopher]

Oh, so did I. I was assembly programming 8086s then - writing BIOS
extensions or complete BIOSES for them,

> https://www.youtube.com/watch?v=WUVZbBBHrI4

Or interfacing with special hardware people had built that needed
drivers written.
But that line of business dried up, so I ended up doing yet another
career shift into running IT businesses.


--
Climate Change: Socialism wearing a lab coat.

[Ahem A Rivet's Shot]

IBM simply provided the same level of documentation for the PC as
they did for their mainframes as a matter of habit I think. That made life a
lot easier for the clone makers who popped up like mushrooms almost
immediately.

[Ahem A Rivet's Shot]

On 5 Aug 2023 10:17:23 +1000

n...@telling.you.invalid (Computer Nerd Kev) wrote:

> BogoMIPS on ARM is known to show incorrect results. The CPU tricks
> the routine that Linux uses to calculate it, so at least it's
> meaningless compared to the number given on architectures like x86,
> or actual Million Instructions Per Second. Somehow the Linux
> developers haven't been able to fix this.

MIPs was a pretty useless concept long before Linux appeared.