nvme SSD and poor performance

[Pierre Willaime]

Hi,

I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
(bulleye now).

For a long time, I suffer poor I/O performances which slow down a lot of
tasks (apt upgrade when unpacking for example).

I am now trying to fix this issue.

Using fstrim seems to restore speed. There are always many GiB which are
reduced :

# fstrim -v /
/ : 236,7 GiB (254122389504 octets) réduits

then, directly after :

# fstrim -v /
/ : 0 B (0 octets) réduits

but few minutes later, there are already 1.2 Gib to trim again :

# fstrim -v /
/ : 1,2 GiB (1235369984 octets) réduits


/Is it a good idea to trim, if yes how (and how often)?/

Some people use fstrim as a cron job, some other add "discard" option to
the /etc/fstab / line. I do not know what is the best if any. I also
read triming frequently could reduce the ssd life.



I also noticed many I/O access from jbd2 and kworker such as :

# iotop -bktoqqq -d .5
11:11:16 364 be/3 root 0.00 K/s 7.69 K/s 0.00 % 23.64 %
[jbd2/nvme0n1p2-]
11:11:16 8 be/4 root 0.00 K/s 0.00 K/s 0.00 % 25.52 %
[kworker/u32:0-flush-259:0]

The percentage given by iotop (time the thread/process spent while
swapping in and while waiting on I/O) is often high.

I do not know what to do for kworker and if it is a normal behavior. For
jdb2, I have read it is filesystem (ext4 here) journal.

I added the "noatime" option to /etc/fstab / line but it does not seem
to reduce the number of access.

Regards,
Pierre


P-S: If triming it is needed for ssd, why debian do not trim by default?

[Dan Ritter]

Pierre Willaime wrote:
> Hi,
>
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).
>
> I am now trying to fix this issue.

The first question is, how slow is this storage?


Here is a good article on using fio:
https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/

> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
>
> # fstrim -v /
> / : 236,7 GiB (254122389504 octets) réduits
>
> then, directly after :
>
> # fstrim -v /
> / : 0 B (0 octets) réduits

That's normal.

> but few minutes later, there are already 1.2 Gib to trim again :
>
> # fstrim -v /
> / : 1,2 GiB (1235369984 octets) réduits

That's high but not impossible.

> /Is it a good idea to trim, if yes how (and how often)?/

I trim in a cron job once per week on high activity systems and
once per month on lower activity systems.

> I also noticed many I/O access from jbd2 and kworker such as :
>
> # iotop -bktoqqq -d .5
> 11:11:16 364 be/3 root 0.00 K/s 7.69 K/s 0.00 % 23.64 %
> [jbd2/nvme0n1p2-]
> 11:11:16 8 be/4 root 0.00 K/s 0.00 K/s 0.00 % 25.52 %
> [kworker/u32:0-flush-259:0]
>
> The percentage given by iotop (time the thread/process spent while swapping
> in and while waiting on I/O) is often high.

These are the kernel processes responsible for writing to disk.

> I added the "noatime" option to /etc/fstab / line but it does not seem to
> reduce the number of access.

It does, but not much more than the Debian default.

Test your drive's speed, then come back with the results.

-dsr-

[Pierre Willaime]

Le 17/08/2021 à 14:02, Dan Ritter a écrit :
> The first question is, how slow is this storage?
>
>
> Here is a good article on using fio:
> https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/

Thanks for the help.

Here are the output of fio tests.


Single 4KiB random write process test:

WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s),
io=12.0GiB (12.9GB), run=62271-62271msec

https://pastebin.com/5Cyg9Xvt


16 parallel 64KiB random write processes (two different results, further
tests are closer to the second than the first):

WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s),
io=35.5GiB (38.1GB), run=63568-69533msec

WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s),
io=14.3GiB (15.3GB), run=60871-72618msec


https://pastebin.com/XVpPpqsC
https://pastebin.com/HEx8VvhS


Single 1MiB random write process:

WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s),
io=16.0GiB (17.2GB), run=60722-60722msec

https://pastebin.com/skk6mi7M

[Linux-Fan]

Pierre Willaime writes:

> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).
>
> I am now trying to fix this issue.
>
> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :

[...]

> but few minutes later, there are already 1.2 Gib to trim again :
>
> # fstrim -v /
> / : 1,2 GiB (1235369984 octets) réduits
>

> Is it a good idea to trim, if yes how (and how often)?
>

> Some people use fstrim as a cron job, some other add "discard" option to the
> /etc/fstab / line. I do not know what is the best if any. I also read
> triming frequently could reduce the ssd life.

I do `fstrim` once per week by a minimalistic and custom script as cron job:
https://github.com/m7a/lp-ssd-optimization

There is no need for custom scripts anymore, you can nowdays enable the
timer from `util-linux` without hassle:

# systemctl enable fstrim.timer

This will perform the trim once per week by default.

When the use of SSDs increased people tried out the `discard` options and
found them to have strange performance characteristics, potential negative
effects on SSD life and in some cases even data corruption (?) Back then it
was recommended to use the periodic `fstrim` instead. I do not know if any
of the issues with discard are still there today.

> I also noticed many I/O access from jbd2 and kworker such as :
>
> # iotop -bktoqqq -d .5
> 11:11:16 364 be/3 root 0.00 K/s 7.69 K/s 0.00 % 23.64
> % [jbd2/nvme0n1p2-]
> 11:11:16 8 be/4 root 0.00 K/s 0.00 K/s 0.00 % 25.52
> % [kworker/u32:0-flush-259:0]

[...]

> I do not know what to do for kworker and if it is a normal behavior. For
> jdb2, I have read it is filesystem (ext4 here) journal.

I highly recommend you to find out what exactly is causing the high number
of I/O operations. Usually there is a userspace process responsible (or a
RAID resync operation) for all the I/O which is then processed by the kernel
threads you see in iotop.

I usually look at `atop -a 4` (package `atop`) for half a minute or so to
find out what processes are active on the system. It is possible that
something is amiss and causing an exceedingly high I/O load leading to the
performance degradation you observe.

[...]

> P-S: If triming it is needed for ssd, why debian do not trim by default?

Detecting reliably if the current system has SSDs that would benefit from
trimming AND that the user has not taken their own measures is difficult. I
guess this might be the reason for there not being an automatism, but you
can enable the systemd timer suggested above with a single command.

HTH
Linux-Fan

öö

[Marco Möller]

I have no experience with SSD, but running my Debian Desktop from a USB
Memory Stick since years, please allow me to share information which
supports the suggestion of Linux-Fan to also investigate if there is
extraordinary I/O taking place and maybe could be avoided:
In the past I found extreme(!) I/O to be produced by Firefox, when it is
writing its cache, and when it is writing its session restore
information. These writes to my observation occure all the time, kind of
nonstop! I could get it very satisfactorily reduced by applying a tool
called "Profile Sync Daemon" (psd) from package "profile-sync-daemon".
While it is packaged for Debian, as a starting point to study its
documentation I suggest to better look it up in the Arch Linux Wiki.

Good Luck!
Marco

[Dan Ritter]

Pierre Willaime wrote:
> Le 17/08/2021 à 14:02, Dan Ritter a écrit :
> > The first question is, how slow is this storage?
> >
> >
> > Here is a good article on using fio:
> > https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
>
> Thanks for the help.
>
> Here are the output of fio tests.
>
> Single 4KiB random write process test:
>
> WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s),
> io=12.0GiB (12.9GB), run=62271-62271msec
>

> 16 parallel 64KiB random write processes (two different results, further
> tests are closer to the second than the first):
>
> WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s),
> io=35.5GiB (38.1GB), run=63568-69533msec
>
> WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s),
> io=14.3GiB (15.3GB), run=60871-72618msec
>

> Single 1MiB random write process:
>
> WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s),
> io=16.0GiB (17.2GB), run=60722-60722msec

These numbers look reasonable for a low-end SSD that
manufacturers throw into their laptops -- which is what your
drive is, as far as I can tell. I can't find specs, but I
suspect it is a QLC set with a small DRAM cache.

I don't think you have a significant performance problem, but
you are definitely feeling some pain -- so can you tell us more
about what feels slow? Does it happen during the ordinary course
of the day?


-dsr-

[Stefan Monnier]

> P-S: If triming it is needed for ssd, why debian do not trim by default?

AFAIK trimming is not needed. It can be beneficial in some cases, but
as a general rule, the SSDs should be able to provide great performance
without it.


Stefan

[Christian Britz]

On 17.08.21 at 15:30 Linux-Fan wrote:
> Pierre Willaime writes:
>

>> P-S: If triming it is needed for ssd, why debian do not trim by default?
>
> Detecting reliably if the current system has SSDs that would benefit
> from trimming AND that the user has not taken their own measures is
> difficult. I guess this might be the reason for there not being an
> automatism, but you can enable the systemd timer suggested above with
> a single command.

I am pretty sure that I have never played with fstrim.timer and this is
the output of "systemctl status fstrim.timer" on my bullseye system:

● fstrim.timer - Discard unused blocks once a week
     Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor
preset: enabled)
     Active: active (waiting) since Tue 2021-08-17 14:10:17 CEST; 3h
5min ago
    Trigger: Mon 2021-08-23 01:01:39 CEST; 5 days left
   Triggers: ● fstrim.service
       Docs: man:fstrim

So it seems this weekly schedule is enabled by default on bullseye.
BTW, if my system is not online at that time, will it be triggered on
next boot?

[Linux-Fan]

Nice, thanks for sharing :)

> BTW, if my system is not online at that time, will it be triggered on next
> boot?

Yes, I would think so.

A systemd timer can be configured to run if its schedule is missed by
`[Timer] Persistent=true` which on my machine is configured for
`fstrim.timer` (check `systemctl cat fstrim.timer`).

See also: https://jeetblogs.org/post/scheduling-jobs-cron-anacron-systemd/

Of course, it is also possible to find out experimentally. Check
`systemctl list-timers` to find out about all registered timers and when
they ran last.

HTH
Linux-Fan

öö

[David Christensen]

On 8/17/21 2:54 AM, Pierre Willaime wrote:
> Hi,
>
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).
>
> I am now trying to fix this issue.
>

> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
>
> # fstrim -v /
> / : 236,7 GiB (254122389504 octets) réduits
>
> then, directly after :
>
> # fstrim -v /
> / : 0 B (0 octets) réduits
>

> but few minutes later, there are already 1.2 Gib to trim again :
>
> # fstrim -v /
> / : 1,2 GiB (1235369984 octets) réduits
>
>

> /Is it a good idea to trim, if yes how (and how often)?/
>

> Some people use fstrim as a cron job, some other add "discard" option to
> the /etc/fstab / line. I do not know what is the best if any. I also
> read triming frequently could reduce the ssd life.
>
>
>

> I also noticed many I/O access from jbd2 and kworker such as :
>
> # iotop -bktoqqq -d .5
> 11:11:16 364 be/3 root 0.00 K/s 7.69 K/s 0.00 %
> 23.64 % [jbd2/nvme0n1p2-]
> 11:11:16 8 be/4 root 0.00 K/s 0.00 K/s 0.00 %
> 25.52 % [kworker/u32:0-flush-259:0]
>
> The percentage given by iotop (time the thread/process spent while
> swapping in and while waiting on I/O) is often high.
>

> I do not know what to do for kworker and if it is a normal behavior. For
> jdb2, I have read it is filesystem (ext4 here) journal.
>

> I added the "noatime" option to /etc/fstab / line but it does not seem
> to reduce the number of access.
>

> Regards,
> Pierre
>
>
> P-S: If triming it is needed for ssd, why debian do not trim by default?

Thank you for posting your fio(1) runs on pastebin -- it is far easier
to comment on real data. :-)


It would help if you told us:

1. Make and model of computer (or motherboard and chassis, if DIY).

2. Make and model of CPU.

3. Quantity, make, and model of memory modules, and how your memory
slots are populated.

4. NVMe drive partitioning, formatting, space usage, etc..


STFW "CAZ-82512-Q11 NVMe LITEON 512GB", that looks like a decent desktop
NVMe drive.


Looking at your fio(1) runs:

1. It appears that you held 6 parameters constant:

--name=random-write
--ioengine=posixaio
--rw=randwrite
--runtime=60
--time_based
--end_fsync=1

2. It appears that you varied 4 parameters:

run bs size numjobs iodepth
1 4k 4g 1 1
2 64k 256m 16 16
3 64k 256m 16 16
4 1m 16g 1 1

3. Runs #2 and #3 had the same parameters, but had very different
operation and results. Why did run #3 layout 16 IO files, but run #2
did not? I suspect that experimental prodedure and/or system loading
were not the same during the two runs (?).

4. The best way to eliminate system loading variances is to create your
own Debian USB live stick. At the "Choose software to install" screen,
choose "SSH server" and "standard system utilities". Once the system is
running, install the tools you want. Use script(1) to capture console
sessions.

5. Pick one set of benchmark tool parameters and do several runs. You
want to gain enough skill running the tool so that the runs have little
variance.

6. In addition to running the benchmark tool the same way every time,
you need to do all the other steps the same way every time -- rebooting,
clearing kernel buffers, erasing files, trimming SSD's, enabling/
disabling processes, whatever.

7. Document your experimental procedure such that you and others can
duplicate your work and compare results.

8. For each set of runs, throw out the top third of results, throw out
the bottom third of results, and use the middle third of results to make
decisions.

9. Pick one parameter to sweep, keep all other parameters fixed, and do
a set of runs for each combination. I would do --bs=64k, --size=20%,
--numjobs equal to the number of cores in your CPU, and sweep --iodepth
with a binary exponential progression -- e.g. 1, 2, 4, 8, 16, etc..
Once I found the sweet spot for --iodepth, I would fix --iodepth at that
value, and sweep --bs from 4k to 1m.


David


References:

[1]
https://www.harddrivebenchmark.net/hdd.php?hdd=CX2-8B512-Q11%20NVMe%20LITEON%20512GB

[2]
https://www.notebookcheck.net/Lite-on-CX2-8B512-Q11-256-GB-Benchmarked.458645.0.html

[3] https://linux.die.net/man/1/fio

[4] https://linux.die.net/man/8/fstrim

[5] https://linux.die.net/man/1/script

[6] https://linux.die.net/man/1/df

[Jochen Spieker]

Pierre Willaime:

>
> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
>
> # fstrim -v /
> / : 236,7 GiB (254122389504 octets) réduits

This is probably the total amount of unused space on that SSD. The first
fstrim run after a reboot always trims all free space, as far as I
understand. After that, only space that was freed since the last fstrim
run will be trimmed.

> then, directly after :
>
> # fstrim -v /
> / : 0 B (0 octets) réduits

See above, this is expected.

> but few minutes later, there are already 1.2 Gib to trim again :
>
> # fstrim -v /
> / : 1,2 GiB (1235369984 octets) réduits

It is odd that something wrote and deleted 1.2 GB in "a few minutes". I
would run iotop to see what process is responsible.

> /Is it a good idea to trim, if yes how (and how often)?/

See the other answers. Usually once per week is enough, but that assumes
that the free space is not overwritten and freed again several times
each day.

On the other hand, personally I have never felt a difference in
performance before and after an fstrim run.

> Some people use fstrim as a cron job, some other add "discard" option to the
> /etc/fstab / line. I do not know what is the best if any. I also read
> triming frequently could reduce the ssd life.

Stay away from the "discard" option and do not worry about SSD life.
Even with heavy usage you should not exceed the expected lifetime
writes. Do worry about data loss in general and create a viable backup
strategy, independent of the storage medium (or service) that you are
using.

> I also noticed many I/O access from jbd2 and kworker such as :
>
> # iotop -bktoqqq -d .5
> 11:11:16 364 be/3 root 0.00 K/s 7.69 K/s 0.00 % 23.64 %
> [jbd2/nvme0n1p2-]
> 11:11:16 8 be/4 root 0.00 K/s 0.00 K/s 0.00 % 25.52 %
> [kworker/u32:0-flush-259:0]

jbd2 is the journalling service of the filesystem and makes sure that
your filesystem does not become corrupt in case of power failures.

kworkers are general purpose kernel threads doing kernel stuff, for
example disk I/O. Seeing these is not a problem in itself.

>

--
I spend money without thinking on products and clothes that I believe
will enhance my social standing.
[Agree] [Disagree]
<http://archive.slowlydownward.com/NODATA/data_enter2.html>

[Anssi Saari]

Jochen Spieker <m...@well-adjusted.de> writes:

> Stay away from the "discard" option and do not worry about SSD life.

What's the issue with the discard option? AFAIK, there may have been
issues with it in the decade before last but again AFAIK, today some
distros enable discard and some run fstrim on timer, both work.

But please share.

[Pierre Willaime]

Thanks all.

I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

But I do not think my issue is trim related after all. I have always a
lot of I/O activities from jdb2 even just after booting and even when
the computer is doing nothing for hours.

Here is an extended log of iotop where you can see jdb2 anormal
activities: https://pastebin.com/eyGcGdUz

I cannot (yet) find what process is generating this activities. I tried
to kill a lot of jobs seing in atop output with no results.

> I don't think you have a significant performance problem, but
> you are definitely feeling some pain -- so can you tell us more
> about what feels slow? Does it happen during the ordinary course
> of the day?

Program are slow to start. Sometimes there is a delay when I type
(letters are displayed few second after typing). Apt unpack take forewer
(5 hours to unpack packages when upgrading to bulleye).

The computer is a recent Dell precision desktop with i9-9900 as CPU, an
NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The
nvme SSD is supposed to be a decent one. This desktop is yet a lot
slower than my (more basic) laptop.

Complete system info: https://pastebin.com/zaGVEpae

--
Pierre Willaime - CNRS
Archives Henri Poincaré - Nancy

[Christian Britz]

Am 20.08.21 um 10:50 schrieb Pierre Willaime:

>
> I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).
>

Interesting. I am almost 100% certain that it was enabled for me by
default on bullseye. Maybe that behaviour changed during the release
process.

Best Regards,
Christian

[piorunz]

On 17/08/2021 15:03, Marco Möller wrote:
> I have no experience with SSD, but running my Debian Desktop from a USB
> Memory Stick since years, please allow me to share information which
> supports the suggestion of Linux-Fan to also investigate if there is
> extraordinary I/O taking place and maybe could be avoided:
> In the past I found extreme(!) I/O to be produced by Firefox, when it is
> writing its cache, and when it is writing its session restore
> information. These writes to my observation occure all the time, kind of
> nonstop! I could get it very satisfactorily reduced by applying a tool
> called "Profile Sync Daemon" (psd) from package "profile-sync-daemon".
> While it is packaged for Debian, as a starting point to study its
> documentation I suggest to better look it up in the Arch Linux Wiki.

Firefox setting which determines how often write session data to disk is:
browser.sessionstore.interval

Default setting is value of 15000 = 15 seconds, not that bad. But still
I changed that to 10 minutes (value of 600000). If I lose an open tab
once in a year when Firefox crashes, so be it. To be honest, I haven't
seen Fx crash in a year, or more.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[Marco Möller]

browser.sessionstore.interval
Storing the session is not the only data written to the disk. If it
would only be this data, then indeed setting a higher interval would be
sufficient. But there is much more going on. Especially the cache seems
to be a cause for the extreme high I/O of Firefox. That's why I finally
decided to go for the psd tool, and this tool made it for me. Well, I
assume it has good reasons that someone programmed psd, probably the
author also noticed that with Firefox parameters alone it often is not
enough to get the I/O to the HDD caused by Firefox significantly reduced?

I am curious to see what's producing the high jdb2 traffic on the system
of Pierre, and if this can be stopped and if this finally can be
attributed to have been the cause for the trouble.

Unfortunately I did not make good notes on how I searched for what was
causing the traffic on my system, I only remember to have monitored my
system with iotop and inotifywait.

---
Always stay in good spirits!
Marco

[Linux-Fan]

Pierre Willaime writes:

> Thanks all.
>
> I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

You're welcome :)

> But I do not think my issue is trim related after all. I have always a lot
> of I/O activities from jdb2 even just after booting and even when the
> computer is doing nothing for hours.
>
> Here is an extended log of iotop where you can see jdb2 anormal activities:
> https://pastebin.com/eyGcGdUz

According to that, a lot of firefox-esr and dpkg and some thunderbird
processes are active. Is there a high intensity of I/O operations when all
Firefox, Thunderbird instances and system upgrades are closed?

When testing with iotop here, options `-d 10 -P` seemed to help getting a
steadier and less cluttered view. Still, filtering your iotop output for
Firefox, Thunderbird and DPKG respectively seems to be quite revealing:

| $ grep firefox-esr eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:51 3363 be/4 pierre 0.00 K/s 1811.89 K/s 0.00 % 17.64 % firefox-esr [mozStorage #3]
| 10:39:58 5117 be/4 pierre 0.00 K/s 1112.59 K/s 0.00 % 0.37 % firefox-esr [IndexedDB #14]
| 10:41:55 3363 be/4 pierre 0.00 K/s 6823.06 K/s 0.00 % 0.00 % firefox-esr [mozStorage #3]
| 10:41:55 3305 be/4 pierre 1469.88 K/s 0.00 K/s 0.00 % 60.57 % firefox-esr [QuotaManager IO]
| 10:41:55 3363 be/4 pierre 6869.74 K/s 6684.07 K/s 0.00 % 31.96 % firefox-esr [mozStorage #3]
| 10:41:56 6752 be/4 pierre 2517.19 K/s 0.00 K/s 0.00 % 99.99 % firefox-esr [Indexed~Mnt #13]
| 10:41:56 6755 be/4 pierre 31114.18 K/s 0.00 K/s 0.00 % 99.58 % firefox-esr [Indexed~Mnt #16]
| 10:41:56 3363 be/4 pierre 9153.40 K/s 0.00 K/s 0.00 % 87.06 % firefox-esr [mozStorage #3]
| 10:41:57 6755 be/4 pierre 249206.18 K/s 0.00 K/s 0.00 % 59.01 % firefox-esr [Indexed~Mnt #16]
| 10:41:57 6755 be/4 pierre 251353.11 K/s 0.00 K/s 0.00 % 66.02 % firefox-esr [Indexed~Mnt #16]
| 10:41:58 6755 be/4 pierre 273621.58 K/s 0.00 K/s 0.00 % 59.51 % firefox-esr [Indexed~Mnt #16]
| 10:41:58 6755 be/4 pierre 51639.70 K/s 0.00 K/s 0.00 % 94.90 % firefox-esr [Indexed~Mnt #16]
| 10:41:59 6755 be/4 pierre 113869.64 K/s 0.00 K/s 0.00 % 79.03 % firefox-esr [Indexed~Mnt #16]
| 10:41:59 6755 be/4 pierre 259549.09 K/s 0.00 K/s 0.00 % 56.99 % firefox-esr [Indexed~Mnt #16]
| 10:44:41 3265 be/4 pierre 1196.21 K/s 0.00 K/s 0.00 % 20.89 % firefox-esr
| 10:44:41 3289 be/4 pierre 3813.36 K/s 935.22 K/s 0.00 % 4.59 % firefox-esr [Cache2 I/O]
| 10:44:53 3363 be/4 pierre 0.00 K/s 1176.90 K/s 0.00 % 0.00 % firefox-esr [mozStorage #3]
| 10:49:28 3363 be/4 pierre 0.00 K/s 1403.16 K/s 0.00 % 0.43 % firefox-esr [mozStorage #3]

So there are incredible amounts of data being read by Firefox (Gigabytes in
a few minutes)? Does this load reflect in atop or iotop's summarizing lines
at the begin of the respective screens?

| $ grep thunderbird eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:43 2846 be/4 pierre 0.00 K/s 1360.19 K/s 0.00 % 15.51 % thunderbird [mozStorage #1]
| 10:39:49 2873 be/4 pierre 0.00 K/s 4753.74 K/s 0.00 % 0.00 % thunderbird [mozStorage #6]
| 10:39:49 2875 be/4 pierre 0.00 K/s 19217.56 K/s 0.00 % 0.00 % thunderbird [mozStorage #7]
| 10:39:50 2883 be/4 pierre 0.00 K/s 18014.56 K/s 0.00 % 29.39 % thunderbird [mozStorage #8]
| 10:39:50 2883 be/4 pierre 0.00 K/s 3305.94 K/s 0.00 % 27.28 % thunderbird [mozStorage #8]
| 10:39:51 2883 be/4 pierre 0.00 K/s 61950.19 K/s 0.00 % 63.11 % thunderbird [mozStorage #8]
| 10:39:51 2883 be/4 pierre 0.00 K/s 41572.77 K/s 0.00 % 27.19 % thunderbird [mozStorage #8]
| 10:39:52 2883 be/4 pierre 0.00 K/s 20961.20 K/s 0.00 % 65.02 % thunderbird [mozStorage #8]
| 10:39:52 2883 be/4 pierre 0.00 K/s 43345.16 K/s 0.00 % 0.19 % thunderbird [mozStorage #8]
| 10:42:27 2846 be/4 pierre 0.00 K/s 1189.63 K/s 0.00 % 0.45 % thunderbird [mozStorage #1]
| 10:42:33 2846 be/4 pierre 0.00 K/s 1058.52 K/s 0.00 % 0.31 % thunderbird [mozStorage #1]
| 10:47:27 2846 be/4 pierre 0.00 K/s 2113.53 K/s 0.00 % 0.66 % thunderbird [mozStorage #1]

Thunderbird seems to write a lot here. This would average at ~18 MiB/s of writing
and hence explain why the SSD is loaded continuously. Again: Does it match the
data reported by atop? [I am not experienced in reading iotop output, hence
I might interpret the data wrongly].

By comparison, dpkg looks rather harmless:

| $ grep dpkg eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:25 4506 be/4 root 0.00 K/s 4553.67 K/s 0.00 % 0.26 % dpkg --status-fd 23 --no-triggers --unpack --auto-deconfigure --force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:38:33 4506 be/4 root 7.73 K/s 4173.77 K/s 0.00 % 1.52 % dpkg --status-fd 23 --no-triggers --unpack --auto-deconfigure --force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:38:49 4506 be/4 root 7.74 K/s 3770.01 K/s 0.00 % 1.41 % dpkg --status-fd 23 --no-triggers --unpack --auto-deconfigure --force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:39:01 4506 be/4 root 0.00 K/s 5529.57 K/s 0.00 % 5.85 % [dpkg]
| 10:39:29 4664 be/4 root 0.00 K/s 5480.00 K/s 0.00 % 3.43 % dpkg --status-fd 23 --configure --pending

This would explain that you are seeing slow progress...

[...]

HTH
Linux-Fan

öö

[Tixy]

It's enabled on my PC which was a fresh install in December using a
Bullseye release candidate installer.

--
Tixy

[Alexander V. Makartsev]

On 20.08.2021 13:50, Pierre Willaime wrote:

Program are slow to start. Sometimes there is a delay when I type (letters are displayed few second after typing). Apt unpack take forewer (5 hours to unpack packages when upgrading to bulleye).

That looks abnormal to me. Have you tried to update SSD firmware¹ and BIOS for your Dell PC?
LiteOn Storage branch was bought by KIOXIA (Toshiba), so I can't find any official information about technical specifications of SSD you have, but just by looking at its photos, I assume it is an average one, with external cache and TLC NAND chips.
The performance of a NVMe looks like it working at slower x2 speed, instead of utilizing full x4 lanes. But it could be also its normal working state and delays\slowdowns could come from parts unrelated to NVMe, like SATA HDDs.
A "smartctl" output for all storage devices could be useful.

The computer is a recent Dell precision desktop with i9-9900 as CPU, an NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The nvme SSD is supposed to be a decent one. This desktop is yet a lot slower than my (more basic) laptop.

Complete system info: https://pastebin.com/zaGVEpae

[1] https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4

-- 
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀ 

[piorunz]

On 20/08/2021 18:11, Marco Möller wrote:
> browser.sessionstore.interval
> Storing the session is not the only data written to the disk. If it
> would only be this data, then indeed setting a higher interval would be
> sufficient. But there is much more going on. Especially the cache seems
> to be a cause for the extreme high I/O of Firefox. That's why I finally
> decided to go for the psd tool, and this tool made it for me.

That's why you can tune disk cache in Firefox, disable it, make is
smaller and so on. By definition disk cache for a browser is a hot cache
where everything lands and it's being taken from again instead of being
downloaded from internet. On my PC I have /home on RAID10 made of 4
spinning drives, so all programs can cache and thrash on the storage all
day. If you, or anyone else in this topic, have /home on SSD and/or
Firefox/TB data churn is a concern, you could disable Fx disk cache
altogether.
I am glad that solution you have chosen, that psd tool (I don't know it)
work for you.

[David Christensen]

On 8/17/21 2:54 AM, Pierre Willaime wrote:
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).

On 8/20/21 1:50 AM, Pierre Willaime wrote:
> Thanks all.
>
> I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).
>
> But I do not think my issue is trim related after all.

I agree.

> I have always a
> lot of I/O activities from jdb2 even just after booting and even when
> the computer is doing nothing for hours.
>
> Here is an extended  log of iotop where you can see jdb2 anormal
> activities: https://pastebin.com/eyGcGdUz

Was your command something like this?

# iotop -o -k -t -q -q -q > iotop.out

> I cannot (yet) find what process is generating this activities. I tried
> to kill a lot of jobs seing in atop output with no results.

Analyzing the first ten minutes worth of data with an improvised Perl
script:

index of field 'read_bw'
1256840.19 firefox-esr
77926.07 apt
316.08 perl
22.74 dpkg
15.27 [kworker/6:0-events]
index of field 'write_bw'
220512.79 thunderbird
29613.87 firefox-esr
23221.20 dpkg
15211.66 [jbd2/nvme0n1p2-]
5529.57 [dpkg]
4148.09 systemd-journald
1699.13 perl
533.28 mandb
507.15 apt
145.61 rsyslogd
131.59 atop
115.77 syncthing
46.35 xfce4-terminal
38.60 smartd
15.48 Xorg
15.44 NetworkManager
7.64 bash
index of field 'swap_percent'
index of field 'io_wait_percent'
12427.58 [jbd2/nvme0n1p2-]
1334.15 firefox-esr
568.91 dpkg
385.31 thunderbird
293.57 mandb
99.99 syncthing
64.82 [kworker/13:3-events_freezable_power_]
63.86 smartd
55.12 [kworker/u32:3+flush-259:0]
38.64 [kworker/u32:2-flush-259:0]
25.27 [kworker/u32:3-events_unbound]
23.13 [kworker/4:0-events_freezable_power_]
22.68 [kworker/u32:2-events_unbound]
21.55 apt
12.51 [kworker/u32:1-ext4-rsv-conversion]
9.87 [kworker/u32:2-ext4-rsv-conversion]
9.63 [kworker/9:1-events]
8.90 [kworker/u32:1-flush-259:0]
8.58 perl
8.11 [kworker/9:1-events_freezable_power_]
5.85 [dpkg]
4.33 [kworker/u32:3-ext4-rsv-conversion]
4.26 NetworkManager
3.57 [kworker/13:3-mm_percpu_wq]
2.85 [kworker/9:1-mm_percpu_wq]
2.71 [kworker/4:0-mm_percpu_wq]
0.40 [kworker/13:3-events] [kworker/4:0-events]
0.38 [kworker/6:1-events]
0.36 [kworker/9:1-rcu_gp]
0.30 [kworker/u32:3-flush-259:0]
0.26 [kworker/6:0-events_freezable_power_]
0.16 systemd-journald
0.03 [kworker/6:0-events]


I appears:

- firefox-esr used the most read bandwidth -- 1256840.19 K/s total

- thunderbird used the most write bandwdith -- 220512.79 K/s total

- No processes were swapping.

- jbd2/nvme0n1p2- waited the longest for I/O -- 12427.58 % total


Both apt(8) and dpkg(1) were also running and using a small amount of
I/O. While I may leave Firefox and Thunderbird running when installing
a package or two, I shut them down for updates and upgrades. Was the
iotop data collected during a long-running upgrade?


AIUI the jbd2/nvme0n1p2 I/O corresponds to the bottom half of the kernel
(e.g. device driver interface, DDI) in response to I/O via the top half
of the kernel (e.g. application programming interface, API). The way to
reduce jdb2 I/O is to reduce application I/O.

> On 8/17/21 7:07 AM, Dan Ritter wrote:
>> I don't think you have a significant performance problem, but
>> you are definitely feeling some pain -- so can you tell us more
>> about what feels slow? Does it happen during the ordinary course
>> of the day?

> Program are slow to start. Sometimes there is a delay when I type
> (letters are displayed few second after typing). Apt unpack take forewer
> (5 hours to unpack packages when upgrading to bulleye).
>
> The computer is a recent Dell precision desktop with i9-9900 as CPU, an
> NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The
> nvme SSD is supposed to be a decent one. This desktop is yet a lot
> slower than my (more basic) laptop.
>
> Complete system info: https://pastebin.com/zaGVEpae

That's a good workstation. :-)


Firefox and Thunderbird are habitual trouble-makers on my Debian
desktops. I run Xfce with the CPU Graph panel applet. If I leave
Firefox or Thunderbird open long enough, eventually I will see a core
pegged at 100% and the CPU fan will spin up. Both applications keep
working in this state; but a swirling toilet bowl mouse pointer in
Thunderbird is a danger sign -- I have lost e-mail messages when moving
a message produced that symptom. The only cure is to close the
offending program(s) and implement additional measures to duplicate
incoming and outgoing e-mail. My conclusion is that both Firefox and
Thunderbird have multi-threaded programming bugs. The next time either
malfunctions, I may capture some iotop(8) data and analyze it.


I have also noticed that Linux is not very good at sharing I/O between
processes and/or drives. If one process is hammering I/O, any other
processes that attempts I/O will stutter. I have seen this both when
two processes access the same drive and when two processes access
different drives.


The last consideration is that Debian 11.0 has just been released. I
have encountered many problems with Debian *.0 releases over the years,
and typically wait several months and a couple of minor version upgrades
before doing a fresh install and migrating services and data. (The
Debian 10 nouveau driver did not like the NVIDIA Optimus graphics in my
Dell Latitude E6520 laptop until recently -- Debian 10.9?)


I suggest that you do a fresh install of Debian 10.10 and evaluate that.
If you still see problems, try Debian 9.13.


David

[Sven Hartge]

This may be part cargo culting and part real problem.

In the past it was shown that many SSD needed to flush their whole queue
whenever a TRIM command was received and the Kernel was really zealos of
sending those for every block freed.

This basically neutered many SSD and even caused corruption issues on
some especially shoddy ones.

Some firmwares got fixed and newer drives don't suffer from the same
problems, but even now end then you can find some devices on the lower
end of the consumer spectrum behaving poorly when bombarded with TRIM
commands.

So in the end the consensous was to just play it safe and batch-TRIM
once a week or day, because TRIMming on demand hasn't been shown to be a
big win.

It is far easier to fall back on that option than trying to
good/bad-list specific drive or firmware versions and select the
appropriate TRIM method.

Grüße,
Sven.

--
Sigmentation fault. Core dumped.

[Pierre Willaime]

Le 20/08/2021 à 19:45, Alexander V. Makartsev a écrit :

> That looks abnormal to me. Have you tried to update SSD firmware¹ and
> BIOS for your Dell PC?
>

> [1] https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4
Thanks all again for your precious answers.

I will try first to update Bios and SSD firmwares.

For the ssd, instructions (see link below) are for windows only.

Is there a way to update dell ssd firmware without taking the ssd out
and plug it in into a windows machine?

[Alexander V. Makartsev]

In this particular case, I don't thinks so, because it looks like SSD firmware and flasher are fused together into a single PE file and I don't think WINE will be able to handle it properly.
There is a possibility that extracted "23201112_WIN.EXE" file will work under WinPE / WinRE environments booted from a USB flash drive, but I wouldn't count on it.

If you don't have a NVMe-capable Windows PC at hand, you can download an evaluation version of Windows 10 from Microsoft¹ and install it on separate storage device, just to update SSD firmware.
Fill the form with gibberish. Microsoft Product Key or Activation are not required. Evaluation copy will work for 90 days.


[1] https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise

[piorunz]

On 23/08/2021 13:09, Alexander V. Makartsev wrote:
> If you don't have a NVMe-capable Windows PC at hand, you can download an
> evaluation version of Windows 10 from Microsoft¹ and install it on
> separate storage device, just to update SSD firmware.
> Fill the form with gibberish. Microsoft Product Key or Activation are
> not required. Evaluation copy will work for 90 days.

It can be even easier:

Download Windows 10 ISO from here
https://www.microsoft.com/en-gb/software-download/windows10iso

put on USB stick, do 10 minutes install, do you thing and nuke it. No
need to register anything. Do it while internet is disconnected, if will
install even faster.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀