WhenI update my system I get this error on random packages. Last time I updated a couple weeks ago this did not happen. It also breaks my system because vmlinuz and initramfs are not no longer on the boot partition because of pacman erroring partway through the update. I can boot again if I reinstall the linux package from the Arch installer and then revert my system via Timeshift, but the error happens every time I try to update and the package it fails on is inconsistent.
My 4TB drive has almost 3TB of free space. The only partition that is not my main one is the boot partition and it's nowhere near full either. I've got 64GB of RAM, so it's definitely not that either. This error happens if I chroot into my install as well.
I'll give manually installing the package a try later today, but since it happens on random packages and sometimes the one it fails on works on another try, I'm not sure that's going to help. I've tried deleting all of the cached packages too, but redownloading them doesn't fix this issue for me.
I didn't mean to manually install the package.
I meant to try to decompress the file in the cache that caused the issue with "zstd -d" to see whether this is only a problem in the context of pacman decompression.
If yes, try "--single-thread", though afaiu that only applies to compression.
I have a WD_BLACK SN850X NVMe M.2 2280 4TB if that helps. I noticed this error complains about me having the resume service disabled. I'm guessing that's not the issue, but I'll give enabling it a try again (EDIT: Doing this didn't fix my issue, as expected). I previously enabled it and the other two needed services to try Wayland, but ran into several problems so I've just been using X.
Ftr, the other thread was about multithreaded xz compression, not zstd decompression. And from the install iso.
You could still test "maxcpus=1" (but while a tolerable workaround for the installation, that's oc no solution to operate the system)
I gave this a try as well. I disabled gdm, set maxcpus=1 in my /boot/loader/entries/arch.conf file, and restarted. Tried updating AND IT WORKED?! Obviously not a longterm solution, but glad I could at least update.
Did you run that test with maxcpus=1 or not? maxcpus=1 is a really weird workaround. So either a cpu core is misbehaving (bad cpu cache? power supply? no idea) or some obscure kernel bug? You could also try with linux-lts kernel...
I'm pretty much unable to help, sorry. There should be a better tool for testing CPU, including special cases like testing different class cores (P-Cores vs. E-Cores). Or maybe you could turn them off in Bios (if at all possible) and set a power limit or something, just to see if it changes anything... no idea really...
I started doing some research about other people with instability issues with an 13900k. I found somebody recommending setting all of your P cores to a specific frequency. I set mine to 5.5Ghz. I've tested a full package reinstall multiple times using
and it never fails anymore. I'm not sure why this is suddenly a problem for me since I built this computer about a year ago. Somehow, the cores hitting 5.8Ghz causes instability. Annoying since I'm using defaults. Oh well. Just glad I found a decent solution.
This is not an "issue" what I'm posting here, instead, it's a solution. A patch for the Linux kernel that might solve the SQUASHFS problem for some devices [don't expect this to be a magic solution if your hardware is actually faulty].
I'm a developer, I started my journey by developing for AOSP when I was younger (like when I was 15 or 16). I may not be the smartest or the wiser, neither the more expert. However, I do have a strong willpower. My device, the EA6350v3, works fine. It's a nice device, some issues with the VLAN and the switch (which I've fixed myself) and with the fact that ath10k is far from perfect. Still, a nice experience.
It worked flawlessly and amazingly running Linux 4.14 and Linux 4.19, but that quickly changed when Linux 5.4 came to master. Since then, when using the USB port and the WiFi intensively at the same time (say, using it as a NAS), and particularly with Samba since the binary weigths like 30MB, the device will spam the log like this:
Spamming the whole kernel log is not the actual issue: the process needing the resources from the ROM will eventually crash, and it can be init, since it may need a shared library, some code swapped-out of memory, from the ROM, since init must live in the flashable SQUASHFS partition; or to run some busybox tool, and if init crashes, the kernel will immediately panic.
So, the problem is not phyisical. The problem only happens when using Linux 5.4, it's a software problem, it's a issue related to how the CPU, the RAM, the DMA and the respective drivers [ath10k, usbcore, ubi/ubiblock/ubifs, spi-qup, mtd/mtdblock] interact, but I can't find out what's wrong.
Investigating about UBI, I found that UBI is resistant to errors, it can detect and correct many problems with actual bad flash chips [not the issue here], and indeed UBI detects a failure and tries again as the log clearly states, and most of the times it succeed. This means that programs or drivers that talk directly to the UBI layer succeed, like the UBIFS for example, and the ubiupdatevol and many more. SQUASHFS should do the same.
This means: SQUASHFS needs to be fixed, since UBI corrects the error, and it's transparent to the upper layer, why isn't SQUASHFS satisfied? Because when UBI fixes the error, the data is correct in the memory, it should be transparent to SQUASHFS, what's the deal?
The Linux kernel will not allow a corrupted page in the memory. It will fight back with any driver attempting to corrupt the kernel's memory [corrupting the memory on purpose it's another story], including the SPI and the MTD and NAND drivers. UBI itself taints it's buffer, tries to read, and perform a checksum over the buffer. This means that the data, after UBI retries, is indeed present in memory, and it's correct, but SQUASHFS is unable to detect it as such. Instead, SQUASHFS will enter in a fail loop, even when the data is technically correct
This is because somehow the page cache (a section of memory used by Linux to optimize disk usage) keep the "corrupted" pages in memory. Almost all file systems in the kernel know how to deal with that situation, like F2FS in a bad flash or EXT4 with a faulty data cable, but SQUASHFS doesn't. From this point and onwars, "corrupted" in quotes means that it looks corrupted to SQUASHFS but no corruption is present as it's guaranteed by UBI, it's synonymous when I write "poison".
This is the first step to fix SQUASHFS: invalidate the pages and fragments that are "known to be corrupted", making them never stay in the "front" cache. The squashfs_cache_get function in the cache.c file contains the "front" cache verification logic of SQUASHFS, it contains the call that, eventually, will read the data into memory and decompress it either to a buffer or directly to the page cache [not discussed here].
If you see the logs closely, SQUASHFS only attempts to decompress once. This is because the data goes along to SQUASHFS's "front" cache even if it does not work. This means that upon the decompression failure, the cache will remain poissoned until Linux needs to free some RAM and the page it's lucky enough to get evicted, and only then it's when Linux will attempt to read again.
Invalidating the "front" cache entries, by changing their block property to the SQUASHFS_INVALID_BLK will force SQUASHFS to try to read and decompress the data again, otherwise, the page marked as faulty will stay in RAM for an indefinite amount of time. However, this does not resolve any poisioning in the "back" cache, and if it's poisoned, it will fail over and onver again.
This alone may fix some problems, but the problem with doing only this is that the "corrupted" data will be used by all waiting processes at least once. It will be attempted again only when another process waits for the same data, but after all the original waiters have already used the corrupted data, or received the SIGBUS killing signal from the kernel, or as the page is not held corrupted in the page cahce, it will enter in contention. A better solution is try to avoid poisoning the "front" cache in the first place.
SQUASHFS manages it's "front" cache (the data decompressed and actually used) in the function described above. The front cache can be invalidated and it will force SQUASHFS to try to read the backing device and decompress again. Here it's the second cache: the "back" cache which is the cache that belongs to the raw SQUASHFS image, the compressed data stored in the disk. SQUASHFS uses the ll_rw_block function (not anymore in the mainstream kernel) to read pages from the backing device. Linux optimizes these calls by putting the data, again, into it's page cache.
So, the very first time, the data will be actually read from disk, but the second and subsequent calls, even when it's requested to do so, will not; and the "corrupted" compressed data will remain in memory indefinitely. Two things need to be done: implement the retry logic for the squashfs_read_data function, so it retries without poisoning the "front" cache and evict the data from RAM, so the function ll_rw_block may attempt to re-fetch the data from disk.
The first one is trivial: change the name of the original function to __squashfs_read_data, make it static inline, wrap it in a fake squashfs_read_data with the same signature that will loop n times when the returning code of the real function is error. This allows retrying without poisoning the "front" cache.
3a8082e126