Total removal of swap files from qubes as an installation option

[Tai...@gmx.com]

Considering most people will be using a device with 16GB+ RAM I believe
that the default should be no swap files at all and at the very least an
option checkbox to disable it

Reasons:
Even older devices like the circa 2011 thinkpad laptops support 16GB
(x220) and 32GB (W520)

The last and best owner controlled x86 devices support 16GB (AMD FT3
pre-PSP laptops like the popular for qubes G505S) and 192GB
(workstations/servers like the kgpe-d16) + I see absolutely zero reason
to have swap on my 64GB RAM workstation.

Pros:
No extra SSD wear
No risk of passwords/data written to disk by accident.

Objections? Thoughts?

I disabled mine in dom0 with no issues on a 8GB RAM laptop, sometimes I
must shut down a VM to start another (I can't have tons of idling VM's)
but other than that no issues.

I also disabled mine in the AppVM's with no issues - although for some
reason it magically re-activates when I update which is aggravating and
I was wondering why that is if anyone can answer me.

[Jean-Philippe Ouellet]

FWIW, I hit swap even with 16gb.

I prefer the current behavior of things swapping and my system
becoming slow and unresponsive until I recover it than the alternative
of processes being killed.

Consider a DispVMs in which you are doing some important work. When it
becomes starved for memory and no swap is available, the OOM-killer
kicks in, and the process with the highest oom-score will probably be
the browser the DispVM is waiting on. When that gets killed, the VM
closes and you lose everything, all without warning, and with no hint
of why this happened to the user.

As much as swapping is a bad user experience, the alternative seems worse IMO.

[Kelly Dean]

Tai...@gmx.com writes:

> I also disabled mine in the AppVM's with no issues - although for some
> reason it magically re-activates when I update which is aggravating and
> I was wondering why that is if anyone can answer me.

How are you disabling it? Instead of editing /etc/sysctl.conf, just make a new file, e.g. /etc/sysctl.d/noswap.conf, with:
vm.swappiness = 1

That will avoid losing your setting if sysctl.conf is overwritten during system update.

Setting swappiness to 0 (to disable swapping) is a bad idea, since it will subject you to the OOM killer, which is a result of the memory-overcommitment brain damage (conflating allocation of virtual address space, physical memory, and swap space) that Linux and its ecosystem inherited from Unix and POSIX. The vm.overcommit_memory option can't fix that, but using 1 instead of 0 for swappiness will swap only as much as necessary to avoid the OOM killer.

[Tai...@gmx.com]

On 08/16/2018 08:21 PM, Kelly Dean wrote:
>
> Tai...@gmx.com writes:
>
>> I also disabled mine in the AppVM's with no issues - although for some
>> reason it magically re-activates when I update which is aggravating and
>> I was wondering why that is if anyone can answer me.
>
> How are you disabling it? Instead of editing /etc/sysctl.conf, just make a new file, e.g. /etc/sysctl.d/noswap.conf, with:
> vm.swappiness = 1

I edit /etc/fstab to remove the swap partition, it sticks in dom0 but
doesn't stick in the VM's every time I update the template it re-enables.

> That will avoid losing your setting if sysctl.conf is overwritten during system update.
>
> Setting swappiness to 0 (to disable swapping) is a bad idea, since it will subject you to the OOM killer, which is a result of the memory-overcommitment brain damage (conflating allocation of virtual address space, physical memory, and swap space) that Linux and its ecosystem inherited from Unix and POSIX. The vm.overcommit_memory option can't fix that, but using 1 instead of 0 for swappiness will swap only as much as necessary to avoid the OOM killer.
>

If I have swap enabled I will ruin my SSD...so I will pass.

Like I said on a workstation with 64GB RAM there is absolutely zero
reason to have swap I have never even used half of that RAM and even on
my 8GB laptop I have never had any issues.

[brenda...@gmail.com]

On Friday, August 17, 2018 at 5:47:14 PM UTC-4, Tai...@gmx.com wrote:
> If I have swap enabled I will ruin my SSD...so I will pass.

I hope you have a contemporary SSD from a quality manufacturer, because if so, you're fear isn't really warranted: contemporary Samsung or Crucial/Micron drives have high TBW ratings ... and astonishingly-high tested TBW before failure, e.g.:

https://www.ontrack.com/uk/blog/pieces-of-interest/how-long-do-ssds-really-last/

> One of the Samsung SSD 850 PRO disk achieved a figure of 9.1 Petabyte of data
> written! That´s 60 times the TBW figure Samsung promises on its data sheets.
> The cheaper Samsung product – Samsung SSD 750 Evo was able to write 1.2
> Petabyte of data, which equals in theory to more than 80 years of constant
> disk writing. However, the pro models showed why their price is higher: None
> of them wrote less than 2.2 Petabyte of data.
>
> The test clearly proves that the fear of a limited lifespan is highly
> exaggerated in most aspects.

Brendan

PS - of course, backups are also smart. :)

[Marcus Linsner]

> > One of the Samsung SSD 850 PRO disk achieved a figure of 9.1 Petabyte of data
> > written! That´s 60 times the TBW figure Samsung promises on its data sheets.
> > The cheaper Samsung product – Samsung SSD 750 Evo was able to write 1.2
> > Petabyte of data, which equals in theory to more than 80 years of constant
> > disk writing. However, the pro models showed why their price is higher: None
> > of them wrote less than 2.2 Petabyte of data.

I'm not trusting Samsung when it comes to SSDs anymore, ever since they botched the firmware for 840 EVO where it periodically(actually at least once per day!) rewrites everything in order to improve read speeds(except when this rewriting takes place and lasts for hours, during which the random read speed for 1 thread 4K is no bigger than 2 megs per sec), no doubt at the cost of disk life and yet without updating the S.M.A.R.T. LBAs written counter: https://www.anandtech.com/show/9196/samsung-releases-second-840-evo-fix
There used to be a system-wise OS freeze whenever the SSD just passed another 1TB of bytes written which took like 1-2 minutes, and those are now happening at least once a week. The bad thing is that you can't roll back to any previous firmware version and there's no way to stop this automatic rewriting from happening.

[Chris Laprise]

This could be a reason to distrust Samsung based on competency or poor
hardware characteristics.

However, I feel that with their push for microphone-equipped audience
monitoring TVs, there is reason to distrust their products based on
their motives and lack of respect for people's privacy.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Marcus Linsner]

On Tuesday, August 21, 2018 at 6:21:22 PM UTC+2, Marcus Linsner wrote:
> > > One of the Samsung SSD 850 PRO disk achieved a figure of 9.1 Petabyte of data
> > > written! That´s 60 times the TBW figure Samsung promises on its data sheets.
> > > The cheaper Samsung product – Samsung SSD 750 Evo was able to write 1.2
> > > Petabyte of data, which equals in theory to more than 80 years of constant
> > > disk writing. However, the pro models showed why their price is higher: None
> > > of them wrote less than 2.2 Petabyte of data.

Actually I have to correct the record here. Basically I was wrong about the cause of this 2MB/sec write speed being caused by that new Samsung firmware update:

> I'm not trusting Samsung when it comes to SSDs anymore, ever since they botched the firmware for 840 EVO where it periodically(actually at least once per day!) rewrites everything in order to improve read speeds(except when this rewriting takes place and lasts for hours, during which the random read speed for 1 thread 4K is no bigger than 2 megs per sec), no doubt at the cost of disk life and yet without updating the S.M.A.R.T. LBAs written counter: https://www.anandtech.com/show/9196/samsung-releases-second-840-evo-fix
> There used to be a system-wise OS freeze whenever the SSD just passed another 1TB of bytes written which took like 1-2 minutes, and those are now happening at least once a week. The bad thing is that you can't roll back to any previous firmware version and there's no way to stop this automatic rewriting from happening.

The fact is, I don't know when that rewriting/refreshing takes place(it could be even more often than I thought, if judging by the SSD temperature being usually 44-47 Celsius, and only sometimes 32-35 Celsius)!

What I was seeing as 2MB/sec was in fact a quirk of my Lenovo Ideapad Z575 laptop whereby after one(and any subsequent) Windows 7 Restart (ie. Start->Restart) the random write speed would be capped(I don't know why) to that 2MB/sec and only a Sleep or Shutdown would bring it back to normal! And this only happens when the SSD is connected via a drive caddy in place of the optical disk drive(ODD), alone with nothing else connected (on ESATA, or main drive bay).
But how do I know it's not the SSD still? Because I added a Kingston SSD on the ESATA port, in addition to the Samsung SSD which sits in the ODD bay, and the speeds for both are normal until after the first reboot from Linux after which the Kingston's random write speed is at around 2 MB/sec, whilst the Samsung's is just normal. Proof of that here: https://gist.github.com/constantoverride/7fc7974eb0d5dd48f4ebe5bc84e623f3#gistcomment-2713671
So that's why I deduced that it cannot be Samsung SSD's fault for this write speed decrease.

The disk life being decreased due to refreshing is just my guess based on my assumption(possibly wrong?) that there's no way to refresh without decreasing cell lifespan. The other one with 1-2minutes of freeze is still true, it was happening even when Samsung SSD was on main drive bay alone(they were just rarer)

[brenda...@gmail.com]

On Saturday, September 22, 2018 at 12:06:04 PM UTC-4, Marcus Linsner wrote:
> Actually I have to correct the record here. Basically I was wrong about the cause of this 2MB/sec write speed being caused by that new Samsung firmware update:
>

> The fact is, I don't know when that rewriting/refreshing takes place(it could be even more often than I thought, if judging by the SSD temperature being usually 44-47 Celsius, and only sometimes 32-35 Celsius)!
> What I was seeing as 2MB/sec was in fact a quirk of my Lenovo Ideapad Z575 laptop whereby after one(and any subsequent) Windows 7 Restart (ie. Start->Restart) the random write speed would be capped(I don't know why) to that 2MB/sec and only a Sleep or Shutdown would bring it back to normal! And this only happens when the SSD is connected via a drive caddy in place of the optical disk drive(ODD), alone with nothing else connected (on ESATA, or main drive bay).

That seems to point to the bios (re-?)initializing the drive into PIO (polled IO) mode instead of DMA mode. Does the BIOS have the sata ports set to AHCI mode? Is the BIOS up to date? Is the drive tray from a 3rd party, not Lenovo?

> The disk life being decreased due to refreshing is just my guess based on my assumption(possibly wrong?) that there's no way to refresh without decreasing cell lifespan. The other one with 1-2minutes of freeze is still true, it was happening even when Samsung SSD was on main drive bay alone(they were just rarer)

Cell lifespan really is no longer an issue for the vast majority of use cases. Mid- to top-tier consumer SSDs (as well as professional/datacenter SSDs) generally have several layers of mitigations in place
to deal with signal loss and cell wear at the flash physical layer:
- Physical storage over-provisioning that is not exposed at the logical level (primarily data-integrity focused).
- Data redundancy schemes, even to the point of data protection when losing multiple silicon layers in the current stacked chip designs. Effectively: matrixed, RAID-like schemes at the slice-of-silicon level.
- Active monitoring and management of the health of blocks of cells as well as "bad block" management layers below the logical data layer exposed through the sata or pcie ports. The amount of insight/signal on this that is exposed via S.M.A.R.T. varies by manufacturer.
- Logical storage over-provisioning that is modifiable by the end-user (primarily performance focused).
- Performance-oriented data storage optimization to keep the pool of available data blocks high.

Some of the data management/redundancy features have strong parallels to approaches used in enterprise-class SANs from the 2000s/2010s...and have now been shrunk down into credit-card sized boards to ensure your baby pictures don't self-immolate.

There are some published ... adverpapers? ... from the manufacturers on these with just enough information to be intriguing, but not enough to easily clone, (plus there are the piles of patents behind them).

Brendan

[Marcus Linsner]

On Monday, September 24, 2018 at 1:13:21 PM UTC+2, Brendan Hoar wrote:
> On Saturday, September 22, 2018 at 12:06:04 PM UTC-4, Marcus Linsner wrote:
> > Actually I have to correct the record here. Basically I was wrong about the cause of this 2MB/sec write speed being caused by that new Samsung firmware update:
> >
> > The fact is, I don't know when that rewriting/refreshing takes place(it could be even more often than I thought, if judging by the SSD temperature being usually 44-47 Celsius, and only sometimes 32-35 Celsius)!
> > What I was seeing as 2MB/sec was in fact a quirk of my Lenovo Ideapad Z575 laptop whereby after one(and any subsequent) Windows 7 Restart (ie. Start->Restart) the random write speed would be capped(I don't know why) to that 2MB/sec and only a Sleep or Shutdown would bring it back to normal! And this only happens when the SSD is connected via a drive caddy in place of the optical disk drive(ODD), alone with nothing else connected (on ESATA, or main drive bay).
>

> That seems to point to the bios (re-?)initializing the drive into PIO (polled IO) mode instead of DMA mode. Does the BIOS have the sata ports set to AHCI mode? Is the BIOS up to date? Is the drive tray from a 3rd party, not Lenovo?

I've come to the conclusion that's not the BIOS itself(at least no alone), because the issue happens only after a Restart from within Windows 7 (or Linux) (so not a restart from BIOS eg. after a Save&Exit, or even after just a ctrl+alt+del at the bios or poweron password dialog). Furthermore, if I place only a Kingston (240G) SSD on ESATA port (with nothing else SATA connected, eg. on main drive bay OR optical drive bay) then it works fine until the first win7 Restart which then brings the random write speed down to 2.7MB/sec (around 670 IOPS). In BIOS mode is set to AHCI (instead of IDE). Now, if I do place a Samsung (840EVO 1TB) SSD on the ESATA port, no matter how many restarts, the speed slowdown doesn't happen! But if I place the Samsung on ODD bay (non-lenovo drive caddy), then after a win7 Restart, I experience the slowdown. But not if I place a Kingston SSD on ODD!
I don't know, it's weird!

BIOS is as up to date as they've released it (2011) https://pcsupport.lenovo.com/us/en/products/LAPTOPS-AND-NETBOOKS/IDEAPAD-Z-SERIES-LAPTOPS/IDEAPAD-Z575/downloads/DS027070

Keeping track of the issue here: https://gist.github.com/constantoverride/7fc7974eb0d5dd48f4ebe5bc84e623f3#gistcomment-2713815

Though I've stopped testing stuff for now, until (if ever) I get some better ideas.

Thanks for all the info, Brandan! Appreciate it.