Do we know the current ZRAM settings?

[John Moser]

Cross-posted from Chromebook Central Help Forum, as they directed me here.

Do we know the current ZRAM settings?  The only discussion I can find is from 2014, and suggests that they just set it to expose 1.5 times the physical RAM as a swap area.

ZRAM has a setting (mem_limit) to constrict how much memory to use, e.g. if you tell it to create a 4 terrabyte swap device but only use 200MB of RAM, then it will swap out 600MB at an average 3:1 compression rate and simply report to the kernel that it can't write new data (it behaves as if the device is full).  Thus you'll only end up using that maximum value.

In general, ZRAM gets between 3:1 and 4:1 compression—closer to 3:1.  ZRAM incurs overhead when swapping in or out, and so the performance impact depends on how many passes you make in a working set before swapping it back out (or invalidating it, if it's not been altered).

It can take 10,000 or 40,000 cycles to decompress a page of RAM, depending on how redundant the data is and what algorithm is used.  For example:  Lempil-Ziv decompression such as LZ4 reads a token, then copies a string of some arbitrary length, and so spends roughly 1 cycle per byte of output depending on the length of the entry.  It's the cycles to decide what dictionary entry to copy, followed by a loop of 1-cycle MOV instructions, divided by the number of bytes.  10 cycles and 100 bytes to copy means 1.1 cycles per byte output.  Likewise, this is fed by a Huffman decoder, which adds a few extra cycles.

All this means that spending a lot of time in one area of RAM diminishes the impact of ZRAM (or any swap), and ZRAM is quite cheap.  As you start swapping more, it suddenly gets *expensive*.  With sufficiently-low memory pressure, the performance impact of ZRAM can be fractions of a percent even if the system is swapping quite-frequently in real-time (something about having 6 billion cycles per second does that).  Likewise, with multi-core systems and read-ahead prediction (or madvise(), if your application is smart enough), the Linux kernel can decompress ZRAM pages to a swap cache area a few microseconds before your application actually needs them, incurring a 0% performance hit.

So that's fascinating, but we've already been sold ZRAM.

Here's the point:  eating a significant amount of RAM with a compressed ZRAM area will increase the rate of actual swapping, causing performance issues.  When dealing with the general case, that's the only metric that matters.

The settings I've seen used in major distros currently allow ZRAM to use an unlimited amount of RAM, and restrict how much uncompressed memory you can swap in.  That means if you (somehow) get 1.25:1 compression and they give you an area 1.5x the size of RAM, ZRAM can and will consume 100% of your memory.

I usually consider 4:1 compression a practical upper limit, and 3:1 the average case.  Thus I set mem_limit to 0.5 times physical RAM, and disksize to 2 times physical ram size.  Theoretically, zram will always hit mem_limit before you swap out the full disksize.

I haven't done any performance measuring and tweaking, beyond noting that 50% works and 90% strains the system.  To do so, you would need to deploy at higher mem_limit and disksize values, with a disksize up to 4 times physical RAM and a mem_limit between 50% and 100% of physical RAM.  You'd then have to gather metrics when ZRAM (kswapd) consumes significant average amounts of non-idle CPU, identifying what percentage of RAM is currently occupied by ZRAM (mem_used_total / total physical RAM).  Automatic adjustment based on CPU consumption is also possible, albeit most likely at the driver level.

So, tl;dr:  ZRAM should be restricted by mem_limit, with disksize set to about 4x that.  I haven't been able to determine if that's what ChromeOS does at the moment; all Linux distributions I've examined just set disksize to about 1.5x or 2x RAM size and hope for the best.

[Mike Frysinger]

the answer to "what is my device doing right now" is easy:

(1) open crosh w/ctrl+alt+t

(2) run "swap"

(3) read the output

(4) eat some hákarl to celebrate

the answer to "what settings should i use to make my chromebook run so fast" is easy: let us manage it ;).

if you're experiencing swap storms and would rather let tabs die than swap, you could try "swap stop" to see if it performs better for your use case, and if it does, use "swap disable" to disable it future boots.  see "swap help" for more details.

if you want to talk finer perf numbers, Luigi has dived into that (probably more than he would like).  we have UMA stats to track the effects of ZRAM on devices to try and balance tab kills with swaps.  i think in our experience, people would rather have a slightly degraded experience now and then instead of seeing their tabs get killed out right.

-mike

--
--
Chromium OS discuss mailing list: chromium-os-discuss@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-discuss?hl=en

[Luigi Semenzato]

Thanks.  Let me comment on the forum, since it was asked there.

[John Moser]

Nod.  The answer I'm looking for looks more like this:

/ $ swapon -s

Filename                                Type            Size    Used    Priority

/dev/zram0                              partition       8092260 0       5

/ $ cat /sys/block/zram*/mem_limit

2071621632

/ $ free

              total        used        free      shared  buff/cache   available

Mem:        4046132      868524     1249548       41496     1928060     2839484

Swap:       8092260           0     8092260

This is one of my (small) servers.  In the above configuration, up to 50% (2GB) of RAM will host ZRAM data; after that, regardless of if I've filled the 8GB swap device or not, zram0 will refuse to store additional data until something is freed.

The default setting on this distribution (it's Ubuntu) is to set mem_limit to 0, which means anything less than 2:1 average compression would literally eat 100% of my RAM with ZRAM pages before it stopped sending things to swap.

My point is largely that the compression ratio is variable, and so the amount of memory used at maximum for a given disksize is variable.  So far, none of the distribution scripts I've seen use the mem_limit tunable.  For example, here's the one used by Fedora:

https://github.com/mystilleef/FedoraZram/blob/master/zramstart

That one creates a non-limited ZRAM swap device with a disksize of 1/3 RAM.  If you have 6G RAM, it will swap up to 2G into ZRAM, likely consuming 0.67GB to store it.

The settings I've shown above, if used on a system with 6G, typically would result in instead swapping 9G to RAM (average 3:1 compression) onto a 12G ZRAM device, then not swapping any additional because the device is using 3G to store it all, leaving the other 3G as regular memory.  Notice, crucially, the absolute guarantee of not consuming more than 3G to store ZRAM data, regardless of how much you put into it and at what ratio.

At this point it's an academic question, although if I could find the relevant information myself and discovered that mem_limit is set to 0 I'd have raised the point as an actionable suggestion—because mem_limit should definitely not be 0.

Interesting that you've been able to gather performance statistics on this already.  That's been something I've been interested in, although I don't have the capacity to do it.

I'd actually like to see an in-kernel driver (either an interface or actual logic) that tracks the moving average percentage of blocking CPU use by ZRAM over a past snapshot of non-idle CPU time (e.g. past 15 seconds of 10 CPU-mS) and selects a "shrink" or "grow", or "hold" mode based on the SMA.

For example, when more than 1% of non-idle CPU time is spent blocking a process waiting for ZRAM swap-in OR consuming a CPU for ZRAM swap-out while another process is waiting to run (i.e. it can't run because there are no free cores, and ZRAM is performing a swap-out), switch to "Shrink" mode.  ZRAM refuses new pages (as if mem_limit is hit), and frees pages that are swapped in.  This continues until the statistic falls by some preset proportion, e.g. by 1% (so until the SMA falls below 0.99%).

That would produce a performance bound on ZRAM swapping.

Hmm... maybe I should bring that up to Nitin.  It's been a long while since I spoke with him.

--
--
Chromium OS discuss mailing list: chromium-...@chromium.org

[Mike Frysinger]

if you run "swap" and "free" in crosh, you'll get exactly those details

-mike

--
--
Chromium OS discuss mailing list: chromium-os-discuss@chromium.org

[John Moser]

I don't.  'swap' gives me mem_used_total BUT NOT mem_limit AT ALL ANYWHERE IN ANY FORM.

Provide a method to find mem_limit.

[Luigi Semenzato]

On Thu, Aug 24, 2017 at 5:13 PM, John Moser <john.r...@gmail.com> wrote:
>
> I don't. 'swap' gives me mem_used_total BUT NOT mem_limit AT ALL ANYWHERE IN ANY FORM.
>
> Provide a method to find mem_limit.

"Please" provide a method?

Generally we're reluctant to add this kind of information because

1. there's no end to it
2. the bulk of our customers are not computer scientists
3. those who are can go to developer mode and do whatever they want

The "swap" or "swap status" command prints mm_stat. One of those
fields is mem_limit. The kernel source will tell you which one.

[Luigi Semenzato]

You can also look at the bottom of chrome://system..

[Mike Frysinger]

you need to stop doing that.  it's an easy way to make people start ignoring you.

-mike

--
--
Chromium OS discuss mailing list: chromium-os-discuss@chromium.org

[John Moser]

Mike I was quite clear and repeatedly stated that I was looking at the mem_limit information, and you gave me two flippant responses—the second of which totally-ignored this lovely little gem:

/ $ cat /sys/block/zram*/mem_limit

2071621632

I assumed by your attitude and your responses that your answer amounted to, "lol, you're an idiot, you don't need that information, only this other information!  *pat on the head* now go play".

Now you continue to insult me by continuing to treat me like an idiot.  You need to stop doing that; it's an easy way to make people start ignoring you.

In case that was too much for you to read and process, let me break it down for you:  you're being an asshole.

[John Moser]

Thanks Luigi, you've been ever-helpful. :)

As for dev mode, I tried that—kind of.  I can do it, but it requires powerwashing the Chromebook.

[Mike Frysinger]

you need to stop, take a breath, and re-evaluate what's actually happening here.  your perception is completely wrong, and becoming angry and abusive is just going to get you banned.  ad-hominem attacks are unacceptable here and anywhere else in the Chromium projects.  if i thought you were an idiot, i wouldn't have wasted time responding and just muted.

now, if you actually ran what i suggested and reviewed the code, you would see that my suggestions are exactly what you're looking for.  "swap status" runs (the equiv) of `swap` and then displays every single file under /sys/block/zram0/.  if mem_limit isn't shown, it's because the kernel you're running doesn't provide it, and thus it's impossible to `cat` a file that doesn't exist.

-mike

--
--
Chromium OS discuss mailing list: chromium-os-discuss@chromium.org

[John Moser]

"if you run "swap" and "free" in crosh, you'll get exactly those details"

"now, if you actually ran what i suggested and reviewed the code, you would see that my suggestions are exactly what you're looking for.  "swap status" runs..."

You didn't mention swap status.

*runs swap status*

.... the output doesn't include mem_limit.  It includes the same exact output as `swap`.  So you're wrong.

As per ad-hominem attacks, I grew up dealing with bullies who learned to slight someone subtly, in full view of everyone else, and harass them deliberately in such a way that nobody else would notice even when pointed directly at it.  You're goading me, deliberately.  I asked a highly-technical question about the underlying, non-user tunable facilities in ChromeOS and your answer was....

"the answer to "what settings should i use to make my chromebook run so fast" is easy: let us manage it ;)."

*giggle* *giggle* *giggle* ohhhhh don't worry about that, silly boy, we're smarter than you!

... phrased in such a way as to suggest I was asking how I should tune it, as if I'm so out of touch as to have suggested that as a possibility!

Meanwhile others have responded by providing technical details and actually-interesting discussions (and actually proven to be smarter than me, instead of just implying it).

You continue, even now, to press on me, claiming that I'm simply not thinking, that the answer is right in front of me, and that you've told me how to do it several times—when you haven't.

Now are you going to again suggest you actually gave me an answer and thus imply that I'm simply not grasping what you're saying or am otherwise just a confused soul trying to dig into things beyond my ken?  Are you going to give me another pat on the head and tell me I should really leave thinking about this kind of difficult stuff up to people who have the smarts?  Will there be a lollipop this time?

By all means, tell everyone again that I'm simply not listening and not seeing what's right in front of me.

[John Moser]

And yes, it doesn't output it because it's kernel 3.8 and doesn't expose that tunable—not, as you suggested, because I didn't follow directions.

[Mike Frysinger]

i don't know what you think is happening here, but i can assure you it is not.  these forums are not the place to post rants or attacks, ad-hominem or otherwise.  if you can't abide by our simple community rules, then please cease posting.  further violations will have no choice but to ban you.

please consult this doc for more information:

  https://chromium.googlesource.com/chromium/src/+/master/CODE_OF_CONDUCT.md

-mike

[Luigi Semenzato]

John writes:

> Now are you going to again suggest you actually gave me an answer and thus imply that I'm simply not grasping what you're saying or am otherwise just a confused soul trying to dig into things beyond my ken?  Are you going to give me another pat on the head and tell me I should really leave thinking about this kind of difficult stuff up to people who have the smarts?  Will there be a lollipop this time?

Sorry if I am stating the obvious, but whether you're "right" or not, from a practical perspective, this isn't really buying you any points, is it?  Unless you value venting your spleen so highly that it's worth making enemies among folks who might be your colleagues---or not, if they'd rather not work with you.  The world is smaller than you seem to think.

In any case, so that you know, I also don't think you're "right".  If anything, Mike gave you too much credit.  The "swap" command simply dumps the content of all entries in /sys/block/zram0.  That content changed with the kernel version, and in recent kernels the value of mem_limit is included in mm_stat instead of being listed explicitly.  You seemed to know your way around that code, so Mike assumed you wouldn't need much hand-holding.

Please give some consideration to this feedback.  If folks didn't care, you wouldn't get it.

[Bob P Wilson]

Hi group,

This is such an uninformative forum, especially for users like myself who are actually seeking a SIMPLE answer to the Chromebook, ZRAM Settings question.

I do NOT speak techie, so this entire posting is ridiculous.   Simple users, many are just like me, we have a basic Tech understanding but we also have common sense... all we want to know is simple....   What are the recommended settings for ZRAM based upon the OEM installed RAM of ANY Chromebook?

Example:  If I have an HP Chromebook which has an original, OEM installed RAM of 2GB, how do I calculate the recommended ZRAM settings?  (Provide an easy example, including any simple math needed so any user can calculate the answer.  Thanks.

     The answer should look something like this:

      - Tak 2gb OEM Ram X's your recommendation.  2 x's 1 = 2gb for your ZRAM setting.

Thanks,

Bob

[Mike Frysinger]

as stated earlier in this thread, the recommended settings are to not customize them at all and use the defaults we already set. regular users shouldn't be messing around with these.

-mike

--
--
Chromium OS Discussion mailing list: chromium-...@chromium.org

View archives, change email options, or unsubscribe:

https://groups.google.com/a/chromium.org/group/chromium-os-discuss

[Selden Deemer]

Bob : Since you are a self-described "simple user," Mike's answer is relevant: "regular users shouldn't be messing around with these."

Here is the output from help for the swap command [emphasis , mine]

swap [ enable <size (MB)> | disable | start | stop | status

 | set_margin <discard threshold (MB)> | set_extra_free <amount (MB)>

 | set_min_filelist <amount (MB)> ]  

  Change kernel memory manager parameters

  (FOR EXPERIMENTS ONLY --- USE AT OWN RISK)

  "swap status" (also "swap" with no arguments) shows the values of various

  memory manager parameters and related statistics.

  The enable/disable options enable or disable compressed swap (zram)

  persistently across reboots, and take effect at the next boot.  The enable

  option takes the size of the swap area (in megabytes before compression).

  If the size is omitted, the factory default is chosen.

  The start/stop options turn swap on/off immediately, but leave the settings

  alone, so that the original behavior is restored at the next boot.

  WARNING: if swap is in use, turning it off can cause the system to

  temporarily hang while the kernel frees up memory.  This can take

  a long time to finish.

  The set_margin, set_min_filelist, and set_extra_free options change

  kernel parameters with similar names.  The change is immediate and

  persistent across reboots.  Using the string "default" as the value

  restores the factory default behavior.