Building headless for minimum cpu+mem usage

[Georgios Diamantopoulos]

Hey fellas,

I love this project! Thank you all for working on it. 

I'm looking to have a "farm" of headless agents at the ready to drive my tests, and we'd like to optimize the headless build to minimize the usage of resources (CPU+memory).

We're not particularly interested in anything more than rendering the DOM and executing any JS on the page so it's functional.

Images, fonts, videos etc are uninteresting so disabling loading of images and fonts would be the obvious starting point.

What else can we do to minimize the resource expenditure in this scenario? 

- Are there features we can disable on the code level?

- Can we target x86 to lower the mem usage? (would that be a significant cut?)

- Are there any hacks we can do using the embedder API to optimize the network loop and/or message loop? 

- Should we aim to run multiple agents in a single process or one agent/process and multiple processes?

ANY suggestions very welcome, thank you in advance.

Georgios

[Alex Clarke]

Using either a custom proxy or C++ ProtocolHandlers you could return stub 1x1 pixel images (etc...) or even block them entirely. 

We are working on adding programmatic control over when frames are produced.  Currently headless chrome is still trying to render at 60 fps which is rather wasteful.  Many pages do need a few frames (maybe 10-20) to render properly (due to usage of requestAnimationFrame and animation triggers) but we expect there are a lot of CPU savings to be had here.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/565099f2-5496-45cd-b48b-0ead0626f3a7%40chromium.org.

[Georgios Diamantopoulos]

Network usage is not really a concern, because like you say can be controlled very easily.

CPU is important but secondary - at 70-150MB peak mem usage to load a modern website (tested with CEF .NET with optimized offscreen settings - still figuring out how to do the same test in headless), we're trying to squeeze as many browsers as we can in one box.

Any tips on that? 

I don't know if it's particular to CEF .NET but I observed a behavior that I found strange - the memory usage of loading the same website is nowhere near deterministic. I get 90MB in one test and then 250MB on another!

On Thu, Mar 9, 2017 at 10:46 AM Alex Clarke <alexc...@google.com> wrote:

Using either a custom proxy or C++ ProtocolHandlers you could return stub 1x1 pixel images (etc...) or even block them entirely. 

We are working on adding programmatic control over when frames are produced.  Currently headless chrome is still trying to render at 60 fps which is rather wasteful.  Many pages do need a few frames (maybe 10-20) to render properly (due to usage of requestAnimationFrame and animation triggers) but we expect there are a lot of CPU savings to be had here.

On 8 March 2017 at 17:39, Georgios Diamantopoulos <geor...@gmail.com> wrote:

Hey fellas,

I love this project! Thank you all for working on it. 

I'm looking to have a "farm" of headless agents at the ready to drive my tests, and we'd like to optimize the headless build to minimize the usage of resources (CPU+memory).

We're not particularly interested in anything more than rendering the DOM and executing any JS on the page so it's functional.

Images, fonts, videos etc are uninteresting so disabling loading of images and fonts would be the obvious starting point.

What else can we do to minimize the resource expenditure in this scenario? 

- Are there features we can disable on the code level?

- Can we target x86 to lower the mem usage? (would that be a significant cut?)

- Are there any hacks we can do using the embedder API to optimize the network loop and/or message loop? 

- Should we aim to run multiple agents in a single process or one agent/process and multiple processes?

ANY suggestions very welcome, thank you in advance.

Georgios

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.

[Sami Kyostila]

First, an obvious tip but easy to overlook: make sure you're running a release build instead of a debug one.

Note that measuring Chrome's memory usage reliably is a little complicated. There are some instructions here: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/. This tool should help you determine which component is the biggest consumer of memory in your setup.

- Sami

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAEykOLEpWmAZCrT9FFbPvxF7ATO7_G88JycaZm3LJrGTj%2B%2B-Dw%40mail.gmail.com.

[Georgios Diamantopoulos]

I'm getting very interesting results here - and some that I'm not sure how to explain

First my config (linux):

args.gn

import("//build/args/headless.gn")

is_debug = false

remove_webcore_debug_symbols = true

target_cpu = "x86"

enable_nacl = false

builder

builder.SetWindowSize(gfx::Size(1200, 1000))

    .EnableDevToolsServer(getDevToolsEndpoint())

    .SetIncognitoMode(true)

    .SetCrashReporterEnabled(false)

    .SetDisableSandbox(true)

    .SetUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36");

So far so good. Memory usage is around 46MB for my example site vs anywhere between 90-250MB for chrome.

What's puzzling is that .SetSingleProcessMode(true) actually blows up the memory usage to around 250MB

I'm measuring usage from the RES column in htop (tree mode, taking the top entry).

VIRT also goes up though not as much.

Why would that be??

FInally, disabling GL with .SetGLImplementation("") actually increases mem usage to around 70MB

You received this message because you are subscribed to a topic in the Google Groups "headless-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/headless-dev/f_tQUs__Yqw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to headless-dev...@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAPuLcztkgAhDzf-NDKVLN%2BUN3Vft3dVxv1qEcAHCX8fs4zRixg%40mail.gmail.com.

[Sami Kyostila]

Are you looking at all chrome processes instead just the browser process? Some memory is also shared between different processes in somewhat non-intuitive ways. For this reason I'd recommend using the memory-infra tools to get data about memory usage instead of htop.

- Sami

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAEykOLEhLhZ%2BhuB6zaWjJTL3oLJ%2B0N2EM8pf8Cd6TRvJ4qtZkw%40mail.gmail.com.

[Georgios Diamantopoulos]

Yes, I did read that page but I'm not sure how to get the traces in headless-friendly way. 

In tree mode, if I understand correctly, htop will show you the total memory usage for all children on the parent - but please let me know if I'm missing something here.

[Sami Kyostila]

Try this:

$ headless_shell --remote-debugging-port=9222 --trace-startup=*,disabled-by-default-memory-infra http://www.chromium.org

After 5 seconds open chrometrace.log in about://tracing (in regular Chrome).

- Sami

[Georgios Diamantopoulos]

You're a rockstar, sir. Now I can properly compare the different modes of operandi.

[headless-man]

can you share your updated optimization settings?

[Georgios Diamantopoulos]

If you're referring to me, I haven't found much that is of use to be honest.

It looks like the rendering/javascript is the main cpu/mem hog - if you can disable javascript, memory and cpu usage go down dramatically, as you'd expect. 

Not much else made a significant difference. 

The only other thing of note is, don't ever try to use the single process mode. It will crash on you.

[headless-man]

thanks

[Roger]

This is a great initiative! Are there any available builds can be shared?

[Yizhuo WANG]

Hi, Georgios

I'm looking into reduce the usage of resources too, now, I'm trying Chromium's many --disable-xxx args.

Is there any suggestion you can share? Thanks!

[Georgios Diamantopoulos]

Sadly, no matter what I did, the CPU/mem remained roughly the same.

It's a shame they abandoned the single-process model (crashes all the time) - the multi process makes sense for the desktop but not for applications where resource usage is expensive.

Georgios

[Yizhuo WANG]

I googled many cases that shows chrome take lots resource

But in my server (16-core Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz), it takes around 30% cpu to run 10 pages.

That is too much... can you share your data for reference?

My launch cmd is this:

google-chrome \

--headless \

--no-sandbox \

--disable-background-networking \

--disable-default-apps \

--disable-extensions \

--disable-sync \

--disable-translate \

--hide-scrollbars \

--metrics-recording-only \

--mute-audio \

--no-first-run \

--safebrowsing-disable-auto-update \

--ignore-certificate-errors \

--ignore-ssl-errors \

--ignore-certificate-errors-spki-list \

--user-data-dir=/tmp \

--remote-debugging-port=9222 \

--remote-debugging-address=0.0.0.0 \

--window-size=1920,1080

[guea...@amazon.com]

Chromium is always going to use as much resources as are available to it. If you want to effectively limit it's utilization, you should look into using cgroups:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01

[Joel Griffith]

Alternatively, if docker is an option for you, you could containerize it and use that as a way of constricting resources. I'm a maintainer of service that does this, but has the docker image itself open-sourced here: https://github.com/joelgriffith/browserless.

I've found this to be an effective way of segregating Chrome from an app's infrastructure so it's not tightly coupled to the VM/machine that it's deployed on. If Chrome goes down, or chews through all your memory, your app won't necessarily go down with it.

[Yizhuo WANG]

I'm using docker myself, and I'll look into the browserless. thanks!

[Yizhuo WANG]

I'll check the cgroups out. meanwhile, I'm using docker and open 10 containers to do the test

It I want to limit the resources Chromium use, the limit for opening different pages will be a problem, maybe I will learn something for cgroups. thanks!

[Guenther, Andrew]

I think that you could actually use cgroups to put resource limits per render process. That way, you can open as many tabs as you’d like while keeping resource utilization linear. That’s a bit outside my expertise in cgroups though, so I can’t point you to exactly how to do it.

 

For those suggesting Docker, this is also a great option and actually uses cgroups under the hood, provides a much nicer wrapper, but offers slightly less control. If you do go that route, keep in mind that you will need to increase the shared memory size for your containers “--shm-size” as well as override the default seccomp profile so that sandboxing will still work. You can find a working seccomp profile here: https://github.com/jessfraz/dotfiles/blob/master/etc/docker/seccomp/chrome.json

[ah...@meedan.com]

Hello everyone, 

Did anybody find a workaround for this problem? Running headless chrome is consuming too much CPU+Memory resources and we need to restart our servers couple of times a day.

Kindly if anyone fixed that to share the solution.

Regards,

Ahmad

[nicholas...@gmail.com]

Hi I'm looking for any guides anyone might have come up with a working ways to optimize chromium headless. 

[can...@gmail.com]

Same problem for me, I have to go to htop to clean the processes

[Isaac Dawson]

I've always found it easier to just run N browsers with a few in stand by, then after a page load is completed, kill the tab or process and start a new one. 

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/66b5d584-5b78-4692-9c29-8b6ec0fd93b5%40chromium.org.

[Hamudy Jibbe]

Hello,

Its been 3 years and we never knew if you actually found a solution :D

I am working on a project with the same concept, but the issue is that each tab is consuming a lot of RAM.

I want to reduce this usage so I can achieve more opened tabs.

[Joey Stox]

Yeah ditto had anyone found a way to constrain chrome headless?

[Joey Stox]

Trying to run in a Heroku worker and it absolutely chews thru memory

[Jerry Lee Daniel]

Loans, Project and Digital Investment financing available up to $500m.
Have a Business Plan, Fundable Project and Redeemable Collateral.

Whatsapp: +44 7405 896213

[Jerry Lee Daniel]

Loans, Project and Digital Investment financing available up to $500m.
Have a Business Plan, Fundable Project and Redeemable Collateral.

Whatsapp: +44 7405 896213

On Friday, 26 May 2023 at 01:01:12 UTC+2 Joey Stox wrote: