linux: why last linking step hangs?

[t dance]

Linux

chromium-75.0.3759.0

[36207/36208] AR obj/headless/libheadless_shell_lib.a

next step hangs:

cpu about 0%

mem 80%

only one process running: ninja chrome. Others sleeping

After interrupt i see only:

chrome-wrapper

chrome.rsp

chrome_100_percent.pak

chrome_100_percent.pak.info

chrome_200_percent.pak

chrome_200_percent.pak.info

ninja chrome_sandbox succeeded

Same problem with  75.0.3738.0

args.gn:

blink_symbol_level = 0

enable_nacl = false

ffmpeg_branding = "Chrome"

is_official_build = true

proprietary_codecs = true

use_sysroot = false

use_system_freetype = true

use_system_harfbuzz = true

use_system_lcms2 = true

use_system_libdrm = true

use_system_libjpeg = true

use_system_libopenjpeg2 = true

use_system_libpng = true      

use_system_zlib = true

[Nico Weber]

`is_official_build = true` on Linux has a linker step that takes a long time the first time it runs, iirc something around 12 minutes on a fast machine. Maybe you need to wait longer?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/83dd5499-56aa-4083-b28e-766abff9189a%40chromium.org.

[t dance]

@Nico Weber

But with 75.0.3738.0 it took ours doing nothing

CPU usage almost 0% only occasional top and [rcu_sched] raises CPU to 1-3%

Is that normal?

[Christian Biesinger]

This may be an issue of reading from the disk... maybe it's swapping?
How much RAM do you have? (official builds need a lot of RAM)


Christian

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/ac921d03-ef8a-4d83-8925-8327d6e9fa72%40chromium.org.

[t dance]

@Christian Biesinger

This may be an issue of reading from the disk... maybe it's swapping?
How much RAM do you have? (official builds need a lot of RAM)

I have 8GB RAM. Memory usage 80% no swap used

During that last linking step CPU usage 0-1%. I don't hear disk activity 

Sure there has to be some sign of CPU activity?

I'll try running ninja chrome this night, and report back

[Christian Biesinger]

I find it hard to believe that linking only takes 80% of 8 GB. How do
you measure that?

Christian

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/7aa8abc4-4839-48bd-a43c-dfee1544b1a0%40chromium.org.

[t dance]

@Christian Biesinger

I find it hard to believe that linking only takes 80% of 8 GB. How do
you measure that?

According to /usr/bin/top currently

GiB Mem : 88.4/7.7

GiB Swap:  8.7/5.6

The line running ninja chrome:

  814 tdance    20   0   29.9g   5.6g   0.7  73.4  40:49.75 D .....

=0.7% CPU

[Matthew Cary]

The ninja process just coordinates the build processes, we wouldn't expect it to be doing much work. Is there a clang++,  ld.lld or kswapd process consuming CPU?

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/b9408201-26e4-4005-b46c-14730b396b2a%40chromium.org.

[t dance]

@Matthew Cary

The ninja process just coordinates the build processes, we wouldn't expect it to be doing much work. Is there a clang++,  ld.lld or kswapd process consuming CPU?

In fact it didn't hang. It only looks like it hangs because

CPU 0.7%

Mem 80%

[4/5] STAMP obj/tools/v8_context_snapshot/v8_context_snapshot.stamp

[5/5] LINK ./chrome

$ ls -l obj/tools/v8_context_snapshot/v8_context_snapshot.stamp chrome

-rwxr-xr-x 1 me me 3739025256 Apr  9 15:46 chrome

-rw-r--r-- 1 me me 0 Apr  8 22:42 obj/tools/v8_context_snapshot/v8_context_snapshot.stamp

That is 16-17 hours linking that last step!

[Christian Biesinger]

It looks like your top output looks a little different than mine. I'll
note that your process does have 30 GiB of virtual memory space
mapped. I remain of the opinion that your linking would be a lot
faster if you had more memory. (Maybe it's not swapping, but just
paging mmap-ed files in and out of memory)

Christian

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/0a2c0296-16e2-4ea7-a852-c31563f761b0%40chromium.org.

[t dance]

@Christian Biesinger

I'll
note that your process does have 30 GiB of virtual memory space
mapped. I remain of the opinion that your linking would be a lot
faster if you had more memory. (Maybe it's not swapping, but just
paging mmap-ed files in and out of memory)

Looking again at the table I now also noticed thatt 30GB virt memory. 

That is huge! Does the kernel store that in a swap file?

[Christian Biesinger]

It depends what the virtual space is used for but one way or another
it will be on disk, yeah.

Christian

> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/dc29c8eb-4c9e-42b4-98d0-ba224f327e76%40chromium.org.

[Matthew Cary]

Yup!

Physical memory is the amount of actual RAM you have in your computer. Virtual memory is the address space that programs see. The kernel takes care of mapping virtual memory to physical memory as necessary. Dirty physical memory gets written to a swap file if the kernel needs to make room in physical memory for a program to operate on some other part of virtual memory. Other things may go elsewhere---for example, unmodified memory that is backed by a file (like an executable) can just be discarded if the kernel knows it will always be able to load it back into physical memory from the backing file.

If the working set of a program in virtual memory is much larger than physical memory, the kernel has to keep moving things between the swap file and physical memory. In your case, probably your link step is spending most of its time shuffling things between memory and swap rather than linking, which is why it's taking so long.

Here's a lot more information on the subject: https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/dc29c8eb-4c9e-42b4-98d0-ba224f327e76%40chromium.org.

[Torne (Richard Coles)]

Building a full release official build with all the optimisations enabled unfortunately is just going to take more RAM - you probably want at *least* 32GB for a machine that's going to do this, if not more (64GB would be good).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEAhAGWqgWwnArci3AfGW_8H43zcXFVmXQGZCfXYmAMfg07KgA%40mail.gmail.com.

[t dance]

@Torne (Richard Coles) wrote:

Building a full release official build with all the optimisations enabled unfortunately is just going to take more RAM - you probably want at *least* 32GB for a machine that's going to do this, if not more (64GB would be good).

I believe that is the case, because it is my first time  Is_official_build = true

All previous times only with is_debug = false