The minidump can not always generated by breakpad when there is a crash occurred.

[Liu Xiao]

[Repetition steps]:

1. keep breakpad enabled;

2. use chrome to browse webpage;

3. repeat step 2 until there is a crash occurred.

 

[What is the expected?]:

minidump file will be always generated successfully. 

 

[What went wrong?]:

Sometimes minidump file can not be generated,and we can see

"google-breakpad: Failed to generate minidump." from the log.

 

[Probability]:

However,the probability of this problem is low, it is difficult to reappear.

 

[Question]:

Is it an inherent defect of breakpad ?

What is the root cause of it, and has it been resolved in the new version of chromium ?

 

Thank you very much.

[Liu Xiao]

Version : 63.0.3239.111.

[Torne (Richard Coles)]

On Android the process is usually 32-bit, which means address space is very limited. When the address space is exhausted then you'll get an OOM crash - and if it's bad enough, breakpad will also fail to function correctly because it will not be able to find any free address space to mmap() the things it needs to look at to generate a proper crash report.

If that's what's happening then there's no way to fix this - it's a limitation of breakpad's design, which runs inside the crashed process in a compromised context. The future replacement crash handling system (crashpad) works in a different way that doesn't have this problem but is not yet ready for Android.

On Tue, 17 Jul 2018 at 04:37 Liu Xiao <liuxia...@gmail.com> wrote:

Version : 63.0.3239.111.

--
--
Chromium Discussion mailing list: chromium...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

Thanks for your answer.

If that's what's happening then there's no way to fix this - it's a limitation of breakpad's design, which runs inside the crashed process in a compromised context.

>> It is very likely that the problem is derived from an OOM crash.

>> From the source code, we can see when crashed the crashed process call system function 'clone' to create a child-process which is responsible for minidump.

>> Under this mode (not full out-of-process), the situation "runs inside the crashed process in a compromised context"  will also exist ?

=========================================================================

bool ExceptionHandler::GenerateDump(CrashContext *context) {

  const pid_t child = sys_clone(

      ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL, NULL,

      NULL);

  // Close the read end of the pipe.

  sys_close(fdes[0]);

  // Allow the child to ptrace us

  sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);

  SendContinueSignalToChild();

  int status;

  const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));

  sys_close(fdes[1]);

}

=========================================================================

[Torne (Richard Coles)]

This sys_clone() call is basically fork() - the child process is mostly identical to the parent process, including having all the same address space mappings and not having any free virtual address space as a result. The context is still compromised, because it's the same context.

The only way to be running in a safe environment in the crash handler is to have the dump generation run inside another process that was already running before the crash, or to exec() another binary after forking to actually get a fresh environment, and breakpad cannot do either of those things.

OOM crashes on 32-bit android aren't very interesting in general - address space is very limited and it's quite possible to run out without anything actually being wrong; some devices also have bad graphics drivers that leak address space. It's unlikely you can do anything to address these crashes even if you had a dump.

[Liu Xiao]

Thank you very much.

his sys_clone() call is basically fork() - the child process is mostly identical to the parent process.

>> Yes, creating child-process by sys_clone() is not the full out-of-process mode.

The only way to be running in a safe environment in the crash handler is to have the dump generation run inside another process.

>> Yes, I have got it from the breakpad doc, that is called out-of-process.

>> I want to implement it before, but  it is sensitive for creating a new process, and I am not sure if it will resolve the problem in OOM Crash, what do you think ? 

OOM crashes on 32-bit android aren't very interesting in general

>> Do you mean OOM Crash is a bothersome problem on 32-bit Android?

>> I am using Android 7.0, it is definitely a 32-bit Android, or there may be a 64-bit version ? How to comfirm it ?

[Torne (Richard Coles)]

On Thu, 19 Jul 2018 at 02:05 Liu Xiao <liuxia...@gmail.com> wrote:

Thank you very much.

his sys_clone() call is basically fork() - the child process is mostly identical to the parent process.

>> Yes, creating child-process by sys_clone() is not the full out-of-process mode.

I'm not sure what you're referring to here.

 

The only way to be running in a safe environment in the crash handler is to have the dump generation run inside another process.

>> Yes, I have got it from the breakpad doc, that is called out-of-process.

I'm not aware of breakpad having any other way to operate - we're replacing it with crashpad in part to avoid this limitation.

 

>> I want to implement it before, but  it is sensitive for creating a new process, and I am not sure if it will resolve the problem in OOM Crash, what do you think ? 

I'm not sure what you're asking; as I said we don't use/support any other way to handle crashes on android right now (as we haven't yet moved to crashpad) and I'm not aware of breakpad having any alternative.

 

OOM crashes on 32-bit android aren't very interesting in general

>> Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate OOM crashes because they very rarely lead to finding an actual bug that can be fixed - so we don't consider it a huge problem that breakpad fails to generate a dump in some OOM situations. (though it is still a bit of a problem, and we're moving to crashpad in part to avoid it).

 

>> I am using Android 7.0, it is definitely a 32-bit Android, or there may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is whether the chromium build you are using is 32-bit or 64-bit. By default it's 32-bit, and while it's possible to build a 64-bit build for some configurations (though not all; monochrome is always 32-bit for example) we don't ship the 64-bit version to anyone and generally don't recommend that people use it, because it currently uses quite a lot more memory, to the point that the user experience is significantly worse. Unless you explicitly chose to build the 64-bit version when configuring your build, you are using the 32-bit version.

[Liu Xiao]

Thank you very much.

I want to ask a question that when crashpad can be implemented on Linux and Android ? And it is sure to resolve the problem "The minidump can not always generated by breakpad" ?

[Torne (Richard Coles)]

Crashpad for android is currently being worked on and hopefully will be enabled in the near future. I don't know about the status on Linux.

No crash handling solution originating from inside the app can guarantee that a dump will always be generated perfectly, but crashpad runs in a non-compromised context which makes the current most common failure modes of breakpad impossible.

[Liu Xiao]

Look forward to crashpad release.

Our team is now mainly using "unwind the stack (using conventional libunwind)" for analyzing crash, and it is rarely failed.

Do you think it is a safe method for developer?  

[Liu Xiao]

though not all; monochrome is always 32-bit for example

>> Recently I have seen "monochrome " many times, is it a lite version of chrome ?

>> I can't get the precise definition of it, could you help me?

[Torne (Richard Coles)]

On Tue, 24 Jul 2018 at 22:56 Liu Xiao <liuxia...@gmail.com> wrote:

Look forward to crashpad release.

Our team is now mainly using "unwind the stack (using conventional libunwind)" for analyzing crash, and it is rarely failed.

Do you think it is a safe method for developer?  

In our official builds it's impossible to use libunwind, because official builds don't include unwind tables in the binary on-device, to reduce the size of the binary.

If you're including the unwind tables then libunwind will probably generally work but I have no idea whether that's actually expected/guaranteed to be safe to call in a compromised context.

[Torne (Richard Coles)]

On Wed, 25 Jul 2018 at 02:05 Liu Xiao <liuxia...@gmail.com> wrote:

though not all; monochrome is always 32-bit for example

>> Recently I have seen "monochrome " many times, is it a lite version of chrome ?

 

>> I can't get the precise definition of it, could you help me?

Monochrome is a version of chrome for android that can also function as the device's implementation of WebView, for use in other apps. Unless you are building your own custom android OS image, Monochrome is of no use to you - only apps specifically designated by the OS can be used as a webview implementation.

Monochrome is significantly larger than regular chrome, because it has to include both 32 and 64 bit versions of the webview code (it's just that it's smaller than the *total* size of the separate versions of chrome and webview, so it's a net saving when it replaces those).

[Liu Xiao]

Thank you very much.

In our official builds it's impossible to use libunwind, because official builds don't include unwind tables in the binary on-device, to reduce the size of the binary.

>> Could I understand like this : the minidump generated by breakpad is the only method for analyzing crash on official builds, though there is the defect "The minidump can not always generated" ?

[Torne (Richard Coles)]

Correct.