How to debug sequence check failures?

[Jeroen Dhollander]

Hey Chromium experts,

People on my team regularly run into sequence check failures when refactoring code/developing features.

Unfortunately these failures are very cryptic and don't give us a lot to go on, like this:

2021-01-26T19:16:33.373651Z FATAL chrome[3168999:3169317]: [sequence_checker.h(120)] Check failed: checker.CalledOnValidSequence(). 

#0 0x7f89a965cf29 base::debug::CollectStackTrace()

#1 0x7f89a9568fd3 base::debug::StackTrace::StackTrace()

#2 0x7f89a9587824 logging::LogMessage::~LogMessage()

#3 0x7f89a95880de logging::LogMessage::~LogMessage()

#4 0x7f89a8f6e79b base::ScopedValidateSequenceChecker::ScopedValidateSequenceChecker()

#5 0x7f89a8f77a1f mojo::InterfaceEndpointClient::SendMessage()

 

Anytime we run into an error like that we usually end up losing about a full day (or more) trying to figure out what exactly is going wrong.

We haven't found any good way to get more information.

Would it be possible to enhance the sequence checker so it outputs more useful information? One solution we came up with is giving each sequence an (optional) name when it's created, like this:

scoped_refptr<SequencedTaskRunner> sequenced_task_runner = base::ThreadPool::CreateSequencedTaskRunner("AssistantBackgroundSequence", ...);

 

This would allow the checker to print both the actual and expected sequence, which should make debugging a lot easier.

What do you all think? Is there a better way to debug sequence check failures?

With regards,

Jeroen

[Daniel Cheng]

When debugging these issues, is it hard to understand what sequence the code should be running on, or what sequence is currently on the stack?

With Chrome code, we often try to structure things such that a given class has a well-defined owner sequence. In theory, this should make it easy to understand what sequence an object should be used on when the SequenceChecker complains, by referencing the code and comments of the class that's failing the sequence check.

In theory, it should also usually be easy to understand what sequence is currently running, though this can be complicated by external libraries that make fewer guarantees about threading.

For this particular crash, the stack is incomplete so it's hard to be certain, but we're either calling a method through a Mojo remote on the wrong sequence, or invoking a Mojo reply callback on the wrong sequence. If the rest of the stack were included though, I wouldn't expect diagnosing this failure to take too long

Daniel

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHLDLCnTT8g4cXhLofN1q%3DUng2f_s6tvQKdad5B%3De7b4BY1CvQ%40mail.gmail.com.