Logging extra information in TSAN reports

[markdavi...@gmail.com]

I'm wondering if there's any way to annotate additional information into TSAN report to help me track down the state of the system when a detected issue (e.g., data race) is happening. For example, it would be nice to be able to tell TSAN to print the contents of variables x, y, and z when a data race is detected (examples of this could be the loop iteration, or the current size of a vector, etc.). This would be similar to breakpoint command lists in gdb.

It's possible that gdb is actually part of the solution here. I could create a breakpoint command on a TSAN intercept function and then somehow print out the contents of what I want. I did try that, but gdb didn't know about the relevant TSAN symbols, probably because my compiler (gcc 5.3) doesn't have debug info compiled into it. But, assuming I could set the right breakpoints on TSAN, and then add a command to navigate to the right stack frame, and then print the relevant variables, that still wouldn't really solve my problem as I'd like to see the variable contents in the actual TSAN error report so I can match up the variable contents with each TSAN warning.

My gdb idea above is really just trying to illustrate my general question; are there debugging techniques that people use to print out variable state at the time of data races?

Thank you. 

[Konstantin Serebryany]

The simplest you can do is to run your application with 

environment variable TSAN_OPTIONS=halt_on_error=1:abort_on_error=1 

which will cause tsan to a) halt on the first detected error and b) call abort() instead of exit(). 

Then you can catch abort in gdb and print whatever you like. 

An alternative is to define your own summary handler, like this: 

extern "C" void __sanitizer_report_error_summary(const char *s) {

  fprintf(stderr, "ZZZ\n");  // Print whatever you like. 

}

hth, 

--kcc 

--
You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Davis]

Konstantin, this is exactly the sort of thing I'm looking for. Thanks.
One more question related to the __sanitizer_report_error_summary
approach. This may be more of a C/C++ question than a TSAN question.
The state that I'd like to print out as part of the error summary are
private members of objects of particular class. That is, they are not
globally scoped / named / accessible. Could you offer any suggestions
to allowing __sanitizer_report_error_summary to get access to the
private state of the actual object that's exhibiting the race
condition? I could probably hack something together, but I'm asking as
there may be a recommended approach here.

The TSAN_OPTIONS=halt_on_error=1:abort_on_error=1 should work well,
although I'd like to also add in some extra information using the
__sanitizer_report_error_summary approach as well.

Thank you.

>> email to thread-sanitiz...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to thread-sanitiz...@googlegroups.com.

[Konstantin Serebryany]

On Tue, Aug 16, 2016 at 4:20 PM, Mark Davis <markdavi...@gmail.com> wrote:

Konstantin, this is exactly the sort of thing I'm looking for. Thanks.
One more question related to the __sanitizer_report_error_summary
approach. This may be more of a C/C++ question than a TSAN question.

Indeed :) 

 

The state that I'd like to print out as part of the error summary are
private members of objects of particular class. That is, they are not
globally scoped / named / accessible. Could you offer any suggestions
to allowing __sanitizer_report_error_summary to get access to the
private state of the actual object that's exhibiting the race
condition? I could probably hack something together, but I'm asking as
there may be a recommended approach here.

I would suggest this: 

let tsan identify the data member of a class that is being accessed racey. 

Then, confirm the race by pure C++ means (no tsan) using something like this: 

https://github.com/hellok/data-race-test-/blob/master/race_checker/race_checker.h

--kcc 

 

>> email to thread-sanitizer+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to thread-sanitizer+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "thread-sanitizer" group.

To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitizer+unsubscribe@googlegroups.com.

[Mark Davis]

That helps. Thank you!

>> >> email to thread-sanitiz...@googlegroups.com.

>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "thread-sanitizer" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to thread-sanitiz...@googlegroups.com.

>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "thread-sanitizer" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to thread-sanitiz...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "thread-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to thread-sanitiz...@googlegroups.com.

[b4ni...@gmail.com]

Hi,

I was just wondering at which point does the halt_on_error / abort_on_error is called. I tried with both flags and the program completes the execution on abort_on_error=1 (I have multiple tsan warnings printed as well), but the execution is interrupted on using halt_on_error=1.

Can you kindly tell how exactly these 2 flags work, which signal is passed and at which point of the program are they called (like the moment when tsan warning is generated) ?

Thanks in advance.

To unsubscribe from this group and stop receiving emails from it, send an email to thread-sanitiz...@googlegroups.com.

[Dmitry Vyukov]

As far as I remember, halt_on_error is checked when a warning is printed.
abort_on_error is checked only when we are exiting abnormally (due to
a warning or an internal error).
So if you want abort on warning you need to set both flags.

[Kuba Mracek]

Yup.

halt_on_error means: Stop at the first issue found.

abort_on_error means: When the process tries to exit() and there were some issues found, abort() instead. So the process runs all the way until it's finished and then we crash/abort.

Kuba

[markdavi...@gmail.com]

I'm trying this approach again and having trouble. Specifically, I'm trying to use gdb to inspect the state of the system when the first TSAN race is detected. I've set both of these flags to one and running under gdb with two threads. When I run in gdb, the program just exits as follows. It doesn't stop and let me inspect the state as I want.

[Thread 0x7eff1a09b700 (LWP 13721) exited]

[Thread 0x7eff1a89c700 (LWP 13717) exited]

[Thread 0x7eff1cbff700 (LWP 13710) exited]

[Inferior 1 (process 13691) exited with code 0102]

Note that I'm running with GCC 5.3 and gdb 7.7.1 in case it matters.    

Here are all of my tsan options and flags:

TSAN_FLAGS += -fsanitize=thread -fPIE -fPIC -g -O0 -fno-inline -fno-omit-frame-pointer

        LFLAGS += -pie

        LIBS += -ltsan

export TSAN_OUT_LOGFILE=tsan_out

TSAN_VERBOSITY ?= 2

export TSAN_OPTIONS="suppressions=$(SUPPORT_PATH)/sanitizer/aggressive_supressions log_path=$(SANITIZER_LOG_DIR)/$(TSAN_OUT_LOGFILE) halt_on_error=1 abort_on_error=1 verbosity=$(TSAN_VERBOSITY)"

What am I doing incorrectly here?

[Dmitry Vyukov]

On Wed, Aug 15, 2018 at 6:50 AM, <markdavi...@gmail.com> wrote:
> I'm trying this approach again and having trouble. Specifically, I'm trying
> to use gdb to inspect the state of the system when the first TSAN race is
> detected. I've set both of these flags to one and running under gdb with two
> threads. When I run in gdb, the program just exits as follows. It doesn't
> stop and let me inspect the state as I want.
>
> [Thread 0x7eff1a09b700 (LWP 13721) exited]
> [Thread 0x7eff1a89c700 (LWP 13717) exited]
> [Thread 0x7eff1cbff700 (LWP 13710) exited]
> [Inferior 1 (process 13691) exited with code 0102]
>
> Note that I'm running with GCC 5.3 and gdb 7.7.1 in case it matters.

Hi,

Did tsan print a warning? Because these flags take effect only on a warning.

You may also want to try a newer compiler. The flags were added at
some point, and maybe they are just not present in the older version.

[markdavi...@gmail.com]

I upgraded my compiler to clang 6.0.1 and that fixed this problem -- these options are now working as expected so it seems that your hypothesis, Dmitry, was correct. Note that I also had to change my compiler and linker options to no longer include pic and pie, nor libtsan; I assume this is now ok? Everything seems to work and I'm able to stop upon the error in gdb.

[Dmitry Vyukov]

On Wed, Aug 15, 2018 at 6:48 PM, <markdavi...@gmail.com> wrote:
> I upgraded my compiler to clang 6.0.1 and that fixed this problem -- these
> options are now working as expected so it seems that your hypothesis,
> Dmitry, was correct. Note that I also had to change my compiler and linker
> options to no longer include pic and pie, nor libtsan; I assume this is now
> ok? Everything seems to work and I'm able to stop upon the error in gdb.

Great!
Yes, with clang you just pass -fsanitize=thread.