1500 task limit is not enough: Re: Local run of mochitest under valgrind aborted due to too many tasks. 500 tasks (!?)

[ISHIKAWA,chiaki]

Something is definitely wrong.

I upped the max threads to 1500, and valgrind still blows up.
It seems that the valgrind+TB combination tries to create more than 1500
threads during mochitest, or to be more precise, at the starting phase
of mochitest.

TIA

Chiaki


On 2020/04/13 3:26, ISHIKAWA,chiaki wrote:
> Without valgrind, the use of "no target", i.e., specifying no target
> after |mochitest| command seems to run the all the mochitest
> intended for TB after I |cd|ed to the C-C  ./comm subdirectory of M-C
> ./mozilla directory.
>
> However, with the same target, if I run TB under valgrind, the
> valgrind run blows up saying
> too many tasks (default 500) and this happened even before a SINGLE
> test is run.
>
> So I suspect
> either
> 1. the default mochitest sets up consumes THAT MANY tasks (does anyone
> if this is also the case with valgrind test of FF run that is done a
> few weeks apart or every month? In that case, adding
> "--max-threads=1000 "  /* April, 2020. */
> to valgrind argument seems to be the way to go. But still.)
>
> 2. TB's mochitest invokded inside valgrind does something funny to
> create so many tasks.
>
> 3. Possible valgrind issue:
>
> I notice that valgrind
> printed the following warning during local run, which is known
> valgrind issue.: syscall 315
>
>  0:21.22 GECKO(275735) --275739-- WARNING: unhandled amd64-linux
> syscall: 315
>
> https://github.com/svn2github/valgrind-master-mirror/blob/master/docs/internals/3_12_BUGSTATUS.txt
>
>
> (My valgrind is farily recent.: valgrind --version
> valgrind-3.16.0.GIT )
>
> syscall 315 is |sched_getattr|.
> So it *IS* possible that there could be some issues related to
> But if so, there can and should be a problem with FF run under
> valgrind as well.
> I am a bit puzzled.
>
> The investigation continues, but I will file a bug regarding this with
> an excerpt from the log.
>
> I believe running TB's mochitest under valgrind is a very good way to
> detect memory issues based on my experience of
> uncovering memory issues  by running |make mozmill| under valgrind.
>
> TIA
>
> Chiaki
>
> On 2020/04/10 19:15, ISHIKAWA,chiaki wrote:
>> Hi,
>>
>> Short questions:
>>
>> What is the proper target of mochitest in order to cover all the
>> subtests of mochitest for TB?
>>
>> Aslo, is using 500 threads for mochitest to be expected?
>>
>>
>> Background that led to the questions above.:
>>
>> I upgraded the local C-C and M-C tree a few days ago, and build TB
>> with my local patches.
>>
>> The ordinary run of mochitest ran fine. (Is that I think.)
>>
>> However, I wanted to run mochitest under valgrind.
>> With a trick I mention later, I have run thunderbird under valgrind
>> with xpcshell-tests and |make mozmill| test before.
>>
>> However, when I ran the following command, valgrind blew up saying
>> there are too many threads, 500 threads. Whoa?
>>
>> time /NREF-COMM-CENTRAL/mozilla/do-make-tb.sh --log-no-times
>> mochitest comm/mail/components/extensions/test/browser
>>
>> |do-make-tb.sh| is a script to run |mach| after setting various
>> environment variables and search path, etc. for my local configuration.
>> This has worked for a few years now after I switch to use |mach|
>> instead of invoking |make| directly, too.
>>
>> PLEASE NOTE THE TARGET, however. :
>>
>> comm/mail/components/extensions/test/browser
>>
>> I picked it up from the recent post regarding the test documentation
>> page. [0]
>> The target is mentioned in the URL [1]
>> Maybe I am wrong to use the target?
>>
>> I checked my log and I found that I tried to run mochitest back in
>> February and in that case,
>> I changed the current directory to /comm/ subdirectory under
>> /mozilla/ before invoking |mach|, and simply ran
>> |mach| command WITHOUT ANY TARGET.
>>
>> pwd
>> /NREF-COMM-CENTRAL/mozilla/comm
>> date
>> Sat Feb 22 06:02:34 JST 2020
>>
>> time /NREF-COMM-CENTRAL/mozilla/do-make-tb.sh --log-no-times
>> mochitest $*
>>
>> Now I am not sure if that has run all the mochitest subtests meant
>> for TB.
>>
>> For that matter, the target mentioned in [1] seems to run only the
>> browser functionality of TB?
>> (I don't seem to see addressbook related tests log in the final log,
>> for example. Hmm....)
>>
>> Q-1: Does someone have an opinion of what is the appropriate target
>> (or targets?)  to use in order run all the mochitest subtests for TB
>> in one shot?
>> I saw that I used comm/mail/test once, maybe while I was in /mozilla
>> M-C tree.
>> But I am not sure if it was good enough.
>> At least, I now have a suspicion that the target mentioned in [1] *IS
>> WRONG*.
>>
>> Finally the basic question: # of threads.
>>
>> Q-2: Is the number, more than 500, of threads to be expected for a
>> mochitest of TB?
>> Using 500 threads sounds too excessive for a mailer.
>> Just consider the cache issue... Is this why TB sometimes feels slowish?
>>
>> TIA
>>
>> Chiaki
>>
>> [0] post to tb-planning:
>> https://mail.mozilla.org/pipermail/tb-planning/2020-April/007532.html
>> [1]
>> https://developer.thunderbird.net/thunderbird-development/testing/running-tests
>>
>> PS: The trick to run TB under valgrind for various tests.
>>
>> I replaced |thunderbird| binary under MOZ_OBJECT/dist/bin by a small
>> program
>> that runs |thunderbird-bin| under the same directory under |valgrund|
>> with appropriate options.
>>
>> This has worked fine for many years now.
>> Even after the major source change of the last 12-24 months, for
>> example.
>>
>> I have run xpcshell test under valgrind as late as Jan 31 this year
>> using the trick.
>> It worked just fine and I reported a few bug reports about the memory
>> issues.
>> (However, I have to run the tests sequentially against many test
>> directories. See the script at the end.)
>>
>> I ran |make mozmill| test under valgrind for many years also.
>> But as you may know |make mozmill| is gone.
>>
>> So I adjusted my local debug scripts (including the scripts that pick
>> up interesting warnings/errors) to run mochitest under valgrind locally.
>>
>> So this is probably the first time I run full blown
>> valgrind+mochitest locally.
>> As soon as I invoked the command mentioned above, valgrind aborted
>> saying too many tasks.
>> Unbelievable. There are 500 tasks and valgrind could not accommodate
>> more tasks. Is this to be expected ?
>>
>>  4:53.26 GECKO(171961) Use --max-threads=INT to specify a larger
>> number of threads
>>  4:53.26 GECKO(171961) and rerun valgrind
>>  4:53.26 GECKO(171961) valgrind: the 'impossible' happened:
>>  4:53.26 GECKO(171961)      Max number of threads is too low
>>  4:53.26 GECKO(171961) host stacktrace:
>>
>>
>> I was about to change this little program to run valgrind with
>> appropriate options to bump the # of threads valgrind handles when I
>> though I should consult the mailing list. Something *IS* fishy.
>>
>> ---
>> PPS: Shell script code to run xpcshell tests sequentially. Valgrind
>> outputs must be separated for individual tests.
>>
>> date
>>
>> STARTING_SEC=$(date +"%s")
>>
>> export MALLOC_PERTURB_=0x5A
>>
>> TESTDIRS="
>> comm/chat/components/src/test
>> comm/chat/modules/test
>> comm/chat/protocols/irc/test
>> comm/chat/protocols/skype/test
>> comm/chat/protocols/xmpp/test
>> comm/common/saxparser/test/unit
>> comm/common/test/xpcshell
>> comm/ldap/xpcom/tests/unit
>> comm/mail/base/test/unit
>> comm/mail/components/enterprisepolicies/tests/xpcshell
>> comm/mail/components/extensions/test/xpcshell
>> comm/mail/components/test/unit
>> comm/mailnews/addrbook/test/unit
>> comm/mailnews/base/test/unit
>> comm/mailnews/compose/test/unit
>> comm/mailnews/db/gloda/test/unit
>> comm/mailnews/db/msgdb/test/unit
>> comm/mailnews/extensions/bayesian-spam-filter/test/unit
>> comm/mailnews/extensions/mdn/test/unit
>> comm/mailnews/extensions/newsblog/test/unit
>> comm/mailnews/imap/test/unit
>> comm/mailnews/import/test/unit
>> comm/mailnews/intl/test/unit
>> comm/mailnews/jsaccount/test/unit
>> comm/mailnews/local/test/unit
>> comm/mailnews/mime/jsmime/test
>> comm/mailnews/mime/test/unit
>> comm/mailnews/news/test/unit
>> "
>>
>> set -vx
>>
>> for dir in $TESTDIRS
>> do
>>     bash -vx ~ishikawa/bin/nrun-xpcshell-memcheck.sh --sequential ${dir}
>> done
>>
>> ### nrun-xpcshell-memecheck.sh invokes the xpcshell-tests with
>> --verbose flag after setting up various
>> ### environment variable and paths for my local configuration.
>> ### Without --verbose, I won't get valgrind trace at all.
>>
>> date
>>
>> echo
>> ENDING_SEC=$(date +"%s")
>> echo " * * * * * * * * * * * * * * "
>> echo "It took " $((ENDING_SEC - STARTING_SEC)) " seconds."
>> echo " * * * * * * * * * * * * * * "
>>
>>
>> PPS: Running xpcshell-tests takes about 12-13 hours of wall time on
>> my PC if TB is run under valgrind.
>> |make mozmill| use to take a similar time to finish again if TB is
>> run under valgrind.
>> I am keen to learn if mochitest is significantly faster or not.
>>
>> [end of message]
>>
>> _______________________________________________
>> dev-apps-thunderbird mailing list
>> dev-apps-t...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-apps-thunderbird
>
>
> _______________________________________________
> dev-apps-thunderbird mailing list
> dev-apps-t...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-apps-thunderbird

[ishikawa]

I filed

https://bugzilla.mozilla.org/show_bug.cgi?id=1629433
Running thunderbird binary under valgrind while trying to run mochitest
requires way TOO MANY threads (> 1500) !?

TIA

[ISHIKAWA,chiaki]

Upping the maximum number of threads to 5,000 let the test proceed
further (!).
So we are using somewhere between 1500 and 5000 threads.

This is too excessive for a testing of mere mail client IMHO.

Something is definitely wrong.

valgrind defaults to 500 threads.
The number is increased by --max-threads=NNN option.

I searched for this option by searchfox and did not find it if I am not
mistaken.
So probably running FF under valgrind does not require such option at all.

There may be something wrong in the way the thread creation is done for
mochitest of FF.

Who knows the execution of FF under valgrind in mozilla
compilation/testing CPU farm?
I may need to discuss this issue.
We may see a decrease of CPU and memory usage on testing CPU farm
if we can decrease the # of therads during testing siginificantly
(specially on the lower-end CPU such as ARM64, etc.)

TIA

Chiaki

[ISHIKAWA,chiaki]

On 2020/04/16 3:19, ISHIKAWA,chiaki wrote:
> There may be something wrong in the way the thread creation is done
> for mochitest of FF.

Ouch. I meant to say mochitest of TB.

TIA

Chiaki