[llvm-dev] [clang] Running a single testcase

[Sedat Dilek via llvm-dev]

Hi,

while experimenting with llvmlinux on Debian/testing AMD64 I wanted to
run some x86-64 ASM tests.

I fell over [1] and wanted to run it.

So, I cloned clang from Git...

$ git clone https://github.com/llvm-mirror/clang.git

I looked through some docs where I have seen I need "llvm-lit" or "lit.py".
The Debian package llvm-7-tools from <apt.llvm.org> does ship "lit.py".
Furthermore, I have installed llvm-7 and clang-7 packages.
The version is 1:7~svn330207-1~exp1+0~20180417201234.1709~1.gbp6fb10d.

[5,6] have some examples on running a single testcase:

For example:

python C:\Tools\llvm\utils\lit\lit.py -sv
--param=build_mode=Win32 --param=build_config=Debug
--param=clang_site_config=C:\Tools\build\tools\clang\test\lit.site.cfg
C:\Tools\llvm\tools\clang\test\Sema\wchar.c

I transformed this...

$ python /usr/lib/llvm-7/build/utils/lit/lit.py -sv
/home/sdi/src/linux-kernel/clang/test/Sema/asm.c
lit.py: /usr/lib/llvm-7/build/utils/lit/lit/TestingConfig.py:101:
fatal: unable to parse config file
'/home/sdi/src/linux-kernel/clang/test/lit.cfg.py', traceback:
Traceback (most recent call last):
File "/usr/lib/llvm-7/build/utils/lit/lit/TestingConfig.py", line
88, in load_from_path
exec(compile(data, path, 'exec'), cfg_globals, None)
File "/home/sdi/src/linux-kernel/clang/test/lit.cfg.py", line 25, in <module>
config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
AttributeError: 'NoneType' object has no attribute 'use_lit_shell'

Yes, I have a lit.cfg.py.

$ LC_ALL=C ll clang/test/lit.*
-rw-r--r-- 1 sdi sdi 5979 Apr 25 12:14 clang/test/lit.cfg.py
-rw-r--r-- 1 sdi sdi 1906 Apr 25 12:14 clang/test/lit.site.cfg.py.in

How can I create a lit.site.cfg.py to pass it as
--param=clang_site_config=/path/to/lit.site.cfg.py?

I have attached TestingConfig.py from llvm-7-tools and lit.cfg.py and
lit.site.cfg.py.in from clang-git.

What am I missing?
What do I need to run a single clang testcase?

Thanks in advance.

Regards,
- Sedat -

[1] https://github.com/llvm-mirror/clang/raw/master/test/Sema/asm.c
[2] https://llvm.org/docs/CommandGuide/lit.html
[3] https://llvm.org/docs/TestingGuide.html
[4] https://llvm.org/docs/TestSuiteMakefileGuide.html
[5] http://clang.llvm.org/hacking.html
[6] http://clang.llvm.org/hacking.html#testingCommands

[Brian Cain via llvm-dev]

The simplest way to run a clang test case that I know of is to clone both llvm and clang repos, run all the tests, then run an individual test.

IIRC like so:

git clone llvm ......

cd llvm/tools

git clone clang .....

cd ../../

mkdir build

cd build

cmake ../llvm

ninja check-clang

./bin/llvm-lit -v ./tools/clang/test/Sema/asm.c

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

--

-Brian

[Sedat Dilek via llvm-dev]

On Mon, May 7, 2018 at 5:52 AM, Brian Cain <brian...@gmail.com> wrote:
> The simplest way to run a clang test case that I know of is to clone both
> llvm and clang repos, run all the tests, then run an individual test.
>
> IIRC like so:
>
> git clone llvm ......
> cd llvm/tools
> git clone clang .....
> cd ../../
> mkdir build
> cd build
> cmake ../llvm
> ninja check-clang
> ./bin/llvm-lit -v ./tools/clang/test/Sema/asm.c
>

Hi Brian,

The cmake-line should be...

# apt-get install ninja-build
$ cmake -G Ninja ../llvm

Unfortunately, on the latest Git my build fails which gcc-7 (see
attached file for details).
Do I really need to build clang-from-git?
I have llvm-7/clang-7 binaries installed on my system?

But I run the test after symlinking clang-7 ---> clang...

$ ./bin/llvm-lit -v ./tools/clang/test/Sema/asm.c
llvm-lit: /home/sdi/src/llvm/llvm/utils/lit/lit/llvm/config.py:334:
note: using clang: /usr/bin/clang
llvm-lit: /home/sdi/src/llvm/llvm/utils/lit/lit/llvm/subst.py:126:
note: Did not find c-index-test in
/home/sdi/src/llvm/build/./bin:/home/sdi/src/llvm/build/./bin
llvm-lit: /home/sdi/src/llvm/llvm/utils/lit/lit/llvm/subst.py:126:
note: Did not find clang-check in
/home/sdi/src/llvm/build/./bin:/home/sdi/src/llvm/build/./bin
llvm-lit: /home/sdi/src/llvm/llvm/utils/lit/lit/llvm/subst.py:126:
note: Did not find opt in
/home/sdi/src/llvm/build/./bin:/home/sdi/src/llvm/build/./bin
-- Testing: 1 tests, 1 threads --
PASS: Clang :: Sema/asm.c (1 of 1)
Testing Time: 0.23s
Expected Passes : 1

Looks good?

Thanks.

Regards,
- Sedat -

[Amara Emerson via llvm-dev]

That looks like your system ran out of memory during linking, which on debug builds is possible especially if you use parallel jobs with -j. You can’t use a different clang binary to run tests and expect them to work, the tests are specific to each individual version of the compiler.

Try using -DCMAKE_BUILD_TYPE=Release which requires less memory to link, unless you really need debug info in the binary, in which case try running ninja -j1 to avoid parallel jobs.

Amara

[Sedat Dilek via llvm-dev]

Hi Amara,

yes, that sounds reasonable.

I looked into my old build_llvm-toolchain.sh script and adapted it to
build only the necessary stuff (see below).

First, I thought I can reduce build-time by...

sdi@iniza:~/src/llvm-toolchain/build$ ninja check-llvm-tools-llvm-lit
[1808/1809] Running lit suite
/home/sdi/src/llvm-toolchain/llvm/test/tools/llvm-lit
Testing Time: 0.11s
Expected Failures : 1

The test runs with some warnings...

sdi@iniza:~/src/llvm-toolchain/build$ ./bin/llvm-lit -v
./tools/clang/test/Sema/asm.c
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/config.py:334:
note: using clang: /usr/bin/clang
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/subst.py:126:

note: Did not find c-index-test in

/home/sdi/src/llvm-toolchain/build/./bin:/home/sdi/src/llvm-toolchain/build/./bin
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/subst.py:126:

note: Did not find clang-check in

/home/sdi/src/llvm-toolchain/build/./bin:/home/sdi/src/llvm-toolchain/build/./bin
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/subst.py:126:
note: Did not find clang-diff in
/home/sdi/src/llvm-toolchain/build/./bin:/home/sdi/src/llvm-toolchain/build/./bin
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/subst.py:126:
note: Did not find clang-format in
/home/sdi/src/llvm-toolchain/build/./bin:/home/sdi/src/llvm-toolchain/build/./bin

-- Testing: 1 tests, 1 threads --
PASS: Clang :: Sema/asm.c (1 of 1)

Testing Time: 0.11s
Expected Passes : 1

... but it looks like stuff is missing, so I did...

sdi@iniza:~/src/llvm-toolchain/build$ ninja check-clang
[1012/1013] Running the Clang regression tests
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/config.py:334:
note: using clang: /home/sdi/src/llvm-toolchain/build/bin/clang
Testing Time: 200.66s
Expected Passes : 12028
Expected Failures : 18
Unsupported Tests : 256

sdi@iniza:~/src/llvm-toolchain/build$ ./bin/llvm-lit -v
./tools/clang/test/Sema/asm.c
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/config.py:334:
note: using clang: /home/sdi/src/llvm-toolchain/build/bin/clang

-- Testing: 1 tests, 1 threads --
PASS: Clang :: Sema/asm.c (1 of 1)

Testing Time: 0.10s
Expected Passes : 1

This now looks good to me.

How can I force to use a different clang than
"/home/sdi/src/llvm-toolchain/build/bin/clang"?
I have no symlink to /usr/bin/clang.

Thanks.

Regards,
- Sedat -

P.S.: New build-script

[build_clang-with-ninja.sh ]

#!/bin/sh

export LANG=C
export LC_ALL=C

cd build

CMAKE="cmake"

CMAKE_OPTS="-G Ninja -DCMAKE_C_COMPILER=clang-7
-DCMAKE_CXX_COMPILER=clang++-7 -DLLVM_PARALLEL_COMPILE_JOBS=2
-DLLVM_PARALLEL_LINK_JOBS=1 -DCMAKE_BUILD_TYPE=RELEASE
-DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=X86"

$CMAKE ../llvm $CMAKE_OPTS

ninja check-clang
- EOF -

[Brian Cain via llvm-dev]

On Mon, May 7, 2018 at 10:13 AM, Sedat Dilek <sedat...@gmail.com> wrote:

...

 

sdi@iniza:~/src/llvm-toolchain/build$ ./bin/llvm-lit -v
./tools/clang/test/Sema/asm.c
llvm-lit: /home/sdi/src/llvm-toolchain/llvm/utils/lit/lit/llvm/config.py:334:
note: using clang: /home/sdi/src/llvm-toolchain/build/bin/clang
-- Testing: 1 tests, 1 threads --
PASS: Clang :: Sema/asm.c (1 of 1)
Testing Time: 0.10s
  Expected Passes    : 1

This now looks good to me.

How can I force to use a different clang than
"/home/sdi/src/llvm-toolchain/build/bin/clang"?
I have no symlink to /usr/bin/clang.

 Can you give a little more context?  What is your ultimate goal for this activity?  Why would you want to run the test in this fashion and what meaning would you attribute to the test passing or failing?

-Brian

[Sedat Dilek via llvm-dev]

When compiling a Linux-kernel I hit a problem in the
KVM/paravirtualization area and "clobbering RCX" on Debian/testing
AMD64.
The question came up, is this is a compiler bug or not.
Details see [1].

Thus, I was looking for a reproducible testcase.
I have seen that clang ships some tests.
So, I was curious to run a single one.
Not sure if clang/test/Sema/asm.c is suitable here to see "clobbering
RCX" on Debian/x86-64.

On my host I have installed a clang-7 package from <apt.llvm.org>.
I wanted to test against that version.

Does this help you?

- Sedat -

[1] https://lkml.org/lkml/2018/4/24/1227

[Brian Cain via llvm-dev]

On Mon, May 7, 2018 at 10:46 AM, Sedat Dilek <sedat...@gmail.com> wrote:

On Mon, May 7, 2018 at 5:19 PM, Brian Cain <brian...@gmail.com> wrote:
>
>

...

 

>  Can you give a little more context?  What is your ultimate goal for this
> activity?  Why would you want to run the test in this fashion and what
> meaning would you attribute to the test passing or failing?
>

When compiling a Linux-kernel I hit a problem in the
KVM/paravirtualization area and "clobbering RCX" on Debian/testing
AMD64.
The question came up, is this is a compiler bug or not.
Details see [1].

Thus, I was looking for a reproducible testcase.
I have seen that clang ships some tests.
So, I was curious to run a single one.
Not sure if clang/test/Sema/asm.c is suitable here to see "clobbering
RCX" on Debian/x86-64.

Yes, I agree that it might not be the right test.

 

On my host I have installed a clang-7 package from <apt.llvm.org>.
I wanted to test against that version.

Does this help you?

- Sedat -

[1] https://lkml.org/lkml/2018/4/24/1227

I see, yes, this is clearer now.

IIUC, this test is testing clang's behavior regarding inline assembly and probably won't verify things like which registers are saved and restored.

I think you might get resolution on this issue faster if you reduce native_save_fl() and its caller to a simple example that shows RCX not being saved.  Bonus points if you can identify any conditions that change this behavior -- or versions of clang that behave differently.  Then share your test and ask on llvm-dev and/or cfe-dev whether or not this behavior is expected.

--

-Brian

[Sedat Dilek via llvm-dev]

Matthias answered in [1].

[1] https://lkml.org/lkml/2018/5/7/534

[George Karpenkov via llvm-dev]

Hi,

> On May 6, 2018, at 5:10 AM, Sedat Dilek via llvm-dev <llvm...@lists.llvm.org> wrote:
>
> What am I missing?
> What do I need to run a single clang testcase?

I might be missing some context here, but to address a very similar problem I have introduced an environment variable LIT_FILTER.
So if you want to run the test blah.test, you could run

$ env LIT_FILTER=blah.test ninja check-clang

I have found that much easier than trying to invoke clang directly.

George

>
> Thanks in advance.
>
> Regards,
> - Sedat -
>
> [1] https://github.com/llvm-mirror/clang/raw/master/test/Sema/asm.c
> [2] https://llvm.org/docs/CommandGuide/lit.html
> [3] https://llvm.org/docs/TestingGuide.html
> [4] https://llvm.org/docs/TestSuiteMakefileGuide.html
> [5] http://clang.llvm.org/hacking.html
> [6] http://clang.llvm.org/hacking.html#testingCommands

> <TestingConfig.py><lit.cfg.py><lit.site.cfg.py.in>_______________________________________________