[LLVMdev] [Patch] Adding unit tests to LLVM
Talin
This patch adds a unit test framework to LLVM, along with a sample unit test for DenseMap. I don't expect this patch to be accepted as-is, this is mainly a trial balloon and proof of concept.
-- Talin
--
-- Talin
Keir Mierle
(Forwarding this to llvm-dev)
This patch adds a unit test framework to LLVM, along with a sample unit test for DenseMap. I don't expect this patch to be accepted as-is, this is mainly a trial balloon and proof of concept.Some notes about the patch:
1) For the testing framework, I went with Google Test, since it's the one I have the most experience with. I fully expect an extended bikeshed discussion to result from this.
+1.
Google test is pretty nice (not that I'm unbiased, as I am a gtest contributor :)
2) Both the test framework and the tests are optional build targets, they will not be built with the normal "make all". To build and run the unit tests, use "make check-unit".
"make unittest"? I would never guess check-unit.
3) I did not actually include the testing framework in the patch; It will need to be checked in separately. There are two approaches to this. One approach is to use the svn:external feature to create a link to the googletest svn repository from the LLVM svn repository. The other approach is to take a snapshot of googletest and check it in to the LLVM repository.The GoogleTest tar archive is here: http://code.google.com/p/googletest/downloads/list. I've located it within the LLVM source tree in the location "third-party/googletest".
I suggest checking a specific version of Google test into the source. There isn't much clear benfit to tracking upstream gtest. On the other hand, there is benefit to not having mysterious breakages due to upstream (as unlikely as that is due to gtest API guarentees). Also, for projects adding the gtest source directly to their tree, there is a special gtest-all.c which contains the entire gtest codebase in a single file (it's not that much code).
Keir
4) I've included makefiles for building the test framework and unit tests. I had to do a bit of makefile hacking to get this to work, suggestions on how to improve this are welcome.
--
-- Talin
--
-- Talin
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Talin
willing to substantially rework things if requested.
I should note that part of my motivation for doing this is that I have
been wanting to do some work on the various LLVM container classes;
However, I strongly feel that without some kind of unit test framework
in place such work would mostly be a waste of time.
_______________________________________________
Bill Wendling
I just got around to looking at this. Sorry for the delay.
I like the idea of unit tests *A LOT*. Adequate testing has been a
major sticking point for LLVM for a long time. In my opinion, once the
color of the bike shed is agreed upon ;-), this will be a welcome
addition. I have never worked with Google's unit testing infrastucture
so I don't know how it works or if its license is compatible with
ours, etc.
I did a very quick look at your code and it seems fine for a first
stab. Let's try to get everyone's input before you do a ton of work.
But I for one think that it will be accepted when people are satisfied
with the form it takes.
-bw
Talin
The googletest framework is open source, and licensed under the "new
BSD" license. I'll leave it to the license lawyers to determine
compatibility.
Introductory docs for googletest are here:
http://code.google.com/p/googletest/wiki/GoogleTestPrimer
You might also want to read this (somewhat out of date) comparison of
C++ unit testing frameworks:
"Exploring the C++ Unit Testing Jungle" http://gamesfromwithin.com/?p=29
Note that this doesn't include googletest or UnitTest++, both of which
were developed after the article was written. (The latter was developed
by the author of the above article.) However, the article does give a
good overview of what is desirable in a unit testing framework.
Also note that one additional requirement of a unit test framework for
LLVM is that it should not rely on RTTI or exceptions, since LLVM is
built with both of those features disabled.
Misha Brukman
> (Forwarding this to llvm-dev)
>
> This patch adds a unit test framework to LLVM, along with a sample unit test
> for DenseMap. I don't expect this patch to be accepted as-is, this is mainly
> a trial balloon and proof of concept.
I think this is a great idea! As Keir already noted, I would also
agree with LLVM snapshotting a copy of googletest, but I think it
should live in llvm/test/googletest (rather than top-level), since
it's not going to be linked into anything outside of llvm/test.
On the assumption that we'll end up using googletest, here are some
comments on your patch:
* s/Insure/Ensure/
* LLVM uses "llvm/foo.h" for inclusion rather than <llvm/foo.h>
* You should use the same format for gtest headers
* If reverse iteration isn't supported, you should either have an
ASSERT_DEATH() on the decrement, or not have the code there (that's
commented out) at all.
* Instead of this:
EXPECT_TRUE(uintMap[0] == 1);
you should use EXPECT_EQ()
* Instead of this:
EXPECT_TRUE(uintMap.find(0u) == uintMap.begin());
is it possible to use EXPECT_EQ() as well?
* In this test:
TEST_F(DenseMapTest, IterationTest) {
you use the array "int numbers[100];" as an array of booleans; why not
make it "bool visited[100];" to make clear what it for, how it's used,
and maybe be slightly more efficient?
> 1) For the testing framework, I went with Google Test, since it's the one I
> have the most experience with. I fully expect an extended bikeshed
> discussion to result from this.
>
> 2) Both the test framework and the tests are optional build targets, they
> will not be built with the normal "make all". To build and run the unit
> tests, use "make check-unit".
I am with Keir on this one: "check-unit" isn't the most intuitive name
to me -- "unittest" sounds fine, but this isn't a big deal to me.
I'll also volunteer to help maintain the LLVM branch of googletest and
review the unittests being checked in, as I also have experience with
using googletest, and I think LLVM could derive a lot of value from
unittesting.
Misha
Keir Mierle
If we make useful changes to gtest, I volunteer to push them upstream.
Keir
Mark Kromis
Talin
> On Dec 22, 7:34 pm, Talin <viri...@gmail.com> wrote:
>
>> (Forwarding this to llvm-dev)
>>
>> This patch adds a unit test framework to LLVM, along with a sample unit test
>> for DenseMap. I don't expect this patch to be accepted as-is, this is mainly
>> a trial balloon and proof of concept.
>>
>
> I think this is a great idea! As Keir already noted, I would also
> agree with LLVM snapshotting a copy of googletest, but I think it
> should live in llvm/test/googletest (rather than top-level), since
> it's not going to be linked into anything outside of llvm/test.
>
makefile in that directory contains a whole bunch of rules for
interfacing with DejaGNU. The unit tests don't (and, I think, shouldn't)
require any dependence on DejaGNU -- in fact, I'm hoping it will be
possible to run the unit tests with only the base LLVM package, without
all of the additional package installations required to run the other
LLVM tests. I'm just wondering if it will be a problem organizing the
makefile so that the unit tests don't have any dependence on the other
tests.
> On the assumption that we'll end up using googletest, here are some
> comments on your patch:
>
> * s/Insure/Ensure/
>
> * LLVM uses "llvm/foo.h" for inclusion rather than <llvm/foo.h>
> * You should use the same format for gtest headers
>
> * If reverse iteration isn't supported, you should either have an
> ASSERT_DEATH() on the decrement, or not have the code there (that's
> commented out) at all.
>
> * Instead of this:
> EXPECT_TRUE(uintMap[0] == 1);
> you should use EXPECT_EQ()
>
> * Instead of this:
> EXPECT_TRUE(uintMap.find(0u) == uintMap.begin());
> is it possible to use EXPECT_EQ() as well?
>
you can make anything printable by adding the necessary stream operator
overloads. In this particular case, I decided that the printed
representation of an iterator wouldn't be meaningful, so I didn't bother
defining those overloads.
> * In this test:
> TEST_F(DenseMapTest, IterationTest) {
> you use the array "int numbers[100];" as an array of booleans; why not
> make it "bool visited[100];" to make clear what it for, how it's used,
> and maybe be slightly more efficient?
>
Misha Brukman
Just a curiosity question, why push for gtest vs Boost Test or a different test suite?I normally use Boost, and their test suite, so I'm more familiar with that. So I was wondering is one better then the other, or is it just that someone makes a patch for it?
It seems that the original patch author is most familiar with gtest (as am I). I looked at boost's testing facilities, as described here: http://gamesfromwithin.com/?p=29#boost and it seems to be quite more verbose for setting up test suites, with what looks like a bit of boilerplate code.
Compare this with the way new tests are written in gtest with TEST_F() macro -- see the original patch by Talin. Note that there's no suite-setup code in your unittest file in gtest, which is required in boost. In fact, the only gtest-related code is the #include statement, not counting the common main() that's linked into every unittest binary -- unless you want to do some custom global-setup code, and hence you'll write your own main().
Misha
Misha Brukman
Just a curiosity question, why push for gtest vs Boost Test or a different test suite?I normally use Boost, and their test suite, so I'm more familiar with that. So I was wondering is one better then the other, or is it just that someone makes a patch for it?
I looked more into Boost.Test to see what's in it. Boost.Test doesn't seem to be stand-alone -- I don't see a way to use Boost.Test without importing some other chunks of Boost that the testing library depends on. While Boost is a fine set of libraries, I don't think we want to increase the LLVM distribution by sizeof(Boost) just to enable unittesting, nor do we want to spend the time on maintaining a subset of Boost that's "just enough" to build and use the unittest library, along a modified configure/build process that Boost wants to use (Boost.Build? Boost.Jam?).
Boost also seems to want to use exceptions, and LLVM does not want to. I'm not sure if there would be some difficulties in running a build where some libraries are compiled with no exceptions, some with, and the results are linked together. At the best case, it would complicate our build system to be able to support different set of flags for building LLVM libraries vs. Boost.Test (and the rest of Boost that we import).
Sample usage of Boost.Test: http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp
Note the code at the end setting up the test suite -- this is boilerplate code that I think shouldn't be necessary to setup and run tests.
Google Test, on the other hand, has no external dependencies, and is distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; and doesn't use exceptions or RTTI.
Sample usage of GTest: http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc
GTest-specific LOC besides the #include statement: 0.
Note that I'm not counting main() for either Boost or GTest, because both provide a standard main() for use with almost all test files.
Misha
Gordon Henriksen
2008/12/27 Mark Kromis <grey...@mac.com>Just a curiosity question, why push for gtest vs Boost Test or a different test suite?I normally use Boost, and their test suite, so I'm more familiar with that. So I was wondering is one better then the other, or is it just that someone makes a patch for it?
I looked more into Boost.Test to see what's in it. Boost.Test doesn't seem to be stand-alone -- I don't see a way to use Boost.Test without importing some other chunks of Boost that the testing library depends on. While Boost is a fine set of libraries, I don't think we want to increase the LLVM distribution by sizeof(Boost) just to enable unittesting, nor do we want to spend the time on maintaining a subset of Boost that's "just enough" to build and use the unittest library, along a modified configure/build process that Boost wants to use (Boost.Build? Boost.Jam?).
Mark Kromis
2008/12/27 Mark Kromis <grey...@mac.com>Just a curiosity question, why push for gtest vs Boost Test or a different test suite?I normally use Boost, and their test suite, so I'm more familiar with that. So I was wondering is one better then the other, or is it just that someone makes a patch for it?
I looked more into Boost.Test to see what's in it. Boost.Test doesn't seem to be stand-alone -- I don't see a way to use Boost.Test without importing some other chunks of Boost that the testing library depends on. While Boost is a fine set of libraries, I don't think we want to increase the LLVM distribution by sizeof(Boost) just to enable unittesting, nor do we want to spend the time on maintaining a subset of Boost that's "just enough" to build and use the unittest library, along a modified configure/build process that Boost wants to use (Boost.Build? Boost.Jam?).
Boost also seems to want to use exceptions, and LLVM does not want to. I'm not sure if there would be some difficulties in running a build where some libraries are compiled with no exceptions, some with, and the results are linked together. At the best case, it would complicate our build system to be able to support different set of flags for building LLVM libraries vs. Boost.Test (and the rest of Boost that we import).
Sample usage of Boost.Test: http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp
Note the code at the end setting up the test suite -- this is boilerplate code that I think shouldn't be necessary to setup and run tests.
Google Test, on the other hand, has no external dependencies, and is distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; and doesn't use exceptions or RTTI.
Sample usage of GTest: http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc
GTest-specific LOC besides the #include statement: 0.
Note that I'm not counting main() for either Boost or GTest, because both provide a standard main() for use with almost all test files.
Misha
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Keir Mierle
Including it in the tree is the most reasonable thing to do. No point in inconveniencing the user over tiny libraries with liberal licenses.
My question was not to cause a battle, but I wanted to be sure we were using the right test kit, and not just picking one just because. For example gtest is very light weight test kit, that can do the job, but will the tests outgrow what the test kit can do, and cause a conversion to a more advanced one later?
I am not sure what advanced features you are thinking of that gtest doesn't already offer; it's pretty sophisticated despite it's small size. Can you give an example of something gtest doesn't already support? Newer releases added support for data parameterized tests, which is really useful. Type parameterized tests are being discussed on the mailing list. All while maintaining a small codebase!
One other nice feature of using gtest is that it integrates with gmock, one of the only really good C++ mocking libraries available. It was just recently released.
http://code.google.com/p/googlemock
Keir
Regards,Mark Kromis
Misha Brukman
So are you planning on maintaining whatever test system, or just have them as a pre-requisite. For example are you going to have the gtest incorporated, or have them install it separately first? I was under the impression that the user would have to install gtest first.
The current plan is to check in the unittest library into LLVM and build it as part of the test process, so the user doesn't have to do anything separately.
Boost also seems to want to use exceptions, and LLVM does not want to. I'm not sure if there would be some difficulties in running a build where some libraries are compiled with no exceptions, some with, and the results are linked together. At the best case, it would complicate our build system to be able to support different set of flags for building LLVM libraries vs. Boost.Test (and the rest of Boost that we import).#define BOOST_NO_EXCEPTIONS
Thanks for the pointer.
Sample usage of Boost.Test: http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp
Note the code at the end setting up the test suite -- this is boilerplate code that I think shouldn't be necessary to setup and run tests.My test cases are not that in-depth, I'm much closer to sample 1. I haven't found a reason to go that crazy yet.
You would need to use fixtures if you are testing classes and their interactions, otherwise each of your test functions will re-do everything that's in the fixture's SetUp() method -- why would you want to repeat the same setup code for every test case?
Google Test, on the other hand, has no external dependencies, and is distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; and doesn't use exceptions or RTTI.Gtest is much more lightweight, no comparison there. I know that llvm is not very good with exceptions, but should a test case system support that?
GTest allows testing for exceptions, it just doesn't require them to work properly:
http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Exception_Assertions
Sample usage of GTest: http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc
GTest-specific LOC besides the #include statement: 0.I think it links to a library as well.
Yes, that's true -- just about any unittesting library will have that requirement, that wasn't part of the comparison. I was pointing out how much C++ a user has to write in their test files for a reasonable test with fixtures.
Also for a note of reference, your links to the examples are the most advanced samples. So boost can do more, thus has more weight/bloat behind it.
Gordon Henriksen pointed out earlier in the thread that Boost.Test will require pulling in 500 headers from Boost just to work, so that's pretty heavy weight. Also, I was comparing apples-to-apples, not taking the simplest example of GTest and the most complex of Boost -- the two examples I chose were doing the same thing, i.e., tests with fixtures.
What does Boost provide that GTest does not, that you think LLVM needs, such that it's worth importing the entire Boost distribution into the LLVM tree?
Were the other test kits looked at? Is gtest the best solution for the project.
There are dozens of C++ unittesting libraries out there, I admit I haven't looked at all of them. I don't know which ones are still under active development or use, but I (and a few other posters) are familiar with and have used GTest enough to say that it's very well suited for LLVM.
Is this something your planning as putting in the tree, thus require pulling in changes from google (license allowing), or does user need to have the libraries/headers pre-installed?
I already answered this question above (first option: in the tree). GTest is under the BSD license, which is compatible with LLVM (though I am not a copyright lawyer).
For example gtest is very light weight test kit, that can do the job, but will the tests outgrow what the test kit can do, and cause a conversion to a more advanced one later?
As soon as LLVM outgrows what GTest can do, we can easily write wrapper macros to forward from GTest-style macros to the other, more suitable framework; however, I think you're conflating the two concepts of "light-weight" and "trivial" -- GTest is light-weight, but not trivial.
Misha
Mark Kromis
...snip...
Also for a note of reference, your links to the examples are the most advanced samples. So boost can do more, thus has more weight/bloat behind it.Were the other test kits looked at? Is gtest the best solution for the project.Is this something your planning as putting in the tree, thus require pulling in changes from google (license allowing), or does user need to have the libraries/headers pre-installed?
Including it in the tree is the most reasonable thing to do. No point in inconveniencing the user over tiny libraries with liberal licenses.
My question was not to cause a battle, but I wanted to be sure we were using the right test kit, and not just picking one just because. For example gtest is very light weight test kit, that can do the job, but will the tests outgrow what the test kit can do, and cause a conversion to a more advanced one later?
I am not sure what advanced features you are thinking of that gtest doesn't already offer; it's pretty sophisticated despite it's small size. Can you give an example of something gtest doesn't already support? Newer releases added support for data parameterized tests, which is really useful. Type parameterized tests are being discussed on the mailing list. All while maintaining a small codebase!
One other nice feature of using gtest is that it integrates with gmock, one of the only really good C++ mocking libraries available. It was just recently released.
http://code.google.com/p/googlemock
Keir
Mark Kromis
2008/12/27 Mark Kromis <grey...@mac.com>So are you planning on maintaining whatever test system, or just have them as a pre-requisite. For example are you going to have the gtest incorporated, or have them install it separately first? I was under the impression that the user would have to install gtest first.
The current plan is to check in the unittest library into LLVM and build it as part of the test process, so the user doesn't have to do anything separately.
Boost also seems to want to use exceptions, and LLVM does not want to. I'm not sure if there would be some difficulties in running a build where some libraries are compiled with no exceptions, some with, and the results are linked together. At the best case, it would complicate our build system to be able to support different set of flags for building LLVM libraries vs. Boost.Test (and the rest of Boost that we import).#define BOOST_NO_EXCEPTIONS
Thanks for the pointer.Sample usage of Boost.Test: http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp
Note the code at the end setting up the test suite -- this is boilerplate code that I think shouldn't be necessary to setup and run tests.My test cases are not that in-depth, I'm much closer to sample 1. I haven't found a reason to go that crazy yet.
You would need to use fixtures if you are testing classes and their interactions, otherwise each of your test functions will re-do everything that's in the fixture's SetUp() method -- why would you want to repeat the same setup code for every test case?
Google Test, on the other hand, has no external dependencies, and is distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; and doesn't use exceptions or RTTI.Gtest is much more lightweight, no comparison there. I know that llvm is not very good with exceptions, but should a test case system support that?
GTest allows testing for exceptions, it just doesn't require them to work properly:
http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Exception_Assertions
Sample usage of GTest: http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc
GTest-specific LOC besides the #include statement: 0.I think it links to a library as well.
Yes, that's true -- just about any unittesting library will have that requirement, that wasn't part of the comparison. I was pointing out how much C++ a user has to write in their test files for a reasonable test with fixtures.
Also for a note of reference, your links to the examples are the most advanced samples. So boost can do more, thus has more weight/bloat behind it.
Gordon Henriksen pointed out earlier in the thread that Boost.Test will require pulling in 500 headers from Boost just to work, so that's pretty heavy weight. Also, I was comparing apples-to-apples, not taking the simplest example of GTest and the most complex of Boost -- the two examples I chose were doing the same thing, i.e., tests with fixtures.
What does Boost provide that GTest does not, that you think LLVM needs, such that it's worth importing the entire Boost distribution into the LLVM tree?Were the other test kits looked at? Is gtest the best solution for the project.
There are dozens of C++ unittesting libraries out there, I admit I haven't looked at all of them. I don't know which ones are still under active development or use, but I (and a few other posters) are familiar with and have used GTest enough to say that it's very well suited for LLVM.
Is this something your planning as putting in the tree, thus require pulling in changes from google (license allowing), or does user need to have the libraries/headers pre-installed?
I already answered this question above (first option: in the tree). GTest is under the BSD license, which is compatible with LLVM (though I am not a copyright lawyer).
For example gtest is very light weight test kit, that can do the job, but will the tests outgrow what the test kit can do, and cause a conversion to a more advanced one later?
As soon as LLVM outgrows what GTest can do, we can easily write wrapper macros to forward from GTest-style macros to the other, more suitable framework; however, I think you're conflating the two concepts of "light-weight" and "trivial" -- GTest is light-weight, but not trivial.
_______________________________________________
Misha
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
me22
>
> Just briefly looked at the googlemock site and it sys about needed tr1
> library. Third paragraph in System Requirements. It just struck me as
> amusing that visual c++ people need boost then.
>
They recently shipped TR1 for VS2008:
http://www.microsoft.com/downloads/details.aspx?FamilyId=D466226B-8DAB-445F-A7B4-448B326C48E7&displaylang=en
Misha Brukman
Is this something your planning as putting in the tree, thus require pulling in changes from google (license allowing), or does user need to have the libraries/headers pre-installed?
Including it in the tree is the most reasonable thing to do. No point in inconveniencing the user over tiny libraries with liberal licenses.I didn't get to look at the license.
One other nice feature of using gtest is that it integrates with gmock, one of the only really good C++ mocking libraries available. It was just recently released.
Just briefly looked at the googlemock site and it sys about needed tr1 library. Third paragraph in System Requirements. It just struck me as amusing that visual c++ people need boost then.
This is getting slightly off-topic (we're now talking about mocking libraries, rather than unittesting), but Visual Studio 2008 has a free add-on pack to add TR1 support: http://msdn.microsoft.com/en-us/visualc/default.aspx (see the downloads section).
Talin
> On Dec 27, 2008, at 7:41 PM, Misha Brukman wrote:
>> 2008/12/27 Mark Kromis <grey...@mac.com>
>> Just a curiosity question, why push for gtest vs Boost Test or a
>> different test suite?
>> I normally use Boost, and their test suite, so I'm more familiar with
>> that. So I was wondering is one better then the other, or is it just
>> that someone makes a patch for it?
before I started using gtest for my own work. Since then I've become
quite familiar with gtest's features, which made it a natural choice for
me. I presume that other testing frameworks will have their own
defenders on this list, and I am certainly willing to listen to what
they have to say.
The unit test comparison article mentioned earlier
(http://gamesfromwithin.com/?p=29) is a good starting point for
evaluating different unit test frameworks. Although gtest is too new to
have been mentioned in the article directly, if you actually compare
gtest's features against the various criteria used by the author of the
article, it comes off quite well.
Note that the author of the article later went and created his own unit
test framework (UnitTest++), based on his experience with all of the
other frameworks he tested. This was the only framework that I seriously
looked at other than gtest, and I found it to be a bit too minimal for
what I needed.
>> I looked more into Boost.Test to see what's in it. Boost.Test
>> doesn't seem to be stand-alone -- I don't see a way to use Boost.Test
>> without importing some other chunks of Boost that the testing library
>> depends on. While Boost is a fine set of libraries, I don't think we
>> want to increase the LLVM distribution by sizeof(Boost) just to
>> enable unittesting, nor do we want to spend the time on maintaining a
>> subset of Boost that's "just enough" to build and use the unittest
>> library, along a modified configure/build process that Boost wants to
>> use (Boost.Build? Boost.Jam?).
Although I haven't actually tried Boost.Test, I kind of figured that
this would be the case - that you pretty much have to drink the "Boost
Kool-Aid" in order to use it.
> So are you planning on maintaining whatever test system, or just have
> them as a pre-requisite. For example are you going to have the gtest
> incorporated, or have them install it separately first? I was under
> the impression that the user would have to install gtest first.
So the plan is to take a snapshot of gtest and check it in to the LLVM
tree, rather than have it installed separately. I was able to integrate
gtest into LLVM's build system fairly easily, as gtest is designed to be
integrated into a foreign build system - basically I just ignored the
makefile that comes with gtest, and wrote an LLVM-style makefile rule
for it. There's a special source file in gtest which includes all other
sources that is intended for just such a purpose. I did not need to
modify the gtest sources in any way.
This means that keeping the gtest snapshot up to date will be trivial,
since it will only require copying in the latest gtest snapshot and
checking it in to LLVM - presuming that gtest remains backwards
compatible, which I assume it will.
Licensing-wise, both LLVM and gtest are distributed under a fairly
permissive BSD-style license. I don't know who would make the judgement
call as to whether or not the licenses can co-exist. However, since
neither license is "viral" in the sense of wanting to apply any sort of
restriction on derived works or the "work as a whole", I see no barrier
to shipping a combined product with different portions falling under
different licenses. Thus, the unit tests themselves would still fall
under the LLVM license, and linking the unit tests with gtest would not
violate either license. Of course, IANAL.
From a maintenance standpoint, we have already heard from several
enthusiastic volunteers who are involved in both the development of LLVM
and gtest. So I doubt there will be much problems on that score.
My personal goal is that one should be able to check out the head of
LLVM on a generic Linux/OS-X system, with only the standard development
environment (i.e. make/gcc/etc) and type "make unittest" and have the
tests run.
In the longer term, I'd like to see LLVM have an automated build that
runs the unit tests as part of the build. I noticed that gtest has an
option to output test results in XML form, although I have not played
with this personally, it might be useful in this regard.
> ------------------------------------------------------------------------
Keun Soo Yim
Hi,
This is a simple question about building ARM cross-compiler.
What is the building procedure using LLVM 2.4 and GCC front-end 4.2?
I used these commands for LLVM, and it is okay.
$ ../configure --prefix=/usr/local -target=arm
$ make ENABLE_OPTIMIZED=0
$ make ENABLE_OPTIMIZED=0 install
Then these commands were used for GCC front-end but this got couple of
errors.
$ ../configure --prefix=/usr/local --enable-checking
--enable-llvm=../../llvm/objdir/ --disable-bootstrap --disable-multilib
--program-prefix=llvm- --target=arm-linux-gnu --enable-languages=c,c++,objc
$ make; make install
Compiler errors are caused due to absence of definition of macros:
MACHO_DYNAMIC_NO_PIC_P, TARGET_64BIT, and so on.
When these macros are manually added, it caused another error, like below:
../objdir/./gcc/xgcc -B../objdir/./gcc/ -B/usr/local/arm-linux-gnu/bin/
-B/usr/local/arm-linux-gnu/lib/ -isystem /usr/local/arm-linux-gnu/include
-isystem /usr/local/arm-linux-gnu/sys-include -O2 -O2 -g -O2 -DIN_GCC
-DCROSS_DIRECTORY_STRUCTURE -W -Wall -Wwrite-strings -Wstrict-prototypes
-Wmissing-prototypes -Wold-style-definition -isystem ./include -I. -I.
-I../../gcc -I../../gcc/. -I../../gcc/../include
-I../../gcc/../libcpp/include -I../../gcc/../libdecnumber -I../libdecnumber
-I../objdir//include -../../llvm-2.4/include -g0 -finhibit-size-directive
-fno-inline-functions -fno-exceptions -fno-zero-initialized-in-bss
-fno-toplevel-reorder -Dinhibit_libc \
-c ../../gcc/crtstuff.c -DCRT_BEGIN \
-o crtbegin.o
../objdir/./gcc/as: line 2: exec: -o: invalid option
exec: usage: exec [-cl] [-a name] file [redirection ...]
make[2]: *** [crtbegin.o] Error 1
make[2]: Leaving directory `../objdir/gcc'
As a native GCC, 4.3.0, 3.4.2, and 3.4.6 were used but all have same error.
A similar command except for --target=arm option works correctly.
In addition to that, does any developer succeed in compiling a micro-kernel
(e.g., L4 or Xen/ARM) using LLVM ARM port?
Thanks in advance.
Best regards,
Keun Soo
Mark Kromis
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Misha Brukman
Misha Brukman wrote:So as far as putting things in llvm/test, I have a question - the
> I think this is a great idea! As Keir already noted, I would also
> agree with LLVM snapshotting a copy of googletest, but I think it
> should live in llvm/test/googletest (rather than top-level), since
> it's not going to be linked into anything outside of llvm/test.
makefile in that directory contains a whole bunch of rules for
interfacing with DejaGNU. The unit tests don't (and, I think, shouldn't)
require any dependence on DejaGNU -- in fact, I'm hoping it will be
possible to run the unit tests with only the base LLVM package, without
all of the additional package installations required to run the other
LLVM tests. I'm just wondering if it will be a problem organizing the
makefile so that the unit tests don't have any dependence on the other
tests.
I agree, the unittests and DejaGNU shouldn't depend on each other. If they are different make targets, would that solve the issue? For example, the nightly tester would run "make all" which runs "make check-local", while unittests could under the "unittest" target, and they won't interfere, right?
Eventually, the unittests should also be run by default under the "all" target.
> * Instead of this:In order to use EXPECT_EQ, both arguments have to be printable, although
> EXPECT_TRUE(uintMap.find(0u) == uintMap.begin());
> is it possible to use EXPECT_EQ() as well?
you can make anything printable by adding the necessary stream operator
overloads. In this particular case, I decided that the printed
representation of an iterator wouldn't be meaningful, so I didn't bother
defining those overloads.
OK, makes sense, thanks.
Misha
Misha Brukman
On Dec 27, 2008, at 11:01 PM, Misha Brukman wrote:This is getting slightly off-topic (we're now talking about mocking libraries, rather than unittesting), but Visual Studio 2008 has a free add-on pack to add TR1 support: http://msdn.microsoft.com/en-us/visualc/default.aspx (see the downloads section).
Just wanted to mention that the add-on pack is not applicable to the express editions, nor anything before vc9. So if someone is using the express editions they would need to use boost or something similar. But if you use gcc4 then your set.
Have you tried VS2008 Express with SP1?
Herb Sutter notes that they plan to support TR1 in Express "in a future release" (back in April):
http://herbsutter.wordpress.com/2008/04/07/visual-c-2008-feature-pack-now-available/
This page (VS2008 Express SP1 beta readme) talks about debug builds with TR1 in Express:
http://download.microsoft.com/download/8/F/D/8FD2A9F0-C3D4-4B0A-80AF-88D738DCDDF4/ExpressReadme.htm
Chris Lattner
> Although I haven't actually tried Boost.Test, I kind of figured that
> this would be the case - that you pretty much have to drink the "Boost
> Kool-Aid" in order to use it.
I agree, boost.test seems like a non-starter from many reasons.
>> So are you planning on maintaining whatever test system, or just have
>> them as a pre-requisite. For example are you going to have the gtest
>> incorporated, or have them install it separately first? I was under
>> the impression that the user would have to install gtest first.
> So the plan is to take a snapshot of gtest and check it in to the LLVM
> tree, rather than have it installed separately.
Yes, we'd want to either do this, or have the configure machinery
detect an installed copy and only enable unit testing if present. If
the goal is to make everyone run unit tests, embedding a copy would be
the best way to go.
> This means that keeping the gtest snapshot up to date will be trivial,
> since it will only require copying in the latest gtest snapshot and
> checking it in to LLVM - presuming that gtest remains backwards
> compatible, which I assume it will.
This is only an issue if the latest and greatest versions of gtest add
something we need. My understanding of unit testing is fairly
limited, but I don't forsee major new innovations regularly happening
that we'll need to bring in on a frequent basis. Doing a periodic
synch (once every year or two) will probably be fine.
>
> Licensing-wise, both LLVM and gtest are distributed under a fairly
> permissive BSD-style license. I don't know who would make the
> judgement
> call as to whether or not the licenses can co-exist. However, since
> neither license is "viral" in the sense of wanting to apply any sort
> of
> restriction on derived works or the "work as a whole", I see no
> barrier
> to shipping a combined product with different portions falling under
> different licenses. Thus, the unit tests themselves would still fall
> under the LLVM license, and linking the unit tests with gtest would
> not
> violate either license. Of course, IANAL.
The licenses coexist. Please just add an entry to the llvm/
LICENSES.txt file.
> My personal goal is that one should be able to check out the head of
> LLVM on a generic Linux/OS-X system, with only the standard
> development
> environment (i.e. make/gcc/etc) and type "make unittest" and have the
> tests run.
Yep. BTW, do these tests run in parallel with make -jN? If not,
please make them :)
> In the longer term, I'd like to see LLVM have an automated build that
> runs the unit tests as part of the build. I noticed that gtest has an
> option to output test results in XML form, although I have not played
> with this personally, it might be useful in this regard.
The only thing I'm concerned about here is extending build time. For
example, if we have a lot of unit tests for libsupport (e.g.
densemap), I wouldn't want those tests to be run unless libsupport
changes. It should just be a "small matter of makefile hackery" to
get this going I suppose.
Is there a final proposed version of the patch with the changes Misha
asked for? If so, I'll review and commit it when I have time.
-Chris
Talin
-- Talin
Misha Brukman
I'm working on an update to the patch. The only thing holding me up is trying to come to a final decision as to where all the various pieces should live. Specifically, the Google Test library, and the actual unit tests themselves.
I would recommend putting Google Test in llvm/test/googletest . The unittests should probably go in llvm/test with other tests, in the same directories, e.g., tests of the various analysis algorithms would live in llvm/test/Analysis, 1 file per analysis, unless folks think they should go further into the tree: llvm/test/Analysis/Andersens/, etc., but I think it would be cleaner and easier if the files for a single library were in the same directory, as we would be able to create a single unittest from all of them, to speed up the build/link/test cycle.
Thoughts?
Chris Lattner
-Chris
Talin
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
--
-- Talin
Misha Brukman
Do the existing tests use a wildcard rule to gather all test sources? We would need to insure that the wildcard rule for the unit tests and the large tests are mutually exclusive.
By "large tests", I assume you mean the .ll tests? Those are one-file-per-test. Also, if you use Chris' suggestion on directories, it will separate the unit tests from the large tests.
Also, will the unit tests be one executable or many? They will probably compile/run faster if there is a smaller number of executables.
Yes, I agree, they'll link faster if we have fewer tests. I would not want a single massive test for all of LLVM -- how about 1 test binary per major directory in llvm/lib, unless it's really necessary to split them up, e.g. target-dependent unittests, but each .cpp file would ideally have its own _test.cpp file, so it's easier to keep the tests for different modules separate.
Talin
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
--
-- Talin
Keir Mierle
One nice feature of gtest is automatic test collection. If the llvm tests all use the same main(), and the main() is not repeated in the individual unittest files, then it is trivial to have a) one massive binary that contains all the tests (fast to build & run all unittests pre-checkin) and separate binaries per .cpp (i.e. foo.cc, foo_test.cc, and foo_test binary) for fast iterating which would only be built and run when explicitly called from the makefile (i.e. they would not be run by the build-all-unittest make target).
On the other hand, if separate binaries are built, then a make unittest target could parallelize trivially by adding -jX. I belive fork-level parallelism within a single test binary is in coming to gtest, but is not available yet.
Keir
.
Talin
-- Talin
Misha Brukman
This is a simple question about building ARM cross-compiler.
What is the building procedure using LLVM 2.4 and GCC front-end 4.2?
I used these commands for LLVM, and it is okay.
$ ../configure --prefix=/usr/local -target=arm
$ make ENABLE_OPTIMIZED=0
$ make ENABLE_OPTIMIZED=0 install
Then these commands were used for GCC front-end but this got couple of
errors.
$ ../configure --prefix=/usr/local --enable-checking
--enable-llvm=../../llvm/objdir/ --disable-bootstrap --disable-multilib
--program-prefix=llvm- --target=arm-linux-gnu --enable-languages=c,c++,objc
$ make; make install
Compiler errors are caused due to absence of definition of macros:
MACHO_DYNAMIC_NO_PIC_P, TARGET_64BIT, and so on.
I've reproduced the same error on my x86/Linux system (the first macro is undefined). I suspect if I go ahead and also add it, I'll continue to the next error you found. One might guess that since this cross-tool build is so broken, that it's likely not used by the folks targeting the ARM backend.
The folks developing on ARM instead might be using QEMU to emulate ARM on x86, and run it as a regular (not cross) compiler in that environment. In fact, looking at the nightly tester runs, I've noticed one of the ARM nightly testers seems to run in QEMU:
Nickname:shark-qemu-armv5tel
uname:Linux shark-armel 2.6.24-rc7 #1 Wed Jan 16 02:28:08 CET 2008 armv5tel GNU/Linux
http://llvm.org/nightlytest/machine.php?machine=335
However, some other nightly testers may be running cross-compilers (but are probably still running the tests via QEMU or similar emulator):
Nickname:lauro-armv6-N800
uname:Linux laurov-laptop 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux
http://llvm.org/nightlytest/machine.php?machine=172
Nickname:lauro-armv5te-softfloat
uname:Linux laurov-desktop 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686 GNU/Linux
http://llvm.org/nightlytest/machine.php?machine=121
Perhaps the owner(s) of these nightly testers would know how to do what you're trying to accomplish.
Misha
Keun Soo Yim
I've reproduced the same error on my x86/Linux system (the first macro is undefined). I suspect if I go ahead and also add it, I'll continue to the next error you found. One might guess that since this cross-tool build is so broken, that it's likely not used by the folks targeting the ARM backend.
The folks developing on ARM instead might be using QEMU to emulate ARM on x86, and run it as a regular (not cross) compiler in that environment. In fact, looking at the nightly tester runs, I've noticed one of the ARM nightly testers seems to run in QEMU:
Nickname:shark-qemu-armv5tel
uname:Linux shark-armel 2.6.24-rc7 #1 Wed Jan 16 02:28:08 CET 2008 armv5tel GNU/Linux
http://llvm.org/nightlytest/machine.php?machine=335
However, some other nightly testers may be running cross-compilers (but are probably still running the tests via QEMU or similar emulator):
Nickname:lauro-armv6-N800
uname:Linux laurov-laptop 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux
http://llvm.org/nightlytest/machine.php?machine=172
Nickname:lauro-armv5te-softfloat
uname:Linux laurov-desktop 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686 GNU/Linux
http://llvm.org/nightlytest/machine.php?machine=121
Perhaps the owner(s) of these nightly testers would know how to do what you're trying to accomplish.
Misha