Test Framework and cases
David LaPalomento
unit-test style test cases that verify certain aspects of the code
generation but require manual testing of semantics. This is a good
starting point but far from where I'd like to be. The next testing
priority I had identified was to start compiling a suite of bitcode
test cases which interact directly with javascript objects like the
document and window to actually produce some output. Ideally, this
suite would be semantically verified using techniques similar to
emscripten's test runner with the addition of something along the
lines of env.js. If we can agree on a common interface to the native
js objects from the IR, I imagine this test suite can be completely
shared between the two projects. This would alleviate the semantic
testing gap ljb has today while taking a step towards where I'd like
to go: using high-level languages hosted on LLVM to build web UIs.
Alon, have you started exposing javascript objects up into the IR yet?
Does this sort of testing sound interesting to you?
David
On Thu, Sep 23, 2010 at 9:15 PM, Alon Zakai <aza...@mozilla.com> wrote:
> So, it looks like there are two open source projects that have separately been working on translating LLVM bitcode to JavaScript, (in alphabetical order) emscripten and llvm-js-backend. We got to talking on email, and then thought to move the discussion to the mailing lists of both projects, since other people might find it interesting.
>
> Our approaches are somewhat different, llvm-js-backend is, as the name implies, a JavaScript backend for LLVM. It's written in C++ and integrated in the LLVM codebase. Emscripten on the other hand parses human-readable LLVM bitcode, and is written in JavaScript. So sharing the core code isn't directly possible. However, we would really like to collaborate as much as possible otherwise, and it seems there is a lot of room for that:
>
> 1. Test framework and cases. While the approaches and architectures are different, we can use the same tests for both, since they are trying to do the same thing. One project (emscripten) has automatic tests that compile C/C++ code into LLVM bitcode that is then tested (through a Python test runner), while the other (llvm-js-backend) has test cases written in LLVM bitcode directly (and run using llvm-lit). I think there is a need for both kinds of tests. We could collaborate on the scripts that run the tests, and on test case code (of which we will need a lot!).
>
> 2. Standard library. Both projects need stdlib, math, stdio, etc. libraries. One approach is to write those in JavaScript (Emscripten is trying that, for now), another would be to compile a stdlib written in C/C++, after some porting. We can debate which approach is better, and work together on a single standard library that both projects can use.
>
> 3. Discuss the various issues we need to tackle in order to run LLVM bitcode in JavaScript, and perhaps implement shared solutions. For example, how to implement pointers, loops, etc. etc. It might turn out that we can agree on a certain implementation of pointers, and share the code for it, etc.
>
> We should probably fork into separate threads for each of these topics.
>
> - azakai
>
David LaPalomento
azakai
++ source code files are compiled into LLVM bitcode using Clang and
llvm-gcc, then run through Emscripten, then run in a JavaScript
engine, and the output compared to what the C/C++ code gives when
compiled and run using gcc. This is done separately for both normal
and optimizing mode, and running is checked in both V8 and
SpiderMonkey. So, this is high level (functional) testing,
essentially.
The test cases go from very high level stuff like a complete malloc
implementation, a script engine for another language, and a raytracer,
down to benchmarks like fasta and fannkuch, and finally down into
small 'unit tests' that target specific issues (polymorphism, taking
the address of a value on the stack, sizeof, etc., and various checks
to prevent regressions).
Regarding interacting with the document or window, I tend to think
it's better to focus first on getting self-contained code to work - on
building C/C++ code and seeing that it outputs the right stuff. Since
the Emscripten test setup already exists and does exactly that, we
could integrate it with llvm-js-backend and it would get a lot of test
coverage very quickly. Also I am very curious to compare the generated
code llvm-js-backend produces on those tests with what Emscripten
generates.
I do agree that tests for interacting with the document/window/etc.
are important. I think it's logical though to start with self-
contained code, then once that is solid to move into interaction with
external stuff. I was planning to add such tests to the Emscripten
test runner later on, but we can do that sooner of course.
- azakai