test setup
Joep Suijs
I've finished a test-setup to demonstrate how we could more efficient
handle testboards and testcases. What would be a good place to post
this (in the SVN tree) so you can take a look?
Joep
Sebastien LELONG
better create a /doc directory. Or first post here if you prefer.
Seb
Sébastien LELONG
http://www.sirloon.net
http://sirbot.org
Joep Suijs
Rob Hamerling
Joep Suijs wrote:
I'm curious! What about the tools directory?
Maybe we should also make a list of who is or will be testing what
(device, libraries, etc), such that we have not too much redundant
tests. I have a pretty large collection of PICs (most with DIP-package)
which I usually test on a breadboard.
Regards, Rob.
--
Rob Hamerling, Vianen, NL (http://www.robh.nl/)
Joep Suijs
This setup has two testsets (test_*.jal) and two board-configurations
(board_*.jal).
The board configuration describes a specific hardware setup. It names
the chip, osc frequency, fuse-setup etc. It also defines the
connection to specific IO (connection to led, lcd, etc). The example
are one for a home-made 18-pin board (with 16f648a, but can also be
used with 16f88) and one for a dwarf board.
At the end of this file, you can include any relevant testset
(currently one at the time, but if this is the way we go, we could
concider them to adapt for multiple tests in one hex-file).
The test files are stripped samples (for now).
The big advantage of this setup is that you only need to define the
board-setup once and then you can run all the relevant testsets. And
if there is a new version of a testset, you can run it without
modification (like adapting the io-config).
If a new testset is added, it might be required to add a few constants
to each board file. But with 10 boardfiles and 10 testsets, you only
have to maintain 20 files and get 100 different testcases.
Joep
Sebastien LELONG
This is nice ! There are pros and cons, and I can't make my decision...
What I like is there's one place where fuse, osc, ... are defined. That's a
big advantage, as there are lots duplicated code in samples, and here we have
a kind of standard way to specify these. Each can put its board
definition/config here, it can be documented with links to schematics, etc...
In the test results, we can specify which board setup has been used.
What I'm worry about is I'm not sure you can put *every* possible setup within
one file... without defining procedures (eg. b0 pin is used in LCD test, but
in pwm tests too). Those are only defined here in board setup files, not in
libraries. These procedures are called within tests. Example: set_ldc_io() is
defined in board files, called within the test. Now someone is opening your
lcd test, he can't figure out where this "set_ldc_io()" call comes from
(maybe from delay_any_mc ?). And it does not come from a library. IMHO we
should try very hard to keep samples and test as most explicit as possible,
no magical calls.
I'm not sure putting all the tests as includes at the end of the file is a
good idea too. At least not in board files, which should only set what must
be set to use a given board, without considering the usage. These board files
could be used to run samples, other tests, etc... With such include
statements, you're stuck, you'll need to edit/remove them. I think it's
better put them on a "higher" level file.
Ex:
=> 16f648_testset.jal file
{{{
include board_16f648a_js
-- now define the test suite
include test_blink
-- maybe do something between test, like waiting for user input to continue
include lcd_hello_world
include echo
}}}
That's one more file, but allows more flexibility. Putting all the tests into
a single file is a nice idea, because we can refer to this file in our test
results wiki pages. That's a "living" documentation, which actually is
compiled, run, ... This is closed to what Bert defines with his Mastertest
program (see jallist:
http://tech.groups.yahoo.com/group/jallist/message/25657)
(Maybe we could add a "/test" directory to put all these test setup files ? Or
better, put them in /samples/test_setup/ ?)
jalanice WE (I try again this one, it appears it did not work last WE :))
Seb
> The board configuration describes a specific hardware setup. It names
> the chip, osc frequency, fuse-setup etc. It also defines the
> connection to specific IO (connection to led, lcd, etc). The example
> are one for a home-made 18-pin board (with 16f648a, but can also be
> used with 16f88) and one for a dwarf board.
>
> At the end of this file, you can include any relevant testset
> (currently one at the time, but if this is the way we go, we could
> concider them to adapt for multiple tests in one hex-file).
>
> The test files are stripped samples (for now).
>
> The big advantage of this setup is that you only need to define the
> board-setup once and then you can run all the relevant testsets. And
> if there is a new version of a testset, you can run it without
> modification (like adapting the io-config).
> If a new testset is added, it might be required to add a few constants
> to each board file. But with 10 boardfiles and 10 testsets, you only
> have to maintain 20 files and get 100 different testcases.
>
> Joep
>
>
Joep Suijs
>
>
> What I'm worry about is I'm not sure you can put *every* possible setup within
> one file... without defining procedures (eg. b0 pin is used in LCD test, but
> in pwm tests too).
(yet) that needs a function call to setup.
There are limitations of the test you can run simultanious. These
limits come from the chip and a specific configuration (where you
choose to use specific pins for specific devices) give even more
limits. Try to create a configuration with 16f88 with an lcd (requires
a full nibble IO), analog, pwm and serial config while running at 20
MHz...
> Example: set_ldc_io() is defined in board files, called within the test.
> Now someone is opening your lcd test, he can't figure out where
> this "set_ldc_io()" call comes from (maybe from delay_any_mc ?).
> And it does not come from a library. IMHO we should try very hard to
> keep samples and test as most explicit as possible, no magical calls.
(i'm going to repeat myself ;-)
I agree, but also see that the purpose of samples and tests are different.
Samples are to show anybody how to use a library. They should be as
simple as possible and explicit. 'Magic calls' are not acceptable.
Test, however have an other purpose: verifying if the library does
(only) what it is supposed to do. We still need to keep these as
accessible as possible, but test-efficiency is also an issue.
The LCD lib is special in the way it uses a lot of IO and (this
version) does not handle io-setup by itself. This also has as a
consequence that it's not so easy to re-use these io-lines, like for
an keyboard.
I made the function as a hack to set the proper io. I'll adapt it to
get rid of this function.
> I'm not sure putting all the tests as includes at the end of the file is a
> good idea too.
>...
> I think it's better put them on a "higher" level file.
I agree.
> Putting all the tests into a single file is a nice idea, ..
It is indeed and I've been thinking about this. We need to be aware of:
- specific board configurations may not support all possible tests for a chip.
- not all possible test can be configured at the same time (they could
use the same resources).
- Some libs require the IO to be available (and setup) when it is
included. e.g. lcd.jal initialises the display at include.
> (Maybe we could add a "/test" directory to put all these test setup files ? Or
> better, put them in /samples/test_setup/ ?)
Since tests and samles are not the same, I'd vote for /test
Joep
Sebastien LELONG
> I agree, but also see that the purpose of samples and tests are different.
> Samples are to show anybody how to use a library. They should be as
> simple as possible and explicit. 'Magic calls' are not acceptable.
Yeah I did not forget your point. I'll try to rephrase what's been said.
I understand that tests and samples serve different goals. Let's imagine tests
and samples aren't the same (...). Consider your lcd_hello_world test. If you
put "magic calls" in it, it's ok, since it's a test. Now, where is the lcd
sample ? Is it another file ? Will there be duplication between lcd test and
sample ? Ex: I've written 3 samples for pwm usage, I consider these must be
part as the test suite because they test different pwm usages. Will I have to
write other samples for these ? This is an important point IMHO.
Having such magic calls often shows weird designs. Don't get me wrong, I
understand we have sometime no choice, because we lack tools, features,
etc... Let's just be aware this can lead to confusion. If tests (or samples,
dunno) are to dependent from board_ files, this could also send the
message: "if you want to use this lib, you'll need this board_ file". And
we'd have something like _bert include files, with a lot a weird default
values. Which we don't want (I think).
> Test, however have an other purpose: verifying if the library does
> (only) what it is supposed to do.
This is supposed to be the same for samples: if people write and commit
samples, they should work, right ? And samples are designed to show how to
use a specific lib.
> We still need to keep these as
> accessible as possible, but test-efficiency is also an issue.
I agree. "import this" in python says: "Special cases aren't special enough to
break the rules. Although practicality beats purity." :)
(it also says lots of interesting other things)
> > (Maybe we could add a "/test" directory to put all these test setup files
> > ? Or better, put them in /samples/test_setup/ ?)
> Since tests and samles are not the same, I'd vote for /test
I'm talking about test setup files: board_ files, etc... not samples/tests
file (eg. pwm_led.jal). Do we talk about the same thing ?
Seb
Joep Suijs
>
> > (i'm going to repeat myself ;-)
> > I agree, but also see that the purpose of samples and tests are different.
> > Samples are to show anybody how to use a library. They should be as
> > simple as possible and explicit. 'Magic calls' are not acceptable.
>
> Yeah I did not forget your point. I'll try to rephrase what's been said.
> I understand that tests and samples serve different goals. Let's imagine tests
> and samples aren't the same (...). Consider your lcd_hello_world test. If you
> put "magic calls" in it, it's ok, since it's a test. Now, where is the lcd
> sample ? Is it another file ? Will there be duplication between lcd test and
> sample ? Ex: I've written 3 samples for pwm usage, I consider these must be
> part as the test suite because they test different pwm usages. Will I have to
> write other samples for these ? This is an important point IMHO.
But to create them is less effort then it seems; see how I created the
tests from the existing samples. We do have to maintain both,
however...
>
> Having such magic calls often shows weird designs.
>
I substituted those with logica definitions of the io lines for each lib.
From an example-point-of-view i'd say defining something which is only
used once in the user code (and even at init-time) is not a good idea,
since it adds too much complexity.
We could however move the init-code to the library. This means that in
case of pwm, the user must define the io-pin of the output and
pwmx_on() and pwmx_off() set the io-port direction. This is the way
Stef setup his libraries and I must say it makes sense: why does the
lib take care of all initialisation except for the pin-direction?
> > Test, however have an other purpose: verifying if the library does
> > (only) what it is supposed to do.
>
> This is supposed to be the same for samples: if people write and commit
> samples, they should work, right ? And samples are designed to show how to
> use a specific lib.
Samples demonstrate how to use the library, by using each call once or
twice and yes, they should work.
Ideal tests are much more extensive - they check more combinations of
calls and use different value's (especially near bounderies) for each
call.
We're not there yet, but creating large testprograms with the level of
documentation required by a sample can be quite a lot of work...
> I'm talking about test setup files: board_ files, etc... not samples/tests
> file (eg. pwm_led.jal). Do we talk about the same thing ?
>
You're right, we have to resolve this issue before we can choose the
right place for the files.
Sebastien LELONG
> But to create them is less effort then it seems; see how I created the
> tests from the existing samples. We do have to maintain both,
> however...
Yes we have to maintain both. But, ok, if it's worth, we can do it !
> We could however move the init-code to the library. This means that in
> case of pwm, the user must define the io-pin of the output and
> pwmx_on() and pwmx_off() set the io-port direction. This is the way
> Stef setup his libraries and I must say it makes sense: why does the
> lib take care of all initialisation except for the pin-direction?
I'm not sure it's possible. How would you do that ?? To set the direction, you
must have the pin name, as defined in the device file (eg. pin_b0_direction).
About pwm_hardware lib, Stef has hard-coded the pins (name + direction)
because the lib was supposed to work only with 16f877 and other
pin-compatible chip.
[later]
OK I have a look at serial_software
(http://code.google.com/p/jallib/issues/detail?id=8). Stef defines the
direction within the lib. You say:
{{{
var volatile bit Serial_SW_out_pin is pin_b7
var volatile bit Serial_SW_out_direction is pin_b7_direction
}}}
and in the code serial_sw_init:
{{{
Serial_SW_out_direction = output
}}}
We could do the same (ref for pin *and* pin direction), though I'm not sure
it's worth it. This can be useful when a lib has to switch direction
input<=>output during execution, but here, I'm not sure.
> Samples demonstrate how to use the library, by using each call once or
> twice and yes, they should work.
> Ideal tests are much more extensive - they check more combinations of
> calls and use different value's (especially near bounderies) for each
> call.
> We're not there yet, but creating large testprograms with the level of
> documentation required by a sample can be quite a lot of work...
OK, I understand this. Let's consider samples and tests as different :)
We're not there yet, but we can also add test unit code within tests. This can
be done using picshell. I'm don't know how efficient and helpful it can be,
but this can run tests in-silico (I mean in-PC-silico...) for almost no
costs.
> > I'm talking about test setup files: board_ files, etc... not
> > samples/tests file (eg. pwm_led.jal). Do we talk about the same thing ?
>
> You're right, we have to resolve this issue before we can choose the
> right place for the files.
Yes, that's the problem. I understand now what you plan to do with *tests*
(extensive, combination, etc...). Things we won't do with samples, indeed.
Few questions I have now:
* if we create a /test directory, will there be again the same hierarchy we
have in /include and /sample ? I'm worry about this, having this again and
again is weird.
* let's talk about *samples*. I've created a pwm_led.jal sample for 16f88.
Now someone wants it to run on its 16f877a. Do we create two samples, as for
now (pwm_led_16f88.jal and pwm_led_16f877.jal) ? Or only one is enough ? I
prefer having both, the more we have ready-to-compile code, the better it
is...
* *tests* code is target independent (no device include), but *samples* are
target dependent. Should we organize tests as samples are organized now
(/external, /peripheral, /jal, ...) ? And samples by device ? We could have
something like:
..../sample/
by-device/
16f88/
b16f88.jal
pwm_led.jal
lcd_hello_world.jal
...
16f877a/
b16f877a.jal
pwm_led.jal
pwm_two_channel.jal
lcd_hello_world.jal -- with device include
...
test/
external/
/lcd
lcd_hello_world.jal -- no device include
peripheral/
/pwm
pwm_heavy_test.jal
...
/adc
...
board/
16f877_dwarf.jal
16f648a_js.jal
16f88_js.jal -- tow board available for 16f88
16f88_sb.jal
testsuite/
16877a_dwarf_testsuite.jal
16f88_js_testsuite1.jal
16f88_js_testsuite2.jal
16f88_sb_testsuite1.jal
16f88_sb_testsuite2.jal
/by-device stores all our current samples. This can give a nice overview of
available samples for a given chip. These samples here are
ready-to-be-compiled, with a device include. We can have the same filename in
different directories (eg. pwm_led.jal). This means only /include must have
unique filenames, because it'll be flatten. (Note: while I'm creating my doc
extract script (jalapi,
seehttp://code.google.com/p/jallib/issues/detail?id=15 ), I search for
samples using given libraries. By looking at the jalapi doc page, user will
see the list of sample available for this lib. This will link to here).
/test contains the hierarchy we have now in /sample. If someone wants to have
a look at some code, he can search by fearture, even if the code is a test
code, not a sample code, that is it can be quite complicated. We also
have /board to store boards files. There can be several board file for a
given chip (_sb is for SirBot, that's my boards, as an example).
/testsuite contains the testsuite definition, that is, include a board file
and several test files. Here, we have a problem: if we use the unix command
to build the -s compiler option, lcd_hello_world.jal in by-device (the
sample) could be consider *before* lcd_hello_world.jal in test (the test...),
because .../by-device could appear before .../test in the -s line. The
contrary (compiling a sample would take things from /test) should not be a
problem because test are only included in a testsuite.
Enough talk... :) What do you think about this ? You, all ?
P.S: just a few more words... test, board, testsuite, etc... all of this is
for advanced users. But it must heavily be documented, so we'll be able to
have more and more advanced users !
Cheers,
Eur van Andel
Maybe we should also make a list of who is or will be testing what
(device, libraries, etc), such that we have not too much redundant
tests. I have a pretty large collection of PICs (most with DIP-package)
which I usually test on a breadboard.
---
ir EE van Andel e...@fiwihex.nl http://www.fiwihex.nl
Fiwihex B.V. Wierdensestraat 74, NL7604BK Almelo, Netherlands
tel+31-546-491106 fax+31-546-491107
Sebastien Lelong
So... I had this proposal about modifying the SVN structure in /sample so it includes samples, tests, boards, test suites,... See this post (sorry as usual, I'm quite verbose :)):
http://groups.google.com/group/jallib/msg/440edaacd0fefeac (+ whole topic http://groups.google.com/group/jallib/browse_thread/thread/052c386da4b43c85/440edaacd0fefeac)
The validation process/criteria is discussed here and there, in multiple topics. I'll try to create a wiki page to summarize, but we first need to make a decision...
Thanks for your feedback !
Seb
Rob Hamerling
Sebastien Lelong wrote:
> I'd like to bother you all again with this issue... I think the validation
> process and criteria are coming on the surface again and again, we need to
> make a decision before it starts to be a blocking point. I also think
> without having a decision now, this can waste a lot of time when we'll start
> to test the whole repos (it's already started...).
You are right, but I have had no time to do anything else than testing
my own stuff (and Eur's new requirements, compiler issues, organizing
myself, keeping notes about 'exceptional' PICs, reading datasheets, etc).
> So... I had this proposal about modifying the SVN structure in /sample so it
> includes samples, tests, boards, test suites,...
It looks good, although maybe a little too ambitious. As I said before
we should not focus on JAL-newbies alone, but also make Jallib contents
attractive to more experienced JAL programmers. But I don't think real
experiended Jal programmer are waiting for us.
About testing: we should not put too much (how much???) time _now_ in
intensive testing. We must take some time to think of the near future
but concentrate on workable set of files for the most popular PICs and
most popular applications. When the basic stuff is working we'll get
more time to gradually expand and (hopefully) get help from users we
made enthousiastic for our work.
May I remind you that this is hobby for me and probably for the majority
of us and potential users?
> The validation process/criteria is discussed here and there, in multiple
> topics. I'll try to create a wiki page to summarize, but we first need to
> make a decision...
The proposed directory structure looks OK to me, but to be honest, I
have not thought about the whole validation process very thoroughly.
I'm somewhat tired and spent already too many hours on this project in
the last couple of weeks (according to my wife), I need a break.
Regards, Rob
Sebastien LELONG
> intensive testing. We must take some time to think of the near future
> but concentrate on workable set of files for the most popular PICs and
> most popular applications. When the basic stuff is working we'll get
> more time to gradually expand and (hopefully) get help from users we
> made enthousiastic for our work.
Yes this is what I think too. We'll gradually add more and more tests, but we
need to define the base now. That's why IMHO having constraints such as "must
be tested on at least x different targets, etc..." to be validated may be too
strong right now. Time will do its job, things will get more and more tested
(hopefully)
> May I remind you that this is hobby for me and probably for the majority
> of us and potential users?
Yes, this is a hobby too :) We've spent a lot of hours/work on this, I hope
we'll meet success !
> The proposed directory structure looks OK to me, but to be honest, I
> have not thought about the whole validation process very thoroughly.
> I'm somewhat tired and spent already too many hours on this project in
> the last couple of weeks (according to my wife), I need a break.
Hey, I have same feedback from my wife too :))) I need a break too, I'll be
soon on holidays :)
Thanks for your reply !