few questions

[Glen Ogilvie]

Hi,

I've got a few questions.

Are the points defined in:
set_hash_rects

From top left to bottom right, by pixel?, based on the whole screenshot?


I am having trouble getting the check to not return a overall: fail..

QEMU finished, running final checks
010_consoletest_setup OK
Building PPM C-libraries...
swig -perl5 -c++ tinycv.i
c++ -O3 -c -fPIC tinycv_wrap.cxx `perl -MExtUtils::Embed -e ccopts`
c++ -O3 -fPIC `pkg-config --cflags opencv` -o tinycv.o -c tinycv.cc
rm tinycv_wrap.cxx tinycv_wrap.o

====
consoletest_setup: na
shutdown: na
isosize: ok
overall: fail


Any suggestions?
You can see my code, in:
https://github.com/nelg/os-autoinst

I am working on the mageia distrib.

[Bernhard M. Wiedemann]

Am 06.04.2013 13:07, schrieb Glen Ogilvie:
> Hi,
>
> I've got a few questions.
>
> Are the points defined in:
> set_hash_rects
>
> From top left to bottom right, by pixel?, based on the whole screenshot?

There is currently work happening in
https://github.com/openSUSE-Team/os-autoinst
to drop hash usage altogether in favour of opencv image matching.

for now, set_hash_rects takes an arrayref with [x,y,width,height]
where x,y is the pixel-position of the top-left corner of the rectangle
that shall be hashed.
0,0 would be the very top-left of the screen.

> I am having trouble getting the check to not return a overall: fail..
>
> QEMU finished, running final checks
> 010_consoletest_setup OK
> Building PPM C-libraries...
> swig -perl5 -c++ tinycv.i
> c++ -O3 -c -fPIC tinycv_wrap.cxx `perl -MExtUtils::Embed -e ccopts`
> c++ -O3 -fPIC `pkg-config --cflags opencv` -o tinycv.o -c tinycv.cc
> rm tinycv_wrap.cxx tinycv_wrap.o
>
> ====
> consoletest_setup: na
> shutdown: na
> isosize: ok
> overall: fail

overall is computed from other results in
https://github.com/nelg/os-autoinst/blob/master/distri/mageia/check.pm
so it will be OK, when you make it depend on tests that succeed.

results for single tests are determined in the "check" method in basetest.pm
using either hashes, reference images, OCR or (in case of audio tests)
multimon

Ciao
Bernhard M.
--
(o_
//\
V_/_ http://www.suse.de/
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix
Imend�rffer, HRB 16746 (AG N�rnberg)

[Glen Ogilvie]

On Sunday, April 7, 2013 6:21:32 AM UTC+12, Bernhard M. Wiedemann wrote:

Am 06.04.2013 13:07, schrieb Glen Ogilvie:
> Hi,
>
> I've got a few questions.
>
> Are the points defined in:
> set_hash_rects
>
>  From top left to bottom right, by pixel?, based on the whole screenshot?

There is currently work happening in
https://github.com/openSUSE-Team/os-autoinst
to drop hash usage altogether in favour of opencv image matching.

Hash matching is good for my use case, as right now, openvc does not work on my system.
Will you be merging the openSUSE-Team work into your git tree soon?
 

for now, set_hash_rects takes an arrayref with [x,y,width,height]
where x,y is the pixel-position of the top-left corner of the rectangle
that shall be hashed.
0,0 would be the very top-left of the screen.


> I am having trouble getting the check to not return a overall: fail..
>
> QEMU finished, running final checks
> 010_consoletest_setup OK
> Building PPM C-libraries...
> swig -perl5 -c++ tinycv.i
> c++ -O3 -c -fPIC tinycv_wrap.cxx `perl -MExtUtils::Embed -e ccopts`
> c++ -O3 -fPIC `pkg-config --cflags opencv` -o tinycv.o -c tinycv.cc
> rm tinycv_wrap.cxx tinycv_wrap.o
>
> ====
> consoletest_setup: na
> shutdown: na
> isosize: ok
> overall: fail

overall is computed from other results in
https://github.com/nelg/os-autoinst/blob/master/distri/mageia/check.pm
so it will be OK, when you make it depend on tests that succeed.

results for single tests are determined in the "check" method in basetest.pm
using either hashes, reference images, OCR or (in case of audio tests)
multimon

OK.
Are the hashes that are checked, the ones defined in
sub checklist()

for each test?  Are these hashes of the various stage areas, or of the entire screenshot?
Also, should I only include hashes for the steps I am going to add to the check?

I was wanting to confirm the test result by the outcome of waitserial().  Is this possible?


The other thing I wonder, is when I am finished, should I issue a shutdown to the guest OS, as the last test or is the a
better way of doing things?

Only other question, is have you devised a way to do skip the install, and just boot the last installed image?  This would be
helpful when I am doing tests of the installed image, and want to make sure my tests work, without having to wait for a full
OS install every time I get something wrong in one of my tests which occur after installation?

Right now, the output I get is:

====
||| checking check /home/test/tmp/../os-autoinst/distri/mageia/check.pm
||| checking consoletest_setup /home/test/tmp/../os-autoinst/distri/mageia/consoletest.d/010_consoletest_setup.pm
--- {"audiodumps":[],"md5_result":"unk","name":"consoletest_setup","screenshots":[{"ocr_result":"na","refimg_result":"na"},{"ocr_result":"na","refimg_result":"na"}],"result":"unk"}

consoletest_setup: unk
||| checking shutdown /home/test/tmp/../os-autoinst/distri/mageia/consoletest.d/900_shutdown.pm
--- {"audiodumps":[],"md5_result":"na","name":"shutdown","screenshots":[{"ocr_result":"na","refimg_result":"na"}],"result":"na"}

shutdown: na
isosize: ok
overall: fail

.. I wonder if shutdown should show something other than na?

Thank you for your help so far.    Once I've got this passing, I would like to submit a pull request, if your happy to merge the Mageia stuff into your git tree.

Regards
Glen Ogilvie

[Glen Ogilvie]

On Sunday, April 7, 2013 6:21:32 AM UTC+12, Bernhard M. Wiedemann wrote:

Am 06.04.2013 13:07, schrieb Glen Ogilvie:

results for single tests are determined in the "check" method in basetest.pm
using either hashes, reference images, OCR or (in case of audio tests)
multimon

One more question about this..  With the hashes defined in checklist(), are they based on hashes of the screenshot / stage from anytime during the
running of the test, ie, anytime during 010_consoletest_setup.pm, or are they based on hashes of what is on the screen at the
end of the 010_consoletest_setup.pm run?

Regards
Glen Ogilvie

[Bernhard M. Wiedemann]

Am 07.04.2013 02:16, schrieb Glen Ogilvie:
>
>
> On Sunday, April 7, 2013 6:21:32 AM UTC+12, Bernhard M. Wiedemann wrote:
>
> Am 06.04.2013 13:07, schrieb Glen Ogilvie:
> > Hi,
> >
> > I've got a few questions.
> >
> > Are the points defined in:
> > set_hash_rects
> >
> > From top left to bottom right, by pixel?, based on the whole
> screenshot?
>
> There is currently work happening in
> https://github.com/openSUSE-Team/os-autoinst
> <https://github.com/openSUSE-Team/os-autoinst>
> to drop hash usage altogether in favour of opencv image matching.
>
> Hash matching is good for my use case, as right now, openvc does not
> work on my system.
> Will you be merging the openSUSE-Team work into your git tree soon?

They are putting a lot of effort into it, so sooner or later I will
merge some of their changes. Maybe we could keep hashes as one way to
check results.

> results for single tests are determined in the "check" method in

> basetest.pm <http://basetest.pm>

> using either hashes, reference images, OCR or (in case of audio tests)
> multimon
>
>
> OK.
> Are the hashes that are checked, the ones defined in
> sub checklist()
>
> for each test? Are these hashes of the various stage areas, or of the
> entire screenshot?

They can be either. In consoletests I usually use whole-screen-hash
but e.g. in bootloader or X11, I usually use areas / hash_rects, so that
clock, cursor etc are not included... but reference images are more
powerful in the latter case, because you can do inexact matches at
variable positions.

> Also, should I only include hashes for the steps I am going to add to
> the check?
>
> I was wanting to confirm the test result by the outcome of
> waitserial(). Is this possible?

not yet, because the checking happens in a separate step after the test.
But it would be possible to add.

> The other thing I wonder, is when I am finished, should I issue a
> shutdown to the guest OS, as the last test or is the a
> better way of doing things?

yes, you should have code that does shutdown the VM in some way.

https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_kde.pm
E.g. this example exercises KDE UI code, and the path to shutdown as
non-root
https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_others.pm
just uses the power-button to trigger acpid

> Only other question, is have you devised a way to do skip the install,
> and just boot the last installed image?

yes, you can do
export KEEPHDDS=1 ; export NOINSTALL=1
the second one will skip all tests that use "installstep" as base class.

> .. I wonder if shutdown should show something other than na?

"na" just means that you dont have automatic checks. That is OK for such
steps that are just there to do something.

> Thank you for your help so far. Once I've got this passing, I would
> like to submit a pull request, if your happy to merge the Mageia stuff
> into your git tree.

Yes, sure. I am glad to see more people using automated tests to improve
Linux software quality.

> One more question about this.. With the hashes defined in checklist(), are they based on hashes of the screenshot / stage from anytime during the
> running of the test, ie, anytime during 010_consoletest_setup.pm, or are they based on hashes of what is on the screen at the
> end of the 010_consoletest_setup.pm run?

The hash will match if it occurred at any time in the whole testrun,
either in one of the hash_rects or in the whole-screen MD5-hash.
This is why it is normally not a good idea with the hash checklist to
have a consoletest just print "OK" - because other tests could be added
later that do the same.

[Glen Ogilvie]

Hi,

On Sunday, April 7, 2013 6:23:29 PM UTC+12, Bernhard M. Wiedemann wrote:

Am 07.04.2013 02:16, schrieb Glen Ogilvie:
>
>
> On Sunday, April 7, 2013 6:21:32 AM UTC+12, Bernhard M. Wiedemann wrote:
>
>     Am 06.04.2013 13:07, schrieb Glen Ogilvie:
>     > Hi,
>     >
>     > I've got a few questions.
>     >
>     > Are the points defined in:
>     > set_hash_rects
>     >
>     >  From top left to bottom right, by pixel?, based on the whole
>     screenshot?
>
>     There is currently work happening in
>     https://github.com/openSUSE-Team/os-autoinst
>     <https://github.com/openSUSE-Team/os-autoinst>
>     to drop hash usage altogether in favour of opencv image matching.
>
> Hash matching is good for my use case, as right now, openvc does not
> work on my system.
> Will you be merging the openSUSE-Team work into your git tree soon?

They are putting a lot of effort into it, so sooner or later I will
merge some of their changes. Maybe we could keep hashes as one way to
check results.

It would be nice if it could be kept as an optional feature to use. :)

I think I've found a bug with the md5_result hashes.
Here is a patch:

diff --git a/basetest.pm b/basetest.pm
index ed6e402..d80bd06 100644
--- a/basetest.pm
+++ b/basetest.pm
@@ -136,7 +136,7 @@ sub check(%) {
                $md5_result = 'unk';
                foreach my $h (keys(%$checklist)) {
                        if($hashes->{$h}) {
-                               $checkval = lc $checklist->{$h};
+                               $md5_result = lc $checklist->{$h};
                                last;
                        }
                }

(Commit:  https://github.com/nelg/os-autoinst/commit/74269e751d83d87225222491aa9a71177546f4da  )

Because, as far as I can tell, $checkval is not used anywhere within the code, and even valid tests, still did
not return a valid result

>     results for single tests are determined in the "check" method in
>     basetest.pm <http://basetest.pm>
>     using either hashes, reference images, OCR or (in case of audio tests)
>     multimon
>
>
> OK.
> Are the hashes that are checked, the ones defined in
> sub checklist()
>
> for each test?  Are these hashes of the various stage areas, or of the
> entire screenshot?
They can be either. In consoletests I usually use whole-screen-hash
but e.g. in bootloader or X11, I usually use areas / hash_rects, so that
clock, cursor etc are not included... but reference images are more
powerful in the latter case, because you can do inexact matches at
variable positions.

It looks like for refimages, I would need opencv support.   This will be no problem for the Mageia tests once Mageia 3 is released, as it includes the required opencv version. This is however, about a month away.

 

> Also, should I only include hashes for the steps I am going to add to
> the check?
>
> I was wanting to confirm the test result by the outcome of
> waitserial().  Is this possible?

not yet, because the checking happens in a separate step after the test.
But it would be possible to add.

I think this would be good.   :)   I'll see if i can figure out how.
 

> The other thing I wonder, is when I am finished, should I issue a
> shutdown to the guest OS, as the last test or is the a
> better way of doing things?

yes, you should have code that does shutdown the VM in some way.

https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_kde.pm
E.g. this example exercises KDE UI code, and the path to shutdown as
non-root
https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_others.pm
just uses the power-button to trigger acpid

OK, so I am telling the VM to shutdown, and it does, but in the log, I still get:
 

"ALARM: qemu virtual machine quit! - exiting..."

Is this just expected?
 

> Only other question, is have you devised a way to do skip the install,
> and just boot the last installed image?

yes, you can do
export KEEPHDDS=1 ; export NOINSTALL=1
the second one will skip all tests that use "installstep" as base class.

Thank you, I'll document this, and others, and submit them as a patch to the docs.

It does still try and boot from the CD first.  How do you usually get around this?
It would be nice if the boot order was from HDD first.

Regards
Glen Ogilvie

[Bernhard M. Wiedemann]

> (Commit:
> https://github.com/nelg/os-autoinst/commit/74269e751d83d87225222491aa9a71177546f4da
> )
>
> Because, as far as I can tell, $checkval is not used anywhere within the
> code, and even valid tests, still did
> not return a valid result

great. I had noticed that hash checking was not properly working, but
did not yet have time to track it down. pushed your commit.

> > results for single tests are determined in the "check" method in

> > basetest.pm <http://basetest.pm> <http://basetest.pm>

> > using either hashes, reference images, OCR or (in case of
> audio tests)
> > multimon
> >
> >
> > OK.
> > Are the hashes that are checked, the ones defined in
> > sub checklist()
> >
> > for each test? Are these hashes of the various stage areas, or
> of the
> > entire screenshot?
> They can be either. In consoletests I usually use whole-screen-hash
> but e.g. in bootloader or X11, I usually use areas / hash_rects, so
> that
> clock, cursor etc are not included... but reference images are more
> powerful in the latter case, because you can do inexact matches at
> variable positions.
>
> It looks like for refimages, I would need opencv support. This will be
> no problem for the Mageia tests once Mageia 3 is released, as it
> includes the required opencv version. This is however, about a month away.

There is the "search" function for image-matching in ppm.pm that works
without opencv.
You need to have nicely formatted names as seen in
http://openqa.opensuse.org/openqa/perl/autoinst/testimgs/

> > Also, should I only include hashes for the steps I am going to
> add to
> > the check?
> >
> > I was wanting to confirm the test result by the outcome of
> > waitserial(). Is this possible?
>
> not yet, because the checking happens in a separate step after the
> test.
> But it would be possible to add.
>
>
> I think this would be good. :) I'll see if i can figure out how.
>
>
> > The other thing I wonder, is when I am finished, should I issue a
> > shutdown to the guest OS, as the last test or is the a
> > better way of doing things?
>
> yes, you should have code that does shutdown the VM in some way.
>
> https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_kde.pm
> <https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_kde.pm>
>
> E.g. this example exercises KDE UI code, and the path to shutdown as
> non-root
> https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_others.pm
> <https://github.com/bmwiedemann/os-autoinst/blob/master/x11test.d/900_shutdown_others.pm>
>
> just uses the power-button to trigger acpid
>
>
> OK, so I am telling the VM to shutdown, and it does, but in the log, I
> still get:
>
> "ALARM: qemu virtual machine quit! - exiting..."
>
> Is this just expected?

qemu is expected to quit at that point, so should be OK.

> > Only other question, is have you devised a way to do skip the
> install,
> > and just boot the last installed image?
>
> yes, you can do
> export KEEPHDDS=1 ; export NOINSTALL=1
> the second one will skip all tests that use "installstep" as base
> class.
>
>
> Thank you, I'll document this, and others, and submit them as a patch to
> the docs.

thanks. great.

> It does still try and boot from the CD first. How do you usually get
> around this?
> It would be nice if the boot order was from HDD first.

one way is what we have for ZDUP in
distri/opensuse/inst.d/020_bootloader.pm

in principle, you could change inst/startqemu.pm to have the -boot param
variable, but then it will act different from the VBox backend, unless
we change that as well.


Ciao
Bernhard M.