Whonix tests

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Patrick,

Do you have some automated tests for Whonix gw/ws? In Qubes we have
tests for some basic functionality - both dom0 and templates. I think it
is a good idea to extend those tests to also include some Whonix
specific cases.

Our tests are based on python unittest module, but actually those are
integration tests. Example tests currently implemented:
- create + start VM
- start some GUI application (currently gnome-terminal), check if window appears
- start Disposable VM, edit some file there
- networking (ping), directly to netvm, through proxyvm etc.
- making VMs backup and restoring it - including compatibility with
older backup formats

This is just to give you an idea what type of tests are there.

What do you think of preparing similar tests for Whonix VMs? I guess it
would include things like:
- checking tor connectivity
- checking time sync
- simple leak tests
- VM configuration - especially if all Whonix specific things are set
up (for example "host" as a hostname, UTC timezone, etc).

Do you have something like that already done? If yes, I'd like to hook
it into our test runner - to automatically call those tests when going
through full test run. If not - can you provide some such tests? Or at
least provide an idea what should be tested and _how_ (what command to
execute and what should be expected result).

Tests are located here:
https://github.com/QubesOS/qubes-core-admin/tree/master/tests

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVkVmFAAoJENuP0xzK19csM+wH/1wHpqTn6UhXOdkBqaXibxzg
T88IceZ1JHSQaY3qjZhFyglxG7PnIhohVgq1mRgrEQvPh/uYZ4sDZfKTMoMnZvMI
i/aj22HKp48IWB2hkwhP9+ixXjrvk2UMVQPIrU3aheVnPzjbfrEU01eH087zthiB
/gVixw9wnFBCboPkwzWnVLMG337psT9mp+xflo6k0+k4sLOxaQMteLsLigXuQskW
3Yg9wNasEj41PK4ezXTBJxsabeC7K6HN+OPUSpT13O2Yf7BOYfmT5O5+tn3U3H4C
LphvIowx+yZwvw/P15nMIpSPl8+ykn0j9MxKlZrMItmPWG3a6Wq72SR114mNztQ=
=fraN
-----END PGP SIGNATURE-----

[Jason M]

On Monday, 29 June 2015 10:43:27 UTC-4, Marek Marczykowski-Górecki wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Patrick,

Do you have some automated tests for Whonix gw/ws? In Qubes we have
tests for some basic functionality - both dom0 and templates. I think it
is a good idea to extend those tests to also include some Whonix
specific cases.

Our tests are based on python unittest module, but actually those are
integration tests. Example tests currently implemented:
 - create + start VM
 - start some GUI application (currently gnome-terminal), check if window appears
 - start Disposable VM, edit some file there
 - networking (ping), directly to netvm, through proxyvm etc.
 - making VMs backup and restoring it - including compatibility with
   older backup formats

I perform most of those tests with the salt management tool I am currently working on; the dom0 tests anyway.  Here are the tests that I run in dom0 to test qvm features I implemented: qvm tests.

I was planning on implementing the VM related tests in the as well since salt has a test mode to test various states.  Salt can either be installed on the VM or commands run via a modified salt-ssh (to run over vchan instead of ip) which does not require salt to be installed on VM.

I am not sure of the best way to hook it into the unittest module though; I would need to think about it.  In the tests I gave in above link, you are able to run those tests in either default or test mode.  In default mode, salt would attempt to modify the state of the VM for each state listed. If run in 'test' mode, salt will only report back its result.  For the example of cloning a VM, in default mode, it would actually clone the VM if the VM to clone exists and the target VM did not exist and return a result of pass.  if the same sates were run in 'test' mode it would return a 'pass stating that it can be cloned or is already cloned but would not actually clone it.

Following is an example of states. The top report is run in default mode, while the bottom in test mode.  There are many report styles that can be used; maybe a custom one to work with unittests can be made.  This is what the states do in the example below:

• makes sure my gpg key is imported and installs it if not
• makes sure python-gnupg is installed, and installs it if not
• enables debug mode to show all [PASS] [FAIL] results instead of just last one in qvm related states since some states it has multiple steps like when creating a VM it first checks to see if one already exists, creates it if it does not, and confirms it was created
• makes sure the template is installed, if not will download and install it
• installs and configures sys-whonix-test (Whonix Gateway) if it does not already exist
• ensures locale is set to utf8
  

root@dom0:/home/user# qubesctl state.highstate
local:
----------
          ID: gnupg
    Function: gpg.import_key
      Result: True
     Comment: [PASS] Not actually changed
             
                     [GNUPG:] IMPORT_OK 0 E0E32283FDCAC1A510078F271BB9B1FB5A4C6DAD
                     gpg: key 5A4C6DAD: "Jason Mehring (Qubes OS Signing Key) <nrg...@gmail.com>" not changed
                     gpg: Total number processed: 1
                     gpg:              unchanged: 1
                     [GNUPG:] IMPORT_RES 1 0 0 0 1 0 0 0 0 0 0 0 0 0
     Started: 19:10:01.656697
    Duration: 85.749 ms
     Changes:  
----------
          ID: gnupg
    Function: pkg.installed
        Name: python-gnupg
      Result: True
     Comment: Package python-gnupg is already installed.
     Started: 19:10:01.742726
    Duration: 926.709 ms
     Changes:  
----------
          ID: virtual-machines-debug-id
    Function: test.debug
      Result: True
     Comment: [PASS] Enabled 'ALL'
     Started: 19:10:02.670849
    Duration: 2.15 ms
     Changes:  
----------
          ID: whonix-test_qubes-template-whonix-ws
    Function: pkg.installed
        Name: qubes-template-whonix-ws
      Result: True
     Comment: Package qubes-template-whonix-ws is already installed.
     Started: 19:10:02.673058
    Duration: 0.227 ms
     Changes:  
----------
          ID: whonix-test
    Function: qvm.exists
      Result: True
     Comment: [PASS] /usr/bin/qvm-check whonix-test
                     A VM with the name 'whonix-test' does exist.
     Started: 19:10:02.674707
    Duration: 253.344 ms
     Changes:  
----------
          ID: sys-whonix-test_qubes-template-whonix-gw-experimental
    Function: pkg.installed
        Name: qubes-template-whonix-gw-experimental
      Result: True
     Comment: Package qubes-template-whonix-gw-experimental is already installed.
     Started: 19:10:02.928315
    Duration: 0.609 ms
     Changes:  
----------
          ID: sys-whonix-test
    Function: qvm.vm
      Result: True
     Comment: ====== ['present'] ======
              [PASS] /usr/bin/qvm-check sys-whonix-test
                     A VM with the name 'sys-whonix-test' does not exist in the system!
              [PASS] /usr/bin/qvm-create --proxy --mem 500 --template whonix-gw-experimental --label purple sys-whonix-test
                     --> Using TemplateVM: whonix-gw-experimental
                     --> Converting Appmenu Templates...
                     --> Adding Apps to the Menu...
              [PASS] /usr/bin/qvm-check sys-whonix-test
                     A VM with the name 'sys-whonix-test' does exist.
             
              ====== ['prefs'] ======
              [SKIP] autostart          : False
              [PASS] /usr/bin/qvm-prefs  --set sys-whonix-test netvm "sys-firewall"
             
              ====== ['service'] ======
              [PASS] /usr/bin/qvm-service sys-whonix-test --enable whonix-tor-enable
     Started: 19:10:02.930997
    Duration: 4871.676 ms
     Changes:  
              ----------
              qvm.prefs:
                  ----------
                  qvm.prefs:
                      ----------
                      netvm:
                          ----------
                          new:
                              sys-firewall
                          old:
                              None
              qvm.service:
                  ----------
                  qvm.service:
                      ----------
                      whonix-tor-enable:
                          ----------
                          new:
                              Enabled
                          old:
                              Missing
----------
          ID: en_US.utf8
    Function: locale.system
      Result: True
     Comment: System locale en_US.utf8 already set
     Started: 19:10:07.804455
    Duration: 66.125 ms
     Changes:  

Summary
------------
Succeeded: 8 (changed=1)
Failed:    0
------------
Total states run:     8

These are the same state, but run in test mode right after above test, so whonix-gateway is now installed and configured.

root@dom0:/home/user# qubesctl state.highstate test=true
local:
----------
          ID: gnupg
    Function: gpg.import_key
      Result: None
     Comment: [PASS] Key will be imported
     Started: 19:11:00.813014
    Duration: 21.495 ms
     Changes:  
----------
          ID: gnupg
    Function: pkg.installed
        Name: python-gnupg
      Result: True
     Comment: Package python-gnupg is already installed.
     Started: 19:11:00.834610
    Duration: 715.893 ms
     Changes:  
----------
          ID: virtual-machines-debug-id
    Function: test.debug
      Result: None
     Comment: [PASS] Enabled 'ALL'
     Started: 19:11:01.551931
    Duration: 2.149 ms
     Changes:  
----------
          ID: whonix-test_qubes-template-whonix-ws
    Function: pkg.installed
        Name: qubes-template-whonix-ws
      Result: True
     Comment: Package qubes-template-whonix-ws is already installed.
     Started: 19:11:01.554138
    Duration: 0.232 ms
     Changes:  
----------
          ID: whonix-test
    Function: qvm.exists
      Result: None
     Comment: [PASS] /usr/bin/qvm-check whonix-test
                     A VM with the name 'whonix-test' does exist.
     Started: 19:11:01.555805
    Duration: 251.743 ms
     Changes:  
----------
          ID: sys-whonix-test_qubes-template-whonix-gw-experimental
    Function: pkg.installed
        Name: qubes-template-whonix-gw-experimental
      Result: True
     Comment: Package qubes-template-whonix-gw-experimental is already installed.
     Started: 19:11:01.807793
    Duration: 0.617 ms
     Changes:  
----------
          ID: sys-whonix-test
    Function: qvm.exists
      Result: None
     Comment: [PASS] /usr/bin/qvm-check sys-whonix-test
                     A VM with the name 'sys-whonix-test' does exist.
     Started: 19:11:01.810518
    Duration: 276.596 ms
     Changes:  
----------
          ID: en_US.utf8
    Function: locale.system
      Result: True
     Comment: System locale en_US.utf8 already set
     Started: 19:11:02.088930
    Duration: 30.017 ms
     Changes:  

Summary
------------
Succeeded: 8 (unchanged=4)
Failed:    0
------------
Total states run:     8

[Patrick Schleizer]

Marek Marczykowski-Górecki:

> Do you have some automated tests for Whonix gw/ws?

Yes, see below.

> In Qubes we have
> tests for some basic functionality - both dom0 and templates. I think it
> is a good idea to extend those tests to also include some Whonix
> specific cases.

Yes.

> Our tests are based on python unittest module, but actually those are
> integration tests. Example tests currently implemented:
> - create + start VM
> - start some GUI application (currently gnome-terminal), check if window appears
> - start Disposable VM, edit some file there
> - networking (ping), directly to netvm, through proxyvm etc.
> - making VMs backup and restoring it - including compatibility with
> older backup formats

Awesome stuff.

ping and Whonix:
Whonix-Gateway has no system DNS on purpose. [6]

With default settings (there are options) only user clearnet is allowed
to use ping.

So if you want a ping test, use:
sudo -u clearnet ping ...

> This is just to give you an idea what type of tests are there.
>
> What do you think of preparing similar tests for Whonix VMs? I guess it
> would include things like:
> - checking tor connectivity

Covered by whonixcheck, see below.

> - checking time sync

Covered by timesync, see below.

> - simple leak tests

Covered by whonixcheck, see below.

> - VM configuration - especially if all Whonix specific things are set
> up (for example "host" as a hostname, UTC timezone, etc).

Hostname:
Covered by whonixcheck, see below.

Timezone:
Not checked yet.
TODO: https://phabricator.whonix.org/T368
Patches welcome or I'll do it in time for Whonix 12.

Other stuff:
Might be covered by whonixcheck already. See:
https://www.whonix.org/wiki/Whonixcheck#Checks
Please see what stuff isn't covered. Feel free to mention here and/or
create tickets etc.

> Do you have something like that already done? If yes, I'd like to hook
> it into our test runner - to automatically call those tests when going
> through full test run. If not - can you provide some such tests? Or at
> least provide an idea what should be tested and _how_ (what command to
> execute and what should be expected result).

Yes. Done below. If something is missing, please say so.

> Tests are located here:
> https://github.com/QubesOS/qubes-core-admin/tree/master/tests

TLDR:

Existing tests...

(1) general sanity checks (during / after build) [1]:

run as root:

/usr/lib/anon-dist/chroot-scripts-pre.d/20_sanity_checks

(2) check, that no non-free software has been installed (during / after
build) [2]:

run as root:

/usr/lib/anon-dist/chroot-scripts-post.d/75_vrms

(3) security and anonymity check (after VM start) [3] [4]:

run as user:

whonixcheck

Or.

whonixcheck --verbose

(4) For testing timesync, just try. Run.

timesync

Long timeout: 10 minutes. No reliable exit code yet when it fails (bug).
Output is correct. Rewrite in progress. [7] [8]

More:

All tests exit non-zero if major issues are found. For all scripts I am
working on, I have reliable exit codes in mind.

(1) and (2) are supposed to be run (primarily) during or after build

In Whonix, general sanity checks (1) are run during build. In Qubes, for
some reason, nrgaway can tell, qubes-template-whonix, file
whonix-gateway/02_install_groups_pre.sh uses '--sanity-tests false'. If
I remember right, because that speeds up the build. I don't like this as
default (it's okay for personal, occasionally builds), but most of the
time it's better to keep those tests running.

If there is any issues with the tests in Qubes, the tests should be
fixed in Whonix. Not the tests disabled by qubes-template-whonix.

(3) whonixcheck is supposed to be run after a VM has been started. In a
Whonix-Gateway ProxyVM or Whonix-Workstation AppVM. Not in a TemplateVM.
I was wondering if I modify whonixcheck to be able to run in TemplateVMs
also. With fewer tests perhaps.

whonixcheck runs basic leak tests already. Runs Socks- and TransPort in
Whonix-Workstation. SocksPort on Whonix-Gateway (has no transparent
proxying enabled by default).

Cheers,
Patrick

[1]
https://github.com/Whonix/anon-shared-build-sanity-checks/blob/master/usr/lib/anon-dist/chroot-scripts-pre.d/20_sanity_checks

[2]
https://github.com/Whonix/anon-shared-build-ban-nonfree/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/75_vrms

[3] https://github.com/Whonix/whonixcheck

[4] https://www.whonix.org/wiki/Whonixcheck

[5]
https://github.com/nrgaway/qubes-template-whonix/blob/eb795b00eac7ed21810dda6ce978b5ed0b27681c/whonix-gateway/02_install_groups_pre.sh#L45

[6] https://www.whonix.org/wiki/Whonix-Gateway_System_DNS

[7] https://phabricator.whonix.org/T300

[8] https://www.whonix.org/forum/index.php/topic,1301.0.html