alternative to bloated templates for faster work and minimal boot time/resources used

[travorfi...@gmail.com]

Is it possible to transfer sys-net/sys-usb/sys-vpn/sys-whonix to 100mb templates based on musl/busybox/sysvinit linux ?

use wpa_supplicant instead of network-manager with tons of scripts
use cli

use cli instead of gui (sxiv,st, fff, surf, mpv, ncmpcpp, mutt, weechat etc)

for minimum RAM usage and disk space requirements

it is also discussed here
https://www.reddit.com/r/Qubes/comments/9upqfd/qubes_and_light_weight_distros_fo
r_t400_or_t420/

Thanks.

[799]

Hello,

Am Do., 15. Nov. 2018, 09:44 hat <travorfi...@gmail.com> geschrieben:

Is it possible to transfer sys-net/sys-usb/sys-vpn/sys-whonix to 100mb templates based on musl/busybox/sysvinit linux ?

(...)

my sys-vms are based on fedora-28-minimal templates.

Honestly I like the idea and think smaller is better, but as I am running lots ~8-12 AppVMs when working productively most ressources are used by those VMs.

I don't think that you gain that much ressources by switching sys-vms.

And honestly storage capacity is not a big deal today ;-)

- O

[Chris Laprise]

On 11/15/2018 12:29 PM, 799 wrote:
> Hello,
>
> Am Do., 15. Nov. 2018, 09:44 hat <travorfi...@gmail.com

> <mailto:travorfi...@gmail.com>> geschrieben:

Disk capacity shouldn't be a big issue unless you like to make lots of
template variations.

For RAM efficiency the available templates are already pretty efficient,
but the Qubes RAM allocation algorithms could be better. Manually
setting the maximum RAM has worked great on my 8GB system: about 350MB
for each service VM, 700-900MB for media playback, 1500-2500MB for
browsing and other heavy apps. Finally, I set the dom0 max to 1500MB in
/etc/default/grub. Using debian-9 templates, these ranges result in a
system that is *much* more usable than it would be with the default RAM
allocation; I highly recommend it.



--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[unman]

As already suggested, you can get a long way by taking control of memory
use within Qubes - limiting the amount allocated to dom0, and to qubes
according to their use.
You could look at trying out a unikernel firewall : search the archives
for mirage firewall.
You have to distinguish between working cli and working without X :
they aren't the same.
It's perfectly possible to run headless qubes by stripping out X and
qubes-gui-agent, and there is some memory saving. Downside is that you
need to connect using xl console if there's any configuration that
needs doing.

Your list includes a variety of technologies. I don't think there's any
work on using them in Qubes, which is not to say that you couldn't have a go.
Qubes uses systemd a lot - no reason why you couldn't rework that in
sysvinit. There was some talk about Alpine some years back, but it never
got anywhere, partly because of restrictions of musl.
The short answer is, yes, it's possible (probably), but I'm not convinced
the effort would pay off.

[799]

Hello unman,

Am Fr., 16. Nov. 2018, 17:28 hat unman <un...@thirdeyesecurity.org> geschrieben:

(...)

It's perfectly possible to run headless qubes by stripping out X and
qubes-gui-agent, and there is some memory saving. Downside is that you
need to connect using xl console if there's any configuration that
needs doing.

Can you provide some information how this can be done? 

I have some "service AppVMs" which are providing services for other VMs, for example to provide file storage.

The idea is to separate storage/cloud syncing into three layers:

1) one AppVM stores encrypted data / no internet connectivity

2) one AppVM can mount the encrypted data and data manipulating is done in DVMs

3) one AppVM can synchronize the encrypted data to a cloud service

Those VMs could perfectly run without a GUI/X-Windows.

- O

[unman]

Sorry to have overlooked this.
You probably need to start by upgrading to packages from
stretch-testing. There was an issue where qubes-core-agent incorrectly
had X as a dependency - that's been fixed now, but I think the updated
package is in testing.
Dont forget to set a root password, if you're removing
passwordless-sudo. You can do this, of course by running:
qvm-run -u root -p <qube> passwd
Attach to the template using sudo xl console and then start stripping
down- you can remove a lot of the graphical programs obviously.
You'll almost certainly find strange dependencies that haven't been set
properly.
Use aptitude and keep a close eye on what you're removing.
It pays to take things slowly and to shutdown and clone as you go - that
way if things go bad you can step back instead of having to start from
the beginning.

Alternatively, roll your own template in qubes-builder, by bootstrapping
using debootstrap with the minbase variant and adding only a few select
packages. Make sure that you keep "recommended" off to avoid unnecessary
packages being installed.

[pieter lems]

Hello Chris,

I was wondering, if i run a qube based on a fedora template it uses about 3.5GB of memory when just browsing. I decreased it to 2500 and it works exactly the same. Will there be any negative effects by decreasing the memory manually?

Thanks for the info btw!

Op vr 16 nov. 2018 om 03:27 schreef Chris Laprise <tas...@posteo.net>:

--
You received this message because you are subscribed to the Google Groups "qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users...@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/d272fbef-32ad-5db6-1a7a-9ad012f8e072%40posteo.net.
For more options, visit https://groups.google.com/d/optout.

[Chris Laprise]

On 11/26/2018 06:56 AM, pieter lems wrote:
> Hello Chris,
> I was wondering, if i run a qube based on a fedora template it uses
> about 3.5GB of memory when just browsing. I decreased it to 2500 and it
> works exactly the same. Will there be any negative effects by decreasing
> the memory manually?
> Thanks for the info btw!

Linux tends allocate whatever amount of RAM is given to it, even if that
memory isn't put to use.

The only real negative to reducing VM memory is that it may start to use
swap space if you open a lot of tabs. Light swap use isn't noticeable,
but heavy swap use would make the browser noticeably slower.

You can check swap use from the VM's terminal with the 'free' command.
I'm currently running Firefox with several windows and Thunderbird in
one 2000MB VM now and swap use is still zero. I would expect slowdowns
if swap use went much above 500MB.

[799]

Hello,

Am Sa., 1. Dez. 2018, 05:12 hat Chris Laprise <tas...@posteo.net> geschrieben:

Linux tends allocate whatever amount of RAM is given to it, even if that
memory isn't put to use.

The only real negative to reducing VM memory is that it may start to use

swap space if you open a lot of tabs. [...]

You can check swap use from the VM's terminal with the 'free' command.

[...]

We could write a script which runs in dom0 and runs the 'free' command in every AppVM (qvm-run) which is running and if swap space is used send a notification.

This script could run via a regular cronjob like every few minutes (assuming that swap space will slowly grow and used for a longer time).

I would like to do so, the only missing part is how I can create a qvm-run command which runs on every running qube.

Must be something like FOR ...

... Can someone help building this part, then I'll try to do the rest.

- O

[Chris Laprise]

On 12/12/2018 04:01 PM, 799 wrote:
> Hello,
>
> Am Sa., 1. Dez. 2018, 05:12 hat Chris Laprise <tas...@posteo.net

> <mailto:tas...@posteo.net>> geschrieben:

Keep in mind this is somewhat _risky_ because its parsing data supplied
by appVMs.

But here's how you could start the loop:

qvm-ls --running -O name | (read line; while read line; do qvm-run -p
$line 'your vm command goes here'; done)

There is an extra 'read line' at the start to get rid of the qvm-ls header.

Sanitizing should be done on any results you get back from qvm-run.

[qubenix]

> But here's how you could start the loop:
>
> qvm-ls --running -O name | (read line; while read line; do qvm-run -p
> $line 'your vm command goes here'; done)
>
> There is an extra 'read line' at the start to get rid of the qvm-ls header.

Adding `--raw-data` should avoid the header (untested).


--
qubenix

CODE PGP: FE7454228594B4DDD034CE73A95D4D197E922B20
EMAIL PGP: 96096E4CA0870F1C5BAF7DD909D159E1241F9C54
IRC OTR: DFD1DA35 D74E775B 3E3DADB1 226282EE FB711765

[799]

Hello Chris,

On Wed, 12 Dec 2018 at 22:52, Chris Laprise <tas...@posteo.net> wrote:

But here's how you could start the loop:
qvm-ls --running -O name | (read line; while read line; do qvm-run -p
$line 'your vm command goes here'; done)

[...]

thanks for the hint.

I tried to run the following command which gives me a list of all running VMs (just to get the loop right):

qvm-ls --running -O name --raw-data | (read line; while read line; do echo $line; done)

In my case:

my-untrusted

my-vault

sys-firewall

sys-net

sys-usb

The command to get the information about the swap usage from a VM (here named APPVMNAME can be shown by running the following command in dom0:

echo APPVMNAME = `qvm-run --pass-io APPVMNAME "free" | tail -n +3 | gawk '{ print $3 }'`

But when I bring both peaces together I only get one result back, not the result for all AppVMs

qvm-ls --running -O name --raw-data | (read line; while read line; do echo $line `qvm-run --pass-io $line "free" | tail -n +3 | gawk '{ print $3 }'` ; done)

Output in my case:

my-untrusted 256

So the output is only showing the output for one VM.

I don't understand why this is happening.

- O

[Mike Keehan]

Hi,

I can't figure out why the code above only returns the first VM's
result, but the following code works for me :-

for i in `qvm-ls --running -O name --raw-data|grep -v dom0`;
do
echo $i = `qvm-run --pass-io $i "free"| tail -n +3 | gawk '{print
$3 }'`;
done

Mike.

[Chris Laprise]

Can't use both --raw-data and the first read line. Use only one or the
other.

>
> qvm-ls --running -O name --raw-data | (read line; while read line; do
> echo $line `qvm-run --pass-io $line "free" | tail -n +3 | gawk '{ print
> $3 }'` ; done)
>
> Output in my case:
> my-untrusted 256
>
> So the output is only showing the output for one VM.
> I don't understand why this is happening.
>
> - O
>

[799]

Hello Mike,

On Thu, 13 Dec 2018 at 21:51, Mike Keehan <mi...@keehan.net> wrote:

On Thu, 13 Dec 2018 20:31:29 +0100
799 <one7...@gmail.com> wrote:

> [...]

> But when I bring both peaces together I only get one result back, not
> the result for all AppVMs

> [...]

I can't figure out why the code above only returns the first VM's
result, but the following code works for me :-
for i in `qvm-ls --running -O name --raw-data|grep -v dom0`;
do
   echo $i = `qvm-run --pass-io $i "free"| tail -n +3 | gawk '{print
$3 }'`;
done

with the help from you I was able to write a small script which will generate a notification pop up if swap usage in an appvm reaches a threshold (maxswap).

This script could then be run via cron or maybe watch.

--- 8< --- --- ---

#!/bin/bash

# name : qvm-show-swap.sh

# version: 0.1

# date : 15.12.2018

# author : one7two99 + help of Qubes OS mailinglist ;-)

# define allowed maximum swap usage in Kb

# if more swap space is used in an AppVM a warning notification will pop-up

maxswap=1024

# get a list of all running AppVms (except dom0)

for i in `qvm-ls --running -O name --raw-data | grep -v dom0`;

do

   # get swap usage in the AppVM

   freeswap=`qvm-run --pass-io $i "free" | tail -n+3 | gawk '{print $3}'`;

   # convert swap usage from Kb to Mb

   swapsize=$(expr $freeswap / 1024);

   # Check if swap usage is > than maxswap

   if [ "$freeswap" -gt "$maxswap" ]; then

      # create a PopUp-notification

      notify-send --urgency normal --icon dialog-warning --expire-time=5000 "$i" ".. is using $swapsize Mb swap";

   fi;

done

--- 8< --- --- ---

For a simple test you could run this script every 5 min in dom0 via:

watch -n 300 ./qvm-show-swap.sh

or you setup something like cron in dom0 etc.

Let me know if you think there is a better approach.

- O