Port forwarding to appvm

[auto35...@hushmail.com]

Hi Everyone,

I want to run a server application in one of the appvms but have problems configuring iptables.
What do I need to do to forward a port from the external/"real" interface of netvm via firewallvm to a netvm?

Kind Regards

[Marek Marczykowski]

This isn't trivial... Manual steps:
1. netvm: add DNAT to firewallvm IP to nat table; allow traffic in FORWARD
chain (default aka filter table)
2. firewallvm: add DNAT to appvm IP to nat table; allow traffic in FORWARD
chain (*)
3. appvm: allow traffic in INPUT chain

You can obtain VM addresses from "qvm-ls -n" in dom0.

Currently automatic tool for such configuration doesn't exists in Qubes, but
patches are welcomed :)

(*) FORWARD chain in firewallvm is automatically rewritten at every network
configuration change or VM start/stop. You can add the rule to the hook script
(/rw/config/qubes_firewall_user_script) to have it added after each such
event. All other rules (in netvm and appvm) you can add to /rw/config/rc.local
script to have them at each system start.

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Marek Marczykowski]

On 25.12.2012 16:36, fendi...@gmail.com wrote:
> Hi Experts,
> I am struggling with this same request. It seems to me
> /rw/config/qubes_firewall_user_script is not being called on vm
> stop/start...I made the file with root and +x
> Did I understand correctly and have created the right file? Does it have to
> have #!/bin/bash in the top?

This is called only in firewallvm (at each firewall reload). For normal AppVM
use /rw/config/rc.local instead.

[Joanna Rutkowska]

On 12/27/12 12:34, fendi...@gmail.com wrote:
> Marek,
> Thanks, I have succesfully configured the user script.
>
> A followup question, when I edit my AppVM firewall rules (service http), I
> do see them getting added to the firewall-vm iptables, but not to the AppVM
> iptables.
> Is this expected?
>
>
Yes. Why would you like to also have it applied in the AppVM?

BTW, I assume you already read this (and the referenced message that
explains the f/w thread model in Qubes):
http://wiki.qubes-os.org/trac/wiki/QubesFirewall

joanna.

[bradbury9]

Hi,

could I suggest you, after all the steps are finished and the port forwarding works, to document it down (type what files you created/modified, what values you changed...) and send it to the devel group?

That way someone with access to the wiki could add that info to the info/doc wiki. It looks like a cool (usefull) thing to document.

El jueves, 27 de diciembre de 2012 12:34:44 UTC+1, fendi...@gmail.com escribió:

Marek,
Thanks, I have succesfully configured the user script.

A followup question, when I edit my AppVM firewall rules (service http), I do see them getting added to the firewall-vm iptables, but not to the AppVM iptables.
Is this expected?

[Joanna Rutkowska]

On 12/27/12 13:09, fendi...@gmail.com wrote:
> @joanna: I get it now, I assumed that the Qubes GUI firewall rules would
> allow incoming traffic aswell, my bad.
>
> @bradbury9 & others:
> I wanted an AppVM to use as a server. I created a standalone AppVM for this
> task, and editted /etc/sysconfig/iptables

Any special reason to use a Standalone VM for this? If you only wanted a
custom iptables setup in an AppVM you could just add appropriate
commands to the AppVM's /rw/config/rc.local script.

joanna.

[Joanna Rutkowska]

> On Thursday, December 27, 2012 2:28:10 PM UTC+1, joanna wrote:

On 12/27/12 15:18, fendi...@gmail.com wrote:> I reasoned Standalone VM
would suit better since I started out with Apache2
> and wanted to edit httpd.conf
>
>
>

I see. Just out of curiosity -- can you describe your usage model for
this? I.e. why would anybody want to run a webserver (or any server) on
a laptop or generally on a client computer? Or, alternatively, why would
you like to use Qubes on a server?

joanna.

[Joanna Rutkowska]

On 12/27/12 13:09, fendi...@gmail.com wrote:
> @joanna: I get it now, I assumed that the Qubes GUI firewall rules would
> allow incoming traffic aswell, my bad.
>
> @bradbury9 & others:
> I wanted an AppVM to use as a server. I created a standalone AppVM for this
> task, and editted /etc/sysconfig/iptables

> Additional netvm, firewallvm and appvm config as discussed in this thread:
>
> [user@netvm ~]$ cat /rw/config/rc.local
> iptables -t filter -D FORWARD 4
> iptables -t nat -A PREROUTING -p tcp -d $EXT_IP --dport 80 -j DNAT
> --to-destination $FIREWALL-VM
>
> [user@firewallvm ~]$ cat /rw/config/qubes_firewall_user_script
> iptables -t nat -A PREROUTING -p tcp -d $FIREWALL-VM --dport 80 -j DNAT
> --to-destination $APP-VM
> iptables -P FORWARD ACCEPT
>
... obviously the last command is not recommended, as it globally sets
the ACCEPT policy for the FORWARD chain for all your AppVMs behind this
firewallvm. I would instead recommend something like:

iptable -I FORWARD 2 -d $APP-VM -j ACCEPT

Can you check this works for you? (I'm about to create a wiki page on this).

j.


> [user@APP-VM ~]$ sudo iptables -S
> ...
> -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
> -A INPUT -j REJECT --reject-with icmp-host-prohibited

[Alex Dubois]

On 31 Dec 2012, at 17:55, fendi...@gmail.com wrote:

Joanna,
I editted my firewall-user-script with your improvement. Then I triggered this script by changing some AppVMs firewall rules.
I can still succesfully reach my desired 'server'. Here below a snippet of the current output of my firewallvm:

[user@firewallvm ~]$ sudo iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i vif+ -p udp -m udp --dport 68 -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -d 10.137.2.10/32 -j ACCEPT
-A FORWARD -i vif0.0 -j ACCEPT
-A FORWARD -i vif+ -o vif+ -j DROP
...

Happy to help,
Cas

Could you confirm you are using Qubes2 beta1

In NetVM I have /rw/config/rc.local which does

An iptables -L output to a file

An iptables -N test

The file shows output of blank iptables and my test policy has been overwritten by default qubes's iptables policy. 

So I suspect a script is defining the iptables policy post rc.local

Thanks

Alex

--
 
 

[Tokyo Jeff]

To clarify, NetVm or ProxyVM (e.g. firewallvm)  -- and does this make a difference?

--
 
 

[Alex Dubois]

Sorry I was not very clear.

My problem is that whatever I code on the netVM in /rw/config/rc.local, it gets cleared and overwritten by the default netVM iptables nat and filter policies.

at the moment I've got this on NetVM on /rw/config/rc.local

#!/bin/sh

/bin/echo test > /rw/config/test
/sbin/iptables -t nat -L >> /rw/config/test
/sbin/iptables -t nat -N PR_QBS-USER-SERVICES
/sbin/iptables -t nat -A PREROUTING -j PR_QBS-USER-SERVICES
/sbin/iptables -t nat -A PR_QBS-USER-SERVICES -i eth0 -j DNAT -d x.x.x.x -p tcp --dport xxxxx --to-destination x.x.x.x

/bin/echo test2 > /rw/config/test

and less /rw/config/test post reboot gives me:

test
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination        

Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination        
test2

and iptables -t nat -L -v -n returns the default qubes iptable policy.

Thanks

Alx

[Marek Marczykowski]

/rw/config/rc.local (qubes-misc-post.service) is started asynchronously (in
parallel) to iptables.service so the order is non-deterministic. In your case
most likely qubes-misc-post is always started first.

I've just commited change to force it after network setup (especially
iptables.service). If you don't want to wait for new qubes-core-vm package
version, you can implement this patch manually in your template.

http://git.qubes-os.org/?p=marmarek/core.git;a=commit;h=917ccc2a4b4d9a2943f2d9e1c5ee604886498876

[Alex Dubois]

---------- Forwarded message ----------
From: Alex Dubois <bow...@gmail.com>
Date: Sat, Jan 12, 2013 at 9:22 AM
Subject: Re: [qubes-devel] Port forwarding to appvm
To: Marek Marczykowski <marm...@invisiblethingslab.com>

Great thanks for the heads up (I was deep into reading systemd man pages that I don't know, I was still on sysV).

I was about to say that it fixed my problem but in fact the netvm would not boot fully (could not launch a terminal) after a reboot of template. so I had to change it back.

I ran into difficulties to revert as I could not get a tty.

I did:

qvm-run -u root -p fedora-17-x64 /bin/bash

but as no tty, I was a bit blind...

managed to fix it by doing:

cd /usr/lib/systemd/system/

cp qubes-misc-post.service qubes-misc-post.service.backup

replaced the 3rd line of the file by learning a bit about sed

sed '3 c\

After=qubes-dvm.service' qubes-misc-post.service > qubes-misc-post.service.new

After checking sed did its magic...

cp qubes-misc-post.service.new qubes-misc-post.service

Do you think putting this would fix it?

After=qubes-dvm.service iptables.service

I also thought that maybe you would agree to put the qubes-firewall.service back into systemd for the netvm so that we could use the hook /rw/config/qubes_firewall_user_script

I feel I could help to create some qvm commands to do the exposition of these services or add it to the Qubes VM Manager but as I don't know how it is structured some quick pointers would be good.

the command line could be something like:

qvm-service expose/remove -p tcp --dport 443 --from personalVM --to netVM/devVM

in the case of devVM this would mean in firewallVM:

iptables -t filter -N dev.personal

iptables -t filter -A FORWARD -j dev.personal -i firewallVM.vif_dev -o firewallVM.vif_personal -s dev-ip -d personal-ip

iptables -t filter -A dev.personal -j ACCEPT -p tcp --sport 1024:65535 --dport  80 -m state --state NEW

(I like to do it this way so that when you add logging to the last rule you only log SYNs, similarly if you want to throttle or set time of day limits)

In the case of netVM this would mean routing on top of the above in firewallVM:

iptables -t nat -N eth0.tcp80
iptables -t nat -A PREROUTING -j eth0.tcp80 -i eth0 -p tcp --sport 1024:65535 --dport 80 -d firewallVM.eth0

iptables -t nat -A eth0.tcp80 -j DNAT --to-destination firewallVM.vif_personal

and in netVM:

routing

iptables -t nat -N eth0.tcp80

iptables -t nat -A PREROUTING -j eth0.tcp80 -i eth0 -p tcp --sport 1024:65535 --dport 80 -d netVM.eth0

iptables -t nat -A eth0.tcp80 -j DNAT --to-destination firewallVM.eth0 (not sure how I would pass this to netVM if not launched from dom0)

filtering

iptables -t filter -N out.firewall

iptables -t filter -A FORWARD -j out.firewall -i netVM.eth0 -o netVM.vif_firewall -s netVM.eth0-ip -d firewallVM.eth0-ip

iptables -t filter -A out.firewall -j ACCEPT -p tcp --sport 1024:65535 --dport  80 -m state --state NEW

A little possible bug in this set-up is that if the exposed port is above 1024 (let's say 25565 for minecraft for my kids :)), then the inbound routing rule for the exposed service may prevent outbound traffic from i.e. a personalVM connecting to an external high port (if the masquerade random client port happen to be 25565). I am not sure how iptables behave.

[Marek Marczykowski]

On 12.01.2013 10:22, Alex Dubois wrote:
> ---------- Forwarded message ----------
> From: Alex Dubois <bow...@gmail.com>
> Date: Sat, Jan 12, 2013 at 9:22 AM
> Subject: Re: [qubes-devel] Port forwarding to appvm
> To: Marek Marczykowski <marm...@invisiblethingslab.com>
>
>
> Great thanks for the heads up (I was deep into reading systemd man pages
> that I don't know, I was still on sysV).
>
> I was about to say that it fixed my problem but in fact the netvm would not
> boot fully (could not launch a terminal) after a reboot of template. so I
> had to change it back.

It worked for me...

> I ran into difficulties to revert as I could not get a tty.

Try sudo xl console fedora-17-x64

>
> I did:
> qvm-run -u root -p fedora-17-x64 /bin/bash
>
> but as no tty, I was a bit blind...
> managed to fix it by doing:
> cd /usr/lib/systemd/system/
> cp qubes-misc-post.service qubes-misc-post.service.backup
>
> replaced the 3rd line of the file by learning a bit about sed
> sed '3 c\
> After=qubes-dvm.service' qubes-misc-post.service >
> qubes-misc-post.service.new
>
> After checking sed did its magic...
>
> cp qubes-misc-post.service.new qubes-misc-post.service
>
>
>
> Do you think putting this would fix it?
> After=qubes-dvm.service iptables.service
>
> I also thought that maybe you would agree to put the qubes-firewall.service
> back into systemd for the netvm so that we could use the hook
> /rw/config/qubes_firewall_user_script

qubes-firewall.service applies per-VM rules controlled by VM settings in dom0.
This rules are exposed via xenstore only to firewallvm (or any ProxyVM in
general), so would not work in netvm without exposing the rules also here (see
below).

Anyway you can enable any Qubes service in any VM using qvm-service (you need
to enter exact "qubes-firewall" name). It is also doable in Qubes VM Manager.

> I feel I could help to create some qvm commands to do the exposition of
> these services or add it to the Qubes VM Manager but as I don't know how it
> is structured some quick pointers would be good.
>
> the command line could be something like:
>
> qvm-service expose/remove -p tcp --dport 443 --from personalVM --to
> netVM/devVM

Great!

This would need the following changes:
1. Introduce some property of QubesVm class to storing redirection -> qubes.py
2. Expose the rules to netvm and firewallvm somewhere in xenstore -> qubes.py
3. Apply the rules in VMs (netvm, firewallvm and perhaps dest-vm)

There are still some things to design here, at least:
Ad. 1: Where this property should be: in netvm (as list of port->destvm) or in
dest-vm (simple ports list)?
The first approach can enable user to expose the port only when using one
specific netvm - eg only on standard netvm (connected to wired interface), but
not on usbvm (connected to 3g modem). This way also make easier to detect
conflicts.
The second one is much more intuitive. And perhaps (slightly) easier to
implement - rules are propagated "up", so it is easy to find out which
(proxy/net)VM should have the rules applied - the user can have many NetVMs
and ProxyVMs!.

Ad. 2: The current "qubes_iptables_domainrules" xenstore directory (for
firewallvm) contains only filter table. Some extension/new directory should be
used for the nat table, eg "qubes_iptables_nat". It will also require to
collect/distribute all redirections in netvm and firewallvm (keeping in mind
that someone can have multiple netvms and multiple firewallvms). Assuming the
second approach in previous point, at VM startup (and rules change) propagate
the rules to the "netvm" (QubesVm.netvm is VM to which it is connected, but
this actually can be ProxyVM) and reload the rules there
(write_iptables_xenstore_entry call). Note that we need to know from where the
rule was propagated to make redirection to right IP, so basically it builds
the list from first approach of storing redirections (but here build at
runtime in separate "internal" attribute). Then this should trigger propagate
rules next level up and reload there, recursively up to the final netvm.

Some technical details:
- currently "write_iptables_xenstore_entry" is implemented in ProxyVM, so not
available in NetVM either in AppVM. If you would like to reuse/extend this
code, move this function to NetVM or even base QubesVM (from which ProxyVM
inherits).
- QubesVm attributes are defined in _get_attrs_config function - there is
machinery to deserialize it from qubes.xml ('eval' entry if it isn't string)
and serialize it back ('save' entry - simple 'str()' in most cases). There are
already lists/hashes stored in QubesVM so you can look there for samples (at
least 'services' and 'pcidevs'). Persistent attributes should also be added to
QubesVmCollection (parse_xml_element method). I know, it isn't good design,
but haven't time yet to rewrite this part of code.

Ad. 3: qubes-firewall service can be somehow reused here, for both firewallvm
(where it is started already) and netvm (where should be added to default
services set -> vm-systemd/qubes-sysinit.sh). I think you will figure out how
it works :)
I'm not sure what to do with firewall in dest-vm. Currently INPUT chain is
totally static and basically contains DROP policy. Perhaps it should be left
as task for the user to add some -j ACCEPT there (in /rw/config/rc.local). But
it is up to you if you want automatic rules also in dest-vm or not.

Regarding cmdline tool for it, IMHO it worth new qvm-redirect (or sth) tool.
You can use any exinsting qvm-tool as a template, maybe qvm-service.

>
> in the case of devVM this would mean in firewallVM:
> iptables -t filter -N dev.personal
> iptables -t filter -A FORWARD -j dev.personal -i firewallVM.vif_dev -o
> firewallVM.vif_personal -s dev-ip -d personal-ip
> iptables -t filter -A dev.personal -j ACCEPT -p tcp --sport 1024:65535
> --dport 80 -m state --state NEW

Is --sport really needed here?

> (I like to do it this way so that when you add logging to the last rule you
> only log SYNs, similarly if you want to throttle or set time of day limits)
>
> In the case of netVM this would mean routing on top of the above in
> firewallVM:
> iptables -t nat -N eth0.tcp80
> iptables -t nat -A PREROUTING -j eth0.tcp80 -i eth0 -p tcp --sport
> 1024:65535 --dport 80 -d firewallVM.eth0

Is --sport really needed?

> iptables -t nat -A eth0.tcp80 -j DNAT --to-destination
> firewallVM.vif_personal
>
> and in netVM:
> routing
> iptables -t nat -N eth0.tcp80
> iptables -t nat -A PREROUTING -j eth0.tcp80 -i eth0 -p tcp --sport
> 1024:65535 --dport 80 -d netVM.eth0

> iptables -t nat -A eth0.tcp80 -j DNAT --to-destination firewallVM.eth0 *(not
> sure how I would pass this to netVM if not launched from dom0)*

> filtering
> iptables -t filter -N out.firewall
> iptables -t filter -A FORWARD -j out.firewall -i netVM.eth0 -o
> netVM.vif_firewall -s netVM.eth0-ip -d firewallVM.eth0-ip
> iptables -t filter -A out.firewall -j ACCEPT -p tcp --sport 1024:65535
> --dport 80 -m state --state NEW
>
> A little possible bug in this set-up is that if the exposed port is above
> 1024 (let's say 25565 for minecraft for my kids :)), then the inbound
> routing rule for the exposed service may prevent outbound traffic from i.e.
> a personalVM connecting to an external high port (if the masquerade random
> client port happen to be 25565). I am not sure how iptables behave.

It would not break anything (even without --sport), because masquarade prevent
to using already forwarded/open ports.

[Alex Dubois]

On Sat, Jan 12, 2013 at 4:27 PM, Marek Marczykowski <marm...@invisiblethingslab.com> wrote:

On 12.01.2013 10:22, Alex Dubois wrote:
> ---------- Forwarded message ----------
> From: Alex Dubois <bow...@gmail.com>
> Date: Sat, Jan 12, 2013 at 9:22 AM
> Subject: Re: [qubes-devel] Port forwarding to appvm
> To: Marek Marczykowski <marm...@invisiblethingslab.com>
>
>
> Great thanks for the heads up (I was deep into reading systemd man pages
> that I don't know, I was still on sysV).
>
> I was about to say that it fixed my problem but in fact the netvm would not
> boot fully (could not launch a terminal) after a reboot of template. so I
> had to change it back.

It worked for me...

Maybe I did another typo... I am the expert... too confident in myself. I will give it another go.

> I ran into difficulties to revert as I could not get a tty.

Try sudo xl console fedora-17-x64

OK will do.

 

>
> I did:
> qvm-run -u root -p fedora-17-x64 /bin/bash
>
> but as no tty, I was a bit blind...
> managed to fix it by doing:
> cd /usr/lib/systemd/system/
> cp qubes-misc-post.service qubes-misc-post.service.backup
>
> replaced the 3rd line of the file by learning a bit about sed
> sed '3 c\
> After=qubes-dvm.service' qubes-misc-post.service >
> qubes-misc-post.service.new
>
> After checking sed did its magic...
>
> cp qubes-misc-post.service.new qubes-misc-post.service
>
>
>
> Do you think putting this would fix it?
> After=qubes-dvm.service iptables.service
>
> I also thought that maybe you would agree to put the qubes-firewall.service
> back into systemd for the netvm so that we could use the hook
> /rw/config/qubes_firewall_user_script

qubes-firewall.service applies per-VM rules controlled by VM settings in dom0.
This rules are exposed via xenstore only to firewallvm (or any ProxyVM in
general), so would not work in netvm without exposing the rules also here (see
below).

OK. We will see after agreeing on the design how we go about it. 

Anyway you can enable any Qubes service in any VM using qvm-service (you need
to enter exact "qubes-firewall" name). It is also doable in Qubes VM Manager.

> I feel I could help to create some qvm commands to do the exposition of
> these services or add it to the Qubes VM Manager but as I don't know how it
> is structured some quick pointers would be good.
>
> the command line could be something like:
>
> qvm-service expose/remove -p tcp --dport 443 --from personalVM --to
> netVM/devVM

Great!

This would need the following changes:
1. Introduce some property of QubesVm class to storing redirection -> qubes.py
2. Expose the rules to netvm and firewallvm somewhere in xenstore -> qubes.py
3. Apply the rules in VMs (netvm, firewallvm and perhaps dest-vm)

OK will start to look into it.
 

There are still some things to design here, at least:
Ad. 1: Where this property should be: in netvm (as list of port->destvm) or in
dest-vm (simple ports list)?
The first approach can enable user to expose the port only when using one
specific netvm - eg only on standard netvm (connected to wired interface), but
not on usbvm (connected to 3g modem). This way also make easier to detect
conflicts.
The second one is much more intuitive. And perhaps (slightly) easier to
implement - rules are propagated "up", so it is easy to find out which
(proxy/net)VM should have the rules applied - the user can have many NetVMs
and ProxyVMs!.

I agree second one is more intuitive, and I would expect end user to assume it is built this way.

This made me think we should allow dport natting (off by default). So that if 2 VMs want to expose 80, the user can choose 80 for one and let's say 81 for the other.

I will take Joanna's article as a reference of possible use cases that we should cover:
http://theinvisiblethings.blogspot.co.uk/2011/09/playing-with-qubes-networking-for-fun.html

 

Ad. 2: The current "qubes_iptables_domainrules" xenstore directory (for
firewallvm) contains only filter table. Some extension/new directory should be
used for the nat table, eg "qubes_iptables_nat". It will also require to
collect/distribute all redirections in netvm and firewallvm (keeping in mind
that someone can have multiple netvms and multiple firewallvms). Assuming the
second approach in previous point, at VM startup (and rules change) propagate
the rules to the "netvm" (QubesVm.netvm is VM to which it is connected, but
this actually can be ProxyVM) and reload the rules there
(write_iptables_xenstore_entry call). Note that we need to know from where the
rule was propagated to make redirection to right IP, so basically it builds
the list from first approach of storing redirections (but here build at
runtime in separate "internal" attribute). Then this should trigger propagate
rules next level up and reload there, recursively up to the final netvm.

Yes this looks like what I had in mind too (also with the simple netvm/firewallvm so will take into account more general scenario).

Some technical details:
 - currently "write_iptables_xenstore_entry" is implemented in ProxyVM, so not
available in NetVM either in AppVM. If you would like to reuse/extend this
code, move this function to NetVM or even base QubesVM (from which ProxyVM
inherits).

Seems like moving it to QubesVM makes sense as we want it in netVM and the user may want to expose to both outside and some internalVMs... Not sure all control can be done in FirewallVM.

We will need more that 2 pair of eyes for the Qubes security aspect as I may introduce attack vector.

 

 - QubesVm attributes are defined in _get_attrs_config function - there is
machinery to deserialize it from qubes.xml ('eval' entry if it isn't string)
and serialize it back ('save' entry - simple 'str()' in most cases). There are
already lists/hashes stored in QubesVM so you can look there for samples (at
least 'services' and 'pcidevs'). Persistent attributes should also be added to
QubesVmCollection (parse_xml_element method). I know, it isn't good design,
but haven't time yet to rewrite this part of code.

Ad. 3: qubes-firewall service can be somehow reused here, for both firewallvm
(where it is started already) and netvm (where should be added to default
services set -> vm-systemd/qubes-sysinit.sh). I think you will figure out how
it works :)
I'm not sure what to do with firewall in dest-vm. Currently INPUT chain is
totally static and basically contains DROP policy. Perhaps it should be left
as task for the user to add some -j ACCEPT there (in /rw/config/rc.local). But
it is up to you if you want automatic rules also in dest-vm or not.

I thought that if we were to move the firewall xenstore policy definition is QubesVM, we would be able to control the dest-vm too. Did I miss understood?

Regarding cmdline tool for it, IMHO it worth new qvm-redirect (or sth) tool.
You can use any exinsting qvm-tool as a template, maybe qvm-service.

>
> in the case of devVM this would mean in firewallVM:
> iptables -t filter -N dev.personal
> iptables -t filter -A FORWARD -j dev.personal -i firewallVM.vif_dev -o
> firewallVM.vif_personal -s dev-ip -d personal-ip
> iptables -t filter -A dev.personal -j ACCEPT -p tcp --sport 1024:65535
> --dport  80 -m state --state NEW

Is --sport really needed here?

I had the habit to have it when the state module was not there... I'm not sure. I'll remove unless told it's required.

 

> (I like to do it this way so that when you add logging to the last rule you
> only log SYNs, similarly if you want to throttle or set time of day limits)
>
> In the case of netVM this would mean routing on top of the above in
> firewallVM:
> iptables -t nat -N eth0.tcp80
> iptables -t nat -A PREROUTING -j eth0.tcp80 -i eth0 -p tcp --sport
> 1024:65535 --dport 80 -d firewallVM.eth0

Is --sport really needed?

Based on your explanation on masquerade can be removed.

 

> iptables -t nat -A eth0.tcp80 -j DNAT --to-destination
> firewallVM.vif_personal
>
> and in netVM:
> routing
> iptables -t nat -N eth0.tcp80
> iptables -t nat -A PREROUTING -j eth0.tcp80 -i eth0 -p tcp --sport
> 1024:65535 --dport 80 -d netVM.eth0

> iptables -t nat -A eth0.tcp80 -j DNAT --to-destination firewallVM.eth0 *(not
> sure how I would pass this to netVM if not launched from dom0)*

> filtering
> iptables -t filter -N out.firewall
> iptables -t filter -A FORWARD -j out.firewall -i netVM.eth0 -o
> netVM.vif_firewall -s netVM.eth0-ip -d firewallVM.eth0-ip
> iptables -t filter -A out.firewall -j ACCEPT -p tcp --sport 1024:65535
> --dport  80 -m state --state NEW
>
> A little possible bug in this set-up is that if the exposed port is above
> 1024 (let's say 25565 for minecraft for my kids :)), then the inbound
> routing rule for the exposed service may prevent outbound traffic from i.e.
> a personalVM connecting to an external high port (if the masquerade random
> client port happen to be 25565). I am not sure how iptables behave.

It would not break anything (even without --sport), because masquarade prevent
to using already forwarded/open ports.

Well time for me to get to it then...

I will not start any coding. I am going to take the time to look then come back we what I feel and probably 1 or 2 quetions :)

Thanks for all these pointers.

Alex
 

[Alex Dubois]

On Sat, Jan 12, 2013 at 8:15 PM, Alex Dubois <bow...@gmail.com> wrote:

On Sat, Jan 12, 2013 at 4:27 PM, Marek Marczykowski <marm...@invisiblethingslab.com> wrote:

On 12.01.2013 10:22, Alex Dubois wrote:
> ---------- Forwarded message ----------
> From: Alex Dubois <bow...@gmail.com>
> Date: Sat, Jan 12, 2013 at 9:22 AM
> Subject: Re: [qubes-devel] Port forwarding to appvm
> To: Marek Marczykowski <marm...@invisiblethingslab.com>
>
>
> Great thanks for the heads up (I was deep into reading systemd man pages
> that I don't know, I was still on sysV).
>
> I was about to say that it fixed my problem but in fact the netvm would not
> boot fully (could not launch a terminal) after a reboot of template. so I
> had to change it back.

It worked for me... 

Maybe I did another typo... I am the expert... too confident in myself. I will give it another go.

No... still no luck. And could not identify root cause.

In templateVM I edited:
 /etc/systemd/system/multi-user.target.wants/qubes-misc-post.service

and added network.target as a dependency

After reboot of templateVM

/rw was empty

After reboot of netvm

/rw/config contained only NM-system-connections

Also the following message was on the console:

[25.21xxx] systemd-readahead-collect[213]: open(/etc/NetworkManager/system-connections) failed: Too many levels of symbolic links

After reboot of firewallVM

/rw was empty

Few log files in /var/log/qubes modified recently...

qubes_restore.log (I don't recall requesting a restore :))

pacat.xx.log

guid.xx.log (cannot lookup 0x2e00006 in wid2windowdata probably links to the terminal not appearing)

qrexec.xx.log

It seems that Qubes philosophy is to have VMs in a tree like shape... leaves are the end user VM, he may have set a branch as a tor or vpn, the trunk is the firewallVM and branches to the different NICs. There could be multiple trees but they would be independent.

Finally if a user plays with some server VMs, even if he has pres/app/data, these would either be "leaves" or the pres would be a leave, and we would not manage deeper if he decide to have his appVM having the presVM as netVM.

I've started to draft a response on the requirements definition, but will send it in a future post...

[Marek Marczykowski]

On 14.01.2013 00:34, Alex Dubois wrote:
> On Sat, Jan 12, 2013 at 8:15 PM, Alex Dubois <bow...@gmail.com> wrote:
>> On Sat, Jan 12, 2013 at 4:27 PM, Marek Marczykowski <
>> marm...@invisiblethingslab.com> wrote:
>>
>>> On 12.01.2013 10:22, Alex Dubois wrote:
>>>> ---------- Forwarded message ----------
>>>> From: Alex Dubois <bow...@gmail.com>
>>>> Date: Sat, Jan 12, 2013 at 9:22 AM
>>>> Subject: Re: [qubes-devel] Port forwarding to appvm
>>>> To: Marek Marczykowski <marm...@invisiblethingslab.com>
>>>>
>>>>
>>>> Great thanks for the heads up (I was deep into reading systemd man pages
>>>> that I don't know, I was still on sysV).
>>>>
>>>> I was about to say that it fixed my problem but in fact the netvm would
>>> not
>>>> boot fully (could not launch a terminal) after a reboot of template. so
>>> I
>>>> had to change it back.
>>>
>>> It worked for me...
>>>
>>
>> Maybe I did another typo... I am the expert... too confident in myself. I
>> will give it another go.
>>
>
> No... still no luck. And could not identify root cause.

Check the systemctl output.

> In templateVM I edited:
> /etc/systemd/system/multi-user.target.wants/qubes-misc-post.service
> and added network.target as a dependency

It should be symlink to /lib/systemd/system/qubes-misc-post.service. Perhaps
you've broken this symlink during edit...

> After reboot of templateVM
> /rw was empty
> After reboot of netvm
> /rw/config contained only NM-system-connections
> Also the following message was on the console:
> [25.21xxx] systemd-readahead-collect[213]:
> open(/etc/NetworkManager/system-connections) failed: Too many levels of
> symbolic links
> After reboot of firewallVM
> /rw was empty

/rw is mounted by qubes-misc-post.service, so it looks like it isn't started
at all...
Anyway /rw/config/rc.local should be called as late as possible, but other
things in qubes-misc-post.service (/usr/lib/qubes/init/misc-post.sh) can be
safely called earlier. So I will split out rc.local call to separate service,
called at the end of boot process.

>
> Few log files in /var/log/qubes modified recently...
> qubes_restore.log (I don't recall requesting a restore :))
> pacat.xx.log
> guid.xx.log (cannot lookup 0x2e00006 in wid2windowdata probably links to
> the terminal not appearing)
> qrexec.xx.log

Nothing related to this problem. Perhaps something in
/var/log/xen/console/guest-netvm.log.
You have a list of VM-related logs in qubes manager.

Some constraints are enforced by qubes code:
1. Only NetVM (or its subtype like ProxyVM) can be set as "network provider"
(netvm property). Especially AppVM cannot be such, so can be only a leaf in
this tree.
2. Any VM can be connected to at most one NetVM.

Of course. Both design and implementation.

This is correct. I've realized that even currently INPUT chain in firewallvm
is populated through the xenstore.
I though more about protection against user errors - to require at least two
steps to drill the hole through all firewalls, so one typo would not expose
unwanted VM service to the outside world.

Joanna, what do you think?

>> I will not start any coding. I am going to take the time to look then come
>> back we what I feel and probably 1 or 2 quetions :)
>>
>> Thanks for all these pointers.

BTW
Perhaps you would have a look at:
http://wiki.qubes-os.org/trac/wiki/DevelopmentWorkflow

[Alex Dubois]

OK it is clear.

Looking at the use cases for chained netVM, it seems that its main use is for outbound tunnelling (tor, vpn), or filtering at higher level than 4 (http proxy). users may want the capability to do the same for inbound traffic, so we may need to have an option to allow a reverse filtering proxy (no prerouting rule/change of dest IP, traffic will be pushed to this proxyVM, INPUT filtering rule instead, and start again prerouting+filtering for the leafs)

Sidenote: thinking about this reverse proxy, it could be used for SSL termination + client cert authentication (or other) but to protect the keys (in case you loose the server). it could be even in 2 reverse proxy if you are not sure you can trust the authentication code and/or loosing the keys would be bad (thinking yubikey).

The user may also wish to specify how deep he wants to drill. For example he has a web server, netvm a reverseproxy, netvm firewall + a radius server, netvm reverseproxy. In this case he may wish to only expose radius to reverse proxy.

I agree with the 2 steps. It could either be step1, inbound routing all the way, step2 filtering or step1 filtering+routing in firewallVM+downstream, step2 appVM itself.

Based on what I tried to explain with the reverse proxy+radius use case, it seems that the first method might be better. here are 3 use cases:

step1 inbound routing
1 qvm_expose_service add radiusVM -p udp --dport radiusPort1,radiusPort2 --depth 1 (optional default being all the way to NIC)

2 qvm_expose_service add webserverVM -p tcp --dport 80 --proxyVMatDepth 1 --proxyExposedPort 443 --netVMInterface eth0,eth1 (if depth is specified, this would apply to the VM at this depth)

step2 filtering

1 qvm_allow_service add -p udp --dport radisuPort1,radiusPort2 --from reverseproxyVM --to webserverVM

2a qvm_allow_service add -p tcp --dport 80 --from reverseproxyVM --to webserverVM

2b qvm_allow_service add -p tcp --dport 443 --to reverseproxyVM --sinterface eth0 -s 192.168.0.0/24

 

A simpler example to expose a minecraft server would be:

step1 inbound routing
1 qvm_expose_service add minecraftVM -p tcp --dport 25565 (all non vif netVMinterfaces would have the port presented)

2 qvm_allow_service add -p tcp --dport 25565 --to minecraftVM (maybe we enforce that you have to specify the interface and/or -s any (by default we expose it to the local network only)?

sidenote1: not sure of the security value but I usually hard code the MAC address of my default gateway

sidenote2: it would be good to have an option to only allow internet traffic to some VMs or to be able to select to have it as default (i.e. dispVM can't reach my LAN) and having to explicitly allow VMs access to the LAN. This is that if one of my VMs get owned, this VMs can't attack my kids PC.

[Marek Marczykowski]

On 14.01.2013 09:21, Alex Dubois wrote:
> On Mon, Jan 14, 2013 at 12:10 AM, Marek Marczykowski <
> marm...@invisiblethingslab.com> wrote:
>
>> On 14.01.2013 00:34, Alex Dubois wrote:
>>> On Sat, Jan 12, 2013 at 8:15 PM, Alex Dubois <bow...@gmail.com> wrote:
>>>> On Sat, Jan 12, 2013 at 4:27 PM, Marek Marczykowski <
>>>> marm...@invisiblethingslab.com> wrote:
>>>>
>>>>> On 12.01.2013 10:22, Alex Dubois wrote:
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Alex Dubois <bow...@gmail.com>
>>>>>> Date: Sat, Jan 12, 2013 at 9:22 AM
>>>>>> Subject: Re: [qubes-devel] Port forwarding to appvm
>>>>>> To: Marek Marczykowski <marm...@invisiblethingslab.com>
>>>>>>
>>>>>>

>>>>> There are still some things to design here, at least:
>>>>> Ad. 1: Where this property should be: in netvm (as list of
>> port->destvm)
>>>>> or in
>>>>> dest-vm (simple ports list)?
>>>>> The first approach can enable user to expose the port only when using
>> one
>>>>> specific netvm - eg only on standard netvm (connected to wired
>>>>> interface), but
>>>>> not on usbvm (connected to 3g modem). This way also make easier to
>> detect
>>>>> conflicts.
>>>>> The second one is much more intuitive. And perhaps (slightly) easier to
>>>>> implement - rules are propagated "up", so it is easy to find out which
>>>>> (proxy/net)VM should have the rules applied - the user can have many
>>>>> NetVMs
>>>>> and ProxyVMs!.
>>>>
>>>>
>>>> I agree second one is more intuitive, and I would expect end user to
>>>> assume it is built this way.
>>>> This made me think we should allow dport natting (off by default). So
>> that
>>>> if 2 VMs want to expose 80, the user can choose 80 for one and let's
>> say 81
>>>> for the other.

I forgot to write it before: maybe in this case it's better to setup the
service on port 81 in dest-vm instead of translating the port in the middle?
This would be much easier to understand and in case of some protocols also
much fault-tolerant.

Imagine the case of HTTP: if some webapp is building absolute URL based on
server settings (server name and possible non-standard port) it will produce
wrong URL if it don't know the real port.

Remember that Qubes is desktop system, not server one... This mechanism
shouldn't be too complicated, same as any other security-critical Qubes code.

Above commands doesn't looks simple for normal user... I think if the user
needs such complicated scenario (which is unlikely on desktop system), he can
do this by writing firewall rules by hand (just as you've done it before).

>
> A simpler example to expose a minecraft server would be:
> step1 inbound routing
> 1 qvm_expose_service add minecraftVM -p tcp --dport 25565 (all non vif
> netVMinterfaces would have the port presented)
>
> 2 qvm_allow_service add -p tcp --dport 25565 --to minecraftVM (maybe we
> enforce that you have to specify the interface and/or -s any (by default we
> expose it to the local network only)?

Source control can be a good addition.

Sidenode: all qubes tools have "-" in names, not "_".

> sidenote1: not sure of the security value but I usually hard code the MAC
> address of my default gateway

If someone can spoof your gateway IP, I see problem with spoofing MAC as well...

> sidenote2: it would be good to have an option to only allow internet
> traffic to some VMs or to be able to select to have it as default (i.e.
> dispVM can't reach my LAN) and having to explicitly allow VMs access to the
> LAN. This is that if one of my VMs get owned, this VMs can't attack my kids
> PC.

This can be done using already exinsting Qubes firewall mechanism. DispVM
inherites its firewall settings from VM starting it.

[Alex Dubois]

Alex

Yes I agree

I think this "toolbox" should not be available by default as I agree with you it is first a desktop and this code will increase the attack surface where most users will not want/need it. A bit like AEM.
So maybe you could advise on how to do it without touching qubes.py etc...

But I think it is simple for simple scenarios (see my last example in previous post)

Yes agree, so maybe just documentation in trac is sufficient.

>
>>
>> A simpler example to expose a minecraft server would be:
>> step1 inbound routing
>> 1 qvm_expose_service add minecraftVM -p tcp --dport 25565 (all non vif
>> netVMinterfaces would have the port presented)
>>
>> 2 qvm_allow_service add -p tcp --dport 25565 --to minecraftVM (maybe we
>> enforce that you have to specify the interface and/or -s any (by default we
>> expose it to the local network only)?
>
> Source control can be a good addition.
>
> Sidenode: all qubes tools have "-" in names, not "_".

OK

>
>> sidenote1: not sure of the security value but I usually hard code the MAC
>> address of my default gateway
>
> If someone can spoof your gateway IP, I see problem with spoofing MAC as well...

The MAC is hard coded. I think documentation is more suitable again.

>
>> sidenote2: it would be good to have an option to only allow internet
>> traffic to some VMs or to be able to select to have it as default (i.e.
>> dispVM can't reach my LAN) and having to explicitly allow VMs access to the
>> LAN. This is that if one of my VMs get owned, this VMs can't attack my kids
>> PC.
>
> This can be done using already exinsting Qubes firewall mechanism. DispVM
> inherites its firewall settings from VM starting it.

Ah OK good. You mean from templateVM. Again I'll look into adding that as best practices documentation.
But don't you think there is benefit in having it set for the entire qubes system, and only allow explicitly LAN access?

[Marek Marczykowski]

I don't know any method of extending python classes from separate file. Do you
know?
This code needs to hook on some events (like firewallvm start/stop) so can't
be done as totally separate code. Qubes currently doesn't have flexible
plugins infrastructure which would make it possible...

>
> But I think it is simple for simple scenarios (see my last example in previous post)
>

>>> sidenote2: it would be good to have an option to only allow internet
>>> traffic to some VMs or to be able to select to have it as default (i.e.
>>> dispVM can't reach my LAN) and having to explicitly allow VMs access to the
>>> LAN. This is that if one of my VMs get owned, this VMs can't attack my kids
>>> PC.
>>
>> This can be done using already exinsting Qubes firewall mechanism. DispVM
>> inherites its firewall settings from VM starting it.
>
> Ah OK good. You mean from templateVM.

No, starting VM. If you click "Open in DispVM" on some file in "work" VM, this
DispVM will have firewall rules of "work" VM. In case of DispVM stared from
dom0 menu it is dom0. Setting firewall rules for dom0 is somehow tricky (you
need to create dir pointed by qvm-prefs), but should work.

> Again I'll look into adding that as best practices documentation.


> But don't you think there is benefit in having it set for the entire qubes system, and only allow explicitly LAN access?

Perhaps this can be achived easily by separate firewallvms with simple rules
in qubes_firewall_user_script. Something like "lan-firewallvm" and
"internet-firewallvm". This will be easy to choose which VM should have LAN
access, without enumerating VM IP in any script. But maybe a little overkill...

Anyway if you want cut of access to LAN for almost all VMs, you can setup
firewall VM setting using qvm-firewall tool with simple bash one-liner to not
retype the same rules many times. Of course you can also add appropriate rules
with qubes_firewall_user_script.

Not sure if it worth additional code in qubes core scripts.

[Joanna Rutkowska]

On 01/14/13 09:21, Alex Dubois wrote:
> Based on what I tried to explain with the reverse proxy+radius use case, it
> seems that the first method might be better. here are 3 use cases:
>
> step1 inbound routing
> 1 qvm_expose_service add radiusVM -p udp --dport radiusPort1,radiusPort2
> --depth 1 (optional default being all the way to NIC)
> 2 qvm_expose_service add webserverVM -p tcp --dport 80 --proxyVMatDepth 1
> --proxyExposedPort 443 --netVMInterface eth0,eth1 (if depth is specified,
> this would apply to the VM at this depth)
>
> step2 filtering
> 1 qvm_allow_service add -p udp --dport radisuPort1,radiusPort2 --from
> reverseproxyVM --to webserverVM
> 2a qvm_allow_service add -p tcp --dport 80 --from reverseproxyVM --to
> webserverVM
> 2b qvm_allow_service add -p tcp --dport 443 --to reverseproxyVM
> --sinterface eth0 -s 192.168.0.0/24
>
> A simpler example to expose a minecraft server would be:
> step1 inbound routing
> 1 qvm_expose_service add minecraftVM -p tcp --dport 25565 (all non vif
> netVMinterfaces would have the port presented)
>
> 2 qvm_allow_service add -p tcp --dport 25565 --to minecraftVM (maybe we
> enforce that you have to specify the interface and/or -s any (by default we
> expose it to the local network only)?

Ok, I got a bit lost while reading this whole thread now -- Alex, can
you explain what would be the difference of exposing vs. allowing the
service?

Also, as I understand you want to cover the two cases:
1) Allowing inbound connections to select AppVMs from the outside world
2) Allowing inter-VM traffic between select AppVMs.

Is that correct?

joanna.

[Alex Dubois]

Alex

Apologies if not very clear. Marek expressed the fact that we may want 2 actions from a user before a service is reachable and I tend to agree (also after through the process of designing these 2 actions maybe a confirmation check box would be OK)
So...
Expose service would code all the pre-routing firewall rules (in the nat table) in the different VMs so that the packets would be routed to the target VM if there was no filtering
Allow service would automate the coding of the firewall rules in the filter table to allow traffic through for the given port.

>
> Also, as I understand you want to cover the two cases:
> 1) Allowing inbound connections to select AppVMs from the outside world
> 2) Allowing inter-VM traffic between select AppVMs.
>
> Is that correct?

Yes, also I have so far many looked in the first use case. The second one should be similar with the following diff:
Inbound routing: Depth will be identified to common NetVMs (of appVM1 and appVM2) - 1
Well, maybe not if 2 tor client vm want to talk to one another, should we go down to firewall vm, I think so...

>
> joanna.
>

[Joanna Rutkowska]

Do we also want to patch /bin/rm so that it always required double
confirmation? How about also patching the "Send" button in the
Thunderbird...? And I also heard that in some .gov agencies they haqve
keyboards without Enter key -- instead it's places 1 meter away next to
a second person who's job is to double check what the first wrote and
press enter only then. No kidding!

In other words, I'm kind of against such superficial (and confusing)
differentiation -- there really are *many* things that can go wrong when
using a computer, and IMHO opening inter VM networking access doesn't
qualify nowhere at the top of the list.

>>
>> Also, as I understand you want to cover the two cases:
>> 1) Allowing inbound connections to select AppVMs from the outside world
>> 2) Allowing inter-VM traffic between select AppVMs.
>>
>> Is that correct?
>
> Yes, also I have so far many looked in the first use case. The second one should be similar with the following diff:
> Inbound routing: Depth will be identified to common NetVMs (of appVM1 and appVM2) - 1
> Well, maybe not if 2 tor client vm want to talk to one another, should we go down to firewall vm, I think so...
>

Why would two tor clients want to talk to each other?

j.

[Alex Dubois]

I'll bind the 2 together. Under the hood it will probably work as expressed before.

 

>>
>> Also, as I understand you want to cover the two cases:
>> 1) Allowing inbound connections to select AppVMs from the outside world
>> 2) Allowing inter-VM traffic between select AppVMs.
>>
>> Is that correct?
>
> Yes, also I have so far many looked in the first use case. The second one should be similar with the following diff:
> Inbound routing: Depth will be identified to common NetVMs (of appVM1 and appVM2) - 1
> Well, maybe not if 2 tor client vm want to talk to one another, should we go down to firewall vm, I think so...
>

Why would two tor clients want to talk to each other? 

Tor was a bad example. Let's not try to extrapolate but rather simplify for now which is in fact your point.

I'll make inter "leaf" communication going at edgeVM -1 (as the edge is a netVM to have NIC driver, and the one above is a firewall VM).

So...

To expose a service in a "leafVM" such as personal the command would be:

qvm-service-expose add personalVM -p tcp --dport 80

which would code for this:

personalVM ==\\

                        \\== firewallVM == netVM =>

otherVM   -------/

optional parameters:

-s sourceIPs,sourceIPs2/Mask

And to expose a service in a "leafVM" to anotherVM it would be:

qvm-service-expose add personalVM -p tcp --dport 80 --toVM clientVM

which would code for this...

personalVM ==\\

                       | -- firewallVM -- netVM

clientVM   <==//

j.

[Joanna Rutkowska]

On 01/14/13 23:17, Alex Dubois wrote:
> Tor was a bad example. Let's not try to extrapolate but rather simplify for
> now which is in fact your point.
>
> I'll make inter "leaf" communication going at edgeVM -1 (as the edge is a
> netVM to have NIC driver, and the one above is a firewall VM).
>
> So...
>
> To expose a service in a "leafVM" such as personal the command would be:
> qvm-service-expose add personalVM -p tcp --dport 80
> which would code for this:
> personalVM ==\\
> \\== firewallVM == netVM =>
> otherVM -------/
> optional parameters:
> -s sourceIPs,sourceIPs2/Mask
>

Why is the arrow in the above diagram pointing outwards, and not
inwards? My understanding is that you're opening a port forwarding *to*
to the personalVM from the outside world in this example, no? If so, the
arrow should be pointing from the outside world to the netvm (and then
to friewallVM and perosnal VM)?

>
> And to expose a service in a "leafVM" to anotherVM it would be:
> qvm-service-expose add personalVM -p tcp --dport 80 --toVM clientVM
> which would code for this...
> personalVM ==\\
> | -- firewallVM -- netVM
> clientVM <==//

And here you're exposing a service in... personalVM or clientVM? The
diagram suggest that in the clientVM, but the command example suggests
otherwise...? Or perhaps I'm completely misunderstanding your intentions
here ;)

joanna.

[Alex Dubois]

On Tue, Jan 15, 2013 at 9:28 AM, Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:

On 01/14/13 23:17, Alex Dubois wrote:
> Tor was a bad example. Let's not try to extrapolate but rather simplify for
> now which is in fact your point.
>
> I'll make inter "leaf" communication going at edgeVM -1 (as the edge is a
> netVM to have NIC driver, and the one above is a firewall VM).
>
> So...
>
> To expose a service in a "leafVM" such as personal the command would be:
> qvm-service-expose add personalVM -p tcp --dport 80
> which would code for this:
> personalVM ==\\
>                         \\== firewallVM == netVM =>
> otherVM   -------/
> optional parameters:
> -s sourceIPs,sourceIPs2/Mask
>

Why is the arrow in the above diagram pointing outwards, and not
inwards? My understanding is that you're opening a port forwarding *to*
to the personalVM from the outside world in this example, no? If so, the
arrow should be pointing from the outside world to the netvm (and then
to friewallVM and perosnal VM)?

You are right. I represented it this way as the tool is qvm-service expose and as the tool will work by propagating from top left to right... but yes the end arrow will be the listener sucking packets to the personalVM

>
> And to expose a service in a "leafVM" to anotherVM it would be:
> qvm-service-expose add personalVM -p tcp --dport 80 --toVM clientVM
> which would code for this...
> personalVM ==\\
>                        | -- firewallVM -- netVM
> clientVM   <==//

And here you're exposing a service in... personalVM or clientVM? The
diagram suggest that in the clientVM, but the command example suggests
otherwise...? Or perhaps I'm completely misunderstanding your intentions
here ;)

You made me smile :) You are right and same explanation. tool propagation follow arrow but first syn packet is in opposite direction.

joanna.

[Joanna Rutkowska]

On 01/15/13 22:17, Alex Dubois wrote:
> On Tue, Jan 15, 2013 at 9:28 AM, Joanna Rutkowska <
> joa...@invisiblethingslab.com> wrote:
>
>> > On 01/14/13 23:17, Alex Dubois wrote:
>>> > > Tor was a bad example. Let's not try to extrapolate but rather simplify
>> > for
>>> > > now which is in fact your point.
>>> > >
>>> > > I'll make inter "leaf" communication going at edgeVM -1 (as the edge is a
>>> > > netVM to have NIC driver, and the one above is a firewall VM).
>>> > >
>>> > > So...
>>> > >
>>> > > To expose a service in a "leafVM" such as personal the command would be:
>>> > > qvm-service-expose add personalVM -p tcp --dport 80
>>> > > which would code for this:
>>> > > personalVM ==\\
>>> > > \\== firewallVM == netVM =>
>>> > > otherVM -------/
>>> > > optional parameters:
>>> > > -s sourceIPs,sourceIPs2/Mask
>>> > >
>> >
>> > Why is the arrow in the above diagram pointing outwards, and not
>> > inwards? My understanding is that you're opening a port forwarding *to*
>> > to the personalVM from the outside world in this example, no? If so, the
>> > arrow should be pointing from the outside world to the netvm (and then
>> > to friewallVM and perosnal VM)?
>> >

>> > You are right. I represented it this way as the tool is qvm-service *
> expose* and as the tool will work by propagating from top left to right...

> but yes the end arrow will be the listener sucking packets to the personalVM
>
>> >
>>> > > And to expose a service in a "leafVM" to anotherVM it would be:
>>> > > qvm-service-expose add personalVM -p tcp --dport 80 --toVM clientVM
>>> > > which would code for this...
>>> > > personalVM ==\\
>>> > > | -- firewallVM -- netVM
>>> > > clientVM <==//
>> >
>> > And here you're exposing a service in... personalVM or clientVM? The
>> > diagram suggest that in the clientVM, but the command example suggests
>> > otherwise...? Or perhaps I'm completely misunderstanding your intentions
>> > here ;)
>> >
> You made me smile :) You are right and same explanation. tool propagation
> follow arrow but first syn packet is in opposite direction.
>
>

It might make more sense, from the user's perspective, to just integrate
this into qvm-firewall tool, I think. Also, we should make sure to
clearly indicate which rules are for outgoing and which for incoming
traffic, in the qvm-firewall output (currently it just outputs a bunch
of rules, and it is assumed that it is obvious that these are rules for
outgoing traffic).

j.

[Alex Dubois]

Hi Marek,

I had to reinstall my system recently. I had to fix systemd config to ensure /rw/config/rc.local custom iptables rules were not overwritten by qubes ones.

/usr/lib/systemd/sys/qubes-misc-post.service
changed from
...
After=qubes-dvm.service
...
to
...
After=qubes-dvm.service iptables.service
...

I suspect I have the issue due to my particular set-up... however I believe this is a little bug which may affect other users.

I don't think it would impact anybody if you were to apply it.

Let me know,
Regards,
Alex

And I want to restate that I appreciate the great work you all do with this project :-)