how to setup proxy in firewall vm

[Igor Bukanov]

I would like to setup a real HTTP proxy in firewall VM for one my app
vm. Is there any script hook in firewallvm that I can use to
start/stop such proxy whenever the app VM starts? Also is it possible
in firewall VM to know which network interface a particular VM would
be bound to?

[Joanna Rutkowska]

Theoretically you could use the /rw/config/qubes_firewall_user_script:

http://git.qubes-os.org/?p=joanna/core.git;a=blob;f=network/qubes_firewall;h=30670b82d41659d071b2acfcd41d12faa2d1daab;hb=HEAD#l52

... that is executed upon every f/w rules reload (which would also
happen whenever a new AppVM starts, assuming it uses this very firewallVM).

The list of domains is exported under qubes_iptables_domainrules
xenstore key in the firewallvm -- e.g. try this:

[root@firewallvm config]# xenstore-list qubes_iptables_domainrules
4
47
5
3
43
45

But unfortunately (for your task) this list only contains Xen temporary
IDs of the AppVMs, and not their names, which is, of course intentional,
as we try no to leak the names of the AppVMs in the system to any other
domain.

On the other hand you should be able to extract the AppVM's IP address
from each rule:

[root@firewallvm config]# xenstore-read qubes_iptables_domainrules/4
*filter\n# 'personal' VM:\n-A FORWARD -s 10.137.2.16 -p udp -d
10.137.1.1 --dport 53 -j ACCEPT\n-A FORWARD -s 10.137.2.16 -p udp -d
10.137.1.254 --dport 53 -j ACCEPT\n-A FORWARD -s 10.137.2.16 -p icmp -j
ACCEPT\n-A FORWARD -s 10.137.2.16 -p tcp -d 10.137.255.254 --dport 8082
-j DROP\n-A FORWARD -s 10.137.2.16 -j ACCEPT\nCOMMIT\n

(Arghh, and just after I pasted this text above, I noticed that we
actually do include the target AppVM's name as a... comment in the
rules... Ok, we will fix it soon (#698) :))

But extracting an IP address by parsing those rules is really a *bad*
idea IMO. I think a much better way could be to create a qrexec service
instead which would be called from the target's AppVM's rc.local...

Perhaps Marek could come up with some better idea, but he won't be
online before the NY.

joanna.

[Igor Bukanov]

On 27 December 2012 22:46, Joanna Rutkowska

<joa...@invisiblethingslab.com> wrote:

> this list only contains Xen temporary
> IDs of the AppVMs, and not their names, which is, of course intentional,
> as we try no to leak the names of the AppVMs in the system to any other
> domain.

How "temporary" is defined here? Are they are reset on each appvm
restart? Or on xen restart?

[Joanna Rutkowska]

Both. These like PIDs in an OS.

j.

[Tokyo Jeff]

The MAC address of a VM is assigned by the configuration (Xen, qvm-prefs).  MAC address could be used as the unique VM identifier for firewall rules across restarts.

TJ

[Joanna Rutkowska]

Yes, and so is the IP address also. Perhaps you could configure your
proxy to detect incoming TCP packets from a fixed IP/MAC.

j.

[Marek Marczykowski]

Fixed.

> But extracting an IP address by parsing those rules is really a *bad*
> idea IMO. I think a much better way could be to create a qrexec service
> instead which would be called from the target's AppVM's rc.local...
>
> Perhaps Marek could come up with some better idea, but he won't be
> online before the NY.

You can check routing table:
[user@firewallvm ~]$ ip route show 10.137.1.19
10.137.1.19 dev vif11.0 scope link

AppVM IP is static (in contrary to interface name).

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Igor Bukanov]

On 5 January 2013 03:46, Marek Marczykowski

<marm...@invisiblethingslab.com> wrote:
> You can check routing table:
> [user@firewallvm ~]$ ip route show 10.137.1.19
> 10.137.1.19 dev vif11.0 scope link
>
> AppVM IP is static (in contrary to interface name).

Thanks, this is what I was looking for. With the static address I can
detect when the VM is up/down in
/rw/config/qubes_firewall_user_script. But now I need to figure out
how to relax the firewall rules to allow the traffic to the proxy from
AppVM.

I tried in firewallVM to allow the traffic to the local socket:

iptables -t filter -A INPUT -s 10.137.2.13/32 -d 10.137.2.1/32 -p tcp
--dport 8085 -j ACCEPT

where 10.137.2.13 is the address of AppVM and 10.137.2.1 is the
firewallvm address.

Then I run in firewallvm socat as a listen server for testing to
output to stdout any input that it receives from the socket:

socat -d -d TCP4-LISTEN:8085,bind=10.137.2.1 -

Yet in AppVM running socat to connect to the server gives:

# socat -d -d - TCP4:10.137.2.1:8085
2013/01/10 10:54:36 socat[930] N reading from and writing to stdio
2013/01/10 10:54:36 socat[930] N opening connection to AF=2 10.137.2.1:8085
2013/01/10 10:54:36 socat[930] E connect(3, AF=2 10.137.2.1:8085, 16):
No route to host
2013/01/10 10:54:36 socat[930] N exit(1)

That is, I got no route to host. I suppose I need to alter iptables in
different ways. Any hints?

[Igor Bukanov]

I figured out the iptables rules. I have to use -I INPUT, not -A
INPUT, in the iptables rule to ensure that the rule will be inserted
before the rule that drops all input packets "-A INPUT -j REJECT
--reject-with icmp-host-prohibited". So after

iptables -t filter -I INPUT -s 10.137.2.13/32 -d 10.137.2.1/32 -p tcp
--dport 8085 -j ACCEPT

the proxy can receive input packets.

The only remaining annoyance is that tinyproxy cannot be configured to
listen on a particular interface so I can not run its 2 instances for
different AppVMs listening on the same port while providing an extra
isolation between VM's, but I can live with different port numbers and
assuming the the firewall rules provide sufficient isolation.

[Igor Bukanov]

On 27 December 2012 22:46, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:

> Theoretically you could use the /rw/config/qubes_firewall_user_script:
>
> http://git.qubes-os.org/?p=joanna/core.git;a=blob;f=network/qubes_firewall;h=30670b82d41659d071b2acfcd41d12faa2d1daab;hb=HEAD#l52
>
> ... that is executed upon every f/w rules reload (which would also
> happen whenever a new AppVM starts, assuming it uses this very firewallVM).

The script is not executed when the VM shuts down, right? I.e. after
the shutdown without custom /rw/config/qubes_firewall_user_script I
still see the old firewall rules with no longer existing ip address.
Is it intentional?

[Marek Marczykowski]

On 10.01.2013 12:47, Igor Bukanov wrote:
> On 27 December 2012 22:46, Joanna Rutkowska
> <joa...@invisiblethingslab.com> wrote:
>> Theoretically you could use the /rw/config/qubes_firewall_user_script:
>>
>> http://git.qubes-os.org/?p=joanna/core.git;a=blob;f=network/qubes_firewall;h=30670b82d41659d071b2acfcd41d12faa2d1daab;hb=HEAD#l52
>>
>> ... that is executed upon every f/w rules reload (which would also
>> happen whenever a new AppVM starts, assuming it uses this very firewallVM).
>
> The script is not executed when the VM shuts down, right? I.e. after
> the shutdown without custom /rw/config/qubes_firewall_user_script I
> still see the old firewall rules with no longer existing ip address.
> Is it intentional?

Not "intentional" in the exact meaning of the word, but it is the fact.
Just haven't found any clean way for VM shutdown hooks in dom0. Most likely
this will end up some day with daemon for monitoring it (with xenstore
watches), but for now it isn't that needed to justify new process in dom0. Old
firewall rules should be harmless (event performance impact isn't noticeable).

If you want notification in firewallvm, you can try some udev rules for vif
device. Place actual rules files in /rw/config and copy/symlink them to /etc
in /rw/config/rc.local at VM startup.

[Igor Bukanov]

On 10 January 2013 12:59, Marek Marczykowski

<marm...@invisiblethingslab.com> wrote:
> If you want notification in firewallvm, you can try some udev rules for vif
> device. Place actual rules files in /rw/config and copy/symlink them to /etc
> in /rw/config/rc.local at VM startup.

I wanted the shutdown hook to shutdown the proxy. But I do not really
need it as at worst it implies that few unnecessary tinyproxy
processes will be running.

The reason I want a real http proxy as the firewall rules based on ip
address just does not work especially if white-listing google or
facebook servers.

For references I attach a script that run the proxy instances and one
of the filter file that white list all the necessary servers for my
social networking VM. I run it from
/rw/config/qubes_firewall_user_script like

/rw/config/proxy/run_proxy update

Here the update command starts/shutdowns all the necessary proxy
instances based on ip route show results. Each instance serves as a
proxy for one vm with rules defined in /rw/config/proxy/filter_<name>
where the name is defined in the script based on the ip address. The
file should follow the filter rules for tinyproxy. To restart the
proxy after updating the file run /rw/config/proxy/run_proxy reload
<name>.

With the proxy I can disable all the networking in Qubus GUI for the
application VM, the only possible connection will be the connection to
the proxy port.

[Marek Marczykowski]

Great work!

Maybe tinyproxy isn't the best choice for this task, as it can be configured
with only one global filter rules, not per-source IP. There are plenty of http
proxy software. Quick yum search reveals that 3proxy can be handy here;
extract from 3proxy.cfg manual:
---
auth <authtype> [...]
Type of user authorization. Currently supported:
none - no authentication or authorization required.
Note: is auth is none any ip based limitation, redirection, etc will
not work.
This is default authentication type
iponly - authentication by access control list with username ignored.
(...)

allow <userlist> <sourcelist> <targetlist> <targetportlist>
<operationlist> <weekdayslist> <timeperiodslist>
deny <userlist> <sourcelist> <targetlist> <targetportlist>
<operationlist> <weekdayslist> <timeperiodslist>
Access control entries. All lists are comma-separated, no spaces
are allowed. Usernames are case sensitive (if used with authtype nbname
username must be in
uppercase). Source and target lists may contain IP addresses (W.X.Y.Z)
or CIDRs (W.X.Y.Z/L). Since 0.6, targetlist may also contain host names,
instead of
addresses. It's possible to use wildmask in the begginning and in the
the end of hostname, e.g. *badsite.com or *badcontent*. Hostname is only
checked if host‐
name presents in request. Targetportlist may contain ports (X) or port
ranges lists (X-Y). For any field * sign means "ANY"
---
It doesn't contain full URL-based filtering (which is the reason why we use
tinyproxy), but in your case it can be no problem.


You may also have a look on this thread:
http://groups.google.com/group/qubes-devel/browse_thread/thread/9e231b0e14bf9d62/643df8c0a03bc546

[Igor Bukanov]

Running separated proxy processes per VM has an advantage that it is
much more difficult to affect other VM if one VM is compromised. Plus
tinyproxy is sufficiently simple and has been around for quite some
time to gain trust in the code.

On 11 January 2013 00:45, Marek Marczykowski

[Abel Luck]

Igor Bukanov:

> Running separated proxy processes per VM has an advantage that it is
> much more difficult to affect other VM if one VM is compromised. Plus
> tinyproxy is sufficiently simple and has been around for quite some
> time to gain trust in the code.
>

This thread looks mighty interesting. I think it would be useful in some
necessary feature additions to TorVM.

Any chance the content of this thread could be organized into a wiki
page so others could cleanly emulate it?

Cheers,

~abel

[Marek Marczykowski]

On 11.01.2013 13:35, Abel Luck wrote:
> Igor Bukanov:
>> Running separated proxy processes per VM has an advantage that it is
>> much more difficult to affect other VM if one VM is compromised. Plus
>> tinyproxy is sufficiently simple and has been around for quite some
>> time to gain trust in the code.
>>
>
> This thread looks mighty interesting. I think it would be useful in some
> necessary feature additions to TorVM.
>
> Any chance the content of this thread could be organized into a wiki
> page so others could cleanly emulate it?

This would be very nice! You can do it yourself :)

[Abel Luck]

Marek Marczykowski:

> On 11.01.2013 13:35, Abel Luck wrote:
>> Igor Bukanov:
>>> Running separated proxy processes per VM has an advantage that it is
>>> much more difficult to affect other VM if one VM is compromised. Plus
>>> tinyproxy is sufficiently simple and has been around for quite some
>>> time to gain trust in the code.
>>>
>>
>> This thread looks mighty interesting. I think it would be useful in some
>> necessary feature additions to TorVM.
>>
>> Any chance the content of this thread could be organized into a wiki
>> page so others could cleanly emulate it?
>
> This would be very nice! You can do it yourself :)
>

I was hoping Igor could, since he has the working proxy setup!

Igor, if you could write maybe just a little howto on how to do your
proxy setup, I could clean it up and expand it into a good wiki article.

What do you think?

[Igor Bukanov]

On 11 January 2013 20:54, Abel Luck <ab...@guardianproject.info> wrote:
> Marek Marczykowski:

> Igor, if you could write maybe just a little howto on how to do your
> proxy setup, I could clean it up and expand it into a good wiki article.

I tried to create a text during the following couple of days.

[Joanna Rutkowska]

On 01/11/13 09:41, Igor Bukanov wrote:
> Running separated proxy processes per VM has an advantage that it is
> much more difficult to affect other VM if one VM is compromised.

Why do you think so? Does the tinyproxy uses some kind of additional
sandboxing? Because if not, then it's trivial to compromise other
processes when one got compromised. I'm sure you saw this:

http://git.qubes-os.org/?p=marmarek/core.git;a=blob;f=misc/qubes.sudoers;h=8087a90a8c0a6f81d4fe45905a6aa9462de2a7ce;hb=HEAD

?


I surely see the usefulness of your scripts, but on the other hand we
should think how to make sure that a compromise of one of the proxy
processes cannot affect the whole firewallvm's integrity (which would
open up avenues to attack other AppVMs). Admittedly it's much easier to
compromise a firewallvm if it exposes the http proxy, then if it exposes
only the TCP/IP stack, right?

joanna.

[Igor Bukanov]

On 14 January 2013 11:55, Joanna Rutkowska

<joa...@invisiblethingslab.com> wrote:
> On 01/11/13 09:41, Igor Bukanov wrote:
>> Running separated proxy processes per VM has an advantage that it is
>> much more difficult to affect other VM if one VM is compromised.
>
> Why do you think so? Does the tinyproxy uses some kind of additional
> sandboxing? Because if not, then it's trivial to compromise other
> processes when one got compromised.

I meant compared with having one global proxy separated proxy
instances provide better isolation against denial of service. Now, as
regarding an explicit attack from compromised AppVM I suppose the
amount of testing that went into the IP stack in Linux is vastly
greater than that of tinyproxy code so the kernel should be more
trustworthy than the proxy even if its source is significantly
smaller.

However consider the attack surface from a compromised netvm against
AppVM. Given the complexity of modern wireless protocols a chance of
having a bug there is just bigger than a bug in tinyproxy. If the
netvm is compromised, then it can inject arbitrary DNS and TCP packets
into a browser running in AppVM. AFAIK browsers are not well-tested
against such attacks and their net handling code is very complex. Now
consider a case of having a proxy in the firewall VM. Even if netvm
manages to compromise the firewall VM through a bug in the proxy or
TCP stack, it would have harder time to infect the browser. The attack
surface through single TCP connection from the proxy to the browser is
just smaller than through arbitrary TCP and DNS packets.

Bottom line: IMO HTTP proxy increases the chance of infecting firewall
VM via compromised AppVM. But it decreases a chance of compromising a
browser running in AppVM from a compromised net or firewall VM. For
this reason currently I run my social, email and banking VM against
firewall VM with the proxy with the strict white-listing rules based
on hosts, the untrusted VMs directly against netvm (I see no point of
having the intermediate firewall VM for AppVMs where I do not use any
firewall rules) and really precious AppVMs connects through ssh proxy
using Qubes RPC without direct network stack exposure.

[Joanna Rutkowska]

On 01/14/13 22:45, Igor Bukanov wrote:
> On 14 January 2013 11:55, Joanna Rutkowska
> <joa...@invisiblethingslab.com> wrote:
>> On 01/11/13 09:41, Igor Bukanov wrote:
>>> Running separated proxy processes per VM has an advantage that it is
>>> much more difficult to affect other VM if one VM is compromised.
>>
>> Why do you think so? Does the tinyproxy uses some kind of additional
>> sandboxing? Because if not, then it's trivial to compromise other
>> processes when one got compromised.
>
> I meant compared with having one global proxy separated proxy
> instances provide better isolation against denial of service. Now, as
> regarding an explicit attack from compromised AppVM I suppose the
> amount of testing that went into the IP stack in Linux is vastly
> greater than that of tinyproxy code so the kernel should be more
> trustworthy than the proxy even if its source is significantly
> smaller.
>
> However consider the attack surface from a compromised netvm against
> AppVM. Given the complexity of modern wireless protocols a chance of
> having a bug there is just bigger than a bug in tinyproxy. If the
> netvm is compromised, then it can inject arbitrary DNS and TCP packets
> into a browser running in AppVM. AFAIK browsers are not well-tested
> against such attacks and their net handling code is very complex.

Sure, but the attacks that can be conducted from a compromised netvm are
equal to those that can be carried by an attacker controlling my WiFi
(so, an attacker sharing a language with me at the airport, or a nearby
room in a hotel) -- those can only be eliminated (and should be) by use
of encrypted protocols, although even those are not a 100% panacea [1].

[1] See slide #54 in this presentation:
http://www.invisiblethingslab.com/resources/2011/SSTIC%202011.pdf

> Now consider a case of having a proxy in the firewall VM. Even if
> netvm manages to compromise the firewall VM through a bug in the
> proxy or TCP stack, it would have harder time to infect the browser.
> The attack surface through single TCP connection from the proxy to
> the browser is just smaller than through arbitrary TCP and DNS
> packets.
>

I'm not quite follow this reasoning? Why do you think the compromised
firewallVM cannot use the same arsenal of attacks as a compromised
netVM? It surely can, it can send whatever eth/IP/TCP/UDP/whatever
packets it wants to any of the connected AppVMs... Just like a netvm.


joanna.

[Joanna Rutkowska]

But then, on the other hand, perhaps having http-level filtering
outweighs the potential problems related to firewallvm compromise? After
all the main goal of having a firewallvm is to prevent stupid user
mistakes in the first place (e.g. clicking on a link embedded in an
email in a domain which should have no random web access)... Only in the
second place we can think of a firewallvm as some (often very
ineffective!) leak prevention mechanism.

Each appvm has its own filtering enabled anyway, which leaves the
attacker (who owns a corresponding firewallvm) with an arsenal of the
same attacks she could fire from the local network (think: airport
lounge, or hotel lobby), which we must always assume and deal with (via
crypto protocols), or alternatively to try some attacks against an
AppVM's eth/TCP/IP stack (specifically some code executed on incoming
packets before they got filtered out), which we normally assume to be
hard (TM). Especially, if an attacker already has a good exploit against
Linux TCP/IP stack (or Xen's eth0 stack), then she could own the netvm
and then firewallvm, and then any Linux-based AppVM *anyway*! (Here, a
nice solution would be to have either netvm or firewallvm based on e.g.
*BSD, instead of Linux, just to force the attacker to have exploits
against two different stacks -- something *very* improbable IMHO).

So, I'm leaning towards providing an option to enable this http-level
filtering. What others think?

joanna.

[Igor Bukanov]

---------- Forwarded message ----------
From: Igor Bukanov <ig...@mir2.org>
Date: 15 January 2013 21:32
Subject: Re: [qubes-devel] how to setup proxy in firewall vm
To: Joanna Rutkowska <joa...@invisiblethingslab.com>


On 15 January 2013 11:23, Joanna Rutkowska

<joa...@invisiblethingslab.com> wrote:
> Sure, but the attacks that can be conducted from a compromised netvm are
> equal to those that can be carried by an attacker controlling my WiFi
> (so, an attacker sharing a language with me at the airport, or a nearby
> room in a hotel) -- those can only be eliminated (and should be) by use
> of encrypted protocols, although even those are not a 100% panacea [1].

I missed that reasoning. Barring a bug in TCP/IP Linux stack a
compromised netvm can only influence open TCP connections or DNS UDP
packets to attack a browser in AppVM, but this is exactly what any
external router can do.

> I'm not quite follow this reasoning? Why do you think the compromised
> firewallVM cannot use the same arsenal of attacks as a compromised
> netVM?

HTTP proxy allows to move the DNS resolver outside the AppVM. This is
a reduction of the attack surface especially if one consider the
complexity of DNSSEC. Another reduction is that HTTP pipe-lining and
aggressive caching can also be implemented outside AppVM in the proxy
reducing exposure of a complex code in the browser without affecting
performance. So if the only connection to the outside world from AppVM
is through HTTP proxy, this minimizes the attack surface against AppVM
compared when the browser connects to the net without the proxy.

Now, after thinking about the default networking setup in Qubes I
realized why shared IP-based firewall makes perfect sense. The default
setup assumes that one can fully trust Linux TCP-IP stack. Under this
assumption the firewall VM can be trusted to implement the *network
isolation* between AppVMs as the only thing that it runs is the
trusted TCP/IP stack plus minimal ethernet drivers. Running a proxy in
the firewall VM violates the trust assumption. Such VM is strictly
more complex and should be less trustworthy than the default firewall
VM when implementing the network isolation.

To properly run the proxy VM one should make a proxy that can connect
between VMs without TCP/IP stack while still implementing connection
filtering. If such setup is sufficiently small, than one can trust it
more than Linux TCP/IP stack. So I have to think more about a proper
HTTP proxy setup.

[Igor Bukanov]

On 15 January 2013 21:17, Joanna Rutkowska

<joa...@invisiblethingslab.com> wrote:
> (Here, a
> nice solution would be to have either netvm or firewallvm based on e.g.
> *BSD, instead of Linux, just to force the attacker to have exploits
> against two different stacks -- something *very* improbable IMHO).

This is not true. Consider that an attacker compromised an AppVM. With
2 TCP-IP stacks that she can reach from the AppVM she has more chances
to find a bug in any of the stacks than if only one stack is
reachable. A bug in any of the stacks allows him to attack any AppVM.

[Joanna Rutkowska]

I can see your point. But I would expect that an attack against a
eth/TCP/IP stack would require direct access (hop-wise) to the victim
VM. In other words I would expect it rather to be very unlikely that one
could run an exploit against the networking stack over one additional
"innocent" hop (i.e. the firewallvm being in the middle between the
AppVM and NetVM). I think so, because I would expect that an attack
against a TCP/IP stack would involve sending some kind of incorrect or
unusual packets, that would likely get discarded or validated (e.g.
reassembled) by the hop in the between. And, of course, attacks on eth
level should be ruled out in such a setup too.

joanna.

[Alex Dubois]

My view is that netVM protect against level 2 and lower
FirewallVM against 3 and 4

If you want to protect higher layers, put the proxy in a VM in front of firewallVM.

AppVM --> AppProxyVM (Http or other) --> firewallVM --> netVM

You should split the appProxyVM per protocol (ie DNS, http, smtp) etc...

If you want to mitigate against issues related to each functional layer (ie attack surface for wireless vs wired) you need to duplicate instances of the functionalVM

Alex

[Igor Bukanov]

Here is HOWTO draft about how to run a HTTP filtering proxy in Qubes
firewall VM.

If it looks OK, where should I put it on the Wiki pages?

Introduction

By default Qubes uses a special firewall VM that sits between the
networking VM and each AppVM. This VM controls the traffic for AppVMs
and can be used to restrict what AppVMs can send or receive. The
traffic rules can be setup using filtering rules GUI in Qubes VM
manager. The manager translates user-defined setup into iptables rules
for the kernel of firewall VM.

The primary goal of the filtering rule setup in firewall VM is to
allow for the user to protect either from own mistakes (like accessing
an arbitrary website from a browser running in a banking VM) or from
mistakes of websites (like a banking website that loads JS code from a
social network operator when the user logs in into the bank).

As the rules in the firewall are IP-based, it has drawbacks. First the
rules cannot be used if one has to use a HTTP proxy to connect for
websites (a common setup on corporate networks). Second the Qubes
resolves DNS names from the firewall rules when the AppVM loads. This
prevents websites that use DNS-based load balancers from working
unless the user reloads the firewall rules (which re-resolve the DNS
names) whenever the balancer transfer her session to another IP. Third
the initial setup of the rules is complicated as the firewall drops
the connection silently. As a workaround on can use browser's network
console to see what is blocked, but this is time-consuming and one can
trivially miss some important cases like including in the firewall
white list sites for OCSP SSL certificate verification.

These drawbacks can be mitigated if one replaces iptable-based rules
with a filtering HTTP proxy. The following describes how to setup
tinyproxy-based proxy in the firewall VM to archive such filtering.

Warning

Running a HTTP proxy in your firewall VM increases the attack surface
against that VM from a compromised AppVM. tinyproxy has a relatively
simple code and a reasonable track record to allow to certain level of
trust. But one cannot exclude bugs especially in the case of a hostile
proxy clients as this is less tested scenario. So it is not advisable
to use the proxy in a shared firewall VM against untrusted AppVM to
black-list some unwanted connection like advertisement sites.

Less problematic setup is to white-list possible connections for
several trusted and semi-trusted AppVMs within one firewall VM. Still
for maximum safety one should consider running a separated firewall VM
per each important AppVMs and run the proxy there.

As a counterweight to this warning it is important to point out that
HTTP proxy decreases attack surface against AppVM. For example, with a
proxy the AppVM does not need DNS connections so a bug in the kernel
or in the browser in that area would not affect the AppVM. Also
browsers typically avoid many latest and greatest HTTP features when
connection through proxies minimizing exposure of new and unproven
networking code.


Setup

1. Copy the attached archive with the proxy control script, default
tinyproxy config and a sample firewall filtering file into the
firewall VM and unpack it in /rw/config folder there as root:

cd /rw/config
sudo tar xzf .../proxy.tar.xz

2. If necessary adjust /rw/config/tinyproxy/config according to the
man page for tinyproxy.conf. The included config file refuses the
connection unless the host is white-listed in the filtering file, so
this can be altered if one wants rather to black-list connection. One
may also specify upstream proxies there. The file is a template file
and the control script will replace {name} constructs in the file with
actual parameters. In general lines with {} should be preserved as is.

3. For each AppVM that one wants to run through the proxy create an
the corresponding filtering file in the /rw/config/tinyproxy
directory. With the default config the filtering file should contain
regular expressions to match white-listed hosts with one regular
expression per line, see the man page for tinyproxy.conf for details.
The file should be named:

name.ip-address-of-app-vm

The name part before the dot can be arbitrary. For convenience one can
use AppVm name here, but this is not required. It is important to get
ip address part right as this is what the control script uses to
determine for which AppVM to apply the proxy rules. One can check the
IP address of AppVM in Qubes VM manager in the VM settings dialog, see
the Networking session under the Basic tab.

The attached archive includes tinyproxy/social.10.137.2.13 file with a
rules for a AppVM allowing connection to google, facebook, linkedin,
livejournal, youtube and few other other sites. One can use it as an
example after changing the the IP address accordingly.

When editing the rules remember to include $ at the end of the host
name and to prefix each dot in the host name with the backslash. This
way the pattern matches the whole host and not just some prefix and
the dot is not interpreted as an instruction to match an arbitrary
character according to regular expression syntax.

4. Check that proxyctl.py script can properly recognize the rule
files. For that run:

sudo /rw/config/tinyproxy/proxyctl.py show

For each rule file it should print the name, ip address, network
interface of the running AppVM if AppVM runs and the id of the
tinyproxy process that proxies that AppVM. The first time each pid
should be --.

5. Now run some AppVM with proxy and then run:

sudo /rw/config/tinyproxy/proxyctl.py update

The update command starts proxy processes and adjusts the iptable
rules to allow for proxy traffic for each running AppVM from the
filtering files list. For each stopped AppVM the proxy is killed.

Check that proxy is started so the pid field of the show command is a number:

sudo /rw/config/tinyproxy/proxyctl.py show

6. Run the browser in the started AppVM and configure it to use the
proxy on the port 8100 running at the IP address of the firewall VM
gateway interface. In Qubes VM manager the address is given after the
Gateway label in the Setting dialog for the firewall VM.

In Firefox go to the Preferences dialog, select Advanced->Network,
click Settings for the Connection section. In the Connection Settings
dialog select Manual proxy configuration. For HTTP Proxy field use the
IP address of the firewall gateway interface. Enter 8100 as the port
and the select the checkbox "Use this proxy server for all protocols".

Go to some site. The browser should either load it if it was
white-listed in the filtering file or show a page generated by
tinyproxy that the page was filtered out.

In the firewall VM see /run/tinyproxy/name/log file. For each filtered
out website it contains an entry and one can adjust the filtering file
to include the corresponding host. After changing the file run either:

sudo /rw/config/tinyproxy/proxyctl.py restart name

to restart proxy with the updated rules file only for the given VM or

sudo /rw/config/tinyproxy/proxyctl.py kill-all-and-restart

to restart all proxy processes.

7. To make sure that the proxy is started automatically when the AppVM
starts change /rw/config/qubes_firewall_user_script to include the
following line:

/rw/config/tinyproxy/proxyctl.py update

If the file does not exist, create it so it looks:

#!/usr/bin/bash

/rw/config/tinyproxy/proxyctl.py update

Make sure that the script is owned by root and executable:

sudo chown root:root /rw/config/qubes_firewall_user_script
sudo chmod 755 /rw/config/qubes_firewall_user_script

8. In Qubes VM manager adjust Firewall rules for each AppVM with a
proxy. In a typical case when only HTTP proxy should be used for
outside connections, simply select Deny network access except..., make
sure that the address list is empty and then unselect Allow ICMP, DNS
and Update proxy checkboxes.

There is no need to add any special entries for the proxy in the GUI
as proxyctl.py adds rules for the proxy traffic itself.

[Alex Dubois]

I strongly believe keeping the attack surface of firewallVM to a minimum is important (even iptables' state module is a risk).

Could we go down the root of having http proxy as an AppVM and use the FirewallSetup page to set-up access to the service?

Joanna, what do you think for the arch, there are 3 options:
- service hosted in proxyVM
- service hosted in AppVM/ServiceVM with service routed via firewallVM
- service hosted in firewallVM
The current implementation is option3. To isolate httpProxy vulnerabilities away from firewallVM the 2 others are an option. Option1 seem to be the standard way. If latter you want to isolate http vulnerabilities from dns-caching vulnerabilities (with a DNS service), I have a pref for option2.


There is one slight problem with option1 or 2: the disk space required.
Here again there are 3 potential options:
- install httpProxy service in base template and launch it, configure, store content in /rw/config (no disk space problem in this case)
- install httpProxy in cloned template, launch, configure, store content in cloned template and run httpProxy in httpProxyVM based on Cloned template. This has the advantage that you can either store cache content in /var/xxx (after each reboot of the proxy, any cache pollution get flushed) or in /rw/xxx
- this option is I believe not possible at the moment and I don't think it is feasible... Same as previous but based on a templateVM which has base templateVM as a template.

Further comments below.

Alex

I have a preference for the service exposed in an ServiceVM routed via firewallVM as explained above. You can always have different httpProxy and firewallVM per security zones

>
> As a counterweight to this warning it is important to point out that
> HTTP proxy decreases attack surface against AppVM. For example, with a
> proxy the AppVM does not need DNS connections so a bug in the kernel
> or in the browser in that area would not affect the AppVM. Also
> browsers typically avoid many latest and greatest HTTP features when
> connection through proxies minimizing exposure of new and unproven
> networking code.

>
>
> Setup
>
> 1. Copy the attached archive with the proxy control script, default
> tinyproxy config and a sample firewall filtering file into the
> firewall VM and unpack it in /rw/config folder there as root:
>
> cd /rw/config
> sudo tar xzf .../proxy.tar.xz

I prefer to use a cloned template and trust RedHat package signatures and updates (If proxy.tar.gz contain the binaries)
I am writing this from my smartphone, so looking forward to use all the work you have done on tinyproxy management.

3128 is quite a standard port for proxy

> --
> You received this message because you are subscribed to the Google Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> <proxy.tar.gz>

[Igor Bukanov]

On 3 March 2013 10:25, Alex Dubois <bow...@gmail.com> wrote:

> I prefer to use a cloned template and trust RedHat package signatures and updates (If proxy.tar.gz contain the binaries)

The archive contains just a Python script to control tinyproxy
processes (tinyproxy is already a part of it), a default config file
for the proxy and a sample filter file.

[Joanna Rutkowska]

Hi Igor, sorry I somehow missed your post and the other in this threat.

What you sent looks cool. I suggest you create a dedicated page for it
and link to it from the section "Advanced Topics" on UserDoc page.

Regarding the attached code -- for any code, even small and simple one,
we always prefer to keep it in a git repo. I think that at this stage,
i.e. when this is still not fully integrated with Qubes (e.g. no ability
to set rules via manager) this should be kept in a separate repo, not in
Qubes core, similarly as is the case with e.g. qubes-tor repo. So it
would be great if you could setup a git repo somewhere, e.g. at github
and link to it from the wiki page.

One more cosmetic comment -- when pasting commands it's good to prefix
them with the original terminal prompt that shows the VM name where the
command is issued, so e.g.:

[user@qubes ~]$ uname -a
Linux qubes 3.7.6-2.pvops.qubes.x86_64 #1 SMP Mon Feb 25 17:42:04 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux

... instead of just

uname -a

joanna.

[Joanna Rutkowska]

On 03/03/13 10:25, Alex Dubois wrote:
> I strongly believe keeping the attack surface of firewallVM to a minimum is important (even iptables' state module is a risk).
>
> Could we go down the root of having http proxy as an AppVM and use the FirewallSetup page to set-up access to the service?
>
> Joanna, what do you think for the arch, there are 3 options:
> - service hosted in proxyVM
> - service hosted in AppVM/ServiceVM with service routed via firewallVM
> - service hosted in firewallVM
> The current implementation is option3. To isolate httpProxy vulnerabilities away from firewallVM the 2 others are an option. Option1 seem to be the standard way. If latter you want to isolate http vulnerabilities from dns-caching vulnerabilities (with a DNS service), I have a pref for option2.
>

The current implementation is just a set if instructions for manual
installation, and so it should be rather easy for users to decide which
specific option from those listed above they really want to implement,
right?

We can think about further integrating Igor's approach with Qubes, in a
way similar to how we now support Tor VM. This would require some
packaging and making a service from it, again similarly like with Tor
VM. It should then be easy for the user to just enable that service in a
specific VM: be that firewallvm, some other proxy VM, or some dedicated
AppVM, and things should generally happen more or less automatically,
except for the need to set up Qubes firewall rules perhaps, and, in the
case of using a dedicated AppVM, setting up custom forwarding rules, as
described in the wiki.

>
> There is one slight problem with option1 or 2: the disk space required.

What do you mean?

joanna.

[Alex Dubois]

OK makes perfect sense like this.
 

>
> There is one slight problem with option1 or 2: the disk space required.

What do you mean?

 

To make good use of a service and "respect" the way the distribution did install it, you have to clone a templateVM. I explain...

If you don't, using the template is great. However for some complex services, the way it is configured (or missed configured) affect security. for example, to that end some distributions have put some effort in setting up file access permissions on conf and data.
A good example is squid.
If you were to use squid from an appVM built on top of a templateVM, any config or data would have to be hosted in a partition which does not get wiped on shutdown (i.e. /rw).
And you may make more mistake than the distro packager in term of file access.

Using a cloned templateVM is better. You configure squid in the clone template (respecting the distro... i.e. settings in /etc). After a restart of the appVM based on it, you are sure your config goes back to last template config.
For the data, you have the choice to leave it where your distro put it (i.e. /var)... I do that for my static apache code (After a restart, even the content is back to last template config, I deploy content to the template, not the app). but you can choose to host all or part of it in /rw.

This is ideal for a service... the only problem is you need a cloned template per service... therefore my comment on disk space.

If we could have 2 level of template that would be ideal, but I suspect you looked into this and there are issues with this.

Or maybe I am not making best use of all Qubes features and there is a better way to leverage on templating.

Alex

> - install httpProxy service in base template and launch it, configure, store content in  /rw/config (no disk space problem in this case)
> - install httpProxy in cloned template, launch, configure, store content in cloned template and run httpProxy in httpProxyVM based on Cloned template. This has the advantage that you can either store cache content in /var/xxx (after each reboot of the proxy, any cache pollution get flushed) or in /rw/xxx
> - this option is I believe not possible at the moment and I don't think it is feasible... Same as previous but based on a templateVM which has base templateVM as a template.
 

joanna.

[Alex Dubois]

Alex

In fact weI would need to explore having a template slot under the cloned template and exposing a ro file system on top of which the cloned templates can add modifications but are all based on. But I instinctively feel their will be issues when patching the base template... Maybe the concept of snapshots used in other virtualisation technologies would be more appropriate: having another type of template than cloned template (full copy), but derived template (snapshot of template, never modified again on which each derived template add modification)...

[Joanna Rutkowska]

Ok, I can see your point now. But why don't just use qvm-backup as a
snapshoting tool for your service vms?

joanna.

[Outback Dingo]

actually, my ideas of a solid proxyvm would be cacheing with Squid, or OOPS with and forward proxying with Varnish, which is also an http accelerator, when i created my torvm, i tool the original arch template, and created an arch-net template, then created the torvm from that, i could exactly do the same for cacheing, to expand on it, im also researching netvm based on FreeBSD or Open/NetBSD with pf or npf and openvswitch in a virtual switch environment.Ideas on the table as further research continues, basically my goal...... 

netvm  - virtual switch

firewallvm - pf based bsd

torvm - security/privacy

dmzvm -> for storagevm

 

cache/proxyvm - web cache/forwarding proxy

hostvms 

all on a laptop :) however it does have 16gb and 1TB disk space