Question to Mirage OS firewall users
jkitt
rtia...@gmail.com
> What's it like to update - is it relatively simple? Would you say it's more secure than Debian or Fedora?
It's easy. Shut down your Mirage OS Firewall VMs, copy over the new kernel files to the relevant directory in /var/lib/qubes/vm-kernels in dom0, and then restart the Mirage firewalls.
However, I don't know if it's more secure than using a Debian or Fedora based sys-firewall; it *might* help guard against a 0 day cascade though.
That said, because the Mirage firewall doesn't seem to work with a dispVM (at least for me, even running the latest code off of github), I still have sys-firewall running in the background anyways. So what I do is run my mirage-firewall behind sys-firewall (which in turn is behind sys-net). I don't know if that's best practice or even has any effect in guarding against a 0-day cascade, but things still work normally for the machines where I don't do any custom vm iptables filter rules and the ram hit isn't too much (I use 32MB).
Note that if you're trying to compile the latest mirage firewall code from github (which isn't reflected on the Release pages yet; there have been some minor changes since the last one), it might be a bit tricky since if you follow the default github instructions, the compilation will eventually fail as mirage-nat tries to pull in older versions of its package dependencies by default.
What I had to do was follow the github instructions until it failed, run 'opam
upgrade' to update what mirage-nat pulled in, then manually install the latest version of the tcpip package by running 'opam install tcpip' and then finally run 'opam install mirage-nat.' After that, following the rest of the github instructions should be fine. That'll work with both the 4.02.3 OCAML compiler, and the 4.03.0+flambda compiler. Compiling mirage-firewall won't work yet with the 4.04 series compilers because the version of mirage-xen in the repository only works with up to version 4.03. The code on mirage-xen's github page has been updated to work with 4.04 a while back, but a release roll up hasn't been pushed out to the repositories yet; not sure when that'll happen.
Chris Laprise
> On Saturday, December 10, 2016 at 6:03:17 AM UTC-7, jkitt wrote:
>> What's it like to update - is it relatively simple? Would you say it's more secure than Debian or Fedora?
> It's easy. Shut down your Mirage OS Firewall VMs, copy over the new kernel files to the relevant directory in /var/lib/qubes/vm-kernels in dom0, and then restart the Mirage firewalls.
>
> However, I don't know if it's more secure than using a Debian or Fedora based sys-firewall; it *might* help guard against a 0 day cascade though.
attack surface. But in this particular role--firewalls--the risk is
fairly low. Qubes configures sys-firewall as 'green' I think for this
reason.
Where this kind of minimization may be more valuable is in running
high-risk VMs like sys-net and sys-usb, but that is understandably more
complex.
Chris
Антон Чехов
Is anyone using the mirage firewall in connection with a proxyVM? How do you configure it properly? Does it handle qubes-firewall-users-scripts?
Reg Tiangha
> Hi!
>
> Is anyone using the mirage firewall in connection with a proxyVM? How do you configure it properly? Does it handle qubes-firewall-users-scripts?
>
firewallVM and they chain together fine. Mirage Firewall in its current
iteration does *not* respect modifications to firewall rules via Qubes
and has to be inputted manually (there are some instructions on how to
do that on the software author's blog). It isn't to say that Mirage
Firewall couldn't do it one day, but I believe the author of the code is
leaving it up as an exercise for the reader. Maybe he'll get around to
implementing it, or maybe not, but from a purely technical standpoint,
there's no reason why it couldn't be modified to work with Qubes
firewall user scripts, it's just that it hasn't been implemented yet.
Note that even if you're running the latest code off of GitHub,
currently, Mirage Firewall still doesn't work correctly with DispVMs (or
at least, I haven't been able to get it to work; the DispVM connects to
it, but there's no traffic), even though there were some minimal fixes
applied to try to handle how it handles IP addresses from a different
pool. Works fine with AppVMs, though, as well as TemplateVMs, at least
in my experience.
WillyPillow
Антон Чехов
Thank you for sharing your experiences and the advice. I will try to wrap my head around this topic.
I have been trying the firewall with an AppVM already and it looked like it was working fine but I have to dig deeper into the process (for my understanding).
Reg Tiangha
it act like a dumb proxy, but if you were looking to do some more
advanced things like blocking or allowing certain ports or domains, it
doesn't interact with the Qubes firewall tools so setting those rules
there would have no effect on the mirage firewall.
If you want to learn more about how mirage firewall works, read up on
the author's blog post about it:
http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/
Thomas Leonard
[user@fedora-23-dvm ~]$ sudo ifconfig eth0 down && sudo ifconfig eth0 up
[user@fedora-23-dvm ~]$ sudo route add $(qubesdb-read /qubes-gateway) dev eth0
[user@fedora-23-dvm ~]$ sudo route add default gw $(qubesdb-read /qubes-gateway)
[user@fedora-23-dvm ~]$ curl http://www.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/?gfe_rd=cr&ei=vKSMWOn7F6vP8Aeg4KeoAQ">here</A>.
</BODY></HTML>
The odd thing is that, as far as I can see, reinitialising the interface is something that only affects Linux (no interaction with the firewall).
(and I'm not sure why my DispVM is Fedora 23 when my default template is Fedora 24, but anyway...)
Thomas Leonard
> On Saturday, December 10, 2016 at 6:03:17 AM UTC-7, jkitt wrote:
> > What's it like to update - is it relatively simple? Would you say it's more secure than Debian or Fedora?
>
> It's easy. Shut down your Mirage OS Firewall VMs, copy over the new kernel files to the relevant directory in /var/lib/qubes/vm-kernels in dom0, and then restart the Mirage firewalls.
>
> Note that if you're trying to compile the latest mirage firewall code from github (which isn't reflected on the Release pages yet; there have been some minor changes since the last one), it might be a bit tricky since if you follow the default github instructions, the compilation will eventually fail as mirage-nat tries to pull in older versions of its package dependencies by default.
It seems to work for me. To make things more predictable, I've added a script to build it with Docker:
sudo yum install docker
sudo systemctl start docker
git clone https://github.com/talex5/qubes-mirage-firewall.git
cd qubes-mirage-firewall
sudo ./build-with-docker.sh
The Dockerfile uses a fixed version of opam-repository, so it shouldn't break even if something gets updated. It also prints out the sha256sum of the binary it built and the expected hash (hard-coded in the file), e.g.
$ sudo ./build-with-docker.sh
[...]
SHA2 of build: f0c1a06fc4b02b494c81972dc89419af6cffa73b75839c0e8ee3798d77bf69b3 mir-qubes-firewall.xen
SHA2 last known: f0c1a06fc4b02b494c81972dc89419af6cffa73b75839c0e8ee3798d77bf69b3
I'd be interested to know if other people get the same hash (of course, the hash will change if you e.g. modify the rules.ml file to change the policy).
Foppe de Haan
Thomas Leonard
> Anyone else tried to use MirageOS i.c.w. a torrent client? I've allocated 60mb ram, but it crashes within 2-8 hours here, which is kind of disappointing.
Do the logs show an out-of-memory error when that happens? I haven't seen one for a long time now, but maybe torrents stress it more than usual.
If so, it could be https://github.com/yomimono/mirage-nat/issues/17 - there's a Mirage hackathon next month and I'm hoping to get some time to work on this there.
Foppe de Haan
Yes. "Fatal error: out or memory. Mirage exiting with status 2"
That said, 2 minutes earlier the log notes that memory use was still only at 16.7/38.2 MB. (Most of the log -- 90-95% -- consists of 'Failed to parse frame' messages, btw.)
Thomas Leonard
> On Tuesday, February 7, 2017 at 5:24:58 PM UTC+1, Thomas Leonard wrote:
> > On Tuesday, February 7, 2017 at 3:55:30 PM UTC, Foppe de Haan wrote:
> > > Anyone else tried to use MirageOS i.c.w. a torrent client? I've allocated 60mb ram, but it crashes within 2-8 hours here, which is kind of disappointing.
> >
> > Do the logs show an out-of-memory error when that happens? I haven't seen one for a long time now, but maybe torrents stress it more than usual.
> >
> > If so, it could be https://github.com/yomimono/mirage-nat/issues/17 - there's a Mirage hackathon next month and I'm hoping to get some time to work on this there.
>
> Yes. "Fatal error: out or memory. Mirage exiting with status 2"
By the way, what version of the firewall are you using?
If it's not qubes-mirage-firewall v0.2 then try upgrading first - there were lots of OOM problems in v0.1.
> That said, 2 minutes earlier the log notes that memory use was still only at 16.7/38.2 MB.
The annoying thing about hashtables is the way they suddenly double in size. Since you're allocating 60 MB to the firewall (I only use 20 MB for mine), you could try adjusting the thresholds at these two lines:
https://github.com/talex5/qubes-mirage-firewall/blob/master/memory_pressure.ml#L41
https://github.com/talex5/qubes-mirage-firewall/blob/master/memory_pressure.ml#L47
Change the 0.9 (allow 90% of memory to be used) to 0.4 in both places. If the NAT table is the cause, that should make the problem go away.
> (Most of the log -- 90-95% -- consists of 'Failed to parse frame' messages, btw.)
"Failed to parse frame" probably means it saw an ICMP (not TCP or UDP) packet and therefore didn't handle it. Another thing I'm hoping to fix soon... https://github.com/yomimono/mirage-nat/issues/15
Foppe de Haan
I built it using docker about 2 days ago. Will do the other things you mentioned, report back when I know more :)
Jean-Philippe Ouellet
> I'm not sure why my DispVM is Fedora 23 when my default template is Fedora 24, but anyway...
[user@dom0 ~]$ qvm-create-default-dvm --default-template
If that does not work it may be a bug worth reporting. As a workaround, try:
[user@dom0 ~]$ qvm-create-default-dvm <custom-template-name>
See https://www.qubes-os.org/doc/dispvm-customization
Thomas Leonard
Thanks! If that is the cause of the memory problem, it should be easy to fix anyway.
Thomas Leonard
> On Sat, Jan 28, 2017 at 9:13 AM, Thomas Leonard wrote:
> > I'm not sure why my DispVM is Fedora 23 when my default template is Fedora 24, but anyway...
>
> If fedora24 is indeed your default template, try:
> [user@dom0 ~]$ qvm-create-default-dvm --default-template
That fixed it - thanks!
Foppe de Haan
It looks stable now (uptime 3-4 days since last reboot, whereas before it only lasted ~8h max).
Thomas Leonard
Thanks for the report! I've now made some updates to the firewall:
- It now uses an LRU-cache to drop old entries, rather than growing until it runs out of memory.
- ICMP queries (e.g. ping) and errors (e.g. Host unreachable) now work (they were dropped before).
- I've ported it to the new Mirage 3 release.
There are quite a lot of changes, so I'd be happy to get reports about whether it works or not (I've just started running the new version on my laptop).
Foppe de Haan
Thomas Leonard
> Stable so far. (Current uptime 12h, it crashed well before that when it wasn't.)
Thanks for testing!
I've made a new release of that version now (identical binary):
https://github.com/talex5/qubes-mirage-firewall/releases/tag/v0.3
Foppe de Haan
Thomas Leonard
> Any clue why Windows 7 won't boot when I have MirageOS selected as the firewall?
I've never tried it. Do the mirage-firewall logs show anything interesting when you try to boot Windows?
Thomas Leonard
Oh, that's more useful than I was expecting! Looks like the Windows boot process starts by running MiniOS! It's hanging at
close network: backend at /local/domain/4/backend/vif/79/0
I guess it asked the firewall to close the network, and never got a reply (because the firewall doesn't have any code to do that).
talex5...@gmail.com
OK, I finally got some time to look into this. I think this patch should fix it (works for Linux HVM anyway):
https://github.com/mirage/mirage-net-xen/pull/67
I also made a patch that seems to let the firewall work with disposable VMs:
https://github.com/mirage/mirage-net-xen/pull/68
Both are based on guesswork though - is the Xen netback protocol documented somewhere?
Jean-Philippe Ouellet
> Both are based on guesswork though - is the Xen netback protocol documented somewhere?
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/netif.h;hb=refs/heads/master
netfront / netback in linux:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/xen-netfront.c
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/xen-netback
And a somewhat outdated but much more approachable introduction in
section 9.2 (starting p.169) of "The Definitive Guide to the Xen
Hypervisor" book in case you have access to it.
Thomas Leonard
Thanks. Is there anything about the setup protocol, though? This file seems less well commented:
BTW, I've updated the Dockerfile to build with the patches applied now, if anyone wants to test it:
https://github.com/talex5/qubes-mirage-firewall/
I've had one report from a Qubes 4.0rc1 user that it now works for them (for HVM Linux).