TorVM on R1

[Abel Luck]

Hi,

Following Joanna's blog post for getting a transparent tor ProxyVM
working, I've ran into a snag.

When first booted, the AnonVM cannot access the network. If I execute
the start_tor_proxy script manually on TorVM, then AnonVM can access the
internet properly proxied.

At first I thought the start_tor_proxy script was not getting ran, but
examining /var/log/messages and iptables --list-rules -t nat revealed
that indeed: tor had started and the forwarding rules were properly set.

For some reason manually re-running start_tor_proxy after the TorVM has
booted fixes the routing.

I've compared the output of the following commands (ran on TorVM) before
AND after running start_tor_proxy manually, and there is no difference:
sudo route, sudo iptables --list-rules, sudo iptables --list-rules -t nat

I'm at a loss as to what the issue is here, or how to proceed in
troubleshooting.

My only guess is the bit in joanna's post regarding qubes_netwatcher and
qubes_firewall. The chkconfig command she provides errors out with:
error reading information on service qubes_firewall: No such file or
directory

Checking the process list, qubes_firewall is running. Killing it doesn't
seem to work, and unchecking it (after +ing it) in the VM settings >
Services dialog does results in no change.

Any thoughts on the issue or how to further debug it?

~abel

[1]:
http://theinvisiblethings.blogspot.com/2011/09/playing-with-qubes-networking-for-fun.html

[Abel Luck]

[Marek Marczykowski]

You can disable those services using qvm-service tool (in dom0). This tool was
introduced after Joanna's post.

qvm-service torvm -d qubes-firewall
qvm-service torvm -d qubes-netwatcher

I'm actually using torvm on most recent Qubes version (setup mostly based on
that blog post), but don't have separate AnonVM. I use DispVM for it:
1. Start torvm.
2. Start firefox in new DispVM (there is menu entry for it)
3. Switch DispVM network to torvm:
qvm-prefs disp3 -s netvm torvm

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Joanna Rutkowska]

Hm... I wonder whether we could somehow automate this 3rd step? Quite
obviously, you would _surely_ forget to do it one day... ;)

joanna.

[Marek Marczykowski]

It can be set for fedora-17-x64-dvm (and then called qvm-create-default-dvm),
but haven't tested it.

[Abel Luck]

Marek Marczykowski:

Would you be interested in having a default TorVM alongside the other
default service VMs? Or maybe at least the option to have it created?

I've improved upon Joanna's original setup, by taking advantage of
stream isolation in tor 0.2.3 [1] to provide some protection against
identity correlation.

The stream isolation defaults are good, but I've additionally opened
some socks port on the TorVM so you can further control the circuits
certain apps use.

~abel

[1]:
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/171-separate-streams.txt

[Joanna Rutkowska]

Yes, it would be great. I think we can just create a default torvm as
part of the standard system setup (i.e. firstboot when use chooses the
default option) -- in the worst case this vm would never be started, so
wouldn't be taking any resources, except a few MBs of disk storage,
which is nothing.

Abel, do you think you could write the required patches? What do you
think would be best method to create such a torvm? Certainly we don't
want to add '--torvm' switch to qvm-create, as this would be too
specific. Perhaps the best solution would be to create a special rpm,
say, qubes-vm-torvm.rpm that, when installed inside a proxyvm would make
it essentially a working torvm (this would involve also installing all
the required 3rd party software, such as vidalia, etc). Then we could
easily create it (e.g. in firstboot) by:

qvm-create --proxy torvm
qvm-run -a torvm 'yum install qubes-vm-torvm'

What do you think?

joanna.

[Marek Marczykowski]

This way will require install qubes-vm-torvm after each torvm reboot (as torvm
isn't StandaloneVM), which isn't convenient. Either creating torvm as
StandaloneVM isn't good idea.

I though about creating this as additional service (qubes-tor-setup), which
can be enabled via qvm-service. Then above torvm setup will look something
like this:

qvm-run -p -a fedora-17-x64 -u root 'yum -y install qubes-vm-tor-setup; poweroff'
qvm-create --proxy torvm
qvm-service -e torvm qubes-tor

> What do you think?
>
> joanna.
>
>
>> I've improved upon Joanna's original setup, by taking advantage of
>> stream isolation in tor 0.2.3 [1] to provide some protection against
>> identity correlation.
>>
>> The stream isolation defaults are good, but I've additionally opened
>> some socks port on the TorVM so you can further control the circuits
>> certain apps use.
>>
>> ~abel
>>
>> [1]:
>> https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/171-separate-streams.txt
>>
>>
>>
>
>

[Joanna Rutkowska]

I like it.

So, Abel, do you feel like contributing patches to implement the above?
(And optionally also a trivial patch for the installer/firstboot to
create it by default)?

joanna.

[Abel Luck]

Joanna Rutkowska:

Agreed.

> Perhaps the best solution would be to create a special rpm,
> say, qubes-vm-torvm.rpm that, when installed inside a proxyvm would make
> it essentially a working torvm (this would involve also installing all
> the required 3rd party software, such as vidalia, etc). Then we could
> easily create it (e.g. in firstboot) by:
>
> qvm-create --proxy torvm
> qvm-run -a torvm 'yum install qubes-vm-torvm'
>
> What do you think?

I like this idea. It could be even further integrated into the GUI by
adding a "Use Tor" check box that appears when clicking the "ProxyVM"
radio button in the "Create New VM" dialog. Or something similar.

The RPM would need to do the following:

* (optional) install Tor's FC repo for latest versions (to get stream
isolation)
* Install Tor package
* Create startup config+scripts
* Execute startup scripts on boot

I'd love to submit this change, however I haven't the faintest clue how
to go from the changes I made on disk to a default ProxyVM to a
standalone rpm.

I'm only familiar with Archlinux and Debian's package formats. Do you
have any docs on how to proceed?

On a related note, it would be great if the developer docs described how
to go about developing Qubes w/out having to reinstall from an ISO after
every change. (i.e., how does the Qubes dev workflow work?)

~abel

[Joanna Rutkowska]

Please see the other email in the thread (Marek's).

> * (optional) install Tor's FC repo for latest versions (to get stream
> isolation)
> * Install Tor package
> * Create startup config+scripts
> * Execute startup scripts on boot
>
> I'd love to submit this change, however I haven't the faintest clue how
> to go from the changes I made on disk to a default ProxyVM to a

> standalone rpm.: via network or USB key, or qvm-copy-to-vm if you do development in one AppVM, and test in another

>
> I'm only familiar with Archlinux and Debian's package formats. Do you
> have any docs on how to proceed?
>

The book on building RPMs:

http://www.rpm.org/max-rpm/

> On a related note, it would be great if the developer docs described how
> to go about developing Qubes w/out having to reinstall from an ISO after
> every change. (i.e., how does the Qubes dev workflow work?)

Hm... something like this:
1) Modify some specific repo
2) Build a new RPM
3) Copy the RPM to a test AppVM or a test system (if developing for Dom0
or hypervisor)

j.

[Marek Marczykowski]

Also qvm-service approach will make easy to enable this from GUI (there is
services tab in VM settings).

>
>> * (optional) install Tor's FC repo for latest versions (to get stream
>> isolation)
>> * Install Tor package
>> * Create startup config+scripts
>> * Execute startup scripts on boot
>>
>> I'd love to submit this change, however I haven't the faintest clue how
>> to go from the changes I made on disk to a default ProxyVM to a
>> standalone rpm.: via network or USB key, or qvm-copy-to-vm if you do development in one AppVM, and test in another
>>
>> I'm only familiar with Archlinux and Debian's package formats. Do you
>> have any docs on how to proceed?
>>
>
> The book on building RPMs:
>
> http://www.rpm.org/max-rpm/

You can also look to already existing rpms. Some assorted hints for torvm:
1. This fit into addons repository
(http://git.qubes-os.org/gitweb/?p=mainstream/addons.git;a=summary)

2. I think it can contain those files:
/usr/lib/qubes-tor/start_tor_proxy.sh (just like in Joanna article)
/lib/udev/rules.d/99-qubes-tor-hook.rules
(based on /etc/udev/rules.d/99-qubes_network.rules - which BTW should also be
in /lib/udev/rules.d...) - instead of qubes_ip_change_hook file
/lib/systemd/system/qubes-tor-setup.service
perhaps some other

3. qvm-service works by creating files in /var/run/qubes-service/, then you
can add simple condition to systemd unit file to get it automatically enabled:
ConditionPathExists=/var/run/qubes-service/qubes-tor

4. Stop action should also terminate tor daemon

5. To disable qubes-netwatcher and qubes-firewall services you can:
a) just do nothing and require the user to manually disable them in VM settings
b) create additional service, started just after qubes-sysinit (which prepare
/var/run/qubes-service dir) and removes qubes-netwatcher and qubes-firewall
files from /var/run/qubes-service. This service can have dependencies
something like:
[Unit]
DefaultDependencies=no
After=qubes-sysinit.service
Before=qubes-network.service qubes-firewall.service

>> On a related note, it would be great if the developer docs described how
>> to go about developing Qubes w/out having to reinstall from an ISO after
>> every change. (i.e., how does the Qubes dev workflow work?)
>
> Hm... something like this:
> 1) Modify some specific repo
> 2) Build a new RPM
> 3) Copy the RPM to a test AppVM or a test system (if developing for Dom0
> or hypervisor)

First of all get qubes-builder :)

[Abel Luck]

Marek Marczykowski:

Thanks for the above. I'll work on it and see what I can come up with.

>>> On a related note, it would be great if the developer docs described how
>>> to go about developing Qubes w/out having to reinstall from an ISO after
>>> every change. (i.e., how does the Qubes dev workflow work?)
>>
>> Hm... something like this:
>> 1) Modify some specific repo
>> 2) Build a new RPM
>> 3) Copy the RPM to a test AppVM or a test system (if developing for Dom0
>> or hypervisor)
>
> First of all get qubes-builder :)

Done, well sorta ;) I'm using qubes-builder to build your git devel
branch and am using the resulting ISO as my main qubes install, though
that was on another vanilla FC17 install. I need to get qubes-builder
running in a qubes-dev VM.

I wish I had a spare machine to test dom0 updates on, but I don't
unfortunately.

[Marek Marczykowski]

On 04.10.2012 00:42, Abel Luck wrote:
> Marek Marczykowski:

You can also create some test standalone VM, prepare files there and when
everything will be working, then pack them into rpm package. Remember to test
package also in fresh standalonevm to spot possibly omitted files.

> I wish I had a spare machine to test dom0 updates on, but I don't
> unfortunately.

Fortunately torvm do not need modification in dom0.
Anyway when I test something in dom0, I have always backup of most critical
files (to always have qvm-start and qvm-run working). This consists of:
/usr/lib64/python2.6/site-packages/qubes
/usr/lib/qubes
/etc/xen
/var/lib/qubes/qubes.xml

You can also keep somewhere original qubes-core-dom0.rpm and if something goes
wrong, just reinstall it (rpm -Uhv --force).

[Abel Luck]

Marek Marczykowski:

>
> You can also look to already existing rpms. Some assorted hints for torvm:
> 1. This fit into addons repository
> (http://git.qubes-os.org/gitweb/?p=mainstream/addons.git;a=summary)
>
> 2. I think it can contain those files:
> /usr/lib/qubes-tor/start_tor_proxy.sh (just like in Joanna article)
> /lib/udev/rules.d/99-qubes-tor-hook.rules
> (based on /etc/udev/rules.d/99-qubes_network.rules - which BTW should also be
> in /lib/udev/rules.d...) - instead of qubes_ip_change_hook file
> /lib/systemd/system/qubes-tor-setup.service
> perhaps some other
>
> 3. qvm-service works by creating files in /var/run/qubes-service/, then you
> can add simple condition to systemd unit file to get it automatically enabled:
> ConditionPathExists=/var/run/qubes-service/qubes-tor
>
> 4. Stop action should also terminate tor daemon
>
> 5. To disable qubes-netwatcher and qubes-firewall services you can:
> a) just do nothing and require the user to manually disable them in VM settings
> b) create additional service, started just after qubes-sysinit (which prepare
> /var/run/qubes-service dir) and removes qubes-netwatcher and qubes-firewall
> files from /var/run/qubes-service. This service can have dependencies
> something like:
> [Unit]
> DefaultDependencies=no
> After=qubes-sysinit.service
> Before=qubes-network.service qubes-firewall.service
>

I'm attempting to add the torproject repos [1]to
/etc/yum.repos.d/torproject.repo (in a template vm) and trying to patch
the update proxy to work correctly with the new urls/hostnames.

When I run 'yum install tor' after adding the repo, I receive many DNS
errors:

$ yum install tor
Loadded plugins: ...
http://linuxdownload.adobe.com/.../repomd.xml: [errno 14] curl#6 -
"Couldn't resolve host"
Trying other mirror
http://mirror.seas.harvard.edu/fedora/linux/updates/17/x86_64/..../repomod.xml:
[errno 14] curl#6 - "Couldn't resolve host"
Trying other mirror
.. and so on

Eventually it tries http://deb.torproject.org/torproject.org/... but it
cannot resolve the hostname.

I thought I needed to add something to the http proxy implemented in
[2], but that doesn't seem to affect DNS.

How do I properly whitelist http://deb.torproject.org/ for the update proxy?

[1]: https://www.torproject.org/docs/rpms.html.en
[2]: http://wiki.qubes-os.org/trac/ticket/568

[Outback Dingo]

>]

I just installed and ran updates and im seeing alot of cant resolv
hosts also.... i rebooted once, it seemed to go away, but really hasnt

[Marek Marczykowski]

It looks like you have some problem with yum proxy setting. This can be the
case just after qubes-core-vm upgrade and before template vm restart.
Look into /etc/yum.conf.d/qubes-proxy.conf - there should be:
proxy=http://10.137.255.254:8082/

> Eventually it tries http://deb.torproject.org/torproject.org/... but it
> cannot resolve the hostname.
>
> I thought I needed to add something to the http proxy implemented in
> [2], but that doesn't seem to affect DNS.
>
> How do I properly whitelist http://deb.torproject.org/ for the update proxy?

If it uses standard yum repo layout, it should work without any additional action.

> [1]: https://www.torproject.org/docs/rpms.html.en
> [2]: http://wiki.qubes-os.org/trac/ticket/568
>
>

[Abel Luck]

Yes, that seemed to be the issue. A template restart fixed the issue.

>> Eventually it tries http://deb.torproject.org/torproject.org/... but it
>> cannot resolve the hostname.
>>
>> I thought I needed to add something to the http proxy implemented in
>> [2], but that doesn't seem to affect DNS.
>>
>> How do I properly whitelist http://deb.torproject.org/ for the update
proxy?
>
> If it uses standard yum repo layout, it should work without any
additional action.

It does, except for the gpgkey= parameter. Rather than whitelisting the
URL to the tor GPG key, I've included it in /etc/pki/rpm-gpg.

Your instructions were very helpful. I have almost everything working now.

The remaining issue is that my qubes-tor service doesn't get executed
automatically on VM boot. I have to execute sudo systemctl start
qubes-tor.service manually.

My systemd unit is as follows:

$ cat /lib/systemd/system/qubes-tor.service
[Unit]
Names=qubes_tor.service
Description=Qubes transprent tor proxy setup
ConditionPathExists=/var/run/qubes-service/qubes-tor
After=network.target

[Service]
Type=forking
PIDFile=/var/run/qubes-tor.pid
RemainAfterExit=no
ExecStart=/usr/lib/qubes-tor/start_tor_proxy.sh
StandardOutput=syslog

[Install]
WantedBy=multi-user.target

The udev rule is:

$ cat /etc/udev/rules.d/99-qubes-tor-hook.rules
SUBSYSTEMS=="xen", KERNEL=="eth*", ACTION=="add",
RUN+="/usr/lib/qubes-tor/start_tor_proxy.sh"

However, I have instructed the vm to autostart the service:
$ qvm-service torvm
qubes-netwatcher : Disabled
meminfo-writer : Enabled
qubes-tor : Enabled
qubes-firewall : Disabled

Any idea why it's not starting up?

~abel

[Marek Marczykowski]

You need to enable the service:
systemctl enable qubes-tor.service

[Hakisho Nukama]

TorVM doesn't provide resolving of hidden services (.onion addresses)
with a setup like Joannas, or does it?

Best Regards,
Hakisho Nukama

[Joanna Rutkowska]

On 10/08/12 12:38, Hakisho Nukama wrote:
> TorVM doesn't provide resolving of hidden services (.onion addresses)
> with a setup like Joannas, or does it?

Can you provide an exemplary URL of such hidden service, so I can test?

joanna.

[Marek Marczykowski]

On 08.10.2012 12:38, Hakisho Nukama wrote:
> TorVM doesn't provide resolving of hidden services (.onion addresses)
> with a setup like Joannas, or does it?

No, it doesn't. But theoretically can be easy extended by enabling socks5
proxy and setting up the application to use it (or via tproxy command). Thus
this will not be transparent to the application.

[Hakisho Nukama]

URL to hidden services can be found on
https://www.torproject.org/docs/tor-hidden-service.

Best Regards,
Hakisho Nukama

[Joanna Rutkowska]

Ah, you're right (just tested).

I thought that the tor DNS resolver (that normally listens on port 53 in
the torvm and to which all the incoming DNS traffic is redirected via
iptables) should be able to detect *.onion requests and resolve them as
if they were normal DNA A queries...

Apparently, however, this is not the case:

[root@torvm ~]# host idnxcnkne4qt76tg.onion 10.137.2.9
Using domain server:
Name: 10.137.2.9
Address: 10.137.2.9#53
Aliases:

Host idnxcnkne4qt76tg.onion not found: 3(NXDOMAIN)

So, why this doesn't work like this?

joanna.

[Joanna Rutkowska]

Ok, I got it :)

[Joanna Rutkowska]

On 10/08/12 12:52, Marek Marczykowski wrote:
> On 08.10.2012 12:38, Hakisho Nukama wrote:
>> > TorVM doesn't provide resolving of hidden services (.onion addresses)
>> > with a setup like Joannas, or does it?
> No, it doesn't. But theoretically can be easy extended by enabling socks5
> proxy and setting up the application to use it (or via tproxy command). Thus
> this will not be transparent to the application.

Abel, any thoughts on this issue? Are you planning to address it in your
patches?


joanna.

[coderman]

On Mon, Oct 8, 2012 at 4:21 AM, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:
>...

> I thought that the tor DNS resolver (that normally listens on port 53 in
> the torvm and to which all the incoming DNS traffic is redirected via
> iptables) should be able to detect *.onion requests and resolve them as
> if they were normal DNA A queries...
>

> So, why this doesn't work like this?

typically this just means you're missing:

AutomapHostsOnResolve 1
and possibly AutomapHosts for .onion

in tor config.

[Abel Luck]

Joanna Rutkowska:

Yup, my patches will include support for resolving .onion addresses from
AppVMs. This is allowed by setting the AutomapHostsOnResolve and
VirtualAddrNetwork config options appropriately.

On another note, my patches will support Tor connections three different
ways:

1. Transparent TCP+DNS proxy (non-DNS UDP rejected)
2. SOCKS proxy on 9050, strict stream isolation settings, most
anonymous, but lower performance
3. SOCKS proxy on 9049, less strict stream isolation, less anonymous,
better performance

I also aim to supply a version of the Tor Browser for use in these VMs.
Using vanilla firefox+tor proxy isn't quite good enough. The TorBrowser
does more to prevent leaks and protect anonymity.

The problem of course is Tor Browser comes with Tor+Vidalia, and running
tor-within-tor isn't recommended [1].

Future work might be running an HTTP proxy on the TorVM as well, though
I haven't quite come up with a use case for this yet. The transparent
proxy should cover all those scenarios.

[1] http://www.alastairhumphreys.com/books/thereareotherrivers/

~abel

[Abel Luck]

Marek Marczykowski:

Alright, my RPM is almost finished.. I just have an RPM question:

How can I get my RPM to install another package from a repo it just created?

I'm installing tor's fedora repo into /etc/yum.repos.d/ and I want to
install the tor package from it. Is this even possible?

Maybe it will require two rpms and two yum commands:

# yum install torproject-repo
# yum install qubes-tor

~abel

[Joanna Rutkowska]

On 10/12/12 19:35, Abel Luck wrote:
> Alright, my RPM is almost finished.. I just have an RPM question:
>
> How can I get my RPM to install another package from a repo it just created?
>
> I'm installing tor's fedora repo into /etc/yum.repos.d/ and I want to
> install the tor package from it. Is this even possible?

You should put "Requires: <some packet>" directive in your rpm's spec
file. Yum will then ensure that <some packet> gets also installed (if
it's not already).

j.

[Abel Luck]

Joanna Rutkowska:

Tried that already, and unfortunately it doesn't work.

The problem is my packages is installing the tor RPM repo from which it
gets the tor package. Since dependency handling occurs before file
installation (duh), I can't depend on a package that doesn't yet exist
in any install repos.

(I'm requiring a specific version of tor, not in the FC repos, hence the
need for a new yum repo)

Anyways, it's no big deal. I've got it working with two packages:

qubes-tor-repo : installs torproject yum repo and gpg key
qubes-tor-init : installs qubes init scripts + requires 'tor' package

Patch incoming :)

[Marek Marczykowski]

On 13.10.2012 15:10, Abel Luck wrote:
> Joanna Rutkowska:
>> On 10/12/12 19:35, Abel Luck wrote:
>>> Alright, my RPM is almost finished.. I just have an RPM question:
>>>
>>> How can I get my RPM to install another package from a repo it just created?
>>>
>>> I'm installing tor's fedora repo into /etc/yum.repos.d/ and I want to
>>> install the tor package from it. Is this even possible?
>>
>> You should put "Requires: <some packet>" directive in your rpm's spec
>> file. Yum will then ensure that <some packet> gets also installed (if
>> it's not already).
>
> Tried that already, and unfortunately it doesn't work.
>
> The problem is my packages is installing the tor RPM repo from which it
> gets the tor package. Since dependency handling occurs before file
> installation (duh), I can't depend on a package that doesn't yet exist
> in any install repos.
>
> (I'm requiring a specific version of tor, not in the FC repos, hence the
> need for a new yum repo)

Is it really required to strict version requirement? IMHO it should depend
just on tor and if the user will have tor repo installed (which would be
recommended in installation instruction) then the newer version will be
installed. Otherwise standard Fedora one.

> Anyways, it's no big deal. I've got it working with two packages:
>
> qubes-tor-repo : installs torproject yum repo and gpg key
> qubes-tor-init : installs qubes init scripts + requires 'tor' package
>
> Patch incoming :)
>

[Abel Luck]

Marek Marczykowski:

> On 13.10.2012 15:10, Abel Luck wrote:
>> Joanna Rutkowska:
>>> On 10/12/12 19:35, Abel Luck wrote:
>>>> Alright, my RPM is almost finished.. I just have an RPM question:
>>>>
>>>> How can I get my RPM to install another package from a repo it just created?
>>>>
>>>> I'm installing tor's fedora repo into /etc/yum.repos.d/ and I want to
>>>> install the tor package from it. Is this even possible?
>>>
>>> You should put "Requires: <some packet>" directive in your rpm's spec
>>> file. Yum will then ensure that <some packet> gets also installed (if
>>> it's not already).
>>
>> Tried that already, and unfortunately it doesn't work.
>>
>> The problem is my packages is installing the tor RPM repo from which it
>> gets the tor package. Since dependency handling occurs before file
>> installation (duh), I can't depend on a package that doesn't yet exist
>> in any install repos.
>>
>> (I'm requiring a specific version of tor, not in the FC repos, hence the
>> need for a new yum repo)
>
> Is it really required to strict version requirement? IMHO it should depend
> just on tor and if the user will have tor repo installed (which would be
> recommended in installation instruction) then the newer version will be
> installed. Otherwise standard Fedora one.

imho, when proxying the amount of traffic that TorVM will, the version
from Tor's repos (>= 0.2.3) is essential.

Identity correlation is a serious risk [1] that is addressed in version
0.2.3. My TorVM implementation uses the isolation features extensively.

If someone is not concerned about identity correlation because they're
just using tor for anonymous browsing, then running TorBrowser in a VM
is the best alternative.

Installing the torprojet repo is a relatively painless part of the
installation process, and given the benefit I just can't recommend not
using it.

~abel

[1]: see thread:
https://lists.torproject.org/pipermail/tor-talk/2012-March/023535.html