Guide: Monero wallet/daemon isolation w/qubes+whonix

[0xB44EFD8751077F97]

I made a guide for Monero to isolate the wallet from the daemon using
Whonix workstations and Qubes qrexec. Curious what qubes-users group thinks:

https://github.com/0xB44EFD8751077F97/monero-site/blob/93996434e2c461feae31956dc587459f59f9d09a/resources/user-guides/cli_wallet_daemon_isolation_qubes_whonix.md

https://github.com/monero-project/monero-site/pull/479

[pauHana]

I have followed your guide and have the wallet-ws and monerod-ws up and running. Something isn't clicking getting them to communicate though. In my wallet I get the cannot connect to monerod daemon. And in monerod-ws i can tail the bitmonero.log and see it working nicely and syncing.

Any troubleshooting steps you would recommend?

[pauHana]

After completing the to VM setups and shutting them down is the intention then to start monero-wallet-ws and interact with the wallet thru this vm as per the usual ./monero-wallet-cli?

[qubenix]

pauHana:

> After completing the to VM setups and shutting them down is the intention then to start monero-wallet-ws and interact with the wallet thru this vm as per the usual ./monero-wallet-cli?
>

Correct. You should be able to run the gui from monero-wallet-ws as
well, but I haven't tried it myself.

--
qubenix
GPG: B536812904D455B491DCDCDD04BE1E61A3C2E500

[pauHana]

How could I test the connection to the other monerod-ws?

One step I have tried is relaunching rc.local on monero-wallet-ws to:

user@host:~$/rw/config/rc.local
2018/04/21 17:22:54 socat[1935] E bind(5, {AF=2 127.0.0.1:18081}, 16): Already in use

So it looks like that is running. I am not sure how to check that the right stuff is going into the pipe from monerod-ws?

[qubenix]

qubenix:

Please don't delete the conversation so the next person that has this
trouble can see everything together.

Is your daemon sync'd fully?

--
qubenix
GPG: B536812904D455B491DCDCDD04BE1E61A3C2E500

[pauHana]

Yes it is. I deleted my two appVMs and redid everything on two new appvms. THis solved my problem, though not that that helps explain what the issue was. Maybe things got out of step because I started the monerod daemon manually before starting the wallet?

[Patrick Schleizer]

I didn't notice this thread until now.

Interesting!

Now reference here:
https://www.whonix.org/wiki/Monero


I am wondering how to save users from as many manual steps as possible.


To save users from having to edit /rw/config/rc.local...

> socat TCP-LISTEN:18081,fork,bind=127.0.0.1 EXEC:"qrexec-client-vm
monerod-ws user.monerod"

Could maybe replaced by file:

/etc/anon-ws-disable-stacked-tor.d/40_monero.conf

content:

$pre_command socat TCP-LISTEN:18081,fork,bind=127.0.0.1
EXEC:"qrexec-client-vm monerod-ws user.monerod"

Should work after reboot (or after "sudo systemctl restart
anon-ws-disable-stacked-tor").

Untested.

Reference:
https://github.com/Whonix/anon-ws-disable-stacked-tor/blob/master/etc/anon-ws-disable-stacked-tor.d/30_anon-dist.conf



/etc/qubes-rpc/policy/user.monerod could maybe become:
/etc/qubes-rpc/policy/whonix.monerod

To have users from manually creating it, could be dropped here:

https://github.com/QubesOS/qubes-core-admin-addon-whonix/tree/master/qubes-rpc-policy

If you like, create a pull request and see what Marek thinks.



/home/user/monerod.service would be better in /rw so only root can write
to it. Even better perhaps systemd user services?

https://www.brendanlong.com/systemd-user-services-are-amazing.html

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820111

[Holger Levsen]

On Tue, Aug 14, 2018 at 07:42:00PM +0000, Patrick Schleizer wrote:
> Now reference here:
> https://www.whonix.org/wiki/Monero
>
>
> I am wondering how to save users from as many manual steps as possible.

since a bit more than 2 weeks monero can be installed on stretch with
'sudo apt install -t stretch-backports monero', so I think this should
work on whonix-14 too.

see https://tracker.debian.org/pkg/monero for more info. currently it's
monero 0.12.3.0


--
cheers,
Holger

-------------------------------------------------------------------------------
holger@(debian|reproducible-builds).org

[0xB44EFD8751077F97]

Patrick Schleizer:

> I didn't notice this thread until now.
>
> Interesting!
>
> Now reference here:
> https://www.whonix.org/wiki/Monero
>
>
> I am wondering how to save users from as many manual steps as possible.
>
>
> To save users from having to edit /rw/config/rc.local...
>
>> socat TCP-LISTEN:18081,fork,bind=127.0.0.1 EXEC:"qrexec-client-vm
> monerod-ws user.monerod"
>
> Could maybe replaced by file:
>
> /etc/anon-ws-disable-stacked-tor.d/40_monero.conf
>
> content:
>
> $pre_command socat TCP-LISTEN:18081,fork,bind=127.0.0.1
> EXEC:"qrexec-client-vm monerod-ws user.monerod"
>
> Should work after reboot (or after "sudo systemctl restart
> anon-ws-disable-stacked-tor").
>
> Untested.
>
> Reference:
> https://github.com/Whonix/anon-ws-disable-stacked-tor/blob/master/etc/anon-ws-disable-stacked-tor.d/30_anon-dist.conf
>

Tested, works on Whonix 14/Qubes 4.0.

Would you consider shipping this as a default Whonix file, or maybe part
of a package? If not, the user will have to put this on the TemplateVM
or config bind-dirs; which are both additional steps.

>
>
> /etc/qubes-rpc/policy/user.monerod could maybe become:
> /etc/qubes-rpc/policy/whonix.monerod
>
> To have users from manually creating it, could be dropped here:
>
> https://github.com/QubesOS/qubes-core-admin-addon-whonix/tree/master/qubes-rpc-policy
>
> If you like, create a pull request and see what Marek thinks.
>

This would be useful. It's on my radar.

>
>
> /home/user/monerod.service would be better in /rw so only root can write
> to it. Even better perhaps systemd user services?
>
> https://www.brendanlong.com/systemd-user-services-are-amazing.html
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820111
>
>

Interesting, I didn't know about this. I don't see how moving the file
from /home/user/ to /home/user/.config/systemd/user is more secure,
though. I think moving it to /rw may be slightly better, but
passwordless sudo kind of negates that.

The best would be to put it on the TemplateVM in /lib/systemd/system/,
but, again, this is more steps for the user.

In regards to monero being in stretch-backports now, I think it might be
an equal number of steps or more than there is now, and more confusing
for the user, to add stretch-backports to the TemplateVM's sources and
install via apt. If it were in stretch this would be no question.

--
- 0xB44EFD8751077F97

[Patrick Schleizer]

https://getmonero.org/resources/user-guides/cli_wallet_daemon_isolation_qubes_whonix.html
is missing how to actually use it.

I guess it is simply: run `monero-wallet-cli` or monero gui in
monero-wallet-ws."

0xB44EFD8751077F97:

> Patrick Schleizer:
>> I didn't notice this thread until now.
>>
>> Interesting!
>>
>> Now reference here:
>> https://www.whonix.org/wiki/Monero
>>
>>
>> I am wondering how to save users from as many manual steps as possible.
>>
>>
>> To save users from having to edit /rw/config/rc.local...
>>
>>> socat TCP-LISTEN:18081,fork,bind=127.0.0.1 EXEC:"qrexec-client-vm
>> monerod-ws user.monerod"
>>
>> Could maybe replaced by file:
>>
>> /etc/anon-ws-disable-stacked-tor.d/40_monero.conf
>>
>> content:
>>
>> $pre_command socat TCP-LISTEN:18081,fork,bind=127.0.0.1
>> EXEC:"qrexec-client-vm monerod-ws user.monerod"
>>
>> Should work after reboot (or after "sudo systemctl restart
>> anon-ws-disable-stacked-tor").
>>
>> Untested.
>>
>> Reference:
>> https://github.com/Whonix/anon-ws-disable-stacked-tor/blob/master/etc/anon-ws-disable-stacked-tor.d/30_anon-dist.conf
>>
>
> Tested, works on Whonix 14/Qubes 4.0.
>
> Would you consider shipping this as a default Whonix file, or maybe part
> of a package?

In package https://github.com/Whonix/qubes-whonix when using socket
activation, yes.

Similar to:

-
https://github.com/Whonix/anon-ws-disable-stacked-tor/blob/master/lib/systemd/system/anon-ws-disable-stacked-tor_autogen_port_9050.socket

-
https://github.com/Whonix/anon-ws-disable-stacked-tor/blob/master/lib/systemd/system/anon-ws-disable-stacked-tor_autogen_port_9050.service

File name should not contain "anon-ws-disable-stacked-tor" / "autogen".

File names...?

/lib/systemd/system/qubes-whonix-monerod.socket
/lib/systemd/system/qubes-whonix-monerod.service

Replace "ExecStart=/lib/systemd/systemd-socket-proxyd 10.152.152.10:9050"

with:

socat TCP-LISTEN:18081,fork,bind=127.0.0.1 EXEC:"qrexec-client-vm
monerod-ws user.monerod"

Untested. Does that work?

Would this break monerod for users not using this Monero wallet/daemon
isolation? I mean, does monerod use local port 18081 by default? In that
case we'd need to change that port.

> If not, the user will have to put this on the TemplateVM
> or config bind-dirs; which are both additional steps.
>>
>>
>> /etc/qubes-rpc/policy/user.monerod could maybe become:
>> /etc/qubes-rpc/policy/whonix.monerod
>>
>> To have users from manually creating it, could be dropped here:
>>
>> https://github.com/QubesOS/qubes-core-admin-addon-whonix/tree/master/qubes-rpc-policy
>>
>> If you like, create a pull request and see what Marek thinks.
>>
>
> This would be useful. It's on my radar.
>
>>
>>
>> /home/user/monerod.service would be better in /rw so only root can write
>> to it. Even better perhaps systemd user services?
>>
>> https://www.brendanlong.com/systemd-user-services-are-amazing.html
>>
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820111
>>
>>
>
> Interesting, I didn't know about this. I don't see how moving the file
> from /home/user/ to /home/user/.config/systemd/user is more secure,
> though.

> I think moving it to /rw may be slightly better, but
> passwordless sudo kind of negates that.

Indeed only useful for users of these:

- https://www.qubes-os.org/doc/vm-sudo/
- https://github.com/tasket/Qubes-VM-hardening

Qubes-VM-hardening will be easily available one day probably.

https://github.com/QubesOS/qubes-issues/issues/2748

I guess password protected sudo will get more and more easy in Qubes so
very much worth going for proper access rights.

> The best would be to put it on the TemplateVM in /lib/systemd/system/,
> but, again, this is more steps for the user.
>
> In regards to monero being in stretch-backports now, I think it might be
> an equal number of steps or more than there is now, and more confusing
> for the user, to add stretch-backports to the TemplateVM's sources and
> install via apt. If it were in stretch this would be no question.
>

And only monerod is in Debian. monero gui is not.

[0xB44EFD8751077F97]

Patrick Schleizer:

> https://getmonero.org/resources/user-guides/cli_wallet_daemon_isolation_qubes_whonix.html
> is missing how to actually use it.
>
> I guess it is simply: run `monero-wallet-cli` or monero gui in
> monero-wallet-ws."

Yes, I aimed for brevity and flexibility, targeting the advanced user in
my first rendition. Looking back on this now I see at least a few
improvements that I intend to make. Thank you for taking an interest in
this!

I wasn't able to get this one working. Are these both to be enabled on
monero-wallet-ws? What port is the socket supposed to be on? Can't be
the same as where socat listens in the service unit.

I'm not familiar with this method, so I most likely made some mistakes.
I won't have much time to play with it until this weekend.

> Would this break monerod for users not using this Monero wallet/daemon
> isolation? I mean, does monerod use local port 18081 by default? In that
> case we'd need to change that port.

By default monerod will use the following ports (depending on what
network you're on):

{1,2,3}8080 = mainnet,testnet,stagenet p2p-bind-port
{1,2,3}8081 = mainnet,testnet,stagenet rpc-bind-port
{1,2,3}8082 = mainnet,testnet,stagenet zmq-rpc-bind-port

We should avoid these ports, as you say.

>
>> If not, the user will have to put this on the TemplateVM
>> or config bind-dirs; which are both additional steps.
>>>
>>>
>>> /etc/qubes-rpc/policy/user.monerod could maybe become:
>>> /etc/qubes-rpc/policy/whonix.monerod
>>>
>>> To have users from manually creating it, could be dropped here:
>>>
>>> https://github.com/QubesOS/qubes-core-admin-addon-whonix/tree/master/qubes-rpc-policy
>>>
>>> If you like, create a pull request and see what Marek thinks.
>>>
>>
>> This would be useful. It's on my radar.
>>
>>>
>>>
>>> /home/user/monerod.service would be better in /rw so only root can write
>>> to it. Even better perhaps systemd user services?
>>>
>>> https://www.brendanlong.com/systemd-user-services-are-amazing.html
>>>
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820111
>>>
>>>
>>
>> Interesting, I didn't know about this. I don't see how moving the file
>> from /home/user/ to /home/user/.config/systemd/user is more secure,
>> though.
>
>> I think moving it to /rw may be slightly better, but
>> passwordless sudo kind of negates that.
>
> Indeed only useful for users of these:
>
> - https://www.qubes-os.org/doc/vm-sudo/
> - https://github.com/tasket/Qubes-VM-hardening
>
> Qubes-VM-hardening will be easily available one day probably.
>
> https://github.com/QubesOS/qubes-issues/issues/2748
>
> I guess password protected sudo will get more and more easy in Qubes so
> very much worth going for proper access rights.
>

Ok, I plan on that.

>> The best would be to put it on the TemplateVM in /lib/systemd/system/,
>> but, again, this is more steps for the user.
>>
>> In regards to monero being in stretch-backports now, I think it might be
>> an equal number of steps or more than there is now, and more confusing
>> for the user, to add stretch-backports to the TemplateVM's sources and
>> install via apt. If it were in stretch this would be no question.
>>
>
> And only monerod is in Debian. monero gui is not.
>

--
- 0xB44EFD8751077F97

[mk]

Hi guys,

Thanks for your work on this.

Setup is working fine for me except that I could not find a way to properly forward i2p port (18080) from outside to monerod-ws when using sys-whonix as the netVM, I had to switch to sys-firewall.

(Script used to setup port forward https://gist.github.com/Joeviocoe/6c4dc0c283f6d6c5b1a3f5af8793292b)

any ideas on how to fix this ?

[qubenix]

mk:

It's not i2p port, it's the p2p (peer to peer) port.

1. You don't need to open that port unless you want to. It's only used
for serving blocks and transactions to other nodes. Monero's
documentation suggests to not open that port when using Tor.

Not to be confused with the rpc port (18081), which is opened when
wanting to create a remote node for light clients to connect to.

2. If you want to keep using sys-whonix as NetVM (better for privacy),
and you still want to open the port, use a Tor hidden service:
https://whonix.org/wiki/Hidden_Services.

You'll just have to advertise your onion address out of band somehow
because Monero doesn't have a setting like externalip= to let peers know
about your onion. Probably not worth it.

3. If you want to use sys-firewall you'll have to remove the
p2p-bind-ip=127.0.0.1 from the /lib/systemd/system/monerod.service file
on the TemplateVM. You will probably also have to allow the port forward
on your home router somehow.

--
qubenix
PGP: 96096E4CA0870F1C5BAF7DD909D159E1241F9C54
OTR: qub...@chat.freenode.net
OTR: DFD1DA35 D74E775B 3E3DADB1 226282EE FB711765

[mk]

qubenix,

I thought I had to run a full node to be synced with main blockchain but I now understand that running a local node is enough

So everything works as expected,

Thanks for you answer and clarification :)

[qubenix]

mk:

It's still a full node because you fully validate all transactions and
blocks as they come in. You just don't send them to new nodes trying to
sync.

Glad it helped.

[mk]

Ok I see,
Thanks again.

[mk]

One last thing,

It should be noted that I still had to remove option "--p2p-bind-ip=127.0.0.1" to let monerod bind on 0.0.0.0 to make it work.

Any security implication about this ? Does all traffic is still routed through Tor network ?

[qubenix]

mk:

> One last thing,
>
> It should be noted that I still had to remove option "--p2p-bind-ip=127.0.0.1" to let monerod bind on 0.0.0.0 to make it work.
>
> Any security implication about this ? Does all traffic is still routed through Tor network ?
>

If you are using gateway sys-whonix then all of your traffic is always
using Tor.

Not sure what you mean about "make it work", but I'm having some
stalling/connection issues[1][2] on the newly released version (v0.13.0).

At first connection issues were solved by adding torsocks to monerod
when using Whonix gateway or binding p2p ip to 0.0.0.0 if not using a
Whonix gateway. Next I started to experience stalls, which I still
haven't found a solution for yet.

[1] https://github.com/monero-project/monero/issues/4468
[2] https://github.com/monero-project/monero/issues/4469

[qubenix]

qubenix:

> mk:
>> One last thing,
>>
>> It should be noted that I still had to remove option "--p2p-bind-ip=127.0.0.1" to let monerod bind on 0.0.0.0 to make it work.
>>
>> Any security implication about this ? Does all traffic is still routed through Tor network ?

Forgot to mention that there is no security problem to bind on 0.0.0.0.

[mk]

>
> If you are using gateway sys-whonix then all of your traffic is always
> using Tor.

Ok ! I had begun to doubt..

> Not sure what you mean about "make it work", but I'm having some
> stalling/connection issues[1][2] on the newly released version (v0.13.0).

Sorry, I was meaning "syncing".

> At first connection issues were solved by adding torsocks to monerod
> when using Whonix gateway or binding p2p ip to 0.0.0.0 if not using a
> Whonix gateway. Next I started to experience stalls, which I still
> haven't found a solution for yet.
>
> [1] https://github.com/monero-project/monero/issues/4468
> [2] https://github.com/monero-project/monero/issues/4469

Exactly the same issue.
Running monerod with torsocks was working for me too.

[qubenix]

> Exactly the same issue.
> Running monerod with torsocks was working for me too.
>

Check my comment here:
https://github.com/monero-project/monero/issues/4468#issuecomment-429671939

Running monerod like that from systemd or the command line (with or
without --non-interactive doesn't matter, only with systemd needs it) is
the best experience I've had with it on Whonix in years.