Idea for (resonable secure) cloud-storage usage with Qubes

[[799]]

Hello,

I thought about how to work with cloud storage under Qubes OS and I'd like to share my idea with you, to provide feedback.

I have already build a prototype that works "reasonable" well, but I am far away from being a security professional, as such I'd like to hear your opion about it.

Assumptions:

You are using cloud storage like Microsoft OneDrive and you would like to do so under Qubes in a more secure way.

Goals:

- all files within onedrive should be encrypted

- files should still be accessible/decryption from other Operating systems

- decrypted data storage and cloud storage access should be separated into two AppVMs

- different AppVMs should have access to data in the cloud storage, but it's impossible for an AppVM to read the data which should be read by other AppVMs (meaning you have the option to create individuall encrypted directories)

- solution should be easy to use and relying on scripts to provide good automation and a good tradeoff between security and user experience.

Idea:

In order to reach the goals, the idea is to work with two AppVMs:

1. "Access+Transfer AppVM" this VM will access the cloud storage provider, provide synchronisation and will always see encrypted data

2. "Storage-AppVM" this VM will receive the encrypted files from the Access+Transfer AppVM and store the files. It will also work as a data-hub to provide access to data to your other AppVMs which you use to manipulate the data within this VM.

As such we have separated:

- Access & Transfer of data from cloud storage provider

- Local data storage

- Data manipulation

Solution Design:

[Access+Transfer AppVM]

Template: fedora-25-minimal

Additional packages:

- OneDrive Freeclient (https://github.com/skilion/onedrive)

- sudo dnf -y install nfsutils

Will be configured to mount a NFS-share from the Storage AppVM and to access OneDrive to synchronize the files

Data will be downloaded and storad in the mounted NFS-Share of the Storage AppVM

[Storage App-VM]

Template: fedora-25-minimal

Additional packages:

- sudo dnf -y nfs-utils encfs

This machine has been setup as a NFS Server.

The /etc/exports file and also the iptables Firewall of this AppVM has been setup, so that the [Access+Transfer AppVM] kann access a certain location.

Within this location all files ENCFS-encrypted.

As such the Access+Transfer AppVM but also the Cloud Storage provider will only see encrypted files.

Additional AppVMs can also mount the main NFS Share/directory.

Those AppVMs can access certain subfolders and mount them via ENCFS to get the unecrypted data.

So the ENCFS decryption are done in those AppVMs.

You could setup various subfolders within your Onedrive directory and each folder could be encrypted within the different AppVMs.

Example:

onedrive\photos --> NFS Share to --> my-photo-appvm

onedrive\work --> NFS Share to --> my-work-appvm

onedrive\media --> NFS Share to --> my multimedia-appvm

Let's look at one AppVM (example my-work-appvm = 10.137.2.25 // storage-appvm = 10.137.2.20)

On sys-firewall there is a rule, so that the work-appvm can access the storage-appvm:

[user@sys-firewall ~]$ sudo iptables -I FORWARD 2 -s 10.137.2.25 -d 10.137.2.20 -j ACCEPT

On the storage appvm:

[user@my-storage ~]$ sudo iptables -I INPUT 5 -i eth0 -s 10.137.2.25 -d 10.137.2.20 -j ACCEPT

The NFS Exports file:

[...]

# 10.137.2.15 = Access+Transfer AppVM

/var/nfs 10.137.2.15(rw,sync,no_subtree_check)

# 10.137.2.25 = Work AppVM

/var/nfs/work 10.137.2.25(rw,sync,no_subtree_check)

[...]

In the Work AppVM you are mounting the NFS Share from the Storage AppVM:

sudo mount 10.137.2.20:/var/nfs/work /mnt/onedrive-work.encfs

In Order to access the files, the NFS share is encfs-mounted:

encfs /mnt/onedrive-work.encfs ~/work

the unencrypted files can be accessed in ~/work.

If saved they will be encfs-encrypted and stored to NFS share of the Storage AppVM.

The Storage AppVM is connected to the Access-Transfer-AppVM which will recognize that an (encrypted) file has changed and will upload it to Onedrive.

As you can guess, you can use different AppVMs, which access different subfolders with different ENCFS-Keys.

For additional security you can also choose to shutdown the Access+Transfer AppVM and disable the NFS Server in the Storage AppVM if you don't need access to the files.

Script to start the NFS Server from dom0

#!/bin/bash

qvm-run my-storage 'xterm -e "sudo systemctl start nfs"'

sleep 2

Scripts to unencrypt the data in an AppVM from dom0:

#!/bin/bash

qvm-run my-work 'xterm -e "encfs /mnt/onedrive-work.encfs ~/work"'

Script to unmount the unencrypted share in an AppVM:

#!/bin/bash

qvm-run my-untrusted 'xterm -e "fusermount -u ~/work"'

I have already a working prototype, regarding the NFS server and ENCFS-part and will now add the onedrive part.

What's your opinion about this approach (I hope I could make clear what the idea is) - am I opening to much attack possibilities because I need to have NFS server running between the AppVMs? Keep in mind, that I am only sharing one directory, which is encrypted and only the AppVM knows how to decrypt the data.

So even if the Storage AppVM gets compromissed, the data should be encrypted (and therof protected).

The password entry within the AppVM to open of the ENCFS-encrypted data could be simplified by using something like a yubikey + short password.

Interested to get your feedback.

[799]

[[799]]

Hello,

I'd like to add, that I know that EncFS seems to have some issues, mentioned here for example:
https://defuse.ca/audits/encfs.htm
While this report is from 2014 and a new version has been issued it seems that encfs 2.x (which should provide better security) is still on its way - if it will come at all.

Unfortunately encfs is the only solution which is cross-platform, if someone is using a Linux only environment, the encryption layer could be replaced with other solutions:
https://www.cryfs.org/comparison

As such the subject should not be...

"Idea for (resonable secure) cloud-storage usage with Qubes"

... but ...

"Idea for (a slightly more secure) cloud-storage usage with Qubes"

[799]

PS: sorry for Top-Posting ;-)
I have a typo in my initial post, replace my-untrusted with my-work (where I unmount the encfs uencrypted directory)

-------- Original-Nachricht --------

[David Hobach]

Hi,

I think you have some misconceptions here - the main one being why
people tend to use Qubes OS:
Segregation of data to application-specific domains, i.e. impact of a
domain compromise is limited.

Your idea however makes your Qubes installation vulnerable to:
- Any attacks originating from that OS ("files should still be
accessible/decryption from other Operating systems").
- Nfs-based attacks (basically all your AppVMs using nfs will be
vulnerable to all nfs vulnerabilities - and there have been quite a few
in the past).
- encfs based attacks which people can even find on wikipedia.

If you don't want to add another idea to the security circus, I'd
reconsider either using Qubes OS, your other OS or your architecture.
The invisible things blog articles should be a good point to start with.

KR
David

>> - OneDrive Freeclient ([https://github.com/skilion/onedrive)](https://github.com/skilion/onedrive)

> <https://github.com/skilion/onedrive>

> --
> You received this message because you are subscribed to the Google
> Groups "qubes-users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to qubes-users...@googlegroups.com
> <mailto:qubes-users...@googlegroups.com>.
> To post to this group, send email to qubes...@googlegroups.com
> <mailto:qubes...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/qubes-users/1AEzAuelWDrVQsdrIbOmPhwI9QwoloMGiX6VcHatwcwHduke7sU-hDiyWwer2ffDpftxICtSWgAMAfUtd4kSus2RzA1v1DSUMOkwdLzkDkE%3D%40protonmail.com
> <https://groups.google.com/d/msgid/qubes-users/1AEzAuelWDrVQsdrIbOmPhwI9QwoloMGiX6VcHatwcwHduke7sU-hDiyWwer2ffDpftxICtSWgAMAfUtd4kSus2RzA1v1DSUMOkwdLzkDkE%3D%40protonmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[[799]]

Hello David,

Thank you for the open feedback.

> I think you have some misconceptions here
> - the main one being why people tend to use
> Qubes OS: Segregation of data to application-
> specific domains, i.e. impact of a domain
> compromise is limited.

You are right, regarding why people use Qubes.
But depending on specific workflows there is a need to either work with cloud storage for collaboration or to switch the OS completely for this use case.
Think about a (cloud based or on premise) storage service which is used by several people.
My goal is to work 100% in Qubes and I think that splitting access of data and local storage offers a better security than having the data synced and stored in one AppVM.
And I tried to build something that makes it easier to access data from various VMs in an easy way (knowing that it is less secure than using qvm-copy-to-vm).
But using some scripts we can reduce the attack surface on nfs in such a way, that we only enable NFS/open ports when access is needed.
I can't see how this approach is less secure than using one VM for syncing/storing/accessing the data?

> Your idea however makes your Qubes
> installation vulnerable to: - Any attacks
> originating from that OS ("files should still be
> accessible/decryption from other Operating
> systems")

True, but wouldn't this mean that the AppVM which is working as NFS Client must be compromised before NFS is attacked?

> Nfs-based attacks (basically all your AppVMs
> using nfs will be vulnerable to all nfs
> vulnerabilities

NFS access to the server is allowed on a per VM basis (firewall allow per IP), shouldn't this be enough to reduce NFS attack surface?

> encfs based attacks which people can even
> find on wikipedia.

Yes true, it is a shame, that we still don't have a multiplatform open source encryption standard that could maybe also be adapted by cloud storage providers.
But as mentioned the idea could also be implemented with other encryption solutions like CryFS, ...

> if you don't want to add

> another idea to the security circus

> I'd reconsider either using Qubes OS, your
> other OS or your architecture.

Hmm ..., why should I abandon Qubes and use a much more less secure OS just because working with cloud/external storage is part of some (!) of my workflows?

Even if all VMs which I use in the described solution are compromised, I can still have other VMs which are fine.
So basically it's one more reason to use Qubes ;-)

[799]

[David Hobach]

Hi again,

On 10/15/2017 08:37 PM, '[799]' via qubes-users wrote:
>> I think you have some misconceptions here
>> - the main one being why people tend to use
>> Qubes OS: Segregation of data to application-
>> specific domains, i.e. impact of a domain
>> compromise is limited.
>
> You are right, regarding why people use Qubes.
> But depending on specific workflows there is a need to either work with cloud storage for collaboration or to switch the OS completely for this use case.

Ok, that's something I can understand. So far I was under the impression
that all of your VMs were using that cloud backed storage.

> Think about a (cloud based or on premise) storage service which is used by several people.
> My goal is to work 100% in Qubes and I think that splitting access of data and local storage offers a better security than having the data synced and stored in one AppVM.
> And I tried to build something that makes it easier to access data from various VMs in an easy way (knowing that it is less secure than using qvm-copy-to-vm).
> But using some scripts we can reduce the attack surface on nfs in such a way, that we only enable NFS/open ports when access is needed.
> I can't see how this approach is less secure than using one VM for syncing/storing/accessing the data?

The point here is that it's not much more secure neither. In fact you
might even introduce unwanted mistakes (mistakenly opening ports to one
of your other VMs e.g.), which ultimately could lead to the compromise
to one of your other VMs.

Attacking a nfs implementation shouldn't be too hard for a dedicated
attacker, i.e. you can bet that a compromise of any of your
nfs-connected VMs would lead to a compromise of _at least_ all of your
nfs connected VMs. Which is equal or worse than what you had without
that idea.

So the standard attack path would be:
other OS --> nfs-client VM --> other nfs VMs

>> Your idea however makes your Qubes
>> installation vulnerable to: - Any attacks
>> originating from that OS ("files should still be
>> accessible/decryption from other Operating
>> systems")
>
> True, but wouldn't this mean that the AppVM which is working as NFS Client must be compromised before NFS is attacked?

Yes, also holds for the standard Qubes OS model though (you running your
nfs client in the same domain where you have your nfs data).

>> Nfs-based attacks (basically all your AppVMs
>> using nfs will be vulnerable to all nfs
>> vulnerabilities
>
> NFS access to the server is allowed on a per VM basis (firewall allow per IP), shouldn't this be enough to reduce NFS attack surface?

No. Protocol & implementation vulnerabilities exist.

>> encfs based attacks which people can even
>> find on wikipedia.
>
> Yes true, it is a shame, that we still don't have a multiplatform open source encryption standard that could maybe also be adapted by cloud storage providers.
> But as mentioned the idea could also be implemented with other encryption solutions like CryFS, ...

I don't know that one.

Anyway file-based encryption suffers from revealing meta data such as
file access timestamps, number of files, work activity, maybe even
folder structures.

Volume-based encryption doesn't tend to have these issues. The
containers of the truecrypt successor should also be supported by
cryptsetup if I recall correctly.

Assuming the other OS is Qubes OS you could do one encrypted
voloume/container per Qubes domain and do an implementation as follows:
- mount the remote fs in some "distributor" appVM, e.g. using sshfs
- use qvm-block from dom0 to attach the encrypted containers from the
distributor VM to the respective target domains
- decrypt the containers in the respective domains using keys that can
only be found there

That implementation still suffers from parsing attacks on cryptsetup,
but the others should be identical to attacking Qubes OS itself.

It might be possible to mitigate potential cryptsetup issues by writing
an own qrexec service, but that should be left to the pros...

The performance should be roughly as good as reading & writing from a
network backend is in general.

For non-Qubes OS systems I don't see the point of separating domains
though. The other OS doesn't do it neither.

KR
David

[Ron Qubed]

On Saturday, October 14, 2017 at 5:54:28 PM UTC-6, [799] wrote:
...

> Solution Design:
>
> [Access+Transfer AppVM]
>
> Template: fedora-25-minimal
>
> Additional packages:
>
> - OneDrive Freeclient (https://github.com/skilion/onedrive)
>
> - sudo dnf -y install nfsutils
>
> Will be configured to mount a NFS-share from the Storage AppVM and to access OneDrive to synchronize the files
>
> Data will be downloaded and storad in the mounted NFS-Share of the Storage AppVM

...
> In the Work AppVM you are mounting the NFS Share from the Storage AppVM:
>
> sudo mount 10.137.2.20:/var/nfs/work /mnt/onedrive-work.encfs
>
>
>
> In Order to access the files, the NFS share is encfs-mounted:
>
> encfs /mnt/onedrive-work.encfs ~/work

...

> What's your opinion about this approach (I hope I could make clear what the idea is) - am I opening to much attack possibilities because I need to have NFS server running between the AppVMs? Keep in mind, that I am only sharing one directory, which is encrypted and only the AppVM knows how to decrypt the data.
>
> So even if the Storage AppVM gets compromissed, the data should be encrypted (and therof protected).

...

> Interested to get your feedback.
>
>
> [799]

Have you considered using SSHFS rather than NFS? I'm no security expert, but it would seem to me to be more secure than NFS.

Ron

[[799]]

Hello Ron,

Thank you for the feedback.

> Have you considered using SSHFS rather than
> NFS? I'm no security expert, but it would
> seem to me to be more secure than NFS.

Actually yes, I thought about it after other mentioned that enabling NFS would offer another attack window.
Even when I am unsure as I have but some encryption and firewall restrictions in place.
The Access&Transfer VM is the only one connected to the internet and the NFS Storage VM.
The other AppVMs who will connect to the storage VM don't have an online connection.
From my understanding an attacker must come through the Access&Transfer VM and then attack the Storage VM.

Unfortunately I don't how those attacks take place and how much time is necessary. It could be possible to launch the access&transfer VM only periodically just to sync the data.
Keep in mind, that all data is encrypted from the view of the access+transfer VM.

I'd like to setup firewall rules, which will only allow traffic from the access+transfer VM to the cloud storage provider, but this need some further investigation.
As far as I understand Qubes Firewall GUI will not work with domain names but with IPs.

Regarding sshfs I will give it a try, as ssh is used to connect remotly I am (reasonable) sure that it has less attack possibilities than NFS.

Even when enabling inter-VM networking I feel more secure when I can keep my data encrypted+synced and have the data access separated in different VMs.

[799]

[rysiek]

Dnia Monday, October 16, 2017 8:20:40 PM CEST Ron Qubed pisze:

> Have you considered using SSHFS rather than NFS? I'm no security expert, but
> it would seem to me to be more secure than NFS.

For what it's worth, we're using (not with Qubes, just generally) a system of
LUKS volumes in large (hundreds of GiB) files on SSHFS-mounted volumes (for
backups), and we're quite happy with that set-up.

Everything works great as long as the connection is stable; if the connection
breaks, stuff needs to be remounted, but we have not had any data corruption
(so far, ~15TiB of data pushed through this).

Perhaps this helps someone.

--
Pozdrawiam,
Michał "rysiek" Woźniak

Zmieniam klucz GPG :: http://rys.io/pl/147
GPG Key Transition :: http://rys.io/en/147

[[799]]

Hello rysiek,

> For what it's worth, we're using (not with Qubes, just generally) a system of

> LUKS volumes in large (hundreds of GiB) files on SSHFS-mounted volumes (for

> backups), and we're quite happy with that set-up.

thanks for the info, good to know that sshfs seems to work on a daily basis.

I have migrated from NFS to SSHFS already.

I'm using certificates for additional security and have restricted firewall rules so that only transfer between both VMs TC/port 22 is allowed.

Seems like a solution that is ok, even when I'll not out the "save-the-world-formula" there (as encfs seems to be the weakest link in my setup).

Regarding my specific use case I would like to synchronize the data to keep a copy at another location.

Using LUKS images can cause a problem depending on the transfer mechanism, as I need to use a mechanism which will only transfer the qctual changed blocks not the whole image.

As such I'd like to use an encryption which works with file based encryption - knowing that this has reduced security as metadata etc. can be used to attack the encryption.

See my other mail, how my solution with SSHFS looks like.

[799]

[Outback Dingo]

I think we might be better to look at say encrypted S3 buckets, as S3 clients can run from any AppVM. long and short, i have storage resources, so im inclined to test a remote backup scenerio via S3 buckets

[rysiek]

Hey,

Dnia Saturday, October 21, 2017 9:48:56 AM CET [799] pisze:

> Regarding my specific use case I would like to synchronize the data to keep
> a copy at another location. Using LUKS images can cause a problem depending
> on the transfer mechanism, as I need to use a mechanism which will only
> transfer the qctual changed blocks not the whole image. As such I'd like to
> use an encryption which works with file based encryption - knowing that
> this has reduced security as metadata etc. can be used to attack the
> encryption.

But that's also what we're doing with LUKS+SSHFS. The LUKS volume is
cryptsetup luksOpened and mounted on the *client*, not on the (SSH) *server*,
meaning the (SSH) server only has access to encrypted data.

Then we're doing regular file-based operations, like rsync or whatnot. Only
modified bytes of the LUKS image seem to be actually transferred.

We're not transferring the whole images back and forth. That would defeat the
purpose.