Re: Incremental backups

[Joanna Rutkowska]

On 05/14/14 12:37, Joanna Rutkowska wrote:
> On 05/13/14 19:46, Davíð Steinn Geirsson wrote:
>>>>
>>>> If I remember correctly, David was working on a PAM module that
>>>> integrated with Qubes RPC to give Dom0-based user confirmation upon
>>>> sudo request. I think that's the way to go...
>>>>
>>
>> Yes, that's one of the things I would like to see in qubes, and I might
>> implement it in the future, but I'm not currently working on it. If
>> anyone else feels like giving it a shot, please don't stop on my
>> account. :)
>>
>> Right now, the time I have to work on qubes is directed at completing
>> the debian template. There are some other things I would like to work
>> on. Improving the backups system is #2 on the list - I would like to
>> see if it's possible to do incremental backups, whether file-level in
>> the VM, or block-level on the VM image without sacrificing the security
>> isolation of the current backup process. Offsite backup space is very
>> expensive, and taking only full backups makes it prohibitively
>> expensive for me.
>>
>
> To implement incremental backups securely, the actual backup diff'ing
> would need to be done by each of the VM itself. But this, I think, would
> require that we will need to expose to each such VM its previous backup
> snapshot to allow it to do a meaningful diff. I think this is doable,
> but such backing up procedure would be *very* slow and also might be
> *very* disk-consuming.
>

Just in case, I created a ticket for this in our "Milestone R2.1", where
we keep tickets for many other nice-to-have-features that we hope (some
of which at least) might be provided by the community at some point.

http://wiki.qubes-os.org/trac/ticket/858

If you decide to start working on this, do let us know and we will
assign the ticket to you then.

Also, moving this to qubes-devel, please respond there.

joanna.

[Marek Marczykowski-Górecki]

On 14.05.2014 12:41, Joanna Rutkowska wrote:
> On 05/14/14 12:37, Joanna Rutkowska wrote:
>> On 05/13/14 19:46, Davíð Steinn Geirsson wrote:
>>>>>
>>>>> If I remember correctly, David was working on a PAM module that
>>>>> integrated with Qubes RPC to give Dom0-based user confirmation upon
>>>>> sudo request. I think that's the way to go...
>>>>>
>>>
>>> Yes, that's one of the things I would like to see in qubes, and I might
>>> implement it in the future, but I'm not currently working on it. If
>>> anyone else feels like giving it a shot, please don't stop on my
>>> account. :)
>>>
>>> Right now, the time I have to work on qubes is directed at completing
>>> the debian template. There are some other things I would like to work
>>> on. Improving the backups system is #2 on the list - I would like to
>>> see if it's possible to do incremental backups, whether file-level in
>>> the VM, or block-level on the VM image without sacrificing the security
>>> isolation of the current backup process. Offsite backup space is very
>>> expensive, and taking only full backups makes it prohibitively
>>> expensive for me.
>>>
>>
>> To implement incremental backups securely, the actual backup diff'ing
>> would need to be done by each of the VM itself. But this, I think, would
>> require that we will need to expose to each such VM its previous backup
>> snapshot to allow it to do a meaningful diff. I think this is doable,
>> but such backing up procedure would be *very* slow and also might be
>> *very* disk-consuming.

Not really, many backup tools can create some "state file" and based on this
create next incremental backup. Simplest such solution would be keeping
previous backup date and include in incremental backup only files modified later.
In fact I don't know any backup solution which require direct access to
previous backup to create incremental one.

Also making backup using some VM service would have another nice feature -
will not require shutting down all the VMs.

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Joanna Rutkowska]

Ah, so in that case each backup contains always either a new full file
(if the backup program determined the file has changed since the last
time), or nothing, never "deltas"? Hm, probably make sense, and indeed
simplifies this ticket.

> Also making backup using some VM service would have another nice feature -
> will not require shutting down all the VMs.
>

Yes. Cool.

joanna.

[Olivier Médoc]

On 05/14/14 13:15, Joanna Rutkowska wrote:
> On 05/14/14 13:09, Marek Marczykowski-Górecki wrote:
>> On 14.05.2014 12:41, Joanna Rutkowska wrote:
>>> On 05/14/14 12:37, Joanna Rutkowska wrote:

But it will require starting up all the VMs ? At least one by one.

One of the thing that I'm afraid is restoring such backups. It will need
partitionning, formatting, and restoring the initial backup with qubes
agent installed for each VM ?

Or do you plan to make a first standard backup of the whole vm, while
simulating the initial iteration in order to create the initial .snar
file (if tar is used). Then next backups can be purely based on tar
using the .snar file.

[Marek Marczykowski-Górecki]

Perhaps "clean" VM can be created, started and restored using some VM service.
This will not work for template/standalone VM though, but IMO not a big
problem (just backup them using current scheme).

> Or do you plan to make a first standard backup of the whole vm, while
> simulating the initial iteration in order to create the initial .snar file (if
> tar is used). Then next backups can be purely based on tar using the .snar file.

Some details needs more thinking. But IMO its better to discuss it later,
after final R2 release.

[cprise]

I don't like this service-based idea either. Apart from the question of
starting the vms, grabbing the image files from the dom0 side is much
cleaner and sure-footed.

What we need is a way to break the image files into many segments, or at
least a way to track many segments within a single image file. OSX
implements this as Sparsebundles which allows Time Machine to do
efficient backups no matter how large the data files you're updating.
Some of us have large multi-gigabyte databases, for instance, which get
changed daily. A Sparsebundle format allows these systems to be
efficiently backed-up and /also/ allows efficient retirement of parts of
the set (i.e. deleting the oldest backups) without affecting the backups
that remain. All the backup utility has to do is look at the file
modification date on each segment to determine if it needs to be backed
up; the segments that don't have a newer date get hardlinked to the
corresponding segments in the prior set instead of being copied.

FWIW, I did survey the backup tools (and disk image) landscape a few
months ago with Qubes in mind. I found a number of interesting options
(though nothing that magically endows Xen or Linux with sparsebundles
AFAIK)... I'll describe them in this thread when I get a chance later today.

[Davíð Steinn Geirsson]

Regarding just running the whole backup in dom0 on the image files,
this works very well with tools like bup or obnam. It's a bit faster
than the default qubes backup, and extremely space-efficient as the
qubes VMs dedup very well. If we had something like btrfs as default in
qubes, or stored the VM images as LVM LVs, we could even snapshot the
AppVMs prior to backup and hence be able to backup whether the machine
was running or shutdown with no risk of filesystem inconsistency.

I didn't like obnam because it uses MD5 hashes for its deduplication
tables... it just feels wrong, even though in practice you're extremely
unlikely to hit any collisions. I believe bup uses SHA1.

I'm currently using bup for backups in dom0. It's great, the only
downside is that there is no way to delete backups from a backupset.
You have to start a new one. In practice this is not a big deal, you
can just consider each backupset as one full backup + n incrementals,
and size your backupsets based on how frequently you want to do full
backups.

Of course, since bup does not do encryption itself, we would need some
external encryption. Currently I do these backups to a LUKS container
which I sync over to my offsite backup. This could be much more
efficient as I need to transfer the whole thing every time. File-level
encryption would probably solve this, as bup only adds packfiles, it
never modifies them. So we could just use rsync or whatever on the
underlying encrypted files. But then the question is, what is the
current security level of encfs or eCryptfs? I seem to recall news of
some not-so-encouraging security audits of these projects, but I
haven't really looked into them.

Perhaps something like this is enough? It doesn't feel in the "qubes
spirit" so to speak to do the whole thing in dom0, but really it's just
reading the VM images as a byte stream and packing them...
theoretically the attack surface should not be too big. Joanna and
Marek, do you have some thoughts on this? :)

-Davíð

[Joanna Rutkowska]

How do you transfer your backups from dom0 to your backup storage medium?

j.

[Davíð Steinn Geirsson]

Currently I just rsync the encrypted LUKS container file over ssh to the
remote server. This is really inefficient as I have to copy the whole
LUKS container every time, but it works for me for now. This is a
two-step process, I don't necessarily update the offsite backup every
time I do a new bup run.

This is the main problem with this method that I see (definately not
good enough for a ready-made backup solution), hence my ponderings
about eCryptfs/encfs...


>
> j.
>

[Joanna Rutkowska]

How can you rsync anything from Dom0, if Dom0 has no networking...?

j.

[Davíð Steinn Geirsson]

On Wed, 14 May 2014 23:15:31 +0200

Ah, I guess I skipped a step in my description :)

I have a second hard drive in my laptop. On that drive I have one or
more files that are LUKS containers. I luksOpen the most recent file,
mount it and do the backup, unmount and luksClose. Then I assign the
internal drive to a dedicated backup VM (not used for anything else,
with firewall rules that only allow talking to the remote backup
server on the SSH port), mount it and copy the newest file to offsite.
So the backup VM never sees the LUKS passphrase.


>
> j.
>
>

-Davíð

[Axon]

Davíð Steinn Geirsson:

This sort of setup concerns me, since sharing your HDD back and forth
between your dom0 and your backupvm seems to essentially escalate your
backupvm to the level of dom0.

This was also discussed in a past thread:
https://groups.google.com/d/topic/qubes-users/M8F8xDttOwA/discussion

[Olivier Médoc]

Do you know that you can directly send a backup through SSH ? You can
simply use a command instead of a file as your backup target.

For instance, I use something like ssh myaccount@myserver -c 'cat >
/home/myaccount/backups/mybackup-2014-05-10'

I use that to share my templates backups on a ssh share with my collegues.

The same works normally for restoring backups: ssh myaccount@myserver -c
'cat /home/myaccount/backups/mybackup-2014-05-10'

[Joanna Rutkowska]

Yeah, connecting untrusted device (your backup disk USB device) to Dom0
is dangerous -- see [1]. If your backup device is USB-based you're
opening a significant amount of physical attack surface this way (if not
doing anything else then by just keeping USB controllers assigned to
Dom0[*], at least for all sorts of physical attacks).

Some of the attacks described there apply even if you use a PCI-based
SATA disk. For example the disk you're shuffling back and forth would be
having a partition table that is under the control of the untrusted
backup vm, and which is later non-trivially parsed by the Dom0 kernel,
once you plug the HDD there.

We've designed and implemented this new backup scheme for Qubes
(introduced in R2B3) that allows to send backups securely to untrusted
AppVMs not without a good reason!

joanna.


[1]
http://theinvisiblethings.blogspot.com/2011/06/usb-security-challenges.html

[*] True, Qubes OS installer does leave USB controllers assigned to Dom0
by default[**], but we're considering to change this default device
partitioning in the future releases. In any case it's trivial for the
user to create a usbvm with just a few clicks in the Qubes Manager.

[**] ... which we do mostly to allow smooth Qubes installation to those
poor people who use laptops with USB-based keyboards (notably Macbooks).

[Davíð Steinn Geirsson]

On Thu, 15 May 2014 12:07:09 +0200

Thanks Axon and Joanna, I was aware of these issues already though.

I think we are getting sidetracked here. There are two discussions
going on:

1) Whether it's feasible to implement a new backup system for qubes
that does not sacrifice any security benefits of the current system,
but could add nice modern backup features like incrementals,
deduplication and so on.

This is an interesting discussion, and I hope to find a solution in
cooperation with other qubes users and devs. I think it would really be
a useful addition to qubes.

My question initially was very restricted in scope: Do others here see
any issue with using a backup system like bup in dom0 to index and pack
the VM images? This question is unrelated to how we move the
encrypted backups out of dom0 into an AppVM, and subsequently how the
user might choose to move the encrypted backup from the AppVM to an
external backup server (this is really easy, and I'm not sure it should
be in scope for the backup system - people have different backup
strategies and not everyone will want to use it). I certainly never
suggested that my current solution, which is very much a compromise,
would become an official part of the qubes backup solution. Before any
implementation takes place, we will have to find a solution to these
issues, but I don't think they will be showstoppers.

2) What backup system I (Davíð) am currently using.

This discussion seems quite offtopic for qubes-devel. It's a personal
compromise having to do with lots of things that have nothing to do
with qubes development (cost of offsite backup, perceived risk of
someone compromising all three of my remote backup server, my dedicated
BackupVM ssh client and the linux kernel handling of partition tables
or filesystems for dom0, and the currently limited capacity of my
secondary disk which is just temporary until I get a bigger disk to
stage backups before uploading to avoid sharing the disk in this way).

I think it servers no purpose to discuss 2. When and if I or someone
else fixes 1, 2 will follow. ;)

[Davíð Steinn Geirsson]

Yes, I know of this, but it will not work for bup since it does not
write a single serialized byte stream out that can be easily piped, but
rather it works with backupsets that are a collection of index and
packfiles. We would have to use something more elaborate to avoid the
middle stage of writing the backup to dom0 disk and then copying the
resulting backup to an AppVM (which doubles the required space).

[cprise]

On 05/15/14 12:28, Davíð Steinn Geirsson wrote:

[...]

My question initially was very restricted in scope: Do others here see
any issue with using a backup system like bup in dom0 to index and pack
the VM images? This question is unrelated to how we move the
encrypted backups out of dom0 into an AppVM, and subsequently how the
user might choose to move the encrypted backup from the AppVM to an
external backup server (this is really easy, and I'm not sure it should
be in scope for the backup system - people have different backup
strategies and not everyone will want to use it). I certainly never
suggested that my current solution, which is very much a compromise,
would become an official part of the qubes backup solution. Before any
implementation takes place, we will have to find a solution to these
issues, but I don't think they will be showstoppers.

I haven't used bup, but I do know it uses a rolling checksum technique to detect changes. Zbackup and ddar are two more utils that use rolling checksums and promise global de-duplication. (Two others, rdiff-backup and duplicity, are popular and create deltas for each file but do not deduplicate those files globally.)

Personally, I think you would do better with Zbackup or ddar, as they allow pruning of old backups to reclaim space (this isn't implemented in Zbackup yet, however, so ddar would be the natural choice unless you don't mind writing your own pruning script). AFAIK, bup cannot prune. ddar in particular is made for piping tar files and might even fit into the current native Qubes backup scheme without too much trouble.

This is basically the extent of the disk image-friendly backup utils I found in my initial search, although there are others which are based on the same ideas and/or lacking features. The (IMO, big) downside of all the above backup utils is they need to scan /all/ source data during each backup in order to find the deltas. That CPU/disk overhead is too much for my liking.

I have come to the conclusion that in order to have really efficient incremental backups it is necessary to have a block layer that provides more information about file modification times, breaking them down somehow into blocks or segments. This is, no doubt, the reason Apple created sparsebundles for use with Time Machine, but there is no actual analog to this in Linux itself apart from md which is not designed to handle thousands of segments per volume.

In another thread I'll describe the block layer alternatives I found.

[Andrew B]

On 05/13/14 19:46, Davíð Steinn Geirsson wrote:

> Hi,
>
> On Tue, 13 May 2014 13:32:47 +0200
> Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:
>
>> On 05/13/14 13:25, Andrew B wrote:
>>> On 05/13/14 13:16, Joanna Rutkowska wrote:
>>>> On 05/13/14 02:57, Marek Marczykowski-Górecki wrote:
>>>>> On 12.05.2014 22:41, Joonas Lehtonen wrote:
>>>>>>> How do I stop tor from auto-starting in an AppVM? I added
>>>>>>> both tor and qubes-tor to in the "Services" tab of AppVM and
>>>>>>> unchecked the boxes, but tor still boots when I start the
>>>>>>> AppVM.
>>>>>>
>>>>>>> In addition to using unnecessary resources, this is setting
>>>>>>> up tor-over-tor circuits - which can't be good.
>>>>>>
>>>>>> By coincidence I noticed that my firewallVM (and all other
>>>>>> AppVMs based on the default template) are running a tor process
>>>>>> as well. This reminded me of this qubes-users thread.
>>>>>>
>>>>>> I believe everyone following the steps in [1] will end up with
>>>>>> such a setup but probably no one will notice it because
>>>>>> everything is working fine (and it does no harm, no it doesn't
>>>>>> cause a tor-over-tor setup)
>>>>>>
>>>>>> The qubes-tor package depends on the actual "tor" package from
>>>>>> torproject.org (it gets installed as a dependency).
>>>>>>
>>>>>> 'qvm-service torvm -e qubes-tor' makes sure that qubes-tor will
>>>>>> only be started on torvm, but the "other" tor is started on all
>>>>>> other VMs anyway.
>>>>>>
>>>>>> You can easily tell whether you see a qubes-tor tor process or
>>>>>> a "vanilla" tor process by looking at the owner. qubes-tor's
>>>>>> tor runs as root, while the native tor is running as an
>>>>>> unprivileged user (_tor). @devs: please consider using an
>>>>>> unprivileged user as well (even if modifications don't survive
>>>>>> a reboot ;)
>>>>>>
>>>>>> The "native" tor from torproject.org apparently doesn't use
>>>>>> systemd style configs.
>>>>>>
>>>>>> If you want to get rid of the unnecessary second tor instance
>>>>>> you can run this in your templateVM:
>>>>>>
>>>>>> chkconfig --del tor
>>>>>>
>>>>>> (this removes the symlinks in /etc/rcX.d folders for you)
>>>>>
>>>>> I've added both of above suggestions to the package: 1. The
>>>>> service is started as '_tor' user. 2. Native tor is disabled on
>>>>> new installation (if you had previous version installed, you need
>>>>> to disable it manually, just as you've described).
>>>>>
>>>>> New package uploaded to unstable repository.
>>>>>
>>>>>> [1] http://qubes-os.org/trac/wiki/UserDoc/TorVM
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Just don't get too excited about running as _tor user instead of:
>>>>
>>>> http://git.qubes-os.org/?p=joanna/core-agent-linux.git;a=blob;f=misc/qubes.sudoers;h=8087a90a8c0a6f81d4fe45905a6aa9462de2a7ce;hb=HEAD
>>>>
>>>>
>>>>
>> :)
>>>>
>>>> But sure, it might make some sense for hygiene reasons, and maybe
>>>> one day we will change this sudoers config, or somebody might build
>>>> a better (community) template.
>>>>
>>>> joanna.

>>>>
>>>
>>> If I remember correctly, David was working on a PAM module that
>>> integrated with Qubes RPC to give Dom0-based user confirmation upon
>>> sudo request. I think that's the way to go...
>>>
>
> Yes, that's one of the things I would like to see in qubes, and I might
> implement it in the future, but I'm not currently working on it. If
> anyone else feels like giving it a shot, please don't stop on my
> account. :)

After looking around the PAM docs for a few minutes, this looked pretty easy, so I coded up a simple implementation (attached). Once again I'm not very interested in making a proper package, so it includes a README file with very simple installation instructions.

This service processes no data whatsoever in Dom0; it merely invokes 'echo 1'. On the AppVM, it implements an extremely simple PAM authentication scheme which merely forks an invocation of the Dom0 RPC service with a local handler of just "grep -q ^1$". When grep's return value indicates a match, the user must have authorized the authentication request in Dom0. Note that I have not considered vulnerability to, e.g. races, where some nefarious program already running in the AppVM somehow intercepts the RPC result or directly modifies memory in the PAM object. I suppose it's probably secure, but outside review is obviously welcome.

Andrew

[Marek Marczykowski-Górecki]

If this is such simple, perhaps even pam_exec will work here?

[Andrew B]

Good idea, I didn't look long enough to see pam_exec.so. It works just fine. The entire configuration/installation instructions can now be concisely written as:

----------
Set up Dom0 RPC service:
=====
(all done as root)

1) Create the file /etc/qubes-rpc/qubes.VMAuth with the contents:
/usr/bin/echo 1
2) Create the file /etc/qubes-rpc/policy/qubes.VMAuth with the contents:
$anyvm dom0 ask


Set up TemplateVM:
=====
1) Modify the file /etc/pam.d/system-auth to have the contents:
#%PAM-1.0
auth [success=done default=die] pam_exec.so seteuid /usr/lib/qubes/qrexec-client-vm dom0 qubes.VMAuth /usr/bin/grep -q ^1$

account required pam_unix.so

password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password sufficient pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password required pam_deny.so

session optional pam_keyinit.so revoke
session required pam_limits.so
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so

2) Edit the /etc/sudoers.d/qubes file to require authentication:
user ALL=(ALL) ALL
----------

The seteuid parameter should ensure AppVM software running as 'user' cannot intercept the RPC result (I think). Of course this is still vulnerable to kernel or PAM exploits; it's only an added line of defense. Also, this allows or denies authentication for _any user_ via Dom0. That obviously fits with Qubes' status as a single-user system, but it's something to be aware of, I guess. Annoying nag prompts for sudo authorization can be turned off on an individual AppVM basis by editing Dom0's /etc/qubes-rpc/policy/qubes.VMAuth file.

Andrew

[Joonas Lehtonen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi Andrew,

thanks for putting this together, see my commends inline.

> 1) Create the file /etc/qubes-rpc/qubes.VMAuth with the contents

What about using 'qubes.VMsudo' for its name?
(I used "qubes.VMsudo" for it, since I only modified /etc/pam.d/sudo)

> 1) Modify the file /etc/pam.d/system-auth to

What about only modifying /etc/pam.d/sudo (since this is the part that
we actually want to change?)


Additionally, I set the sudo timeout to 0 via:
(to require dom0 consent every time)

/etc/sudoers:

Defaults env_reset,timestamp_timeout=0

and locked down 'su' via pam_wheel:

/etc/pam.d/su:

auth required pam_wheel.so use_uid

while ensuring that the wheel group has no members.


Whould be nice if the default templateVM would come with this out of
the box ;)

-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJTndxeAAoJEG58zmw5nc+vPKwP/AxhKuy9FUK/sUAxP8OTRlIQ
rZuPcxGLNnbmtuqK6bH/XQVP0JHPRGwAMKjy3ui0MQ/j62/LxL7Y6TgfqoI0vlT7
TwFR3Fp9b9L3wUR6uQzswt9o3+dUc/JJQ3OOr2BDrD5aVl8kl5pcBooPnRNwOb0Q
tn6xa1niq3ay5An2oLvZVv6krnsQ2LBMnIttGLM1sqzjJzPBsLaQUd2HR2bkqoyw
hWngs0aGIwbaLlto0lb1Deb7b6u0EA6PABYdkU7hpjehjjCCa3gPUidpmdc048CO
T/e4AiDIQWJl8GvogTT9caRzslBWckhPuctQvDAcuyWBpkRLlOLdl0B+t5wmXZi8
WrZmfZJjBlLYDY9LXIZwv6rY/DBpS4YvNPfEJxJJw7bnODJYlrnZfnOntb3kv+aB
fYAwaz9g9luVNFE3wkYZ+P6znKo7Mk7yvQhlM4z0KCyZchV7GBsLWjPAmDR2uJGA
03bu0zWRgBuN/LHCvVWbPKgn3rp6SSVAbbS6LzF+Hy4FYUSbe/66ul+fLfu/WOPl
T6IXo9d0m66Wv2X4TSjjrro80KjeCmx0EaOzb2Gw09fyzN2oBBFeEnGgL6DcYfYn
eEqb8MOX4HuMz/JK0/nHsOHGqXDCaC/bYM0HrS7EcjGPfhZ9PN75cAHsLlRjf1P2
wkoMFCm6qb9SdnjMnjWM
=lNjs
-----END PGP SIGNATURE-----

[rashc...@gmail.com]

> What we need is a way to break the image files into many segments, or at
least a way to track many segments within a single image file.

I didn't understood from discussion, was it solved or not, but it seems that BigBrotherFS from FUSE tutorial does exactly this. It logs all operations, including write, with offset and size.

Sorry for necrobump.