bugreport: backup doesn't accept symlinks

[Oleg Artemiev]

Hello.

I've made a storage split for hdd/ssd storage - appvms private.img
were moved to hdd and simlink is placed in each appvm folder.


When private.img has been moved to some other place and the simlink is
created there the VM is running OK, but the backup shows wrong size
for VM with such an image - 4.0Kib instead of real size of private.img
.


Also after selecting such a VM in a backup the backup finally fails to
run showing no output in its window.

Additional technical information:

pwd
/var/lib/qubes/appvms/myvmname

ls myvmname
total 111940
drwxrwxr-x 2 myuid myuid 4096 Jan 28 01:44 apps
drwxrwxr-x 2 myuid myuid 4096 Jan 28 01:44 apps.icons
lrwxrwxrwx 1 myuid myuid 54 Nov 1 23:54 icon.png ->
/usr/share/icons/hicolor/128x128/devices/appvm-red.png
-rw-rw-r-- 1 myuid myuid 918 Nov 1 23:58 myvmname.conf
lrwxrwxrwx 1 root root 61 Jan 28 03:26 private.img ->
/mypath/mypath2/myvmname/private.img
-rw-rw-r-- 1 myuid myuid 11811160064 Nov 2 00:11 volatile.img
-rw-rw-r-- 1 myuid myuid 56 Nov 1 23:54 whitelisted-appmenus.list

ls -ld /mypath/
drwxr-xr-x. 23 myuid myuid 4096 Jan 28 00:34 /mypath/

ls -ld /mypath/mypath2/
drwxr-xr-x 10 myuid myuid 4096 Jan 28 03:25 /mypath/mypath2/

ls -ld /mypath/mypath2/myvmname/
drwxr-xr-x 2 root root 4096 Jan 28 03:26 /mypath/mypath2/myvmname/

ls -ld /mypath/mypath2/myvmname/private.img
-rw-rw---- 1 myuid myuid 2147483648 Nov 2 00:11
/mypath/mypath2/myvmname/private.img


--
Bye.Olli.
gpg --search-keys grey_olli
Key fingerprint = 9901 6808 768C 8B89 544C 9BE0 49F9 5A46 2B98 147E
Blog keys (mostly in russian): http://grey-olli.livejournal.com/tag/

[Axon]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I'm not sure if this helps at all, but I have several AppVMs where
I've moved the entire VM directory (not just private.img) to a
secondary drive and created a symlink back to /var/lib/qubes/appvms. I
haven't noticed any of the backup problems you mention with respect to
these AppVMs.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJUyHk5AAoJEJh4Btx1RPV86IUQAKwlPuBfXdCl9L8Bet/yvHct
pZAjT2RrC6TVwJfLYMoo6+b59fB/gvyb3a/w2JrotBir6nSqGAf5JaYFegQcX16M
6GUlpgllwWaWQgdFRBWqBnqyLl98yrazW4GP1p7KVr9OqLzy3VLGX2fyGlJ/LuTD
+cGcXCm3BCgqgpO0ashl4cYjc931qZBIc0TI20xdNYngEFpA0o9dxCHNiiqL2978
eSoKXcmHIE0Wf7jJHqSkDyBj50M8nLe6nMyeGHhHo2wsZPaCUrVuT3mKsPhipFrq
ZEAV/MM7STtLpO/X8RMa/D/D/qN00KwtjHUPwT9iMCO+69Zsn/XBBOQo6XKWtFFV
zW42tZCxgHVpIndySt6Oqu3+FWZDwREzYQsnoTDTL8UQIfEKYFE7y9DvO9bukVzQ
2Udny77YLVJPHbuBXZQROCKd7quKUhyay6p57fl9NjB938LZixmJQTvtwSVpVkQP
uWpq7R82e5r/unT2diJtxzO4mtXfwF7PjxWAqAuEO0O0oNvckQVEZjLo4Wqvpmb+
XyLhLn/a+NC3pVi8ddwH/mitXW+OBZ9zJ9RX0cwgsCD9LYcECC4VulgyGGWpEl69
o2IPcQKzo60//Mfu/awtQTS6meWqhuTk13LKxsI83TBCdVo6To6sTqk2W/88Ew77
kESBvuV29omFJuORgqa+
=ive+
-----END PGP SIGNATURE-----

[Oleg Artemiev]

>> I've made a storage split for hdd/ssd storage - appvms private.img
>> were moved to hdd and simlink is placed in each appvm folder.
>>
>>
>> When private.img has been moved to some other place and the simlink
>> is created there the VM is running OK, but the backup shows wrong
>> size for VM with such an image - 4.0Kib instead of real size of
>> private.img .
>>
>>
>> Also after selecting such a VM in a backup the backup finally fails
>> to run showing no output in its window.
>>

[]

> I'm not sure if this helps at all, but I have several AppVMs where
> I've moved the entire VM directory (not just private.img) to a
> secondary drive and created a symlink back to /var/lib/qubes/appvms. I
> haven't noticed any of the backup problems you mention with respect to
> these AppVMs.

thank you for information, but I need main image be on ssd and private
imge be on hdd -
just to make ssd live longer. If I move the dir - both will get the same place.

[Marek Marczykowski-Górecki]

On Wed, Jan 28, 2015 at 10:33:28PM +0400, Oleg Artemiev wrote:
> >> I've made a storage split for hdd/ssd storage - appvms private.img
> >> were moved to hdd and simlink is placed in each appvm folder.
> >>
> >>
> >> When private.img has been moved to some other place and the simlink
> >> is created there the VM is running OK, but the backup shows wrong
> >> size for VM with such an image - 4.0Kib instead of real size of
> >> private.img .
> >>
> >>
> >> Also after selecting such a VM in a backup the backup finally fails
> >> to run showing no output in its window.
> >>
> []
>
> > I'm not sure if this helps at all, but I have several AppVMs where
> > I've moved the entire VM directory (not just private.img) to a
> > secondary drive and created a symlink back to /var/lib/qubes/appvms. I
> > haven't noticed any of the backup problems you mention with respect to
> > these AppVMs.
> thank you for information, but I need main image be on ssd and private
> imge be on hdd -
> just to make ssd live longer. If I move the dir - both will get the same place.

What do you mean by "main image"? VM root filesystem is from template,
so you can simply keep templates on your SSD. Unless you are using
standalone VMs (not based on any template) - then indeed such VMs have
both root.img and private.img in the same directory.

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[cprise]

On 01/28/15 13:33, Oleg Artemiev wrote:

I've made a storage split for hdd/ssd storage - appvms private.img
were moved to hdd and simlink is placed in each appvm folder.


When private.img has been moved to some other place and the simlink
is created there the VM is running OK, but the backup shows wrong
size for VM with such an image - 4.0Kib instead of real size of
private.img .


Also after selecting such a VM in a backup the backup finally fails
to run showing no output in its window.

[]

I'm not sure if this helps at all, but I have several AppVMs where
I've moved the entire VM directory (not just private.img) to a
secondary drive and created a symlink back to /var/lib/qubes/appvms. I
haven't noticed any of the backup problems you mention with respect to
these AppVMs.

thank you for information, but I need main image be on ssd and private
imge be on hdd -
just to make ssd live longer. If I move the dir - both will get the same place.

FWIW, SSDs no longer appear to have some of the reliability quirks they had circa 2009. The units made today can even outlast HDs.

[Oleg Artemiev]

On Wed, Jan 28, 2015 at 9:39 PM, Marek Marczykowski-Górecki
<marm...@invisiblethingslab.com> wrote:
> On Wed, Jan 28, 2015 at 10:33:28PM +0400, Oleg Artemiev wrote:
>> >> I've made a storage split for hdd/ssd storage - appvms private.img
>> >> were moved to hdd and simlink is placed in each appvm folder.
>> >>
>> >>
>> >> When private.img has been moved to some other place and the simlink
>> >> is created there the VM is running OK, but the backup shows wrong
>> >> size for VM with such an image - 4.0Kib instead of real size of
>> >> private.img .
>> >>
>> >>
>> >> Also after selecting such a VM in a backup the backup finally fails
>> >> to run showing no output in its window.
>> >>
>> []
>>
>> > I'm not sure if this helps at all, but I have several AppVMs where
>> > I've moved the entire VM directory (not just private.img) to a
>> > secondary drive and created a symlink back to /var/lib/qubes/appvms. I
>> > haven't noticed any of the backup problems you mention with respect to
>> > these AppVMs.

see below about most possible cause of problem - symlink was owned by root.

>> thank you for information, but I need main image be on ssd and private
>> imge be on hdd -
>> just to make ssd live longer. If I move the dir - both will get the same place.

In the appvm directory there're also some stuff that's rewrited rarely:

apps/* apps.icons/* *.xml *.png *.list

I'd prefer these to stay on ssd. Moving only volatile.img and private.img .

> What do you mean by "main image"? VM root filesystem is from template,
> so you can simply keep templates on your SSD. Unless you are using
> standalone VMs (not based on any template) - then indeed such VMs have
> both root.img and private.img in the same directory.

Thanks for reminding. I meant standalone VMs at the same time wrongly
talking about AppVMs.

As about bugreport:

clarification: the backupping fails only when symlinks has been made by root.

If the symlink is owned by the user that owns the image - image size
to backup is still displayed incorrectly as size of symlink,
but backup works okay.

[Marek Marczykowski-Górecki]

And the backup contains only a *symlink*, not the actual file... So for
now you need to backup your HDD manually.
Perhaps we will solve this problem in the future, but simply following the
symlinks isn't the best option, because we need to properly backup
symlinks in some places (at least icon.png).

[Axon]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Marek Marczykowski-Górecki wrote:
> On Fri, Jan 30, 2015 at 02:30:45AM +0400, Oleg Artemiev wrote:

[...]

>> clarification: the backupping fails only when symlinks has been
>> made by root.
>>
>> If the symlink is owned by the user that owns the image - image
>> size to backup is still displayed incorrectly as size of
>> symlink, but backup works okay.
>
> And the backup contains only a *symlink*, not the actual file... So
> for now you need to backup your HDD manually. Perhaps we will solve
> this problem in the future, but simply following the symlinks isn't
> the best option, because we need to properly backup symlinks in
> some places (at least icon.png).
>

Maybe I'm misunderstanding you, Marek, but I just tried restoring one
of my backed up AppVMs, the directory of which was symlinked to a
secondary drive at the time of the backup, and the restored AppVM
contains all of my user data. (So, it appears the backup was not only
a symlink; rather, it contained my actual data). Does the issue you
are describing arise when only the private.img file is symlinked
(rather than the whole AppVM directory being symlinked)?
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJUy9rmAAoJEJh4Btx1RPV8NQQP/jp52yp0aW+SKOpuBrNzYvYv
xcQWB6s73J9Yg7W/UFmuMiLjcKS92wy1U29WygcmoczsPkme/bb6q9rprmHkCt5U
2khRhslYmsYgR2fLu6ZX/uAmcxeT15JI1Laa40xeru1+oXAEMZBc3rz074rJyyfE
jfdZ8eqWOc2IoZC7bGOvrzASvZWcaQ8iWXeqaTorIZAz1+F1GVx9LXEaVDCGvnG8
8q2EmZx3nZf0ADLw69joknpkPyBXoiBdqNjy0JBR+zCNBtPR8xtFrJ3+eqTlX49o
EVULdLkuns/qCmi2txABFqAYCWFf9BwzPwIiDakzNW/F1ZANduyx+oPCNcbynJQw
iXg10h6oLWmHAQX3mUasjRfsBdGsB+z57hU1PaoYZ5wzhUXRN/NHyfAVXdGHCbvX
ZJJWpF8GziA7AsM2ia/zGUk0WOmT2PukQJXEjm8y2blBjWkPSNfMvNPIVipsuX/V
dvA2Jqhw44/jv5rn9h1oEvz2wPo+bEjfZ6XD8x6D72917WhYv7eSqINS1y4V8AV6
rCGMYHCDMB4JXVLYkO6TIjM3spAb2pFdyAt2qwx3/FIcB6twYSPDd96PbI0R3jFU
YA4vcLy72EejoI78XoZtDKQ3gFRRFpLgDcn2OsgsR1kIAMKbIfFiw/uK/RSU1/ni
FpeCFQOvrJ26Zhn3mdiP
=4mTL
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

On Fri, Jan 30, 2015 at 07:26:37PM +0000, Axon wrote:
> Marek Marczykowski-Górecki wrote:
> > On Fri, Jan 30, 2015 at 02:30:45AM +0400, Oleg Artemiev wrote:
> [...]
> >> clarification: the backupping fails only when symlinks has been
> >> made by root.
> >>
> >> If the symlink is owned by the user that owns the image - image
> >> size to backup is still displayed incorrectly as size of
> >> symlink, but backup works okay.
> >
> > And the backup contains only a *symlink*, not the actual file... So
> > for now you need to backup your HDD manually. Perhaps we will solve
> > this problem in the future, but simply following the symlinks isn't
> > the best option, because we need to properly backup symlinks in
> > some places (at least icon.png).
> >
>
> Maybe I'm misunderstanding you, Marek, but I just tried restoring one
> of my backed up AppVMs, the directory of which was symlinked to a
> secondary drive at the time of the backup, and the restored AppVM
> contains all of my user data. (So, it appears the backup was not only
> a symlink; rather, it contained my actual data). Does the issue you
> are describing arise when only the private.img file is symlinked
> (rather than the whole AppVM directory being symlinked)?

Yes, when only private.img is symlinked, only symlink will be stored in
backup. Then the whole directory is symlinked, it can work correctly.
BTW If you try your approach (to symlink a VM directory) on a
*TemplateVM*, there could be the same problem - only symlink will be
saved in the differently.

[Axon]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Ah, I understand now. That's very good to know; thank you!
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJUy/TkAAoJEJh4Btx1RPV8ZXIP/jLz8zaVFxBFO/zAhnLZxUHN
R1I4kstsCkSl2Zg9Guiw8YBrPP388liPdvO2nNJzDgrSAxyBS6vLzpKKW4N+smaE
osHEDGqSG5GDcnwX21q8zv3l39lZP1SINqzisp8dofY9kAMZrwn0Xf/O1osE7EoN
a0C0s85DfH85CTnyc9xV/4elQirx7HaI8iQh89S266mHSiEScsZCEcGIofKqgKDx
j9/bEZePQVJpQPxJIqKk5RiN6u6Luh1DjEtQbDpzdYLNNO8X3WHblu3S2NjyUbU2
/jhtJvITKiSilxTQANX/M5XQAfr9T2zJLasSC0CmRbgTGPBViQAiHh664DZQVqIZ
WMqSQF9R2yaa4XYybTv/8AXRNbbSZK7Nd+2QMWNU9GGH0TNd2rH0qlQodBRdrxD8
l7ExhGdrXLHwXbKRNHvlA/FajJMxbASf7URXzR+yT4PZeGZYaJPlkh4E1S+GMDdh
3kiQQJf3Kt6S9b45hD8NYvIv3wk2muGL5svFvrytdbE5ZgP4QsUBknSfCNNzRwZP
Go2IsvbTGK9G2ubeouVs7p803gZCtIddYOmfbI28z7NhOKHxbKS0Z22WCkMi4ENi
TkMiok0MbP6f45CiFKRjGCFbe+fyeuVYflHlt9rvtqGk2kfWo9+DoRhNOAX8v9Ge
FzAUh//YEBRnge7n72fy
=yVAj
-----END PGP SIGNATURE-----