qubes-updates-cache, a Squid-based package update cache
213 views
Skip to first unread message
Rusty Bird
Jun 6, 2016, 7:14:04 PM6/6/16
to qubes...@googlegroups.com
Hi folks,
Updating my various templates was getting a little annoying with all the
redundant network traffic, so I set up a qubes-updates-cache modeled on
qubes-updates-proxy, using standalone Squid (no Apache etc. involved).
https://github.com/rustybird/qubes-updates-cache
One caveat: If you want to use this, you need to (run a provided script
to) modify your client templates' repo configuration URLs so that they
explicitly point to the cache, and in some cases to a specific mirror
instead of the mirror rotation, e.g.
Before: http://download.fedoraproject.org/...
After: http://10.137.255.254:8083/https://dl.fedoraproject.org/...
As you can see, the advantage here is being able to download over HTTPS
(for privacy, not integrity) while still caching the downloaded files,
without any convoluted MitM setup. The provided sed script switches
several of the standard Fedora/Debian/Whonix repositories to HTTPS.
The downside is that those modified URLs may be overwritten during some
system updates, at which point you'd have to rerun the sed script; which
should be idempotent though. Eventually, a dnf/apt plugin modifying URLs
"just in time" without clobbering configuration files would be good.
Installation
Create a new ProxyVM, which currently should be based on Fedora 23
(minimal is okay). Ensure it has a netvm and enable the updates cache
service:
[dom0] $ qvm-create --proxy --label red --template fedora-23 squidp
[dom0] $ qvm-prefs --set squidp netvm default # or sys-whonix etc.
[dom0] $ qvm-service squidp --enable qubes-updates-cache
Copy this directory (containing the README you're reading) into your new
ProxyVM's template, carefully inspect the install-server script there
and:
[squidp's template] # dnf install squid
[squidp's template] # ./install-server
[squidp's template] # poweroff
For each of the client VMs whose package updates you want to cache (in
this example, a Debian template), copy the install-client script into
the client, carefully inspect it there -- the comments explain what it
does -- and:
[debian-8] # ./install-client
[debian-8] # poweroff
Disable the (non-caching) updates proxy setup services on the client,
and make your new ProxyVM the client's netvm:
[dom0] $ qvm-service debian-8 --disable updates-proxy-setup
[dom0] $ qvm-service debian-8 --disable yum-proxy-setup
[dom0] $ qvm-start squidp
[dom0] $ qvm-prefs --set debian-8 netvm squidp
If, like in this example, your client is a template (as opposed to a
StandaloneVM), then change its firewall settings to deny absolutely all
access except TCP connections to 10.137.255.254:8083:
[dom0] $ qvm-firewall debian-8 --policy deny
[dom0] $ qvm-firewall debian-8 --icmp deny
[dom0] $ qvm-firewall debian-8 --dns deny
[dom0] $ qvm-firewall debian-8 --yum-proxy deny
[dom0] $ qvm-firewall debian-8 --add 10.137.255.254 tcp 8083
[dom0] $ qvm-firewall debian-8 --list --numeric # all good?
That's it! Up to 4 GiB of package updates will be cached to squidp's
volatile storage in /var/lib/qubes/vm-updates/. If you really want to
keep them across reboots, bind mount a directory in /rw/ owned by
squid:squid to that destination.
Rusty
Updating my various templates was getting a little annoying with all the
redundant network traffic, so I set up a qubes-updates-cache modeled on
qubes-updates-proxy, using standalone Squid (no Apache etc. involved).
https://github.com/rustybird/qubes-updates-cache
One caveat: If you want to use this, you need to (run a provided script
to) modify your client templates' repo configuration URLs so that they
explicitly point to the cache, and in some cases to a specific mirror
instead of the mirror rotation, e.g.
Before: http://download.fedoraproject.org/...
After: http://10.137.255.254:8083/https://dl.fedoraproject.org/...
As you can see, the advantage here is being able to download over HTTPS
(for privacy, not integrity) while still caching the downloaded files,
without any convoluted MitM setup. The provided sed script switches
several of the standard Fedora/Debian/Whonix repositories to HTTPS.
The downside is that those modified URLs may be overwritten during some
system updates, at which point you'd have to rerun the sed script; which
should be idempotent though. Eventually, a dnf/apt plugin modifying URLs
"just in time" without clobbering configuration files would be good.
Installation
Create a new ProxyVM, which currently should be based on Fedora 23
(minimal is okay). Ensure it has a netvm and enable the updates cache
service:
[dom0] $ qvm-create --proxy --label red --template fedora-23 squidp
[dom0] $ qvm-prefs --set squidp netvm default # or sys-whonix etc.
[dom0] $ qvm-service squidp --enable qubes-updates-cache
Copy this directory (containing the README you're reading) into your new
ProxyVM's template, carefully inspect the install-server script there
and:
[squidp's template] # dnf install squid
[squidp's template] # ./install-server
[squidp's template] # poweroff
For each of the client VMs whose package updates you want to cache (in
this example, a Debian template), copy the install-client script into
the client, carefully inspect it there -- the comments explain what it
does -- and:
[debian-8] # ./install-client
[debian-8] # poweroff
Disable the (non-caching) updates proxy setup services on the client,
and make your new ProxyVM the client's netvm:
[dom0] $ qvm-service debian-8 --disable updates-proxy-setup
[dom0] $ qvm-service debian-8 --disable yum-proxy-setup
[dom0] $ qvm-start squidp
[dom0] $ qvm-prefs --set debian-8 netvm squidp
If, like in this example, your client is a template (as opposed to a
StandaloneVM), then change its firewall settings to deny absolutely all
access except TCP connections to 10.137.255.254:8083:
[dom0] $ qvm-firewall debian-8 --policy deny
[dom0] $ qvm-firewall debian-8 --icmp deny
[dom0] $ qvm-firewall debian-8 --dns deny
[dom0] $ qvm-firewall debian-8 --yum-proxy deny
[dom0] $ qvm-firewall debian-8 --add 10.137.255.254 tcp 8083
[dom0] $ qvm-firewall debian-8 --list --numeric # all good?
That's it! Up to 4 GiB of package updates will be cached to squidp's
volatile storage in /var/lib/qubes/vm-updates/. If you really want to
keep them across reboots, bind mount a directory in /rw/ owned by
squid:squid to that destination.
Rusty
Iestyn Best
Jun 7, 2016, 10:16:04 PM6/7/16
to qubes-devel, rust...@openmailbox.org
Hi Rusty,
Thank you for all your work, sounds interesting.
I am not currently setup to use Qubes-OS at the moment but I did use it in the past and looking to switch back as soon as I can.
I do remember how annoying it could be downloading the same packages over and over for the different VMs.
Thanks again.
Regards,
Iestyn Best
Rusty Bird
Jun 7, 2016, 10:35:26 PM6/7/16
to qubes-devel, Iestyn Best
Hi Iestyn,
> I do remember how annoying it could be downloading the same packages over
> and over for the different VMs.
And imagine using Qubes on multiple notebooks, OMG... I might even set
up a second-level Squid cache on a Raspberry Pi or something.
Rusty
> I do remember how annoying it could be downloading the same packages over
> and over for the different VMs.
up a second-level Squid cache on a Raspberry Pi or something.
Rusty
Iestyn Best
Aug 2, 2016, 11:48:47 PM8/2/16
to qubes-devel, rust...@openmailbox.org
Hi Rusty,
I just tried to set this up on my current install of R3.2 rc-2 and it seems to be half working.
Your install guide here is different to your install guide on github. Which should I follow?
If I am to follow this one, I cannot find the files to download to perform the install on both the proxyVM and the client VM.
I followed most of the guide except running the client install script and I seem to be unable to connect to repositories from a debian template.
Please let me know if there is anything I should do.
Regards,
Iestyn Best
Rusty Bird
Aug 3, 2016, 4:35:29 AM8/3/16
to qubes-devel, Iestyn Best
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi Iestin,
> Your install guide here is different to your install guide on
> github. Which should I follow?
Oh sorry, I knew I should have sent an update here... The qubes-devel
post is outdated, please follow the instructions in the GitHub repo.
> I followed most of the guide except running the client install
> script and I seem to be unable to connect to repositories from a
> debian template.
If you've followed the old instructions, this is how you can undo the
last few steps:
[dom0] $ qvm-service <client-template> --default updates-proxy-setup
[dom0] $ qvm-service <client-template> --default yum-proxy-setup
[dom0] $ qvm-firewall <client-template> --del 10.137.255.254 tcp 8083
[dom0] $ qvm-firewall <client-template> --yum-proxy allow
Then restart <client-template>.
Does that fix the connection problems for you? I haven't tried
qubes-updates-cache on R3.2 yet, but it should work. If not, try
running "sudo journalctl -u qubes-updates-cache" in the squidp VM,
maybe it says something interesting.
Rusty
-----BEGIN PGP SIGNATURE-----
iQJ8BAEBCgBmBQJXoazBXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfnjEP/RAlOS4apnsPg24rch4fpH25
VdAREBdZd2QZRO9/q/jKJiW5Z0hF7kkIkmlMUl7IxZiOm+4yWtdpThH4hH3wfe7X
/RMCRrpHzJNptNelnbjywruVBv2qnHFxdiZ6jWxAIJmhdInfRGbtpnfjPDKBNIRn
liD2kBMKS+3lsD9wj3EWOaaYVjPmfMv6zdLKp+H2WLgo5N3OTeNOrOAKS4Kjh65N
210lmK4Gov2WqbCeUG8UV1vWpHn0RF2rL+1L6NdUZAE8YxggnHypvHklZ1R/tPFY
WqCEzL1Arqg5QmWotqjVu0iUty3XikjHUv9pbQnYUu9iABuvJprxealrc7BFMqj2
26eqvLt7MuU2so73GAnrKkmHL/ZtYSfFbiRqDFeIaSh299Fg3cIaBdfU3x41CW42
C9H139fVR7y0tMYy0cY6Fc8dZVCE4ZnKp1jWHx7TBRzFeOniETtnB+RjlvBRxaEN
MpZNfoHwAvHGygLLhe0KDrR6UvdypicLx3U9M/Vl/EzUO8SEkqRJSKvpKGmTL0lc
Tmy11M9z/8BT1QcyBk9DG0rOdcHIEP/nk0LS+ExoQm73m9XXZE6oBIwsnZ9R8ztM
nLfr0qwOf3zcL9Zd70+5OBKBTUWVK6OyILG86TZwWAlbepV+lqqG29rq1tt9KXha
6KL3aoUIJPXcYLsOjcUU
=VwtI
-----END PGP SIGNATURE-----
Hash: SHA512
Hi Iestin,
> Your install guide here is different to your install guide on
> github. Which should I follow?
post is outdated, please follow the instructions in the GitHub repo.
> I followed most of the guide except running the client install
> script and I seem to be unable to connect to repositories from a
> debian template.
last few steps:
[dom0] $ qvm-service <client-template> --default updates-proxy-setup
[dom0] $ qvm-service <client-template> --default yum-proxy-setup
[dom0] $ qvm-firewall <client-template> --del 10.137.255.254 tcp 8083
[dom0] $ qvm-firewall <client-template> --yum-proxy allow
Then restart <client-template>.
Does that fix the connection problems for you? I haven't tried
qubes-updates-cache on R3.2 yet, but it should work. If not, try
running "sudo journalctl -u qubes-updates-cache" in the squidp VM,
maybe it says something interesting.
Rusty
-----BEGIN PGP SIGNATURE-----
iQJ8BAEBCgBmBQJXoazBXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfnjEP/RAlOS4apnsPg24rch4fpH25
VdAREBdZd2QZRO9/q/jKJiW5Z0hF7kkIkmlMUl7IxZiOm+4yWtdpThH4hH3wfe7X
/RMCRrpHzJNptNelnbjywruVBv2qnHFxdiZ6jWxAIJmhdInfRGbtpnfjPDKBNIRn
liD2kBMKS+3lsD9wj3EWOaaYVjPmfMv6zdLKp+H2WLgo5N3OTeNOrOAKS4Kjh65N
210lmK4Gov2WqbCeUG8UV1vWpHn0RF2rL+1L6NdUZAE8YxggnHypvHklZ1R/tPFY
WqCEzL1Arqg5QmWotqjVu0iUty3XikjHUv9pbQnYUu9iABuvJprxealrc7BFMqj2
26eqvLt7MuU2so73GAnrKkmHL/ZtYSfFbiRqDFeIaSh299Fg3cIaBdfU3x41CW42
C9H139fVR7y0tMYy0cY6Fc8dZVCE4ZnKp1jWHx7TBRzFeOniETtnB+RjlvBRxaEN
MpZNfoHwAvHGygLLhe0KDrR6UvdypicLx3U9M/Vl/EzUO8SEkqRJSKvpKGmTL0lc
Tmy11M9z/8BT1QcyBk9DG0rOdcHIEP/nk0LS+ExoQm73m9XXZE6oBIwsnZ9R8ztM
nLfr0qwOf3zcL9Zd70+5OBKBTUWVK6OyILG86TZwWAlbepV+lqqG29rq1tt9KXha
6KL3aoUIJPXcYLsOjcUU
=VwtI
-----END PGP SIGNATURE-----
Andrew
Aug 3, 2016, 12:29:20 PM8/3/16
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Rusty Bird:
Hash: SHA512
test it, but it's a great idea.
It seems to me that most of the problems described in your post are
the results of hacking this on top of a system that's trying to fight
you. If Qubes integrates this, then everything should Just
Work (tm), right?
Are there any arguments for *not* integrating this
into the base system?
Andrew
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJXohvTAAoJEKJgxxXeQHZV2KMQAIRtz8qRD0NPAyRnDM3Z4jIC
auzeQoAIFMiE1VqYhUjTQMHHmmTDo/UUoTG7G4V0pty4nUtM7aqIcBi82N+vVzZO
Tfxh4xXvdPcYZt76XiYb2V0OnG92OkEMUmErpbkatTPUbN0AovANFcCMVus9pfg8
c1HCMzRD/h3iPuQMkBml8ouFODH/Hj0j5sklt8xQe6u/xZLW9UwrSxAXh9O4Fw5C
iIBPYuCwnMl75ug1wETzTRqiBOn4t7n6xhNj0MwOxP3nu8pBUXM88y1OEzOLh07G
NCdXo1ozkO2PRaeuiiZnzFXOWSCdXEY1VxuFLOj1cBxGn1dkFCQNlEMAvnzmNSft
oh/3n5Ue/6BKyG5+4xTkD4FsXbkrgzLH9BtYi7CNNH20rhasJ4pGYSGQ3AmoPUH1
pm+feYe2ZjKWWOKyqc3F2uLEAq9tz1nnLoquAQ2KwJrZhDCOVoXL92OOac0IjpM2
eXpW9rQ8H4u2TkA43amWsz8xPr1yQAMKAb3aY/qmeBWA3q6Ig5SKmiOvtl0gf5z9
ZGe6sjBac1dn5Tt2xhvCBBs85JcO8Y18nhweF8w3Osu2OrZL/d10Lnbg2D969pzE
42uBaGmSp/+QsMlRjEXYbER0QaAe6VARWl3nS02lZ3bxo0lagGPGRbjMt1tklDDM
EteZBfWBJZrnPFq8mZYI
=ZQn9
-----END PGP SIGNATURE-----
Andrew David Wong
Aug 3, 2016, 10:09:41 PM8/3/16
to Andrew, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hash: SHA512
On 2016-08-03 09:29, Andrew wrote:
> I don't know how I missed this before, but great work. I have yet to test
> it, but it's a great idea.
>
> It seems to me that most of the problems described in your post are the
> results of hacking this on top of a system that's trying to fight you. If
> Qubes integrates this, then everything should Just Work (tm), right?
>
> Are there any arguments for *not* integrating this into the base system?
>
> Andrew
>
Not as far as I know. I think Marek just needs a chance to review the code,
> I don't know how I missed this before, but great work. I have yet to test
> it, but it's a great idea.
>
> It seems to me that most of the problems described in your post are the
> results of hacking this on top of a system that's trying to fight you. If
> Qubes integrates this, then everything should Just Work (tm), right?
>
> Are there any arguments for *not* integrating this into the base system?
>
> Andrew
>
then I think it'll be time to integrate it into Qubes.
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJXoqPbAAoJENtN07w5UDAwLZYP/RB941i2cW3DHY2jtsPz7pPC
BiN7gAtu5s1r+O1kyt1MZ8Zx4LtwGQpRhDyfLcVDhWx54w/0GrlmQPT4ufKcYhgM
JBzZImJ//a+wfBTZlF3GjvU7uNHMT06Zk15xjxON+VCgnSFf5ZWYEJqizFzK+NoE
s2Qk/q6h7BaGZO6dmst6v6kjl/qVhcOvuH/I2JfEP33Cyb+5lgDo50Og+EsE16zi
5A2k43a/vaXIf/WdTC0JRby0ZJRNQc/CvCybZ36IxeahEDraUpG35H5ISqaLWCk6
ikonn37aouG19gv0YTRSMvhGt32UuqvapBBsFYscU+Wgl4+TGlj5IsJTHtHDGhNz
OIKWOc0ceXUx7v+GlNILwMSR4pRmNZ7plhxxiiQEByUBKmJc8+ly2J6qm/t8Jb8x
v/PDyGEgAv54VKq1gG4eshtuwlVmMvwWUWQ0rOjxtFhwZJJzJe5F+PZWhvjcZgZ9
6he9+TCDcROCrcmpE8HKZEux13Hq65yB/Yky6jfuRqb+bB7DGI2Nz/8VCMiwvZe6
0i0HzpsW9Rm23X/mPxCuSJgI0XZwHXjXEBjHeo+zzQhQK5/HjlIIMkgTeO06xbiC
I5osoPqhz+7mgH9X4CUwtwenSB7FSrA0XM+84+ebfmd/roR+d9bXOBjUx3wJGOG6
C14VfmKN2/SRMxDOEKVX
=jEkj
-----END PGP SIGNATURE-----
Iestyn Best
Aug 8, 2016, 7:21:16 PM8/8/16
to qubes-devel, vfre...@gmail.com, rust...@openmailbox.org
Hi Rusty,
I tried running those commands on 2 of my client VMs (1 fedora-23 and 1 debian-8) but it still did not seem to work.
I setup the cache VM from a debian-8 template, could that make any difference. I did notice that you updated your scripts to support debian installs.
I tried retrieving the journal for qubes-updates-cache but it only had one line that described the timeframe for the journal.
Please let me know your thoughts.
@Andrew, it would be interesting to know your results with configuring this. Please let us know when you have a chance to test.
Regards,
Iestyn Best
Rusty Bird
Aug 14, 2016, 9:25:59 AM8/14/16
to qubes-devel, Iestyn Best
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi Iestyn,
Hash: SHA512
> I tried retrieving the journal for qubes-updates-cache but it only
> had one line that described the timeframe for the journal.
Rusty
-----BEGIN PGP SIGNATURE-----
iQJ8BAEBCgBmBQJXsHFdXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfnlUP/3K9WJY7iRcuxos+ub7DD0+y
xot5Z0HMQYM+j8tvhni+yIPK3zaiqEoh0S1LWoRbDJToCunojFmcYG7mgniWy4wq
+Z75ZTjfse203P3mF2/XaXv/QDWmkgMSN5Jc4Y68sNod17GcoU3FrYJ1UcReHLrj
YsglL6+0la+xo/KWkLTXa+7jVF3RQ+r1g+DT7huBrbsD6PYQTjoVjEM6He1m45ty
e4oE6UDoLr4gQUyeO4ib4+yVEMdEa8Ik6i3qV+uADInypl6Ub9FMBP9Lh393REaZ
cXxPxPM6x9izVVCDqkiuIHcL2zYDFP8kxrY7mWSjwTiGYvLDTDxYjMZ56GQs93yP
36Kumdwy1y5u54I0+ZuKMWOW/voLHQ/sPrwqQZ/8FvQ6CBrK4rzOuaquaxPTFbay
QknivCOcUVm3zbkXR+8Xoh3Gg/HcnVjDC08mqaukYn4kCU2EATWmruEYT0Ye3oxr
JYgrTO9pbNSytGPQvBS2kO6JMc1uYx+++kb6uN2tz0fdS+VizyyLuYvhdtqT44io
QZ+GdMt9ROj3poaMiI+q5mkSo9UZQiuofbtZeQUYK0ZtYzEOsny63bVigZLaEYlV
eSwc45NMmGM8aepRknSW5hrH9I3kg/Iyxt7Ks8dkbe1p5zSxH4b3jd9A4zT970Ui
94Jx/5bSKMixRqtfJLoO
=nVzK
-----END PGP SIGNATURE-----
Reply all
Reply to author
Forward
0 new messages