Archlinux: How to allow the pacman updates thru the proxy update FWsetting

[Tim W]

I have successfully installed the Archlinux template into 3.1rc1.  Most everything seems to be working without issue.   The issue I am having is that I have to open up the firewall settings on the template to allow for updating or installing as the typical proxy updating setting thru the qubes fw setting does not seem to allow the pacman updates connect.  How do I config to allow the updating.  If the addresses have to be static I could not allow mirror repo and only use the offical site.  

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

(copying response to better titled thread)

Pacman needs to be configured to use http proxy at
http://10.137.255.254:8082/ (if
/var/run/qubes-service/updates-proxy-setup exists).

Further details:
https://www.qubes-os.org/doc/software-update-vm/#tocAnchor-1-1-7

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWiuN3AAoJENuP0xzK19csgEAIAIupuN9AMouwRXZQBA1oFfpB
uJnt1rxPe/QKur9LWUWHcW9k27Qu6BcO330UHZzkn6/ptSLwUbMVXLUDxRA1aKEv
mzHRhU9aSlNI0NQ1o4XK0FqWMVT1XJolxbHreQn0N4yVExPkVN7DJi6OQuQeg4hv
uOTgejcAmWbVqUTG+VrUlANgg9PWQ/IvD8uOL55IusAL+/mpOwDmFQRjGOnPomBx
WjhoJeU5iZCpyF1pQUJFY6lPPHqzp+ey93EDSR6OgjniaKG+jN/jflTfJJTFDoXx
kyoJnlfGiJbd9ksqPJVnmaFETSUUIOGarZVByAa9LbXKnVqeqlkzKL6K3q1RTXs=
=n6QE
-----END PGP SIGNATURE-----

[Tim W]

Thanks Marek I will look into it and see if I can get it working.

[Tim W]

Marek,

What is suppose to be in /var/run/qubes-service/updates-proxy-setup?

I checked it and the file is there but I got a empty file with: $ sudo cat /var/run/qubes-service/updates-proxy-setup

Also yum and update check files are also empty in qubes-service directory.

I tired adding to $ /etc/pacman.conf

ProxyServer = 10.137.255.254
ProxyPort = 8082

Unless I am missing something, which is highly possible, I am starting to think, looking at the various forums for info, that pacman is not proxy aware.   Do I need to uncomment  curl or wget as a front downloader as they can have proxy config thru there own conf files?

But it errors with it not recognizing those lines.   This pacman thing while a neat setup is well not so neat.   I did not realize that they still may not have database signing which supposedly was in the process of being done over 3 yrs ago.

I could script a export http_proxy='http://  ftp_proxy='ftp:// to run each time from a shell or put it in profile.d for each use of the shell.   I just feel I am missing something basic as I thought most default package managers allowed proxy setting

[sudod...@gmail.com]

That pacman does not support HTTP proxies surprises me too but it's probably true:
https://wiki.archlinux.org/index.php/Pacman#pacman_does_not_honor_proxy_settings

What about just granting the archlinux template access to your mirror of choice with the firewall? (Thats the 'Deny network access except...' option in the firewall tab)

I don't know how the update proxy works in detail. But it somehow filters HTTP traffic that does not look like yum traffic. Pacman traffic might look different from yum traffic. Does the update proxy still let pacman traffic through? I didn't try.
Another issue will be HTTPS mirrors. There is no way for the update proxy to know whether the traffic is legitimate pacman traffic or not.
Some users might prefer TLS connections to the mirror.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Jan 04, 2016 at 11:54:23PM -0800, Tim W wrote:
>
>
> On Tuesday, January 5, 2016 at 12:12:08 AM UTC-5, Tim W wrote:
> >
> >
> >
> > On Monday, January 4, 2016 at 4:26:21 PM UTC-5, Marek Marczykowski-Górecki
> > wrote:
> >>
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA256
> >>
> >> On Sun, Jan 03, 2016 at 07:37:47PM -0800, Tim W wrote:
> >> > I have successfully installed the Archlinux template into 3.1rc1. Most
> >> > everything seems to be working without issue. The issue I am having
> >> is
> >> > that I have to open up the firewall settings on the template to allow
> >> for
> >> > updating or installing as the typical proxy updating setting thru the
> >> qubes
> >> > fw setting does not seem to allow the pacman updates connect. How do I
> >> > config to allow the updating. If the addresses have to be static I
> >> could
> >> > not allow mirror repo and only use the offical site.
> >>
> >> (copying response to better titled thread)
> >>
> >> Pacman needs to be configured to use http proxy at
> >> http://10.137.255.254:8082/ (if
> >> /var/run/qubes-service/updates-proxy-setup exists).
> >>
> >> Further details:
> >> https://www.qubes-os.org/doc/software-update-vm/#tocAnchor-1-1-7
> >>
> >
> >

> > Thanks Marek I will look into it and see if I can get it working.
> >
>
>
>
> Marek,
>
> What is suppose to be in /var/run/qubes-service/updates-proxy-setup?
>
> I checked it and the file is there but I got a empty file with: $ sudo cat
> /var/run/qubes-service/updates-proxy-setup

Yes, this file is empty. This is flag file, which signal if package
manager should actually use the proxy (if file present), or connect
directly (if absent).

> Also yum and update check files are also empty in qubes-service directory.
>
> I tired adding to $ /etc/pacman.conf
>
> ProxyServer = 10.137.255.254
> ProxyPort = 8082
>
> Unless I am missing something, which is highly possible, I am starting to
> think, looking at the various forums for info, that pacman is not proxy
> aware. Do I need to uncomment curl or wget as a front downloader as they
> can have proxy config thru there own conf files?

I don't know. But when you find the right configuration, take a look at
/usr/lib/qubes/update-proxy-configs, which configures the proxy in
various package managers. You can simply add pacman things there (the
file is in git in qubes-core-agent-linux/network/).

> But it errors with it not recognizing those lines. This pacman thing
> while a neat setup is well not so neat. I did not realize that they still
> may not have database signing which supposedly was in the process of being
> done over 3 yrs ago.

Really? What a shame...

> I could script a export http_proxy='http:// ftp_proxy='ftp:// to run each
> time from a shell or put it in profile.d for each use of the shell. I
> just feel I am missing something basic as I thought most default package
> managers allowed proxy setting

Exporting http_proxy globally isn't a good idea - that updates proxy is
meant to be used only by package manager. The same goes for curl/wget -
it shouldn't be configured globally for proxy, only from package
manager. So if going with curl/wget, probably some command line argument
is the way to go.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWi8w4AAoJENuP0xzK19cs4XIH/35usKEcUvGbpvPfWfrgWj5p
D5+f7CrXSqkv/7Jvswk1pSv0N5/ybdpU5Xr+0rghAMbPDfQl/t05wJPySGi18IiW
+QpLCBlZZhd99GNFXNvd/kwH/0zGo00H3GLPYZh7MZ9AaXRcy+IoJnsYXORLmr6H
+TYfVn+sPdxmKo762Wso/xLwKEiY0aURFa02G0lCa8DAX4exL+bQsIAO033igun8
iAsWacfpJ5e2q3d7ejhMqy6Jj1+HCnkrytHmA9XPmlGmCubxMgsfLg+hl7ui+f3r
ufXWGjvZE/gdfYKqsX0BApqYtD+MMoWCZquk+IcBL9V54oQxhLGlPOOVatyYBzk=
=Qo84
-----END PGP SIGNATURE-----

[Tim W]

Well as was stated by sudod...@gmail.com I too have confirmed that pacman has no way to directly set a proxy.  The only way is thru wget or curl.  I just can not understand that given I have never heard of a PM that did not have a way to config that.


sudod...@gmail.com,

I hate to stick to just one static or a few static mirrors.  The reason for this is archlinux has hundreds of mirrors and you use reflector to set the list periodically to make sure A they are sync'd to the current and B they are the faster ones.   These things change over time.  I do not want to be on a mirror that while still up happens to not be in sync with primary.

Marek,

Is there some way firewall could pull from a file list of ips for allow.  I am guessing likely not but ideally if FW could look at mirrorlist and have them in the allow http ftp that would likely solve it.  Otherwise it looks like the simplest solution is to set a single ip to the direct archlinux repo server in FW for fttp ftp and still disallow everything else and use it.      I personally would never use a app in a template.  The only net connectivity I do is for install and update period. So those and gpg download of associated sig keys is it.

What is your suggestion?

I will keep archlinux and see how it goes but the more I research it there are some real concerns I am starting to have over its security.  While greatly minimize by Qubes still with all the choices out there..... 

I was hoping this would fill what I wanted from gentoo as a template without all the extra ground up work making it a pvm but I think I am still going to work on that. Protage has no issue setting proxy in its portage/make.conf file

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Jan 05, 2016 at 09:21:22PM -0800, Tim W wrote:
> Is there some way firewall could pull from a file list of ips for allow. I
> am guessing likely not but ideally if FW could look at mirrorlist and have
> them in the allow http ftp that would likely solve it. Otherwise it looks
> like the simplest solution is to set a single ip to the direct archlinux
> repo server in FW for fttp ftp and still disallow everything else and use
> it. I personally would never use a app in a template. The only net
> connectivity I do is for install and update period. So those and gpg
> download of associated sig keys is it.

If you want to go that way, you can do that in firewallvm in
/rw/config/qubes-firewall-user-script. There is no way to do that from
Qubes Manager.

> What is your suggestion?

Is is possible to set custom wget/curl command line arguments? If so,
you can specify proxy address there.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWjQiZAAoJENuP0xzK19csy80H/1t2W38b1w2UP5Y7pjN3l+ru
CG3LThKub2452wmpkO7anzH7rcsQOFdUCxVi5Q62JWsu/0Kt6OxCKdtTcQ+izQ74
8KwQciOvbbE5jVjr48Z+QzFFADoM8ieh7VpD8MINFE3P4Wzw9U5zm+eKGklF+dwq
KIxwqChQAtCi85vKJ6UHfXoECwTPDi6COrsL8yfMH4WnHTufsiROLTYGBizYlKfE
v1TS6FeSGVLvKHWos+39HyyxGc1w0ko5N5vQtb+/YI2xOhJb1rm7z3lJRgb+U36J
QjK2F+XXHvtk9WTRriS6KaKJMOQdWnJNSSyw0C7cMvOeOrUJ48AzJCqKLoQTeqw=
=NO4X
-----END PGP SIGNATURE-----

[Tim W]

Ok I think I have found a way to run proxy using a pacman wrapper:  xyne powerpill

I have configure and tested it and it runs fine and Qubes updates proxy allows it all to pass without issue.

THe other benefit is the powerpill allows for much faster downloads as it can download in parallel and  in segements.  Various tests have shown it to be a good bit faster than basic pacman.  The huge benefit here though is we have a  config dedicated to powerpill that allows for proxy's to be set.  


Here is what I did and I combining the edits all together for each file rather than following in the order I had to figure them out.

Edit pacman.conf:  /etc/pacman.conf

Need to first turn on sig required for packages for each repo as the global setting at the top creates issues and needs to be commented out.

Here is how the pacman.conf should look.   I have highlighted those edited and or added in bolded blue.

pacman.conf:

#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
GPGDir      = /etc/pacman.d/gnupg/
HoldPkg     = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta    = 0.7
Architecture = auto

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg   =
#IgnoreGroup =
#NoUpgrade   =
NoUpgrade = /etc/X11/xinit/xinitrc.d/pulseaudio
#NoExtract   =

# Misc options
#UseSyslog
#Color
#TotalDownload
CheckSpace
#VerbosePkgLists

# By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages.
#SigLevel    = Required DatabaseOptional
LocalFileSigLevel = Optional
#RemoteFileSigLevel = Required
 
# NOTE: You must run `pacman-key --init` before first using pacman; the local
# keyring can then be populated with the keys of all official Arch Linux
# packagers with `pacman-key --populate archlinux`.

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#   - URLs will have $repo replaced by the name of the current repo
#   - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.

#[testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

#[community-testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

# If you want to run 32 bit applications on your x86_64 system,
# enable the multilib repositories as required here.

#[multilib-testing]
#Include = /etc/pacman.d/mirrorlist

#[multilib]
#Include = /etc/pacman.d/mirrorlist

# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#SigLevel = Optional TrustAll
#Server = file:///home/custompkgs

[multilib]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

#[qubes]
#commented out as it errors and is not current
#Server = http://olivier.medoc.free.fr/archlinux/pkgs/

[xyne-x86_64]
# Added to download powerpill app
# A repo for Xyne's own projects: http://xyne.archlinux.ca/projects/
# Packages for the "x86_64" architecture.
# Note that this includes all packages in [xyne-any].
SigLevel = Required
Server = http://xyne.archlinux.ca/repos/xyne

#end of file

Set the firewall in archlinux template to allow temporarily all traffic.  I just used the count down time set to 30 min but whatever way just make sure not to leave it open.

Stupidly pacman is set to get package sig keys as optional meaning unless you setup gpg and edit the pacman.conf as above you are open to a unsigned malicious download.

We will just use the root repo server for now and update the mirrorlist below for more regular use.

Next initialize the gpg keyring:

$ sudo pacman-key --init

Now populate that keyring with archlinux master keys:

$ sudo pacman-key --populate archlinux

Make sure to compare the keys with those of on archlinux: https://www.archlinux.org/master-keys/

For more info on pacman key signing: https://wiki.archlinux.org/index.php/Pacman/Package_signing

***Of note archlinux still does not require database files to be signed.  They started converting over to signed DB in 2012 and yet still have not enforced it.  Sad Sad Sad    This is also why the pacman.conf is not set to mandatory DB signing and creates errors if you do.

Go ahead and run a basic update to ensure everything is updated:

$ sudo pacman -Syu

Next install powerpill

$ sudo pacman -S powerpill

Another app to install is reflector.  It scripts mirror updating.  Grabbing the most up to date gen mirror list.  It ranks them by most recently sync'd.  Then ranks them on fastest speed.

$ sudo pacman -S reflector

You can combine the install with:

$ sudo pacman -S powerpill reflector


Next we backup the mirrorlist to run reflector to update the active mirrorlist with those you want to use and that are insync and fastest.  Look to this page for various configs of the list: https://wiki.archlinux.org/index.php/Reflector  

Mirrorlist can be found @ /etc/pacman.d/mirrorlist  back it up to be safe.

$ cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.bkup

Now run whatever reflector string that gives you the mirrorlist you would like to use.

Example for someone want the 5 fastest synced mirrors:

$ sudo reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist

The above ranks all the most update and sorts then in the 5 fastest.

Update the repo databases:

$ sudo pacman -Syy

Configure powerpill file to use the Qubes proxy:

Qubes Proxy: 10.137.255.254:8082

Edit powerpill.  (powerpill no longer uses a .conf file it used the following):

/etc/powerpill/powerpill.json


Part of powerpill download is aria2  in the powerpill.json file you will see the aria2 section using the args arguements
You need to add to the bottom of that section:  (the " " and , are needed and should be included in the string below:

"--all-proxy=10.137.255.254:8082",


Here is the copy of the powerpill.json file with the additions in bolded blue

powerpill.json:

{
  "aria2": {
    "args": [
      "--allow-overwrite=true",
      "--always-resume=false",
      "--auto-file-renaming=false",
      "--check-integrity=true",
      "--conditional-get=true",
      "--continue=true",
      "--file-allocation=none",
      "--log-level=error",
      "--max-concurrent-downloads=100",
      "--max-connection-per-server=5",
      "--min-split-size=5M",
      "--remote-time=true",
      "--show-console-readout=true",
      "--all-proxy=10.137.255.254:8082"   
    ],
    "path": "/usr/bin/aria2c"
  },
  "pacman": {
    "config": "/etc/pacman.conf",
    "path": "/usr/bin/pacman"
  },
  "pacserve": {
    "server": null
  },
  "powerpill": {
    "select": true,
    "reflect databases": false
  },
  "reflector": {
    "args.unused": [
      "--protocol",
      "http",
      "--latest",
      "50"
    ]
  },
  "rsync": {
    "args": [
      "--no-motd",
      "--progress"
    ],
    "db only": true,
    "path": "/usr/bin/rsync",
    "servers": []
  }
}

--------------------------------

Time to test the config.  As powerpill is a pacman wrapper you can pass the same cmds used in pacman to powerpill.

First make sure that the archlinux firewall settings are now back to the only things that are checked are:

Deny Network Access Except: checked
All connections to Updates Proxy: checked

Now run a basic update command:

$ sudo powerpill -Syu


You should get a output similar to this:


archlinux% sudo powerpill -Syu                                                 

01/07 02:01:12 [NOTICE] GID#907683b79b918aea - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db

01/07 02:01:12 [NOTICE] GID#3ad61df1a92605a5 - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db.sig

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db.sig

01/07 02:01:12 [NOTICE] GID#190847ee8efbf461 - Download has already completed: /var/lib/pacman/sync/multilib.db

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/multilib.db

01/07 02:01:12 [NOTICE] GID#048356b3cc7d9185 - Download has already completed: /var/lib/pacman/sync/core.db

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/core.db
[DL:1.4MiB][#7fd54b 864KiB/3.1MiB(26%)][#68c1c0 672KiB/1.7MiB(38%)]                                         
01/07 02:01:14 [NOTICE] Download complete: /var/lib/pacman/sync/extra.db

01/07 02:01:14 [NOTICE] Download complete: /var/lib/pacman/sync/community.db

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
907683|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db
3ad61d|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db.sig
190847|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
048356|OK  |       0B/s|/var/lib/pacman/sync/core.db
68c1c0|OK  |   1.1MiB/s|/var/lib/pacman/sync/extra.db
7fd54b|OK  |   1.6MiB/s|/var/lib/pacman/sync/community.db

Status Legend:
(OK):download completed.
:: Starting full system upgrade...
 there is nothing to do
archlinux%

Cheers,

Tim