Google Groupes

Re: [qubes-users] Archlinux: How to allow the pacman updates thru the proxy update FWsetting


Tim W 6 janv. 2016 23:15
Envoyé au groupe : qubes-users


On Wednesday, January 6, 2016 at 7:29:19 AM UTC-5, Marek Marczykowski-Górecki wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Jan 05, 2016 at 09:21:22PM -0800, Tim W wrote:
> Is there some way firewall could pull from a file list of ips for allow.  I
> am guessing likely not but ideally if FW could look at mirrorlist and have
> them in the allow http ftp that would likely solve it.  Otherwise it looks
> like the simplest solution is to set a single ip to the direct archlinux
> repo server in FW for fttp ftp and still disallow everything else and use
> it.      I personally would never use a app in a template.  The only net
> connectivity I do is for install and update period. So those and gpg
> download of associated sig keys is it.

If you want to go that way, you can do that in firewallvm in
/rw/config/qubes-firewall-user-script. There is no way to do that from
Qubes Manager.

> What is your suggestion?

Is is possible to set custom wget/curl command line arguments? If so,
you can specify proxy address there.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWjQiZAAoJENuP0xzK19csy80H/1t2W38b1w2UP5Y7pjN3l+ru
CG3LThKub2452wmpkO7anzH7rcsQOFdUCxVi5Q62JWsu/0Kt6OxCKdtTcQ+izQ74
8KwQciOvbbE5jVjr48Z+QzFFADoM8ieh7VpD8MINFE3P4Wzw9U5zm+eKGklF+dwq
KIxwqChQAtCi85vKJ6UHfXoECwTPDi6COrsL8yfMH4WnHTufsiROLTYGBizYlKfE
v1TS6FeSGVLvKHWos+39HyyxGc1w0ko5N5vQtb+/YI2xOhJb1rm7z3lJRgb+U36J
QjK2F+XXHvtk9WTRriS6KaKJMOQdWnJNSSyw0C7cMvOeOrUJ48AzJCqKLoQTeqw=
=NO4X
-----END PGP SIGNATURE-----




Ok I think I have found a way to run proxy using a pacman wrapper:  xyne powerpill

I have configure and tested it and it runs fine and Qubes updates proxy allows it all to pass without issue.

THe other benefit is the powerpill allows for much faster downloads as it can download in parallel and  in segements.  Various tests have shown it to be a good bit faster than basic pacman.  The huge benefit here though is we have a  config dedicated to powerpill that allows for proxy's to be set.  


Here is what I did and I combining the edits all together for each file rather than following in the order I had to figure them out.

Edit pacman.conf:  /etc/pacman.conf

Need to first turn on sig required for packages for each repo as the global setting at the top creates issues and needs to be commented out.

Here is how the pacman.conf should look.   I have highlighted those edited and or added in bolded blue.

pacman.conf:

#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
GPGDir      = /etc/pacman.d/gnupg/
HoldPkg     = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta    = 0.7
Architecture = auto

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg   =
#IgnoreGroup =
#NoUpgrade   =
NoUpgrade = /etc/X11/xinit/xinitrc.d/pulseaudio
#NoExtract   =

# Misc options
#UseSyslog
#Color
#TotalDownload
CheckSpace
#VerbosePkgLists

# By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages.
#SigLevel    = Required DatabaseOptional
LocalFileSigLevel = Optional
#RemoteFileSigLevel = Required
 
# NOTE: You must run `pacman-key --init` before first using pacman; the local
# keyring can then be populated with the keys of all official Arch Linux
# packagers with `pacman-key --populate archlinux`.

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#   - URLs will have $repo replaced by the name of the current repo
#   - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.

#[testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

#[community-testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

# If you want to run 32 bit applications on your x86_64 system,
# enable the multilib repositories as required here.

#[multilib-testing]
#Include = /etc/pacman.d/mirrorlist

#[multilib]
#Include = /etc/pacman.d/mirrorlist

# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#SigLevel = Optional TrustAll
#Server = file:///home/custompkgs

[multilib]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

#[qubes]
#commented out as it errors and is not current
#Server = http://olivier.medoc.free.fr/archlinux/pkgs/


[xyne-x86_64]
# Added to download powerpill app
# A repo for Xyne's own projects: http://xyne.archlinux.ca/projects/
# Packages for the "x86_64" architecture.
# Note that this includes all packages in [xyne-any].
SigLevel = Required
Server = http://xyne.archlinux.ca/repos/xyne


#end of file

Set the firewall in archlinux template to allow temporarily all traffic.  I just used the count down time set to 30 min but whatever way just make sure not to leave it open.

Stupidly pacman is set to get package sig keys as optional meaning unless you setup gpg and edit the pacman.conf as above you are open to a unsigned malicious download.

We will just use the root repo server for now and update the mirrorlist below for more regular use.

Next initialize the gpg keyring:

$ sudo pacman-key --init

Now populate that keyring with archlinux master keys:

$ sudo pacman-key --populate archlinux

Make sure to compare the keys with those of on archlinux: https://www.archlinux.org/master-keys/

For more info on pacman key signing: https://wiki.archlinux.org/index.php/Pacman/Package_signing

***Of note archlinux still does not require database files to be signed.  They started converting over to signed DB in 2012 and yet still have not enforced it.  Sad Sad Sad    This is also why the pacman.conf is not set to mandatory DB signing and creates errors if you do.

Go ahead and run a basic update to ensure everything is updated:

$ sudo pacman -Syu

Next install powerpill

$ sudo pacman -S powerpill

Another app to install is reflector.  It scripts mirror updating.  Grabbing the most up to date gen mirror list.  It ranks them by most recently sync'd.  Then ranks them on fastest speed.

$ sudo pacman -S reflector

You can combine the install with:

$ sudo pacman -S powerpill reflector


Next we backup the mirrorlist to run reflector to update the active mirrorlist with those you want to use and that are insync and fastest.  Look to this page for various configs of the list: https://wiki.archlinux.org/index.php/Reflector  

Mirrorlist can be found @ /etc/pacman.d/mirrorlist  back it up to be safe.

$ cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.bkup

Now run whatever reflector string that gives you the mirrorlist you would like to use.

Example for someone want the 5 fastest synced mirrors:

$ sudo
reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist

The above ranks all the most update and sorts then in the 5 fastest.

Update the repo databases:

$ sudo pacman -Syy

Configure powerpill file to use the Qubes proxy:

Qubes Proxy: 10.137.255.254:8082

Edit powerpill.  (powerpill no longer uses a .conf file it used the following):

/etc/powerpill/powerpill.json


Part of powerpill download is aria2  in the powerpill.json file you will see the aria2 section using the args arguements
You need to add to the bottom of that section:  (the " " and , are needed and should be included in the string below:

"--all-proxy=10.137.255.254:8082",


Here is the copy of the powerpill.json file with the additions in bolded blue

powerpill.json:

{
  "aria2": {
    "args": [
      "--allow-overwrite=true",
      "--always-resume=false",
      "--auto-file-renaming=false",
      "--check-integrity=true",
      "--conditional-get=true",
      "--continue=true",
      "--file-allocation=none",
      "--log-level=error",
      "--max-concurrent-downloads=100",
      "--max-connection-per-server=5",
      "--min-split-size=5M",
      "--remote-time=true",
      "--show-console-readout=true",
      "--all-proxy=10.137.255.254:8082"   
    ],
    "path": "/usr/bin/aria2c"
  },
  "pacman": {
    "config": "/etc/pacman.conf",
    "path": "/usr/bin/pacman"
  },
  "pacserve": {
    "server": null
  },
  "powerpill": {
    "select": true,
    "reflect databases": false
  },
  "reflector": {
    "args.unused": [
      "--protocol",
      "http",
      "--latest",
      "50"
    ]
  },
  "rsync": {
    "args": [
      "--no-motd",
      "--progress"
    ],
    "db only": true,
    "path": "/usr/bin/rsync",
    "servers": []
  }
}

--------------------------------

Time to test the config.
  As powerpill is a pacman wrapper you can pass the same cmds used in pacman to powerpill.

First make sure that the archlinux firewall settings are now back to the only things that are checked are:

Deny Network Access Except: checked
All connections to Updates Proxy: checked

Now run a basic update command:

$ sudo powerpill -Syu


You should get a output similar to this:


archlinux% sudo powerpill -Syu                                                 

01/07 02:01:12 [NOTICE] GID#907683b79b918aea - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db

01/07 02:01:12 [NOTICE] GID#3ad61df1a92605a5 - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db.sig

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db.sig

01/07 02:01:12 [NOTICE] GID#190847ee8efbf461 - Download has already completed: /var/lib/pacman/sync/multilib.db

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/multilib.db

01/07 02:01:12 [NOTICE] GID#048356b3cc7d9185 - Download has already completed: /var/lib/pacman/sync/core.db

01/07 02:01:12 [NOTICE] Download complete: /var/lib/pacman/sync/core.db
[DL:1.4MiB][#7fd54b 864KiB/3.1MiB(26%)][#68c1c0 672KiB/1.7MiB(38%)]                                         
01/07 02:01:14 [NOTICE] Download complete: /var/lib/pacman/sync/extra.db

01/07 02:01:14 [NOTICE] Download complete: /var/lib/pacman/sync/community.db

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
907683|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db
3ad61d|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db.sig
190847|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
048356|OK  |       0B/s|/var/lib/pacman/sync/core.db
68c1c0|OK  |   1.1MiB/s|/var/lib/pacman/sync/extra.db
7fd54b|OK  |   1.6MiB/s|/var/lib/pacman/sync/community.db

Status Legend:
(OK):download completed.
:: Starting full system upgrade...
 there is nothing to do
archlinux%

Cheers,

Tim