My new aptly repository management script - written in bash

[Daniel Corrigan]

Here is my script that I am using for repository management for my company.

It is brand new and will need some work.. but it's written in bash, so it should remain simple.

I appreciate any feedback.

Thanks,

Dan

TL;DR

A few things it will do now:

update apt mirrors from remote

create new snapshots
switch existing pubished repos to new snapshots
clean up old snapshots

https://github.com/dancorrigan1/aptly-scripts/blob/master/main.sh

To setup aptly to use this script:

0. aptly's rootDir is /vol1/aplty and homedir is /home/aptly and today's date is 20170606

1. Install aptly.

2. Set up initial /etc/aptly.conf 

The only setting I've changed:

  "rootDir": "/vol1/aptly",

3. Mirror all distros you will be hosting. (i.e. trusty, xenial, zesty) 

aptly@aptly-server1:~/bin$ ./main.sh initialmirrors

This will run several functions in succession. 

create_initial_mirrors() {

   distros=(trusty xenial)

   for distro in ${distros[@]}; do

      aptly mirror create -architectures="amd64" ${distro}-main http://us.archive.ubuntu.com/ubuntu/ ${distro} main restricted universe multiverse

      aptly mirror create -architectures="amd64" ${distro}-updates http://us.archive.ubuntu.com/ubuntu/ ${distro}-updates main restricted universe multiverse

      aptly mirror create -architectures="amd64" ${distro}-security http://security.ubuntu.com/ubuntu/ ${distro}-security main restricted universe multiverse

      aptly mirror create -architectures="amd64" ${distro}-backports http://us.archive.ubuntu.com/ubuntu/ ${distro}-backports main restricted universe multiverse

   done

}

update_from_remote_mirrors() {

   for mirror in `aptly mirror list -raw`; do

      aptly mirror update ${mirror}

   done

}

### This will be called as  update_dev new 

update_dev() {

   new_or_existing=$1

   distros=(trusty xenial)

   for distro in ${distros[@]}; do

      # create todays snapshots

      aptly snapshot create ${distro}-main-${date} from mirror ${distro}-main

      aptly snapshot create ${distro}-updates-${date} from mirror ${distro}-updates

      aptly snapshot create ${distro}-security-${date} from mirror ${distro}-security

      aptly snapshot create ${distro}-backports-${date} from mirror ${distro}-backports

      # merge todays snapshots into common "final" repo

      aptly snapshot merge -latest ${distro}-final-${date} ${distro}-main-${date} ${distro}-updates-${date} ${distro}-security-${date} ${distro}-backports-${date} 

      if [[ $new_or_existing == "existing" ]]; then

         # switch published repos to new snapshot

         aptly publish switch -passphrase="${passphrase}" ${distro} dev ${distro}-final-${date}

      elif [[ $new_or_existing == "new" ]]; then

         # create new published repo

         aptly publish snapshot -passphrase="${passphrase}" -distribution="${distro}" ${distro}-final-${date} dev

      else

         exit 1

      fi

   done

}

update_prod() {

   dev_current_publish_date=`aptly publish list|grep dev|egrep -o "xenial-final-[0-9]{8}"|awk -F"-" '{print $3}'`

   distros=(trusty xenial)

   for distro in ${distros[@]}; do

      aptly publish switch -passphrase="${passphrase}" ${distro} prod ${distro}-final-${dev_current_publish_date}

   done

}

newgraph() {

   aptly graph -layout="vertical" -format="png" -output="/vol1/aptly/public/current.png"

}

What you should end up with is:

aptly@aptly-server1:~/bin$ aptly mirror list

List of mirrors:

 * [trusty-backports]: http://us.archive.ubuntu.com/ubuntu/ trusty-backports

 * [trusty-main]: http://us.archive.ubuntu.com/ubuntu/ trusty

 * [trusty-security]: http://security.ubuntu.com/ubuntu/ trusty-security

 * [trusty-updates]: http://us.archive.ubuntu.com/ubuntu/ trusty-updates

 * [xenial-backports]: http://us.archive.ubuntu.com/ubuntu/ xenial-backports

 * [xenial-main]: http://us.archive.ubuntu.com/ubuntu/ xenial

 * [xenial-security]: http://security.ubuntu.com/ubuntu/ xenial-security

 * [xenial-updates]: http://us.archive.ubuntu.com/ubuntu/ xenial-updates

aptly@aptly-server1:~/bin$ aptly snapshot list -raw

trusty-backports-20170606

trusty-final-20170606

trusty-main-20170606

trusty-security-20170606

trusty-updates-20170606

xenial-backports-20170606

xenial-final-20170606

xenial-main-20170606

xenial-security-20170606

xenial-updates-20170606

aptly@aptly-server1:~/bin$ aptly publish list

Published repositories:

  * dev/trusty [amd64, i386] publishes {main: [trusty-final-20170607]: Merged from sources: 'trusty-main-20170607', 'trusty-updates-20170607', 'trusty-security-20170607', 'trusty-backports-20170607'}

  * dev/xenial [amd64, i386] publishes {main: [xenial-final-20170607]: Merged from sources: 'xenial-main-20170607', 'xenial-updates-20170607', 'xenial-security-20170607', 'xenial-backports-20170607'}

  * prod/trusty [amd64, i386] publishes {main: [trusty-final-20170606]: Merged from sources: 'trusty-main-20170606', 'trusty-updates-20170606', 'trusty-security-20170606', 'trusty-backports-20170606'}

  * prod/xenial [amd64, i386] publishes {main: [xenial-final-20170606]: Merged from sources: 'xenial-main-20170606', 'xenial-updates-20170606', 'xenial-security-20170606', 'xenial-backports-20170606'}

DEV and PROD repositories that you can now easily cycle whenever you want. 

Each distro has a DEV and PROD repo.

DEV and PROD for trusty are currently pointed at the snapshot:  trusty-final-20170607

DEV and PROD for xenial are currently pointed at the snapshot:  xenial-final-20170607

To update DEV repositories: 

aptly@aptly-server1:~/bin$ ./main.sh updatedev

This will result in new snapshots for DEV and re-pointing the DEV repositories to the new snapshots, e.g. trusty-final-20170702 (If ran on July 2nd)

PROD would at trusty-final-20170607.

To mirror PROD repositories from latest published DEV repository: 

aptly@aptly-server1:~/bin$ ./main.sh updateprod

This will determine the current snapshot that DEV is using and re-point PROD to this DEV snapshot.

DEV and PROD would now BOTH be at  trusty-final-20170702

Once you are no longer using a certain dated snapshot. i.e. a published repo is no longer relying on a snapshot from a certain date (20170606)

aptly@aptly-server1:~/bin$ ./main.sh removesnapshot 20170606

This will remove all 20170606 snapshots.

I hope this all makes sense. =)

To setup an infrastructure to use this script:

1. Classify all Linux servers as either DEV or PROD

Assign either a DEV or PROD repository to the machine.

I am considering DEV to include any staging server with a prod counterpart or any non-critical servers.

I am considering PROD to include any server IN production or deemed critical. 

All PROD servers SHOULD have a DEV counterpart.

2. Assign all Ubuntu 14.04 and 16.04 servers to use either a DEV or PROD repository.

e.g. deb https://aptly.goempyrean.com/dev/ xenial main

3. Example patching schedule

Patching window is on the 2nd Thursday of every month.

DEV patching window, July 6th 2017,  8pm CST  - 11pm CST

Latest patches from upstreams are downloaded (ahead of time) and a date-stamped snapshot is created of that certain set of available packages. 

Snapshots are published and DEV repository switched over to the latest snapshot of packages.

Patches are applied to all DEV servers and any necessary  reboots will be performed.

Patches are tested for 1 month.

PROD patching window, August 10th 2017,  8pm CST  - 11pm CST

PROD repo is updated to match DEV's 1 month old date-stamped snapshot.

Patches are applied to all PROD servers and any necessary  reboots will be performed.

Rinse, repeat.

--

Thanks,

Dan

[Andrey Smirnov]

Thanks, Dan, that looks like really good workflow!

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Corrigan]

Thank You Andrey.

So far in my quest to find an apt repo management tool that fits my model, Aptly has been the best choice.

I greatly appreciate the work you put into it.

Dan

[Michael Hofer]

Hi Dan

On Wed, 14 Jun 2017 05:47:20 -0700 (PDT)
Daniel Corrigan <dancor...@gmail.com> wrote:
> Thank You Andrey.
>
> So far in my quest to find an apt repo management tool that fits my model,
> Aptly has been the best choice.

[...]

Looks nice! We use pyaptly, which is a wrapper around aptly to make scheduled
snapshots and publishing easier:

https://github.com/adfinis-sygroup/pyaptly

This way you can define your complete structure in a simple yaml file.

Cheers

Michael

[Joost Ringoot]

Very nice, thanks,

But the graph, how do you do that?

When I try this:

aptly graph -layout="vertical"

I get this:

flag provided but not defined: -layout
Usage: aptly graph

aptly graph - render graph of relationships


Options:
  -architectures="": list of architectures to consider during (comma-separated), default to all available
  -config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
  -dep-follow-all-variants=false: when processing dependencies, follow a & b if dependency is 'a|b'
  -dep-follow-recommends=false: when processing dependencies, follow Recommends
  -dep-follow-source=false: when processing dependencies, follow from binary to Source packages
  -dep-follow-suggests=false: when processing dependencies, follow Suggests
  -format="png": render graph to specified format (png, svg, pdf, etc.)
  -output="": specify output filename, default is to open result in viewer
ERROR: unable to parse flags

# aptly version
aptly version: 0.9.7

[Andrey Smirnov]

You might want to upgrade aptly to more recent version (www.aptly.info/download)