keeping old packages in a mirror

[John Leach]

Hi,

I'm mirroring a Launchpad ppa - launchpad removes the old packages when an updated one is uploaded.

I want to mirror it with aptly but keep all old versions of the packages it provides, even after they are removed from the upstream repository.

I could then republish the mirror elsewhere and people would be able to install specific older versions of the packages.

This is non-standard Debian/Ubuntu behaviour of course, but it's common (for example, the puppet apt repositories have every old version of puppet ever uploaded and it's easy to pin apt to use specific versions).

Is this possible? It looks a bit like "snapshot pull" with --no-remove might be the key here but I can't quite figure it out.

Thanks,

John.
--
https://www.brightbox.com

[Andrey Smirnov]

Hi John!

The easiest way to do that most probably is to implement the following script:

1. aptly repo create all-versions (create local repo which would hold all the versions)

2. aptly mirror update ppa-mirror (update mirror, pull any new versions)

3. aptly repo import all-versions ppa-mirror Name (this would copy all the packages from the mirror to local repo, overwrites are no-op)

So over time "all-versions" local repo would accumulate all the package versions from mirror "ppa-mirror" even if "ppa-mirror" drops older versions. 

Hope that helps.

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Lang]

I was under the impression that Aptly included only the latest version of a
package when you have multiple versions in snapshots that you are merging.

David Lang

[Andrey Smirnov]

David, 

First of all what I described doesn't need snapshot yet (it might be created, but all the work happens in the local repo).

But even with snapshot merging, you can choose what you want: keep only latest or keep all the packages.

[John Leach]

On Mon, 2016-05-16 at 08:53 +0000, Andrey Smirnov wrote:
> Hi John!
>
> The easiest way to do that most probably is to implement the
> following script:
>
> 1. aptly repo create all-versions (create local repo which would hold
> all the versions)
> 2. aptly mirror update ppa-mirror (update mirror, pull any new
> versions)
> 3. aptly repo import all-versions ppa-mirror Name (this would copy
> all the packages from the mirror to local repo, overwrites are no-op)
>
> So over time "all-versions" local repo would accumulate all the
> package versions from mirror "ppa-mirror" even if "ppa-mirror" drops
> older versions. 
>
> Hope that helps.

Hi Andrey,

yes, excellent thanks. That appears to work.

For the record, the "repo import" command takes src first, then
destination:

aptly repo import ppa-mirror all-versions Name


my problem now is, I have some snapshots of a mirror with the old
packages in and I want to import them into the repo (since I've long
since already updated my mirror). But "repo import" doesn't seem to
take a mirror snapshot as a source.

I can't find a way to promote a snapshot to a repo or something (at
least not without publishing it to local filesystem and then importing
it again, which seems a bit daft!)

Any thoughts? Any reason I wouldn't be able to patch aptly to allow
imports from snapshots?

Thanks,

John.

[christop...@gmail.com]

I'm not happy with this answer. I think we can do better.

My organization relies heavily on docker - mostly the stable images of Debian.

We also pin dependencies tightly in our Dockerfiles; up to and including their minor version.

We do this, because even minor version changes in a .deb package tend to break images (often).

Our tight pinning creates the problem, that typically, older package versions get deleted from an index as soon as a new minor version is released.

I.e. apt throws `E: Version '1.1.1c-1' for 'libssl-dev' was not found`, because it's `1.1.1d-0+deb10u1` now, apparently.

The proposed answer in this thread means:

  a. Either we have to

      ...mirror the whole upstream repositor, e.g. `http://cdn-fastly.deb.debian.org/debian` (which takes up a lot of disk space)

      ...or cherry-pick the packages needed by our docker images, manually adding them to the repo one by one

  b. Then we have to update the mirror daily (or so) and import the new packages into the repo, with no guarantee that we didn't miss a package version because update frequency was to low

My ideal solution would be some sort of cach-only option for a mirrow:

It gets a request for a package. If a specific version was requested and it has that version of the package from a previous request, it sends the package. If not, it downloads the package form upstream. It only ever stores packages that were requested at least once - not the whole index.The caching morror index would keep all old versions available, provided they were requested at least one and not explicitly deleted. Otherwise it would act like a regular mirror.

apt-cacher-ng does most of this already, tragically it does not keep old packages on the index (why they are still on disk but not apt-installable is beyond me).

I've seen many requests for this feature research tools (e.g. apt-mirror, apt-cacher-ng, debmarshal and Sonartype Nexus). No one seems to have it.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-...@googlegroups.com.

[Andrey Smirnov]

yep, aptly doesn't support that flow because once mirror is updated, it can be used to create snapshots and do other fun stuff, so packages should be downloaded.

Can't say that it's a solution, but one approach might work:

1. update mirror (yep, it takes a lot of space, but filtering out something you don't need like desktop apps might help)

2. snapshot it (to keep package versions referenced)

3. merge snapshots with -no-remove to create a mega-snapshot of packages ever there

4. use that snapshot for cherry-picking or aptly snapshot pull which does that automatically 

at some point it might make sense to keep only X daily snapshots to reduce disk usage

not perfect, but should work

пн, 7 окт. 2019 г. в 17:37, <christop...@gmail.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aptly-discuss/f60e935c-84a2-41d8-8950-960ca5486356%40googlegroups.com.

--

Andrey Smirnov,
http://smira.ru/