Errors while using aptly to mirror an aptly repository

[Carlos González]

Hi Everyone,

We are experimenting with a hierarchy of multiple aptly servers and we noticed that we are getting the following error while trying to update the multiverse mirror on the slave:

root@tst-ubuntu-repo02:~# aptly mirror update trusty-multiverse-mirror
Downloading http://<master server>/ubuntu/latest/dists/trusty/InRelease...
gpgv: Signature made Wed 17 May 2017 12:07:08 PM GMT using RSA key ID 286E910B
gpgv: Good signature from "<signature>"
Downloading & parsing package files...
Downloading http://<master server>/ubuntu/latest/dists/trusty/multiverse/binary-i386/Packages.bz2...
ERROR: unable to update: no candidates for http://<master server>/ubuntu/latest/dists/trusty/multiverse/debian-installer/binary-i386/Packages found
root@tst-ubuntu-repo02:~#

I noticed "debian-installer" directory is not being created on multiverse, why?

This is our configuration:

On the "master aptly server" are using multi-component repository publishing so we created 4 mirrors:
trusty-main-mirror
trusty-multiverse-mirror
trusty-restricted-mirror
trusty-universe-mirror

And we also took the following 4 snapshots:
trusty-main-snapshot-2017-05-17
trusty-multiverse-snapshot-2017-05-17
trusty-restricted-snapshot-2017-05-17
trusty-universe-snapshot-2017-05-17

And we publish them using the "latest" prefix:
# aptly publish show trusty latest
Prefix: latest
Distribution: trusty
Architectures: amd64 i386
Sources:
  restricted: trusty-restricted-snapshot-2017-05-17 [snapshot]
  multiverse: trusty-multiverse-snapshot-2017-05-17 [snapshot]
  main: trusty-main-snapshot-2017-05-17 [snapshot]
  universe: trusty-universe-snapshot-2017-05-17 [snapshot]
root@tst-ubuntu-repo01:~#

Im attaching the output of aptly mirror/snapshot show from both servers.

Regards,

Carlos

[Andrey Smirnov]

Hey Carlos, 

Do you have `-with-udebs` option enabled while creating your mirror from upstream?

By default aptly doesn't mirror .udeb packages.

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carlos González]

Hi Andrey,

That is correct, for example on the mirror_snapshot_details_repo01.txt you can see that it shows "Download .udebs: yes":

Begin command output: ----------------------------
Name: trusty-main-mirror
Archive Root URL: http://mirrors.mit.edu/ubuntu/
Distribution: trusty
Components: main
Architectures: i386, amd64
Download Sources: no
Download .udebs: yes
Last update: 2017-05-17 12:00:06 UTC
Number of packages: 14287

Information from release file:
Architectures: amd64 arm64 armhf i386 powerpc ppc64el
Codename: trusty
Components: main restricted universe multiverse
Date: Thu, 08 May 2014 14:19:09 UTC
Description:  Ubuntu Trusty 14.04

Label: Ubuntu
Origin: Ubuntu
Suite: trusty
Version: 14.04
....

Regards,

Carlos

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

[Andrey Smirnov]

I'm not sure what might the problem here. You might want to check whether `.udeb` packages are really part of snapshot. Publish should automatically create `debian-installer` hierarchy once it hits at least one `.udeb` package.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

[Carlos González]

Hi Andrey,

Yeah, it does looks like a problem. All the repo's mirror have udeb:

root@tst-ubuntu-repo01:/servers/ubuntu# aptly mirror list | egrep 'trusty-(main|multiverse|restricted|universe)'
 * [trusty-main-mirror]: http://mirrors.mit.edu/ubuntu/ trusty [udeb]
 * [trusty-multiverse-mirror]: http://mirrors.mit.edu/ubuntu/ trusty [udeb]
 * [trusty-restricted-mirror]: http://mirrors.mit.edu/ubuntu/ trusty [udeb]
 * [trusty-universe-mirror]: http://mirrors.mit.edu/ubuntu/ trusty [udeb]
root@tst-ubuntu-repo01:/servers/ubuntu#

I can see many udeb pacakges:

root@tst-ubuntu-repo01:/servers/ubuntu# find public/latest/pool/ -name '*.udeb' | head
public/latest/pool/main/libb/libbsd/libbsd0-udeb_0.6.0-2ubuntu1_amd64.udeb
public/latest/pool/main/libb/libbsd/libbsd0-udeb_0.8.2-1_amd64.udeb
public/latest/pool/main/libb/libbsd/libbsd0-udeb_0.6.0-2ubuntu1_i386.udeb
public/latest/pool/main/libb/libbsd/libbsd0-udeb_0.8.2-1_i386.udeb
public/latest/pool/main/libd/libdrm/libdrm2-udeb_2.4.52-1_i386.udeb
public/latest/pool/main/libd/libdrm/libdrm2-udeb_2.4.67-1_amd64.udeb
public/latest/pool/main/libd/libdrm/libdrm2-udeb_2.4.67-1_i386.udeb
public/latest/pool/main/libd/libdrm/libdrm2-udeb_2.4.52-1_amd64.udeb
public/latest/pool/main/libd/libdebian-installer/libdebian-installer4-udeb_0.102ubuntu1_amd64.udeb
public/latest/pool/main/libd/libdebian-installer/libdebian-installer4-udeb_0.102ubuntu1_i386.udeb
root@tst-ubuntu-repo01:/servers/ubuntu#

root@tst-ubuntu-repo01:/servers/ubuntu# find public/latest/pool/ -name '*.udeb' | wc -l
1199
root@tst-ubuntu-repo01:/servers/ubuntu#

But for trusty I noticed debian-installer was only created for the main and universe components:

root@tst-ubuntu-repo01:/servers/ubuntu# find public/latest/ -name 'debian-installer' | grep trusty
public/latest/dists/trusty/main/debian-installer
public/latest/dists/trusty/universe/debian-installer
root@tst-ubuntu-repo01:/servers/ubuntu#

Regards,

Carlos

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

[Andrey Smirnov]

Carlos, so what's the issue there? Probably there are no udebs in other components? I'm just trying to understand the problem.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--

You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

[Carlos González]

Hi Andrey,

Probably there are no udebs in other components?

It looks like that is the case, downloading http://mirrors.mit.edu/ubuntu/dists/trusty/multiverse/debian-installer/binary-amd64/Packages.bz2 contains an empty file.

So what's the issue there?
I would expect consistency if aptly is able to pull from a public mirror I would expect to be able to use the same expect to use the same steps to mirror the aptly publishment.

Regards,
Carlos

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

[Andrey Smirnov]

I'm still trying to understand the root cause, but if I got it correctly, aptly creates debian-installer indexes while publishing only for components which have udebs, while mirroring aptly requires every component to have debian-installer index. Is my understanding correct?

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

[Carlos González]

Yeah, that looks like the root cause.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

[Andrey Smirnov]

Would you mind submitting issue at GitHub? Should be pretty easy to fix on mirroring side.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

[Carlos González]

Sure  I create https://github.com/smira/aptly/issues/580

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--

K-lee

[Andrey Smirnov]

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

K-lee

--

K-lee

--

K-lee