RabbitMQServer RPM repositories issues (broken?) for centos7 on packagecloud.io

[Bastian Bretagne]

Hello,

I am trying to mirror the new rpm repository for centos7 for rabbitmq-server packages from https://packagecloud.io/rabbitmq/rabbitmq-server/el/7/$basearch

But it seems that some metadata from the xmls uses relative (../) path and yum-utils/reposync will not sync it.

Example of errors:

WARNING: skipping package rabbitmq-server-3.8.9-1.el7.noarch: remote path "../noarch/rabbitmq-server-3.8.9-1.el7.noarch.rpm" not within repodir, unsafe to mirror locally

Any ideas? Should I open an issue on the github project?

[Artem Sidorenko]

Hello,

you can invoke reposync with ´--allow-path-traversal` by accepting some possibe security risks (from the man page):

       --allow-path-traversal

              Allow packages stored outside their repo directory to be synced.  These are packages that are referenced in metadata by using absolute paths or up-level ".." symbols, and are normally skipped by reposync for security reasons.

              CAUTION: Using this option has potential security implications since, by providing malicious repodata, an attacker could make reposync write  to  arbitrary  locations on the file system that are accessible by the user running it.

I can confirm some problems for new cloudsmith repositories for rabbitmq-server package (erlang works fine).

I can't sync it via reposync as well.

Here a log with URLGRABBER_DEBUG=1 and running reposync with --allow-path-traversal:

* About to connect() to dl.cloudsmith.io port 443 (#0)

*   Trying 13.224.193.77...

* Connected to dl.cloudsmith.io (13.224.193.77) port 443 (#0)

...

> GET /public/rabbitmq/rabbitmq-server/rpm/el/7/x86_64/../noarch/rabbitmq-server-3.8.16-1.el7.noarch.rpm HTTP/1.1

User-Agent: urlgrabber/3.10 yum/3.4.3

Host: dl.cloudsmith.io

Accept: */*

* The requested URL returned error: 404 Not Found

* Closing connection 0

Btw, mirroring via jfrog artifactory doesn't work for both sources: packagecloud, cloudsmith. Probably because of the relative path issue

[Bastian Bretagne]

Hi,

--allow-path-traversal is not an acceptable option unfortunately. Reposync will create a dir "noarch" outside of the folder of the repo that you give because of those "../" path.

example:

-bash-4.2$ reposync -c /etc/reposync/yum.conf -p /d0/centos/7/x86_64 -r rabbitmq-server --allow-path-traversal

Config time: 0.004
repo time: 0.010
Setting up Package Sacks
pkgsack time: 0.004
rpmdb time: 0.000server-3.5.8-1.noarch.rpm ...
(1/66): rabbitmq-server-3.5.8-1.noarch.rpm

[.....]

-bash-4.2$ ls noarch/

rabbitmq-server-3.5.8-1.noarch.rpm

[...]

-bash-4.2$ ls rabbitmq-server
repodata

Hopefully it got fix

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9c3e6b79-888a-4120-b3a5-ed5df5f90e23n%40googlegroups.com.

[Artem Sidorenko]

I agree. 

Currently it's the only one way how I got the rabbitmq-server package mirrored :-(

I'll ask in rabbitmq slack if somebody can have a look at this problem

[M K]

We do not control the layout of the PackageCloud repository. PackageCloud likely do not try to support

mirroring of their repositories (I'm not saying they are trying to break it intentionally, of course).

[M K]

 Mirroring may or may not be a goal of services such as PackageCloud and Cloudsmith.io. My guess is that

being friendly to mirroring is priority number 357 for them or so. I cannot blame them: most of their users likely do not mirror.

RabbitMQ core team publishes *.rpm files for every GitHub release. If you want to maintain your own yum repository,

you can. All the important parts are produced for you under the MPLv2 license.

[Artem Sidorenko]

Are you sure it's not related to some kind of settings?

We are mirroring several repos from PackageCloud (several GitLab repos, GitHub LFS etc) already for a long time without any issues

[Artem Sidorenko]

BTW, mirroring of erlang works just fine

[Bastian Bretagne]

> Are you sure it's not related to some kind of settings?

It surely is, but sadly it's the least import thing for rabbit team to look at I believe - even if it is few click in a GUI

I am gonna make a simple script to download files from github and issue a crearepo after it, until it is fixed (or not)

Anyway thanks Artem for confirming I am not the only one to have this issue.

And thanks Michael for your input, that make sense, unfortunately this hit us in a way that we use mirror to avoid being a victim of a 3rd party rpm repo shortage (and for cost transfers too) - I believe it is very common to mirror repos in VPCs.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f21e834e-88df-4a15-a27e-33a43f40b41en%40googlegroups.com.

[M K]

There aren't many repository settings on PackageCloud, it's a highly opinionated service.

I do see some publicly available mirroring instructions they provide, though, so I stand corrected:

https://packagecloud.io/rabbitmq/rabbitmq-server/mirror

There are no settings or restrictions related to mirroring that I could find anywhere. Again, it's a service wit every few settings

to tweak.

Cloudsmith.io has a lot more settings but I don't see anything RPM-specific (there are Debian-specific settings)

or anything related to the repository layout. The only difference between rabbitmq/rabbitmq-erlang and rabbitmq/rabbitmq-server

have to do with the re-upload policy. There are two other settings that seem to be related to the age of those

repos: NPM and Python package syncing is enabled for one but not the other. We do not distribute any of those

packages and I don't think we ever tweaked them, so they must be the defaults that Cloudsmith.io introduced at some point.

[M K]

We wouldn't mind looking into what may affect mirroring but we need to know what to look for. PackageCloud specifically

is both

 * very opinionated, not a lot of settings, no control over repository layout by design

* provides a mirroring guide of some kind [1] for every repo

Cloudsmith.io has a lot more settings but none of them seem to be RPM-specific, control the layout

or limit access to anything but package stats. No mirroring instructions that I could find.

We are not RPM packaging experts, so we need to understand what to look for.

1. https://packagecloud.io/rabbitmq/rabbitmq-server/mirror

[M K]

I can reproduce the problem with the rabbitmq-server repo on PC and PC's own mirroring instructions. I had to change some reposync flags,

possibly things have changed on modern-ish CentOS 8.x. The repo was set up using the PC-provided shell script.

While googling for the meaning of the error, I've stumbled upon [1]. I see no repository layout settings we can control,

so my best guess is that at some point in the last 5 or so years we have been on PackageCloud, things have changed

and for some repos, paths are reported as relative. I have no idea why.

Obviously relative paths is a potential attack vector [2] but this is a client-side behaviour.

I will try with Cloudsmith.io next.

1. https://bugzilla.redhat.com/show_bug.cgi?id=1898089

2. https://bugzilla.redhat.com/show_bug.cgi?id=1552328

[Artem Sidorenko]

This happens only for rabbitmq-server, eventually it's somehow related to the 'noarch' and 'x84_64' stuff.
But even in this case I expect to get at least working 'noarch' repository :-/

Is the build/publishing process of rabbitmq-server public? Is it somewhere on GitHub?

I would like to reproduce this problem in my PC playground setup and maybe ask support of PC

> You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/yPl0whVvOXo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d03a1678-d9bb-436a-bdca-2a401c36f34en%40googlegroups.com.

[M K]

Cloudsmith.io repos, when set up using the .repo files from our docs [1] (at least with a contribution from Artem — thanks for that!)

sync successfully with

reposync --repoid=rabbitmq_server-noarch -p=$HOME/mirroring/rabbitmq-server --download-metadata

1. https://www.rabbitmq.com/install-rpm.html#cloudsmith

[M K]

All relevant files are public [1]. To produce a package you need a source tarball

and the contents of the repo in [1]. This has changed after all repositories were folded into rabbitmq/rabbitmq-server,

so I'd have to refresh my memory on how to do that. I'll get back to you in this thread.

The build RPMs are then uploaded to PackageCloud and Cloudsmith.io using their respective publishing APIs.

We are quite sure they don't mess with the package itself but sign repository contents using their own repo,

and rebuilding/reindexing of the repo is a black box for us.

Unlike Launchpad, we do not submit source packages to those services. They do not build any packages,

only accept the RPMs you can access and inspect.

1. https://github.com/rabbitmq/rabbitmq-packaging/tree/master/RPMS/Fedora

[M K]

Below is a line from our Concourse job that has built a recent alpha package:

make -C rabbitmq-packaging package-rpm-rhel8 PACKAGES_DIR=/tmp/build/80754af9/PACKAGES RABBITMQ_PACKAGING_REPO=/tmp/build/80754af9/rabbitmq-packaging SOURCE_DIST_FILE=/tmp/build/80754af9/source-archive/rabbitmq-server-3.8.17-alpha.11.tar.xz VERSION=3.8.17-alpha.11 RPM_VERSION=3.8.17~alpha.11 SAVE_CHANGELOG=yes SIGNING_KEY=…

I suspect that SAVE_CHANGELOG and SIGNING_KEY are optional. PACKAGES_DIR is where the artefacts will be stored.

So you primarily need:

 * A full distribution source tarball, specified as SOURCE_DIST_FILE

* A local rabbitmq/rabbitmq-packaging repo clone at RABBITMQ_PACKAGING_REPO

* A version to use

This implies you have all the build time dependencies [1] installed. 

By the way, I'm pretty sure we produce the package on Debian Buster

since that's what our CI jobs use.

1. https://www.rabbitmq.com/build-server.html

[Ludovic]

Sorry to re-up this topic, we have a local CentOS7 package repository in my company and we're also having issues trying to mirror rabbitmq-server with reposync from Cloudsmith or PackageCloud. I don't know if anyone here have found a solution since Michael's last message?

[Bastian Bretagne]

Centos 7 repo remains broken at this day. I couldnt find a solution so we are staying with old 3.8