aptly publish performance

[Ritesh Sarraf]

We have a Debian derivative distribution, for which we use the aptly publisher for repository management.

We have multiple repositories like Debian's main, contrib, non-free etc; but with different names.

In total, we have around 8000 source packages, part of the derivative distribution.

When preparing to publish a repository via aptly, the performance is terrible. It goes into multiples of hours, on average taking 24-36 hours to process the publishing. Is it normal and expected ?

Note that we do heavily make use of the aptly snapshot features, wherein we do take snapshots on final repository publishes.

At this time, when I'm reporting this issue, we have:

* 5 releases of the derivative using the aptly publisher

* 4 repositories per release

* Total of 20 repositories

* Total of 429 snapshots thus far

[Ritesh Sarraf]

It may be worth mentioning that it is the addition of packages that is the most performance hungry operation.

With around 8000 source packages, the resultant binary package number is around: 55000 packages. And then we have multiple architectures, primarily, amd64, arm64 and arm32. Probably that 55k is the sum of the multiple binary architectures + source packages.

On avg, it takes 2 to 3 seconds to process each file addition. Which results in around 45 hrs of processing time.

[Todd Goodman]

We have all the Ubuntu repos mirrored, with snapshots and those snapshots published.

Updating mirrors for the first time can take over three days, but a lot of that is due to the company's poor network setup.

429 snapshots doesn't sound like a lot compared to what we have and isn't generally an issue (as I believe it's just a database update).

We publish from snapshots and it doesn't take anywhere like 24-36 hours.  I don't remember how long the last publication took, but it was certainly less than an hour.

Are you publishing from snapshots?

Publishing creates a lot of hard links, so, if it were ,e, I'd look into my filesystem performance

Todd

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aptly-discuss/3c9d74cc-8a42-419c-bbcc-4178bd2bdc03n%40googlegroups.com.

[Ritesh Raj Sarraf]

Hello Todd,

Sorry for the delayed response. And thank you for responding to my
email.

On Wed, 2023-03-01 at 08:52 -0500, Todd Goodman wrote:
> We have all the Ubuntu repos mirrored, with snapshots and those
> snapshots published.
> Updating mirrors for the first time can take over three days, but a
> lot of that is due to the company's poor network setup.
> 429 snapshots doesn't sound like a lot compared to what we have and
> isn't generally an issue (as I believe it's just a database update).
> We publish from snapshots and it doesn't take anywhere like 24-36
> hours.  I don't remember how long the last publication took, but it
> was certainly less than an hour.
> Are you publishing from snapshots?
> Publishing creates a lot of hard links, so, if it were ,e, I'd look
> into my filesystem performance
> Todd

Thank you for clarifying on that.

Ours is a different setup. We make use of the Open Build Service aka
OBS software for build requirements. We've plugged in the aptly tool as
the publisher of choice. Previously, we were making use of reprepro.

After further investigation, it was the way OBS + Aptly integration is
in our setup, that is causing the insane performance penalty.

Say, you have a replica of the main repository from Debian. You'll end
up with the same on OBS, with a single repository with thousands on
packages. For this conversation, lets say that it is 5000 (source)
packages. They'll reflect as 5000 packages in an OBS repository.


When OBS does a publishing of the repository, the aptly publisher is
invoked, which in turn takes each file as an argument to `aptly add`
So that ends up making it at least 5000 src pkgs * 3 architecutres
(arm32, arm64 and x86_64). Now increase that with each of those source
package's binary packages. Let's take an average of 3 binary package
outputs + 1 source package.

Makes it a total to 60,000 `aptly add` operations. That is where the
bottleneck was/is.

I have now validated that it is not an aplty performance problem
because the usual `snaptshot`, `publish` operations are completing in a
reasonable amount of time.

[Todd Goodman]

Hi Ritesh,

Thank you for letting us all know the results of your investigation!

Best regards,

Todd