"aptly publish switch" extremely slow

[jor...@gmail.com]

Hi everyone!

I've been using aptly for months to keep a local mirror of Ubuntu's repositories without any problems, but yesterday my weekly automated update process stopped because of an out-of-memory error. I tried to close unneeded processes to free memory and relaunching the offending command (aptly publish switch precise-security precise_security_20160214) and while I'm not getting the OOM error anymore, it is taking ages and the CPU is stuck at almost 100% at "Generating metadata files and linking package files". It's not stopped though: I can see the package count going on, even if very slowly.

Here is my aptly configuration for that repository:

# aptly mirror show precise_security
Name: precise_security
Archive Root URL: http://archive.ubuntu.com/ubuntu/
Distribution: precise-security
Components: main, restricted, universe, multiverse
Architectures: i386, amd64
Download Sources: no
Download .udebs: no
Filter: !Name (% *-dbg),!Name (% *lowlatency*),!Name (% *-virtual),!Section (games),!Name (% *nexuiz*),!Name (% *sauerbraten*),!Name (% *redeclipse*),!Name (% *alien-arena*),!Name (% 0ad*),!Name (% *wesnoth*)
Filter With Deps: no
Last update: 2016-02-14 02:40:50 CET
Number of packages: 7804

Information from release file:
Architectures: amd64 armel armhf i386 powerpc
Codename: precise
Components: main restricted universe multiverse
Date: Fri, 12 Feb 2016 8:39:18 UTC
Description: Ubuntu Precise Security

Label: Ubuntu
Origin: Ubuntu
Suite: precise-security
Version: 12.04

# aptly snapshot show precise_security_20160214
Name: precise_security_20160214
Created At: 2016-02-14 02:44:20 CET
Description: Snapshot from mirror [precise_security]: http://archive.ubuntu.com/ubuntu/ precise-security
Number of packages: 7804

And here's how I update it:

aptly mirror update precise_security
aptly snapshot create precise_security_$DATE from mirror precise_security
aptly publish switch precise-security precise_security_$DATE

This is with aptly 0.9.6 on Ubuntu 12.04.5

Any hint on what I could try?

[Andrey Smirnov]

Hi jorilx!

That's most probably Contents index generation, you can disable it by passing `-skip-contents` to publish commands.

--
You received this message because you are subscribed to the Google Groups "aptly-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aptly-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Federico Fanton]

On 15/02/2016 13:25, Andrey Smirnov wrote:
> Hi jorilx!
>
> That's most probably Contents index generation, you can disable it by
> passing `-skip-contents` to publish commands.

Now the command finishes correctly, thanks! What are the downsides of
disabling the Contents index?

[Andrey Smirnov]

No downsides, except for the fact that Contents index won't be generated :) 

aptly caches Contents for the packages, so it's extremely long only for the first time. But still if you don't use it, probably better to skip it.

[David Lang]

what uses the Contents index? How can I tell if I need it or not?

[ch...@cogdon.org]

On Tuesday, March 22, 2016 at 10:32:53 AM UTC-7, David Lang wrote:
> what uses the Contents index? How can I tell if I need it or not?

Contents is a list of each of the packages contents. I.e., what files it has, etcetc. I _believe_ that apt-cache search will look inside the downloaded Contents files if you're searching for a particular file, but I think that's the only real use.

P.s., I found this page because my aptly publish was taking a LONG time, and it appears to be very slow when "processing" large packages, like the linux kernel packages. I was thinking "why on earth does it need to open up every package"? Well, it does, if it needs to know what is inside the package in order to build the Contents file. Makes total sense.

[Bernd Naumann]

If I do not mistaken, `apt-file search <name>` depends also on that
content, which is handy if you search a package where a certain file is
in it.

Do you use the latest version of aptly? I think this issue was resolved
in the pre-last version or so, but is still resource hungry.

[ch...@cogdon.org]

On Thursday, July 7, 2016 at 3:51:26 AM UTC-7, Bernd Naumann wrote:
> If I do not mistaken, `apt-file search <name>` depends also on that
> content, which is handy if you search a package where a certain file is
> in it.

Sounds reasonable!

For the audience: If you're just trying to figure out "what package installed a file on my system" you can use dpkg --search /path/to/your/installed/file . That only checks installed files, and doesn't need the "Contents" file we're talking about.

> Do you use the latest version of aptly? I think this issue was resolved
> in the pre-last version or so, but is still resource hungry.

Latest: 0.9.7. First-time (empty cache) publish, which appears to "at least" be examining the contents of each package to populate the Contents database, runs about 2 times slower than doing a dpkg-deb --contents **with output running to the terminal**. Part of that will be writing the information to the database but I can't imagine what else it might be doing to make it that slow. Is it creating digests of all the package's contents?

From my examination, there's a burst of CPU activity (multi-threaded, too) followed by a longer burst of disk activity. I suspect that after each package it's doing a database commit, which is forcing a disk flush, and the database is probably not particularly performant in that regard. It looks like it's using memdb, but with some extension that allows it to persist to disk, and I'm going to take a guess that the combination of the two isn't particularly high-performance in writes.