/usr/ports/distfiles/go insanely oversized

[Thor Ablestar]

Hello!

I have a habit to keep /usr/ports/distfiles on separate SSD in order to
build packages on many different computers and OSes (12, 13 and now
14.2). Usually i do the following:

# cd /usr/ports/sometheme/somepackage

# make checksum

# make install

so that I supervise the download of source files and then have some rest
while the port compiles.

SSD became full, and I decided to copy data to the different HDD. Then I
have found that the /usr/ports/distfiles/go has an insane number of
subdirectories (some hundreds of thousands). I spent a full day in
attempts to copy them (successfully at last) and only some minutes to
copy the rest.

Then I decided to find a total size of the distfiles with du. Now du
spent 5 minutes of machine time (top listing) and has shown 288 GB in go
subdir and 469 GB total.

Then I have tried to compare some subdirectories in go directory, and it
seems that each subdirectory contains subdirectories of all their
dependencies (I am not sure). For instance,

go/archivers_nfpm/pkg/mod/cache/download/github.com/!masterminds/goutils/@v

contains the same files (name and size) as

go/audio_gonic/pkg/mod/cache/download/github.com/!masterminds/goutils/@v

What to do? How to manage it? If the problem cannot be solved I am going
just to ban the /usr/ports/distfiles/go dependent ports completely and
try to live without them.

I had the same problem 2 years ago but it was not so severe as now.

Sincerely yours,

Anatoly

[Kevin Oberman]

I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. This makes me wonder just why you are doing things like 'make checksum'? 'make install' will do the following:

config, fetch, checksum, depends, extract, patch, configure, build, install.

That normally results in a single file in distfiles. Some ports require multiple dist files, usually for a port which maintains multiple versions or a metaport that contains distfiles for many ports included in the metaport. E.g. gnome. Still, this does not mean that there will be anything like what you are seeing.

Can you explain exactly how you are doing this? And, why? Why not just use packages? Either pull down the packages you need or use poudriere or 'make package' to generate what you need? Either way will be far less resource intensive. Especially pulling down the pre-built packages.

Clearly, something you're doing is not what you want.

--

Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkob...@gmail.com

PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683

[Thor Ablestar]

Hello!

On 6/17/24 10:26, Kevin Oberman wrote:

I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. This makes me wonder just why you are doing things like 'make checksum'? 'make install' will do the following:

config, fetch, checksum, depends, extract, patch, configure, build, install.

"make fetch" and "make checksum" require supervision due to peculiarities of local Internet that I am afraid to discuss. After this, "make install" can be run unattended.

That normally results in a single file in distfiles. Some ports require multiple dist files, usually for a port which maintains multiple versions or a metaport that contains distfiles for many ports included in the metaport. E.g. gnome. Still, this does not mean that there will be anything like what you are seeing.

It is exactly so for everything but NOT for /usr/ports/distfiles/go

I see that "make checksum" fetches and checksums not only the files listed in, say, /usr/ports/archivers/nfpm/distinfo but some (lots of) files placed in /usr/ports/distfiles/go/archivers_nfpm/pkg/mod/cache - and I cannot understand why.

Can you explain exactly how you are doing this?

For instance (I use archivers/nfpm as a testbed for this problem - it's the first one in alphabetical order):

# cd /usr/ports/archivers/nfpm

# make fetch

# make checksum

# make install

And, why?

I am afraid of discussing the "why". Hint: HTTP error 451

Why not just use packages?

I do it often but not always. Sometimes I need ports.

Either pull down the packages you need or use poudriere or 'make package' to generate what you need? Either way will be far less resource intensive. Especially pulling down the pre-built packages.

Clearly, something you're doing is not what you want.

Yours, Anatoly

[Denis Shaposhnikov]

Hi,

On Sun, 16 Jun 2024, at 11:15, Thor Ablestar wrote:
> go/archivers_nfpm/pkg/mod/cache/download/github.com/!masterminds/goutils/@v
>
> contains the same files (name and size) as
>
> go/audio_gonic/pkg/mod/cache/download/github.com/!masterminds/goutils/@v
>
> What to do? How to manage it? If the problem cannot be solved I am going

I use a port overlay and inside it I have custom Mk/Uses/go.mk:
https://github.com/dsh2dsh/freebsd-ports/blob/master/Mk/Uses/go.mk

==========================================================
include "${USESDIR}/go.mk"

if !defined(_DSH_INCLUDE_USES_GO_MK)
_DSH_INCLUDE_USES_GO_MK= yes

if ${go_ARGS:Mmodules}
GO_GOPATH= ${DISTDIR}/go
endif # ${go_ARGS:Mmodules}

endif # !defined(_DSH_INCLUDE_USES_GO_MK)
==========================================================

This change shares mod cache between all go ports, instead of separated mod cache for every go port. Actually I don't understand why go.mk separates go cache for every port, I don't see a reason for that.

BTW, I use forked [poudriere-devel](https://github.com/dsh2dsh/poudriere) which shared go mod cache in DISTFILES_CACHE. As I see upstream's poudriere doesn't copy go dependencies back to DISTFILES_CACHE.