[gentoo-dev] [PATCH v3] go-module.eclass: deprecate EGO_SUM

[William Hubbs]

EGO_SUM can be thousands of lines long in ebuilds, and it leads to
creating Manifest files that are thousands of lines long.
It has been determined that dependency tarballs are a better solution if
upstream doesn't vendor their dependencies.

Signed-off-by: William Hubbs <will...@gentoo.org>
---
eclass/go-module.eclass | 46 ++++++++++++++++++++++++++++++-----------
1 file changed, 34 insertions(+), 12 deletions(-)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
index 635d2b5161a..485f26096c3 100644
--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -25,8 +25,21 @@
#
# If the software has a directory named vendor in its
# top level directory, the only thing you need to do is inherit the
-# eclass. Otherwise, you need to also populate
-# EGO_SUM and call go-module_set_globals as discussed below.
+# eclass. If it doesn't, you need to also create a dependency tarball and
+# host it somewhere, for example in your dev space.
+#
+# Here is an example of how to create a dependency tarball.
+# go-mod is a temporary directory where the dependencies will be stored,
+# but it must be named go-mod to match the GOMODCACHE setting in this
+# eclass.
+#
+# @CODE
+#
+# $ cd /path/to/project
+# $ GOMODCACHE=go-mod go mod download -modcacherw
+# $ tar -acf project-1.0-deps.tar.xz go-mod
+#
+# @CODE
#
# Since Go programs are statically linked, it is important that your ebuild's
# LICENSE= setting includes the licenses of all statically linked
@@ -40,15 +53,9 @@
#
# inherit go-module
#
-# EGO_SUM=(
-# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod"
-# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59"
-# )
-#
-# go-module_set_globals
-#
-# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
-# ${EGO_SUM_SRC_URI}"
+# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
+# Add this line if you have a dependency tarball.
+# SRC_URI+=" ${P}-dep.tar.xz"
#
# @CODE

@@ -99,7 +106,11 @@ QA_FLAGS_IGNORED='.*'
RESTRICT+=" strip"

# @ECLASS-VARIABLE: EGO_SUM
+# @DEPRECATED:
# @DESCRIPTION:
+# This is replaced by a dependency tarball, see above for how to create
+# one.
+#
# This array is based on the contents of the go.sum file from the top
# level directory of the software you are packaging. Each entry must be
# quoted and contain the first two fields of a line from go.sum.
@@ -143,6 +154,7 @@ RESTRICT+=" strip"
# go.sum copy of the Hash1 values during building of the package.

# @ECLASS-VARIABLE: _GOMODULE_GOPROXY_BASEURI
+# @DEPRECATED:
# @DESCRIPTION:
# Golang module proxy service to fetch module files from. Note that the module
# proxy generally verifies modules via the Hash1 code.
@@ -165,6 +177,7 @@ RESTRICT+=" strip"
: "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"

# @ECLASS-VARIABLE: _GOMODULE_GOSUM_REVERSE_MAP
+# @DEPRECATED:
# @DESCRIPTION:
# Mapping back from Gentoo distfile name to upstream distfile path.
# Associative array to avoid O(N*M) performance when populating the GOPROXY
@@ -194,6 +207,7 @@ ego() {
}

# @FUNCTION: go-module_set_globals
+# @DEPRECATED:
# @DESCRIPTION:
# Convert the information in EGO_SUM for other usage in the ebuild.
# - Populates EGO_SUM_SRC_URI that can be added to SRC_URI
@@ -284,6 +298,7 @@ go-module_set_globals() {
}

# @FUNCTION: go-module_setup_proxy
+# @DEPRECATED:
# @DESCRIPTION:
# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
# this function in src_unpack.
@@ -327,11 +342,14 @@ go-module_setup_proxy() {
# @FUNCTION: go-module_src_unpack
# @DESCRIPTION:
# If EGO_SUM is set, unpack the base tarball(s) and set up the
-# local go proxy.
+# local go proxy. Also warn that this usage is deprecated.
# - Otherwise, if EGO_VENDOR is set, bail out.
# - Otherwise do a normal unpack.
go-module_src_unpack() {
if [[ "${#EGO_SUM[@]}" -gt 0 ]]; then
+ eqawarn "This ebuild uses EGO_SUM which is deprecated"
+ eqawarn "Please migrate to a dependency tarball"
+ eqawarn "This will become a fatal error in the future"
_go-module_src_unpack_gosum
elif [[ "${#EGO_VENDOR[@]}" -gt 0 ]]; then
eerror "${EBUILD} is using EGO_VENDOR which is no longer supported"
@@ -342,6 +360,7 @@ go-module_src_unpack() {
}

# @FUNCTION: _go-module_src_unpack_gosum
+# @DEPRECATED:
# @DESCRIPTION:
# Populate a GOPROXY directory hierarchy with distfiles from EGO_SUM and
# unpack the base distfiles.
@@ -387,6 +406,7 @@ _go-module_src_unpack_gosum() {
}

# @FUNCTION: _go-module_gosum_synthesize_files
+# @DEPRECATED:
# @DESCRIPTION:
# Given a path & version, populate all Goproxy metadata files which aren't
# needed to be downloaded directly.
@@ -414,6 +434,7 @@ _go-module_gosum_synthesize_files() {
}

# @FUNCTION: _go-module_src_unpack_verify_gosum
+# @DEPRECATED:
# @DESCRIPTION:
# Validate the Go modules declared by EGO_SUM are sufficient to cover building
# the package, without actually building it yet.
@@ -462,6 +483,7 @@ go-module_live_vendor() {
}

# @FUNCTION: _go-module_gomod_encode
+# @DEPRECATED:
# @DESCRIPTION:
# Encode the name(path) of a Golang module in the format expected by Goproxy.
#
--
2.34.1

[William Hubbs]

EGO_SUM can be thousands of lines long in ebuilds, and it leads to
creating Manifest files that are thousands of lines long.
It has been determined that dependency tarballs are a better solution if
upstream doesn't vendor their dependencies.

Signed-off-by: William Hubbs <will...@gentoo.org>
---
eclass/go-module.eclass | 46 ++++++++++++++++++++++++++++++-----------
1 file changed, 34 insertions(+), 12 deletions(-)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass

index 635d2b5161a..427a61a1b84 100644

--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -25,8 +25,21 @@
#
# If the software has a directory named vendor in its
# top level directory, the only thing you need to do is inherit the
-# eclass. Otherwise, you need to also populate
-# EGO_SUM and call go-module_set_globals as discussed below.
+# eclass. If it doesn't, you need to also create a dependency tarball and
+# host it somewhere, for example in your dev space.
+#
+# Here is an example of how to create a dependency tarball.

+# The base directory in the GOMODCACHE setting must be go-mod in order
+# to match the settings in this eclass.

+#
+# @CODE
+#
+# $ cd /path/to/project

+# $ GOMODCACHE=/path/to/go-mod go mod download -modcacherw
+# $ tar -acf project-1.0-deps.tar.xz /path/to/go-mod \
+# --xform 's,.*go-mod/,go-mod/'

[William Hubbs]

EGO_SUM can be thousands of lines long in ebuilds, and it leads to
creating Manifest files that are thousands of lines long.
It has been determined that dependency tarballs are a better solution if
upstream doesn't vendor their dependencies.

Signed-off-by: William Hubbs <will...@gentoo.org>
---

eclass/go-module.eclass | 45 ++++++++++++++++++++++++++++++-----------
1 file changed, 33 insertions(+), 12 deletions(-)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
index 635d2b5161a..2d42758c6f4 100644
--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -25,8 +25,20 @@

#
# If the software has a directory named vendor in its
# top level directory, the only thing you need to do is inherit the
-# eclass. Otherwise, you need to also populate
-# EGO_SUM and call go-module_set_globals as discussed below.
+# eclass. If it doesn't, you need to also create a dependency tarball and
+# host it somewhere, for example in your dev space.
+#
+# Here is an example of how to create a dependency tarball.
+# The base directory in the GOMODCACHE setting must be go-mod in order
+# to match the settings in this eclass.
+#
+# @CODE
+#
+# $ cd /path/to/project

+# $ GOMODCACHE="${PWD}"/go-mod go mod download -modcacherw
+# $ tar -acf project-1.0-deps.tar.xz go-mod

+#
+# @CODE
#
# Since Go programs are statically linked, it is important that your ebuild's
# LICENSE= setting includes the licenses of all statically linked

@@ -40,15 +52,9 @@

#
# inherit go-module
#
-# EGO_SUM=(
-# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod"
-# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59"
-# )
-#
-# go-module_set_globals
-#
-# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
-# ${EGO_SUM_SRC_URI}"
+# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
+# Add this line if you have a dependency tarball.
+# SRC_URI+=" ${P}-dep.tar.xz"
#
# @CODE

@@ -99,7 +105,11 @@ QA_FLAGS_IGNORED='.*'

RESTRICT+=" strip"

# @ECLASS-VARIABLE: EGO_SUM
+# @DEPRECATED:
# @DESCRIPTION:
+# This is replaced by a dependency tarball, see above for how to create
+# one.
+#
# This array is based on the contents of the go.sum file from the top
# level directory of the software you are packaging. Each entry must be
# quoted and contain the first two fields of a line from go.sum.

@@ -143,6 +153,7 @@ RESTRICT+=" strip"

# go.sum copy of the Hash1 values during building of the package.

# @ECLASS-VARIABLE: _GOMODULE_GOPROXY_BASEURI
+# @DEPRECATED:
# @DESCRIPTION:
# Golang module proxy service to fetch module files from. Note that the module
# proxy generally verifies modules via the Hash1 code.

@@ -165,6 +176,7 @@ RESTRICT+=" strip"

: "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"

# @ECLASS-VARIABLE: _GOMODULE_GOSUM_REVERSE_MAP
+# @DEPRECATED:
# @DESCRIPTION:
# Mapping back from Gentoo distfile name to upstream distfile path.
# Associative array to avoid O(N*M) performance when populating the GOPROXY

@@ -194,6 +206,7 @@ ego() {

}

# @FUNCTION: go-module_set_globals
+# @DEPRECATED:
# @DESCRIPTION:
# Convert the information in EGO_SUM for other usage in the ebuild.
# - Populates EGO_SUM_SRC_URI that can be added to SRC_URI

@@ -284,6 +297,7 @@ go-module_set_globals() {

}

# @FUNCTION: go-module_setup_proxy
+# @DEPRECATED:
# @DESCRIPTION:
# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
# this function in src_unpack.

@@ -327,11 +341,14 @@ go-module_setup_proxy() {

# @FUNCTION: go-module_src_unpack
# @DESCRIPTION:
# If EGO_SUM is set, unpack the base tarball(s) and set up the
-# local go proxy.
+# local go proxy. Also warn that this usage is deprecated.
# - Otherwise, if EGO_VENDOR is set, bail out.
# - Otherwise do a normal unpack.
go-module_src_unpack() {
if [[ "${#EGO_SUM[@]}" -gt 0 ]]; then
+ eqawarn "This ebuild uses EGO_SUM which is deprecated"
+ eqawarn "Please migrate to a dependency tarball"
+ eqawarn "This will become a fatal error in the future"
_go-module_src_unpack_gosum
elif [[ "${#EGO_VENDOR[@]}" -gt 0 ]]; then
eerror "${EBUILD} is using EGO_VENDOR which is no longer supported"

@@ -342,6 +359,7 @@ go-module_src_unpack() {

}

# @FUNCTION: _go-module_src_unpack_gosum
+# @DEPRECATED:
# @DESCRIPTION:
# Populate a GOPROXY directory hierarchy with distfiles from EGO_SUM and
# unpack the base distfiles.

@@ -387,6 +405,7 @@ _go-module_src_unpack_gosum() {

}

# @FUNCTION: _go-module_gosum_synthesize_files
+# @DEPRECATED:
# @DESCRIPTION:
# Given a path & version, populate all Goproxy metadata files which aren't
# needed to be downloaded directly.

@@ -414,6 +433,7 @@ _go-module_gosum_synthesize_files() {

}

# @FUNCTION: _go-module_src_unpack_verify_gosum
+# @DEPRECATED:
# @DESCRIPTION:
# Validate the Go modules declared by EGO_SUM are sufficient to cover building
# the package, without actually building it yet.

@@ -462,6 +482,7 @@ go-module_live_vendor() {

[Sam James]

> On 5 Mar 2022, at 22:17, William Hubbs <will...@gentoo.org> wrote:
>
> EGO_SUM can be thousands of lines long in ebuilds, and it leads to
> creating Manifest files that are thousands of lines long.
> It has been determined that dependency tarballs are a better solution if
> upstream doesn't vendor their dependencies.
>
> Signed-off-by: William Hubbs <will...@gentoo.org>
> ---
> eclass/go-module.eclass | 45 ++++++++++++++++++++++++++++++-----------
> 1 file changed, 33 insertions(+), 12 deletions(-)

lgtm

[Sam James]

> On 5 Mar 2022, at 22:17, William Hubbs <will...@gentoo.org> wrote:
>

> EGO_SUM can be thousands of lines long in ebuilds, and it leads to
> creating Manifest files that are thousands of lines long.
> It has been determined that dependency tarballs are a better solution if
> upstream doesn't vendor their dependencies.
>
> Signed-off-by: William Hubbs <will...@gentoo.org>
> ---

> [snip]

> -#
> -# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
> -# ${EGO_SUM_SRC_URI}"
> +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
> +# Add this line if you have a dependency tarball.
> +# SRC_URI+=" ${P}-dep.tar.xz"

In #gentoo-dev-help, Weiss-Fder[m] pointed out this should be 'deps'.

Best,
sam

[William Hubbs]

This is fixed and the patch is merged.

Thanks,

William

[tastytea]

On 2022-03-05 16:17-0600 William Hubbs <will...@gentoo.org> wrote:

> +# @CODE
> +#
> +# $ cd /path/to/project
> +# $ GOMODCACHE="${PWD}"/go-mod go mod download -modcacherw
> +# $ tar -acf project-1.0-deps.tar.xz go-mod

What's the reason to use `go mod download` here and not `go mod
vendor`? The former seems to always generate much bigger directories.
For www-apps/hugo it is 1.5G vs. 65M and someone on IRC mentioned that
for another ebuild it is >600M vs 22M.

Kind regards, tastytea

[William Hubbs]

You are correct, it does generate bigger directories. "go mod vendor"
doesn't preserve everything needed to make some builds happen, so I had
to go with "go mod download". For more info, see the following upstream
issue (note that several folks have asked for the discussion to be
restarted; I haven't seen any responses yet).

http://github.com/golang/go/issues/26366

Thanks,

William