[PATCH] classes/repository: use the proper filename to find the packages under repo

[srinuv...@siemens.com]

From: srinuvasan <srinuv...@siemens.com>

The very first time when we generate the base-apt the REPO_BASE_DIR
is empty, it doesn't contain any packages, hence repo_contains_package
function returns 2, based on the return value (2), we are adding all the
packages to the empty repo by calling the repo_add_packages function.

After clearing the temporary and cache files, when we retrigger the base-apt
we should skip the repo_add_packages function for all the packages (assume that
repo packages are not contaminated), but we observed repo_contains_function
says some of the packages not available in the existing repo, but really those
packages available in the repo.

Here the issue is, reprepro caching all the packages by skipping the
epoch version if any packages have, and all the packages contain only the
<upstream_version>-<debian_version>.

But in the download folder , few packages contain the epoch version,
f.e: automake, git-man, ssh, now we try to find the packages with
epoch version in APT repo, but APT repos skip the epoch version
during repo generation.

With that wrong finding again we are calling the repo_add_package
function for few packages, even those packages already available in the repo.

Skip the epoch version during finding the packages in the repo, with that we can eliminate
unnecessarily adding the packages to repo.

Signed-off-by: srinuvasan <srinuv...@siemens.com>
---
meta/classes/repository.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 28e712fd..7379658d 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -99,7 +99,8 @@ repo_contains_package() {
local file="$2"
local package

- package=$(find ${dir} -name ${file##*/})
+ file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
+ package=$(find ${dir} -name ${file_name})
if [ -n "$package" ]; then
# yes
cmp --silent "$package" "$file" && return 0
--
2.39.5

[cedric.h...@siemens.com]

Hello Srinu,

I have a different implementation of this function that I was going to
submit after doing some clean-up and running a full CI build

Here's anyhow what it looks like:

# lookup ${file} in the database for the current suite
package=$(reprepro -b ${dir} --dbdir ${dbdir} \
--list-format 'Filename\n' \
listfilter ${distro} \
'Filename (% /'${file##*/}')' | head -n 1)

if [ -n "$package" ]; then
# yes
cmp --silent "$package" "$file" && return 0

# yes but not the exact same file
return 1
fi
# no
return 2

I would suggest you come up with a more concise description of your
change and ideally some way to reproduce and exhibit the problem as I
am not sure I follow your problem statement

Cedric Hombourger
Siemens AG
www.siemens.com

[cedric.h...@siemens.com]

On Sat, 2024-11-16 at 22:58 +0530, srinuv...@siemens.com wrote:

As I was not fully understanding the problem, I checked the packages
you listed and now it is much clearer:

$ find build/base-apt/local/ build/base-apt/downloads/ -name git-man\*
build/base-apt/local/apt/debian/pool/main/g/git/git-man_2.39.5-
0+deb12u1_all.deb
build/base-apt/downloads/deb/debian-bookworm/git-man_1%3a2.39.5-
0+deb12u1_all.deb

Could you reword your commit message to be more concise?

[Arjunan, Srinu]

Thanks cedric for the review, I will refactor the commit message even more concise.

Many thanks,
Srinu

[srinuv...@siemens.com]

From: srinuvasan <srinuv...@siemens.com>

This patch fixes the issue in repo_contians_package function, here we
are adding a few packages to repo again even though the repo already contains those
packages, ideally we should skip those

reproduce the issue:

1. Create the base-apt
2. Again retrigger the base-apt (assume that there is no changes in the previously built base-apt)

here we observed few packages getting added again to the APT, even those

packages already available in the repo.

Basically few packages having the epoch version (automake, git-man, ssh) in the download folder,
but the same packages are cached in the repo without the epoch version, due to this
mismatch, it tries to add the packages again into the repo, to fix this issue,
just find the package name in the repo without the epoch version.

Signed-off-by: srinuvasan <srinuv...@siemens.com>
---
meta/classes/repository.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 28e712fd..7379658d 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -99,7 +99,8 @@ repo_contains_package() {
local file="$2"
local package

- package=$(find ${dir} -name ${file##*/})
+ file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
+ package=$(find ${dir} -name ${file_name})
if [ -n "$package" ]; then
# yes
cmp --silent "$package" "$file" && return 0
--

2.34.1

[Uladzimir Bely]

Not sure if epoch versions more than 9 exist for some packages, but I
would add '\+' here to consider big epoch numbers:

```
$ echo git-man_1%3a2.39.5-0+deb12u1_all.deb | sed 's/[0-9]%3a//g'
git-man_2.39.5-0+deb12u1_all.deb
$ echo git-man_10%3a2.39.5-0+deb12u1_all.deb | sed 's/[0-9]%3a//g'
git-man_12.39.5-0+deb12u1_all.deb
$ echo git-man_10%3a2.39.5-0+deb12u1_all.deb | sed 's/[0-9]\+%3a//g'
git-man_2.39.5-0+deb12u1_all.deb
```

> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0
> --
> 2.34.1
>

--
Best regards,
Uladzimir.

[srinuv...@siemens.com]

From: Srinuvasan A <srinuv...@siemens.com>

This patch fixes the issue in repo_contians_package function, here we
are adding a few packages to repo again even though the repo already contains those
packages, ideally we should skip those

reproduce the issue:

1. Create the base-apt
2. Again retrigger the base-apt (assume that there is no changes in the previously built base-apt)

here we observed few packages getting added again to the APT, even those
packages already available in the repo.

Basically few packages having the epoch version (automake, git-man, ssh) in the download folder,
but the same packages are cached in the repo without the epoch version, due to this
mismatch, it tries to add the packages again into the repo, to fix this issue,
just find the package name in the repo without the epoch version.

Signed-off-by: Srinuvasan A <srinuv...@siemens.com>

---
meta/classes/repository.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass

index 28e712fd..043c8621 100644

--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -99,7 +99,8 @@ repo_contains_package() {
local file="$2"
local package

- package=$(find ${dir} -name ${file##*/})

+ file_name=$(echo "${file##*/}" | sed 's/[0-9]\+%3a//g')

+ package=$(find ${dir} -name ${file_name})
if [ -n "$package" ]; then
# yes
cmp --silent "$package" "$file" && return 0
--

2.30.2

[Uladzimir Bely]

On Wed, 2024-11-20 at 15:02 +0530, srinuv...@siemens.com wrote:
> From: Srinuvasan A <srinuv...@siemens.com>
>
> This patch fixes the issue in repo_contians_package function, here we
> are adding a few packages to repo again even though the repo already
> contains those
> packages, ideally we should skip those
>
> reproduce the issue:
>
> 1. Create the base-apt
> 2. Again retrigger the base-apt (assume that there is no changes in
> the previously built base-apt)
>
> here we observed few packages getting added again to the APT, even
> those
> packages already available in the repo.
>
> Basically few packages having the epoch version (automake, git-man,
> ssh) in the download folder,
> but the same packages are cached in the repo without the epoch
> version, due to this
> mismatch, it tries to add the packages again into the repo, to fix
> this issue,
> just find the package name in the repo without the epoch version.
>
> Signed-off-by: Srinuvasan A <srinuv...@siemens.com>

Hello.

Since now we have https://groups.google.com/g/isar-users/c/NrVHusKyNTA
series merged to next, this patch gets incompatible and likely not
needed anymore.

> ---
>  meta/classes/repository.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/repository.bbclass
> b/meta/classes/repository.bbclass
> index 28e712fd..043c8621 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -99,7 +99,8 @@ repo_contains_package() {
>      local file="$2"
>      local package
>  
> -    package=$(find ${dir} -name ${file##*/})
> +    file_name=$(echo "${file##*/}" | sed 's/[0-9]\+%3a//g')
> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0

--

Best regards,
Uladzimir.