[PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

Hi all,

this series switch Isar internal binary package processing to apt repository.
It performs this following:

1. Create repositories 'tmp/deploy/apt' for all architectures requested by
multiconfig.

2. Generate reprepro database.

3. Put all the newly built packages to apt repository.

4. Pass this Isar repository to image multistrap.

Documentation will be updated after agreement on this implementation.

Changes since v4:
- Drop meta-isar-bin layer.

With best regards,
Alex

Alexander Smirnov (4):
apt: Generate configs for apt
apt: Generate Isar reprepro database
apt: Populate Isar apt
apt: Install packages via multistrap

meta-isar/conf/layer.conf | 13 ++++-
meta-isar/conf/local.conf.sample | 4 ++
.../recipes-core/images/files/distributions.in | 3 +
.../recipes-core/images/files/multistrap.conf.in | 9 ++-
meta-isar/recipes-core/images/isar-image-base.bb | 6 +-
meta/classes/ext4-img.bbclass | 2 +-
meta/classes/image.bbclass | 65 +++++++++++++++++++---
7 files changed, 88 insertions(+), 14 deletions(-)
create mode 100644 meta-isar/recipes-core/images/files/distributions.in

--
2.1.4

[Alexander Smirnov]

Generate reprepro database for Isar apt repository.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---
meta-isar/conf/layer.conf | 3 +++
meta-isar/conf/local.conf.sample | 4 ++++
meta/classes/image.bbclass | 34 ++++++++++++++++++++++++++++++++++
3 files changed, 41 insertions(+)

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index 0950a34..cd42f06 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -22,3 +22,6 @@ DEBDISTRONAME = "isar"

# Path to the Isar apt repository
DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
+
+# Path to the Isar databases used by `reprepro`
+DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index a456b1b..660958f 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
#
# Default parallel jobs for bitbake:
BB_NUMBER_THREADS = "4"
+
+#
+# Number of attempts to try to get reprepro lock for access to apt cache
+REPREPRO_LOCK_ATTEMPTS = "16"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 57705a7..47f53a3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -10,6 +10,31 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"

inherit ${IMAGE_TYPE}

+call_reprepro() {
+ for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
+ # According to `sh` manual page, shell exit statuses fall between
+ # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
+ # integer results value (239).
+ eexist=$(python -c 'import errno; print(256-errno.EEXIST)')
+ retval="0"
+ reprepro $* || retval="$?"
+
+ # If reprepro has failed to get database lock, it returns EEXIST code.
+ # In this case we continue trying to get lock until max amount of
+ # attempts is reached.
+ if [ $retval -eq $eexist ]; then
+ bbwarn "Failed to get reprepro lock, trying again..."
+ sleep 5
+ else
+ break
+ fi
+ done
+
+ if [ $retval -ne 0 ]; then
+ bbfatal "reprepro failed"
+ fi
+}
+
CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
do_cache_config[dirs] = "${CACHE_CONF_DIR}"
do_cache_config[stamp-extra-info] = "${DISTRO}"
@@ -21,6 +46,15 @@ do_cache_config() {
sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
fi
+
+ path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
+ path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
+
+ if [ ! -d "${path_databases}" ]; then
+ call_reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DEBDISTRONAME}
+ fi
}

addtask cache_config before do_fetch
--
2.1.4

[Alexander Smirnov]

Generate reprepro configuration files for apt deployment.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta-isar/conf/layer.conf | 10 ++++++++--
meta-isar/recipes-core/images/files/distributions.in | 3 +++
meta/classes/image.bbclass | 17 ++++++++++++++++-
3 files changed, 27 insertions(+), 3 deletions(-)
create mode 100644 meta-isar/recipes-core/images/files/distributions.in

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index 72a103c..0950a34 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -1,5 +1,5 @@
# This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
+# Copyright (C) 2015-2017 ilbers GmbH

# We have a conf and classes directory, add to BBPATH
BBPATH .= ":${LAYERDIR}"
@@ -13,6 +13,12 @@ BBFILE_PRIORITY_isar = "5"

# This should only be incremented on significant changes that will
# cause compatibility issues with other layers
-LAYERVERSION_isar = "2"
+LAYERVERSION_isar = "3"

LAYERDIR_isar = "${LAYERDIR}"
+
+# Codename of the repository created by the caching class
+DEBDISTRONAME = "isar"
+
+# Path to the Isar apt repository
+DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
diff --git a/meta-isar/recipes-core/images/files/distributions.in b/meta-isar/recipes-core/images/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta-isar/recipes-core/images/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5bf9524..57705a7 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,5 +1,5 @@
# This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
+# Copyright (C) 2015-2017 ilbers GmbH

KERNEL_IMAGE ?= ""
INITRD_IMAGE ?= ""
@@ -10,6 +10,21 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"

inherit ${IMAGE_TYPE}

+CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
+ ${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+}
+
+addtask cache_config before do_fetch
+
do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"

# Install Debian packages, that were built from sources
--
2.1.4

[Alexander Smirnov]

Populate newly built binary packages to Isar apt repository.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta/classes/image.bbclass | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 47f53a3..610dd74 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -64,6 +64,13 @@ do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"

# Install Debian packages, that were built from sources

do_populate() {
if [ -n "${IMAGE_INSTALL}" ]; then
+ for p in ${IMAGE_INSTALL}; do
+ call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
+ --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+ -C main \
+ includedeb ${DEBDISTRONAME} \
+ ${DEPLOY_DIR_DEB}/${p}_*.deb
+ done
sudo mkdir -p ${IMAGE_ROOTFS}/deb

for p in ${IMAGE_INSTALL}; do
--
2.1.4

[Alexander Smirnov]

Install custom packages to target image filesystem using multistrap.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta-isar/recipes-core/images/files/multistrap.conf.in | 9 ++++++++-
meta-isar/recipes-core/images/isar-image-base.bb | 6 +++++-
meta/classes/ext4-img.bbclass | 2 +-
meta/classes/image.bbclass | 11 +----------
4 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..491e5a2 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
[General]
noauth=true
unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
aptsources=##DISTRO##
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
components=##DISTRO_COMPONENTS##
packages=##IMAGE_PREINSTALL##
omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_APT##
+suite=##ISAR_DISTRO_SUITE##
+components=main
+omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index f4b0125..7798b25 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -46,6 +46,9 @@ do_rootfs() {
-e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
-e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+ -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+ -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
+ -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
"${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"

# Create root filesystem
@@ -57,4 +60,5 @@ do_rootfs() {
sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
}

-addtask rootfs before do_populate after do_unpack
+addtask rootfs before do_build after do_populate
+
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
fi
}

-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 610dd74..e1a5c85 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -61,7 +61,7 @@ addtask cache_config before do_fetch

do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"

-# Install Debian packages, that were built from sources
+# Populate Isar apt repository by newly built packages

do_populate() {
if [ -n "${IMAGE_INSTALL}" ]; then

for p in ${IMAGE_INSTALL}; do

@@ -71,15 +71,6 @@ do_populate() {
includedeb ${DEBDISTRONAME} \

${DEPLOY_DIR_DEB}/${p}_*.deb

done
- sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
- for p in ${IMAGE_INSTALL}; do
- sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
- done
-
- sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
- sudo rm -rf ${IMAGE_ROOTFS}/deb
fi
}

--
2.1.4

[Claudius Heine]

Hi,

Thanks!

I would prefer that being handled the same way the DL_DIR and SSTATE_DIR
is handled. So the same variable naming scheme (maybe APT_CACHE_DIR) and
similar default paths (maybe "${TOPDIR}/aptcache"). The deploy dir is
not really the place to do work in.

If you want to have something deployed to the deploy dir then it should
be done via a recipe. I would suggest something like a
"do_deploy_repository_deb" task in image recipes. But thats a separate
patch.

Claudius

--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de

[Claudius Heine]

Hi,

On 10/05/2017 12:08 PM, Alexander Smirnov wrote:

> Generate reprepro database for Isar apt repository.
>
> Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
> ---
> meta-isar/conf/layer.conf | 3 +++
> meta-isar/conf/local.conf.sample | 4 ++++
> meta/classes/image.bbclass | 34 ++++++++++++++++++++++++++++++++++
> 3 files changed, 41 insertions(+)
>
> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
> index 0950a34..cd42f06 100644
> --- a/meta-isar/conf/layer.conf
> +++ b/meta-isar/conf/layer.conf
> @@ -22,3 +22,6 @@ DEBDISTRONAME = "isar"
>
> # Path to the Isar apt repository
> DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
> +
> +# Path to the Isar databases used by `reprepro`
> +DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"

The same comment in the previous patch applies here.

> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index a456b1b..660958f 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
> #
> # Default parallel jobs for bitbake:
> BB_NUMBER_THREADS = "4"
> +
> +#
> +# Number of attempts to try to get reprepro lock for access to apt cache
> +REPREPRO_LOCK_ATTEMPTS = "16"
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 57705a7..47f53a3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -10,6 +10,31 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
>
> inherit ${IMAGE_TYPE}
>
> +call_reprepro() {
> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> + # According to `sh` manual page, shell exit statuses fall between
> + # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
> + # integer results value (239).
> + eexist=$(python -c 'import errno; print(256-errno.EEXIST)')

You can just use the python interpreter that is already running like this:

def _eexist():
import errno
return 265-errno.EEXIST

...

eexist="${@_eexist()}"

[Alexander Smirnov]

This should be in deploy directory, because "${DEPLOY_DIR}/apt" folder
contains self-contained apt repository that could be exported/published
by Isar user's demand, like images.

>
> If you want to have something deployed to the deploy dir then it should
> be done via a recipe. I would suggest something like a
> "do_deploy_repository_deb" task in image recipes. But thats a separate
> patch.

Please see the whole series. Apt repository population is performed in
image recipe by do_populate task.

Alex

[Claudius Heine]

I took a look at how OE creates repositories and its similar to the way
its implemented here.

I am not 100% sure that it makes sense to copy it from OE in this case,
because we are using this repo differently (as a cache) than oe (just as
a output of the package index) but I am willing to accept this solution,
because people coming from OE might be more familiar this way.

[Henning Schild]

Hi,

i just tried it and the task do_cache_config was not executed before
the first do_populate.

I would suggest the following changes:
- create one task in dpkg-base.bbclass that does the following
- config and init reprepro if no other recipe did that before
- add package
- use bitkages task[lockfiles] to deal with races between recipes,
put distro into lockfile-name so we have one lock per distro
- drop do_populate
- call the new task instead of, or in do_deploy_deb

Henning

Am Thu, 5 Oct 2017 13:08:03 +0300
schrieb Alexander Smirnov <asmi...@ilbers.de>:

[Henning Schild]

Am Thu, 5 Oct 2017 13:08:05 +0300
schrieb Alexander Smirnov <asmi...@ilbers.de>:

Now that there are not magic numbers anymore we can skip the verbose
comment.

Henning

[Claudius Heine]

Hi,

On 10/09/2017 02:00 PM, [ext] Henning Schild wrote:
> Hi,
>
> i just tried it and the task do_cache_config was not executed before
> the first do_populate.

Same issue here.

[Alexander Smirnov]

Hi,

> i just tried it and the task do_cache_config was not executed before
> the first do_populate.

Could you please specify more details:

- Build command
- Commit ID

Alex

--
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

[Henning Schild]

On Wed, 18 Oct 2017 14:06:53 +0300
Alexander Smirnov <asmi...@ilbers.de> wrote:

> Hi,
>
> > i just tried it and the task do_cache_config was not executed before
> > the first do_populate.
>
> Could you please specify more details:
>
> - Build command

Well i am using layering, and so is Claudius. So the command wont help
you much. It should basically boil down to

bitbake multiconfig:qemuamd64-stretch:isar-image-base

> - Commit ID

f33e48bd07039032e3

I am guessing you can not reproduce the issue, we should find out why.
But since Claudius confirmend that he can not use these patches either,
and there are still some open points, i think you should remove those
patches from next.

Henning

[Alexander Smirnov]

On 10/18/2017 04:44 PM, Henning Schild wrote:
> On Wed, 18 Oct 2017 14:06:53 +0300
> Alexander Smirnov <asmi...@ilbers.de> wrote:
>
>> Hi,
>>
>>> i just tried it and the task do_cache_config was not executed before
>>> the first do_populate.
>>
>> Could you please specify more details:
>>
>> - Build command
>
> Well i am using layering, and so is Claudius. So the command wont help
> you much. It should basically boil down to
>
> bitbake multiconfig:qemuamd64-stretch:isar-image-base
>
>> - Commit ID
>
> f33e48bd07039032e3
>
> I am guessing you can not reproduce the issue, we should find out why.
> But since Claudius confirmend that he can not use these patches either,
> and there are still some open points, i think you should remove those
> patches from next.
>

At first, as I understand you use private layering. So if something
doesn't work in your custom environment, without details and analysis
it's not the case to reset 'next' branch.

Before perform any merge, apart from my local builds, the reference
build on CI machine is also passed. So the upstream Isar does support
the claimed features.

If I'm not able to reproduce your issue and there is no information
about problem, then I'm not able to fix it. So if you need assistance in
resolving the issue, please provide more details. If the problem will be
found, the fix should be sent.

Regarding new open points appeared after v5, they contain several
proposals what to try, but they doesn't influence on apt feature.

I'll try the things you proposed with multiconfig and if they work, I'll
send updates for current apt implementation. But working feature in
upstream should not be blocked.

Alex

[Alexander Smirnov]

Please try the patch attached.

[Henning Schild]

On Wed, 18 Oct 2017 20:10:49 +0300

I will try to provide you with a way to reproduce the issue and will
look at the patch you attached. Still the whole series went into next
while it was still under review and still had open points. As far as i
can tell the only difference to my build is that BB_NUMBER_THREADS was
1, but i might be missing something here.
That said i do not understand why you refuse to take it back out, i can
not work on next since these patches are in and i can not even test the
new stuff that was merged on top, same might be true for other
developers.
Also the next q is getting pretty long, as far as i am concerned
everything up to 9c6e8a1e2fb6b65 can be rebased on master and master
can be advanced to that point.

Henning

[Baurzhan Ismagulov]

On Thu, Oct 19, 2017 at 10:41:45AM +0200, Henning Schild wrote:
> I will try to provide you with a way to reproduce the issue and will
> look at the patch you attached.

That would be great. You don't even need to spend time providing a minimal use
case; I suggest that you provide your tree(s) as is off-list and we look at
that with high priority.

> Still the whole series went into next while it was still under review and
> still had open points.

The feature was in discussion since 2017-08-27. The last open points -- EEXIST
and meta-isar-bin -- were addressed in v5. After no comments followed, the
series was merged. It's after that that you report a build problem with an
unpublished layer and suggest a completely new architecture.

Under these circumstances, my suggestion is to make use of the existing feature
v1 -- CIP and other people need that -- and implement the new architecture in
v2.

> That said i do not understand why you refuse to take it back out, i can
> not work on next since these patches are in and i can not even test the
> new stuff that was merged on top, same might be true for other
> developers.

Because the series works for all use cases known to us.

You are demanding to block the feature till you resolve the problem. But the
win-win solution is to solve your problem. This is what we should aim at. We
get mails from you since ten days, but what we would like to have is 1. your
tree, as is, and 2. feedback to Alex's patch -- I see that as a homework before
demanding the same for the third time.

> Also the next q is getting pretty long

We had intended to do a merge to master in Sep, but failed to do so. Merging
the queues while short is exactly what we are trying to achieve right now.


With kind regards,
Baurzhan.

[Henning Schild]

On Thu, 19 Oct 2017 11:54:25 +0200
Baurzhan Ismagulov <i...@radix50.net> wrote:

> On Thu, Oct 19, 2017 at 10:41:45AM +0200, Henning Schild wrote:
> > I will try to provide you with a way to reproduce the issue and will
> > look at the patch you attached.
>
> That would be great. You don't even need to spend time providing a
> minimal use case; I suggest that you provide your tree(s) as is
> off-list and we look at that with high priority.

I will have to look into that, one way or another i can not just give
you what i have.

> > Still the whole series went into next while it was still under
> > review and still had open points.
>
> The feature was in discussion since 2017-08-27. The last open points
> -- EEXIST and meta-isar-bin -- were addressed in v5. After no
> comments followed, the series was merged. It's after that that you
> report a build problem with an unpublished layer and suggest a
> completely new architecture.

The series was merged 3h after it was presented for review, my bad for
being so slow.
Issues not found in earlier rounds of reviews are still issues. And i
have heard many comments about that weird retry-loop, unfortunately not
everyone is active on the list.

> Under these circumstances, my suggestion is to make use of the
> existing feature v1 -- CIP and other people need that -- and
> implement the new architecture in v2.

It is not merged yet, it is just in a rebasing branch where you can
just drop it! But if you want to keep a known to be broken v1 go ahead.

> > That said i do not understand why you refuse to take it back out, i
> > can not work on next since these patches are in and i can not even
> > test the new stuff that was merged on top, same might be true for
> > other developers.
>
> Because the series works for all use cases known to us.

I would like to understand how the code expresses that do_cache_config
has to happen before the first do_populate.
As far as i can see it does not do that, so the race is obvious, but
might be hard to reproduce. My first guess is that BB_NUMBER_THREADS=1
might help get you there as well.
However if it is expressed, i will do my homework and find why that
expression does not work in my case.

> You are demanding to block the feature till you resolve the problem.
> But the win-win solution is to solve your problem. This is what we
> should aim at. We get mails from you since ten days, but what we
> would like to have is 1. your tree, as is, and 2. feedback to Alex's
> patch -- I see that as a homework before demanding the same for the
> third time.

1. only works in our intranet, sorry
2. i have provided that feedback several times and suggested
[lockfiles] and [depends] as means to deal with races and express deps

Henning

[Alexander Smirnov]

On 10/19/2017 04:30 PM, Henning Schild wrote:
> On Thu, 19 Oct 2017 11:54:25 +0200
> Baurzhan Ismagulov <i...@radix50.net> wrote:
>
>> On Thu, Oct 19, 2017 at 10:41:45AM +0200, Henning Schild wrote:
>>> I will try to provide you with a way to reproduce the issue and will
>>> look at the patch you attached.
>>
>> That would be great. You don't even need to spend time providing a
>> minimal use case; I suggest that you provide your tree(s) as is
>> off-list and we look at that with high priority.
>
> I will have to look into that, one way or another i can not just give
> you what i have.
>
>>> Still the whole series went into next while it was still under
>>> review and still had open points.
>>
>> The feature was in discussion since 2017-08-27. The last open points
>> -- EEXIST and meta-isar-bin -- were addressed in v5. After no
>> comments followed, the series was merged. It's after that that you
>> report a build problem with an unpublished layer and suggest a
>> completely new architecture.
>
> The series was merged 3h after it was presented for review, my bad for
> being so slow.

The series was sent to the list 05.10.2017 at 12:00 CES. Series was
pushed to upstream 06.10.2017 after 17:00 CES (don't remember exact
local time, sorry). I suppose you look at commit dates, which reflect
the time I call 'git commit' on my machine and have nothing common with
pushing.

So please be more correct next time.

> Issues not found in earlier rounds of reviews are still issues. And i
> have heard many comments about that weird retry-loop, unfortunately not
> everyone is active on the list.

All the issues found in v4 were fixed, features supported by clean Isar
were tested and eventually series was published. If later you propose to
try another way to do the same things, I'll try it and come back with
the results. But upstream tree should not be frozen until all your ideas
and proposals will be evaluated.

>
>> Under these circumstances, my suggestion is to make use of the
>> existing feature v1 -- CIP and other people need that -- and
>> implement the new architecture in v2.
>
> It is not merged yet, it is just in a rebasing branch where you can
> just drop it! But if you want to keep a known to be broken v1 go ahead.
>
>>> That said i do not understand why you refuse to take it back out, i
>>> can not work on next since these patches are in and i can not even
>>> test the new stuff that was merged on top, same might be true for
>>> other developers.
>>
>> Because the series works for all use cases known to us.
>
> I would like to understand how the code expresses that do_cache_config
> has to happen before the first do_populate.
> As far as i can see it does not do that, so the race is obvious, but
> might be hard to reproduce. My first guess is that BB_NUMBER_THREADS=1
> might help get you there as well.
> However if it is expressed, i will do my homework and find why that
> expression does not work in my case.
>

I've listed the tasks chain and noticed that do_populate has no
predecessor what could be the source of your issue. Please confirm if
the change attached fixes your issue while I'll update the respective
patch in v5 series.

Alex

[Alexander Smirnov]

Install custom packages to target image filesystem using multistrap.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---
meta-isar/recipes-core/images/files/multistrap.conf.in | 9 ++++++++-

meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
meta/classes/ext4-img.bbclass | 2 +-
meta/classes/image.bbclass | 13 ++-----------
4 files changed, 15 insertions(+), 14 deletions(-)

index f4b0125..c9e504e 100644

--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -46,6 +46,9 @@ do_rootfs() {
-e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
-e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+ -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+ -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
+ -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
"${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"

# Create root filesystem

@@ -57,4 +60,4 @@ do_rootfs() {

sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
}

-addtask rootfs before do_populate after do_unpack
+addtask rootfs before do_build after do_populate

diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
fi
}

-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass

index 4d3e0a6..82877f4 100644

--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -61,7 +61,7 @@ addtask cache_config before do_fetch

do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"

-# Install Debian packages, that were built from sources
+# Populate Isar apt repository by newly built packages
do_populate() {
if [ -n "${IMAGE_INSTALL}" ]; then
for p in ${IMAGE_INSTALL}; do

@@ -71,17 +71,8 @@ do_populate() {

includedeb ${DEBDISTRONAME} \
${DEPLOY_DIR_DEB}/${p}_*.deb
done
- sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
- for p in ${IMAGE_INSTALL}; do
- sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
- done
-
- sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
- sudo rm -rf ${IMAGE_ROOTFS}/deb
fi
}

-addtask populate before do_build
+addtask populate before do_build after do_unpack
do_populate[deptask] = "do_deploy_deb"
--
2.1.4

[Henning Schild]

On Wed, 18 Oct 2017 20:14:18 +0300

That solved the problem but introduced the new problem that do_rootfs
could start before do_fetch/unpack.

See my fix here:
https://github.com/henning-schild-work/isar/commit/1e1092be974dbffbf87ab943031c44481ac0908c

btw, that branch is all of current next with only that one fix and the
two kernel version detection patches reverted. So you can consider
current next tested by me to that degree.

Now that it is not broken anymore we can delay the additional changes
for the apt stuff to a v2, IMHO..

Henning

[Alexander Smirnov]

Merged to v5, thanks.

Alex

[Christian Storm]

> [...]

> I would suggest the following changes:
> - create one task in dpkg-base.bbclass that does the following
> - config and init reprepro if no other recipe did that before
> - add package
> - use bitkages task[lockfiles] to deal with races between recipes,
> put distro into lockfile-name so we have one lock per distro
> - drop do_populate
> - call the new task instead of, or in do_deploy_deb

This would be quite useful to source builds with dependencies: Consider
three source built packages A, B, and C. A and B are -dev packages to be
built prior to C. Both may have dependencies satisfiable from the
Debian mirrors. Then, when building C, A and B have to be installed as
build-time dependencies into the buildchroot prior to building C. For
this, the DEPENDS= may be used in the recipe of C to install A and B
just like with any other dependencies coming from the Debian mirrors
as they're in the reprepro repository that must be made accessible in the
buildchroot. For this to work, there's an apt get update needed to
refresh the "view" of the reprepro in the buildchroot after package
addition. This update should be restricted to the reprepro and not
update from the Debian mirrors.

Currently, installing A and B into the buildchroot may be done via
via dpkg -i from DEPLOY_DEB_DIR/{A,B} but this doesn't handle
dependencies. Having A and B in the reprepro repository, and this
reprepro repository being available to the buildchroot, an apt-get
install {A,B} derived from DEPENDS= would work.



Kind regards,
Christian

--
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany