[PATCH v2 0/2] FILESEXTRAPATHS

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

This v2 patch has the following changes.

- Modify the correct bitbake.conf file.
- Add a new patch to use FILESEXTRAPATHS in linux-custom.inc,
u-boot-custom.inc and module.inc.

The second patch in the series should also hopefully serve
as a testcase to validate FILESEXTRAPATHS.

Vijai Kumar K (2):
meta: Add support for FILESEXTRAPATHS
Replace FILESPATH with FILESEXTRAPATHS

doc/user_manual.md | 2 ++
meta/classes/base.bbclass | 19 +++++++++++++++++++
meta/conf/bitbake.conf | 3 ++-
meta/recipes-bsp/u-boot/u-boot-custom.inc | 2 +-
meta/recipes-kernel/linux-module/module.inc | 2 +-
meta/recipes-kernel/linux/linux-custom.inc | 2 +-
6 files changed, 26 insertions(+), 4 deletions(-)

--
2.17.1

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

OE-core provides FILESEXTRAPATHS to extend the search path for
files and patches. This is particularly useful when you want to
add more files or replace existing files using bbappend.

Bring in support for FILESEXTRAPATHS from OE-core.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

doc/user_manual.md | 2 ++
meta/classes/base.bbclass | 19 +++++++++++++++++++
meta/conf/bitbake.conf | 3 ++-

3 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index c2657da..e096c24 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -321,6 +321,8 @@ Some other variables include:
- `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
- `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional.
- `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping
+ - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
+ - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.

---

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 8c7b021..0670430 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -19,6 +19,7 @@
# OTHER DEALINGS IN THE SOFTWARE.

THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
+FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}","${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files","${FILE_DIRNAME}"], d)}"

def get_deb_host_arch():
import subprocess
@@ -222,3 +223,21 @@ python do_cleanall() {
except bb.fetch2.BBFetchException as e:
bb.fatal(str(e))
}
+
+# Derived from OpenEmbedded Core: meta/classes/utils.bbclass
+def base_set_filespath(path, d):
+ filespath = []
+ extrapaths = (d.getVar("FILESEXTRAPATHS") or "")
+ # Remove default flag which was used for checking
+ extrapaths = extrapaths.replace("__default:", "")
+ # Don't prepend empty strings to the path list
+ if extrapaths != "":
+ path = extrapaths.split(":") + path
+ # The ":" ensures we have an 'empty' override
+ overrides = (":" + (d.getVar("FILESOVERRIDES") or "")).split(":")
+ overrides.reverse()
+ for o in overrides:
+ for p in path:
+ if p != "":
+ filespath.append(os.path.join(p, o))
+ return ":".join(filespath)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index bca1114..1f20c98 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -23,8 +23,8 @@ B = "${S}"
CVSDIR = "${DL_DIR}/cvs"
DEPENDS = ""
DEPLOY_DIR = "${TMPDIR}/deploy"
-FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
+FILESEXTRAPATHS ?= "__default:"
GITDIR = "${DL_DIR}/git"
P = "${PN}-${PV}"
PF = "${PN}-${PV}-${PR}"
@@ -58,6 +58,7 @@ SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
+FILESOVERRIDES = "${DISTRO_ARCH}:${MACHINE}"

# Setting default QEMU_ARCH variables for different DISTRO_ARCH:
QEMU_ARCH_amd64 = "x86_64"
--
2.17.1

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

The suggested approach to include files & patches from bbappend
files is FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

This is not possible if FILESPATH is modified directly. Use
FILESEXTRAPATHS instead.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/recipes-bsp/u-boot/u-boot-custom.inc | 2 +-
meta/recipes-kernel/linux-module/module.inc | 2 +-
meta/recipes-kernel/linux/linux-custom.inc | 2 +-

3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 8843789..41124fc 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -5,7 +5,7 @@
#
# SPDX-License-Identifier: MIT

-FILESPATH =. "${LAYERDIR_core}/recipes-bsp/u-boot/files:"
+FILESEXTRAPATHS_prepend := "${FILE_DIRNAME}/files:"

DESCRIPTION ?= "Custom U-Boot"

diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 6506e39..d30e799 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -5,7 +5,7 @@
#
# SPDX-License-Identifier: MIT

-FILESPATH =. "${LAYERDIR_core}/recipes-kernel/linux-module/files:"
+FILESEXTRAPATHS_prepend := "${FILE_DIRNAME}/files:"

DESCRIPTION ?= "Custom kernel module ${PN}"

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 5a4f5bf..dc2af1a 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -5,7 +5,7 @@
#
# SPDX-License-Identifier: MIT

-FILESPATH =. "${LAYERDIR_core}/recipes-kernel/linux/files:"
+FILESEXTRAPATHS_prepend := "${FILE_DIRNAME}/files:"

DESCRIPTION ?= "Custom kernel"

--
2.17.1

[Henning Schild]

Hi Kumar,

Good idea, i guess we all stumbled over this several times ;).

This deserves an entry into the RECIPE-API-CHANGELOG.md. To let people
following that know that they can and should migrate their layers.

Henning

Am Wed, 16 Oct 2019 20:07:43 +0530
schrieb <vijaikumar....@gmail.com>:

[Vijai Kumar K]

On Wed, Oct 16, 2019 at 07:05:36PM +0200, Henning Schild wrote:
> Hi Kumar,
>
> Good idea, i guess we all stumbled over this several times ;).

Yep ;)

>
> This deserves an entry into the RECIPE-API-CHANGELOG.md. To let people
> following that know that they can and should migrate their layers.

Sure will add and entry and send v3 for review.

Thanks,
Vijai Kumar K

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

Changes in v3:
- Added entry in RECIPE-API-CHANGELOG.md

This v2 patch has the following changes.

- Modify the correct bitbake.conf file.
- Add a new patch to use FILESEXTRAPATHS in linux-custom.inc,
u-boot-custom.inc and module.inc.

The second patch in the series should also hopefully serve
as a testcase to validate FILESEXTRAPATHS.

Vijai Kumar K (2):
meta: Add support for FILESEXTRAPATHS
Replace FILESPATH with FILESEXTRAPATHS

RECIPE-API-CHANGELOG.md | 10 ++++++++++

doc/user_manual.md | 2 ++
meta/classes/base.bbclass | 19 +++++++++++++++++++
meta/conf/bitbake.conf | 3 ++-
meta/recipes-bsp/u-boot/u-boot-custom.inc | 2 +-
meta/recipes-kernel/linux-module/module.inc | 2 +-
meta/recipes-kernel/linux/linux-custom.inc | 2 +-

7 files changed, 36 insertions(+), 4 deletions(-)

--
2.17.1

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

OE-core provides FILESEXTRAPATHS to extend the search path for
files and patches. This is particularly useful when you want to
add more files or replace existing files using bbappend.

Bring in support for FILESEXTRAPATHS from OE-core.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

RECIPE-API-CHANGELOG.md | 10 ++++++++++
doc/user_manual.md | 2 ++
meta/classes/base.bbclass | 19 +++++++++++++++++++
meta/conf/bitbake.conf | 3 ++-

4 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index bbef1a3..e9072d7 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -176,3 +176,13 @@ Otherwise set a encrypted root password like this:
USERS += "root"
USER_root[password] = "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
```
+### Use FILESEXTRAPATHS to add custom paths to FILESPATH
+
+Direct modification of FILESPATH variable is discouraged. Use FILESEXTRAPATHS
+instead to add a custom search path for files and patches. This makes overriding
+files and patches using bbappend a lot easier.
+
+For example:
+```
+FILESEXTRAPATHS_prepend := "$THISDIR/files:"
+```

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

The suggested approach to include files & patches from bbappend
files is FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

This is not possible if FILESPATH is modified directly. Use
FILESEXTRAPATHS instead.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/recipes-bsp/u-boot/u-boot-custom.inc | 2 +-
meta/recipes-kernel/linux-module/module.inc | 2 +-
meta/recipes-kernel/linux/linux-custom.inc | 2 +-

[Baurzhan Ismagulov]

On Thu, Oct 17, 2019 at 02:58:54PM +0530, vijaikumar....@gmail.com wrote:
> meta: Add support for FILESEXTRAPATHS
> Replace FILESPATH with FILESEXTRAPATHS

The patches look good to me and I'd like to apply them. However, I get the
following unrelated errors in CI:

ERROR: Unable to start bitbake server
ERROR: Last 10 lines of server log for this session (/workspace/build/isar_ibr_devel/65/build/bitbake-cookerdaemon.log):
Traceback (most recent call last):
File "/workspace/build/isar_ibr_devel/65/bitbake/lib/bb/daemonize.py", line 83, in createDaemon
function()
File "/workspace/build/isar_ibr_devel/65/bitbake/lib/bb/server/process.py", line 444, in _startServer
self.cooker = bb.cooker.BBCooker(self.configuration, self.featureset)
File "/workspace/build/isar_ibr_devel/65/bitbake/lib/bb/cooker.py", line 185, in __init__
self.watcher = pyinotify.WatchManager()
File "/workspace/build/isar_ibr_devel/65/bitbake/lib/pyinotify.py", line 1764, in __init__
raise OSError(err % self._inotify_wrapper.str_errno())
OSError: Cannot initialize new instance of inotify, Errno=Too many open files (EMFILE)

Has anyone seen this?

With kind regards,
Baurzhan.

[Henning Schild]

Am Fri, 25 Oct 2019 14:49:50 +0200
schrieb Baurzhan Ismagulov <i...@radix50.net>:

Not yet. I think it is a ulimit thing, maybe caused by jenkins or your
hosts distro.

Or maybe our builds actually keep files open and everyone building in
containers means we do not see it. Try "lsof" and regular reboots ;).

You are truly building that directly on a debian buildhost with the
jenkins user having sudo there?

Henning

> With kind regards,
> Baurzhan.
>

[Baurzhan Ismagulov]

Hello Henning,

thanks for the response.

On Fri, Oct 25, 2019 at 03:22:37PM +0200, Henning Schild wrote:
> Not yet. I think it is a ulimit thing, maybe caused by jenkins or your
> hosts distro.
>
> Or maybe our builds actually keep files open and everyone building in
> containers means we do not see it. Try "lsof" and regular reboots ;).

I'd like to understand the problem before I start playing with the knobs :) .
Moving to a different container solution or rebooting wouldn't solve the
underlying problem. Besides, I'm not sure this problem doesn't affect e.g.
docker, since ulimits are per-process, and this one occurred during bitbake -e
in start_vm.

With kind regards,
Baurzhan.

[Jan Kiszka]

echo 1024 > /proc/sys/fs/inotify/max_user_instances

is required even on my notebook when too many processes watch files,
only some of them being bitbake & Co.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Baurzhan Ismagulov]

On Fri, Oct 25, 2019 at 07:17:57PM +0200, Jan Kiszka wrote:
> echo 1024 > /proc/sys/fs/inotify/max_user_instances
>
> is required even on my notebook when too many processes watch files,
> only some of them being bitbake & Co.

Thanks for the explanation! This worked.

With kind regards,
Baurzhan.

[Baurzhan Ismagulov]

On Thu, Oct 17, 2019 at 02:58:54PM +0530, vijaikumar....@gmail.com wrote:

> meta: Add support for FILESEXTRAPATHS
> Replace FILESPATH with FILESEXTRAPATHS

Applied to next, thanks.

With kind regards,
Baurzhan.

[Henning Schild]

New warnings with these new patches:

builder@51066e51c54c:/out/build$ bitbake -v
multiconfig:qemui386-buster:isar-image-base WARNING:
/this/meta-isar/recipes-kernel/example-module/example-module.bb:
Getting checksum for example-module-${KERNEL_NAME} SRC_URI entry : file
not found except in DL_DIR WARNING:
/this/meta-isar/recipes-kernel/example-module/example-module.bb: Unable
to get checksum for example-module-${KERNEL_NAME} SRC_URI entry src:
file could not be found

Henning

Am Mon, 4 Nov 2019 20:45:02 +0100
schrieb Baurzhan Ismagulov <i...@radix50.net>:

[vijai kumar]

Hi Henning,

I'll look into it. I remember seeing this without the patch. But I might be mistaken. I'll check it out.

Thanks,

Vijai Kumar K

[Jan Kiszka]

On 05.11.19 17:18, vijai kumar wrote:
> Hi Henning,
>
> I'll look into it. I remember seeing this without the patch. But I might
> be mistaken. I'll check it out.

It's clearly related to the patches, seen it here as well, resolved it
be dropping them. Likely it's due to some early variable evaluation that
now breaks.

Jan

>
> Thanks,
> Vijai Kumar K
>
>
> On Tue 5 Nov, 2019, 9:38 PM Henning Schild, <henning...@siemens.com

> <mailto:henning...@siemens.com>> wrote:
>
> New warnings with these new patches:
>
> builder@51066e51c54c:/out/build$ bitbake -v
> multiconfig:qemui386-buster:isar-image-base WARNING:
> /this/meta-isar/recipes-kernel/example-module/example-module.bb

> <http://example-module.bb>:

> Getting checksum for example-module-${KERNEL_NAME} SRC_URI entry : file
> not found except in DL_DIR WARNING:
> /this/meta-isar/recipes-kernel/example-module/example-module.bb

> <http://example-module.bb>: Unable

> to get checksum for example-module-${KERNEL_NAME} SRC_URI entry src:
> file could not be found
>
> Henning
>
> Am Mon, 4 Nov 2019 20:45:02 +0100

> schrieb Baurzhan Ismagulov <i...@radix50.net <mailto:i...@radix50.net>>:

>
> > On Thu, Oct 17, 2019 at 02:58:54PM +0530,
> > vijaikumar....@gmail.com

> <mailto:vijaikumar....@gmail.com> wrote:
> > >   meta: Add support for FILESEXTRAPATHS
> > >   Replace FILESPATH with FILESEXTRAPATHS 
> >
> > Applied to next, thanks.
> >
> > With kind regards,
> > Baurzhan.
> >
>

> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+...@googlegroups.com
> <mailto:isar-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/CALLGG_K9xx2uCci2y4%2B6y%2BjH_isLfR95f2dMphvtd-omcOsDUQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/isar-users/CALLGG_K9xx2uCci2y4%2B6y%2BjH_isLfR95f2dMphvtd-omcOsDUQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[vijai kumar]

On Tue 5 Nov, 2019, 10:37 PM Jan Kiszka, <jan.k...@siemens.com> wrote:

On 05.11.19 17:18, vijai kumar wrote:
> Hi Henning,
>
> I'll look into it. I remember seeing this without the patch. But I might
> be mistaken. I'll check it out.

It's clearly related to the patches, seen it here as well, resolved it
be dropping them. Likely it's due to some early variable evaluation that
now breaks.

Jan

Oh Ok. Thanks Jan.