[PATCH] meta: Add support for FILESEXTRAPATHS

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

OE-core provides FILESEXTRAPATHS to extend the search path for
files and patches. This is particularly useful when you want to
add more files or replace existing files using bbappend.

Bring in support for FILESEXTRAPATHS from OE-core.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---
bitbake/conf/bitbake.conf | 3 ++-
meta/classes/base.bbclass | 19 +++++++++++++++++++
2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
index a460df4..9894ed3 100644
--- a/bitbake/conf/bitbake.conf
+++ b/bitbake/conf/bitbake.conf
@@ -24,12 +24,13 @@ DEPENDS = ""
DEPLOY_DIR = "${TMPDIR}/deploy"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images"
DL_DIR = "${TMPDIR}/downloads"
-FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
+FILESEXTRAPATHS ?= "__default:"
GITDIR = "${DL_DIR}/git"
IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
IMAGE_ROOTFS = "${TMPDIR}/rootfs"
OVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
+FILESOVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
P = "${PN}-${PV}"
PERSISTENT_DIR = "${TMPDIR}/cache"
PF = "${PN}-${PV}-${PR}"
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 8c7b021..0670430 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -19,6 +19,7 @@
# OTHER DEALINGS IN THE SOFTWARE.

THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
+FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}","${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files","${FILE_DIRNAME}"], d)}"

def get_deb_host_arch():
import subprocess
@@ -222,3 +223,21 @@ python do_cleanall() {
except bb.fetch2.BBFetchException as e:
bb.fatal(str(e))
}
+
+# Derived from OpenEmbedded Core: meta/classes/utils.bbclass
+def base_set_filespath(path, d):
+ filespath = []
+ extrapaths = (d.getVar("FILESEXTRAPATHS") or "")
+ # Remove default flag which was used for checking
+ extrapaths = extrapaths.replace("__default:", "")
+ # Don't prepend empty strings to the path list
+ if extrapaths != "":
+ path = extrapaths.split(":") + path
+ # The ":" ensures we have an 'empty' override
+ overrides = (":" + (d.getVar("FILESOVERRIDES") or "")).split(":")
+ overrides.reverse()
+ for o in overrides:
+ for p in path:
+ if p != "":
+ filespath.append(os.path.join(p, o))
+ return ":".join(filespath)
--
2.17.1

[Jan Kiszka]

On 03.10.19 15:54, vijaikumar....@gmail.com wrote:
> From: Vijai Kumar K <Vijaikumar_...@mentor.com>
>
> OE-core provides FILESEXTRAPATHS to extend the search path for
> files and patches. This is particularly useful when you want to
> add more files or replace existing files using bbappend.
>
> Bring in support for FILESEXTRAPATHS from OE-core.

Technically, this is not needed, prepending FILESPATH directly also works. It
might still be a good idea to add this for conformance with OE.

But as this is no a bitbake interface, it should be documented. Also FILESOVERRIDES.

Jan

Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Vijai Kumar K]

On Fri, Oct 04, 2019 at 09:30:17AM +0200, Jan Kiszka wrote:
> On 03.10.19 15:54, vijaikumar....@gmail.com wrote:
> > From: Vijai Kumar K <Vijaikumar_...@mentor.com>
> >
> > OE-core provides FILESEXTRAPATHS to extend the search path for
> > files and patches. This is particularly useful when you want to
> > add more files or replace existing files using bbappend.
> >
> > Bring in support for FILESEXTRAPATHS from OE-core.
>
> Technically, this is not needed, prepending FILESPATH directly also works.
> It might still be a good idea to add this for conformance with OE.

Right now, yes, we might not need it. We could use the FILESPATH_prepend like you
suggested. But one case which came to my mind while testing both the solutions is that
you could manipulate the order of FILESPATH with FILESEXTRAPATHS and FILESOVERRIDES.
This was particularly useful when I want to replace a file only for particular distro
or a particular machine.

Thanks,
Vijai Kumar K

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

OE-core provides FILESEXTRAPATHS to extend the search path for
files and patches. This is particularly useful when you want to
add more files or replace existing files using bbappend.

Bring in support for FILESEXTRAPATHS from OE-core.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

Changes in v2:
- Captured documentation for FILESEXTRAPATHS & FILESOVERRIDES.

bitbake/conf/bitbake.conf | 3 ++-
doc/user_manual.md | 2 ++
meta/classes/base.bbclass | 19 +++++++++++++++++++
3 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
index a460df4..9894ed3 100644
--- a/bitbake/conf/bitbake.conf
+++ b/bitbake/conf/bitbake.conf
@@ -24,12 +24,13 @@ DEPENDS = ""
DEPLOY_DIR = "${TMPDIR}/deploy"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images"
DL_DIR = "${TMPDIR}/downloads"
-FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
+FILESEXTRAPATHS ?= "__default:"
GITDIR = "${DL_DIR}/git"
IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
IMAGE_ROOTFS = "${TMPDIR}/rootfs"
OVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
+FILESOVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
P = "${PN}-${PV}"
PERSISTENT_DIR = "${TMPDIR}/cache"
PF = "${PN}-${PV}-${PR}"

diff --git a/doc/user_manual.md b/doc/user_manual.md
index c2657da..e096c24 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -321,6 +321,8 @@ Some other variables include:
- `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
- `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional.
- `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping
+ - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
+ - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.

---

2.17.1

[Baurzhan Ismagulov]

Hello Vijai Kumar,

On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar....@gmail.com wrote:
> OE-core provides FILESEXTRAPATHS to extend the search path for
> files and patches. This is particularly useful when you want to
> add more files or replace existing files using bbappend.
>
> Bring in support for FILESEXTRAPATHS from OE-core.

Thanks. What are the possibilities of adding a test case for this?

With kind regards,
Baurzhan.

[Vijai Kumar K]

Baurzhan,

Sorry for the late reply. I was on vacation.

Is there some test suite in ISAR to test features like these? Just wondering
where would the test cases go.

Thanks,
Vijai Kumar K
>
> With kind regards,
> Baurzhan.
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20191007093737.5coyplnujrjobtc4%40yssyq.m.ilbers.de.

[Jan Kiszka]

On 14.10.19 08:40, Vijai Kumar K wrote:
> On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
>> Hello Vijai Kumar,
>>
>> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar....@gmail.com wrote:
>>> OE-core provides FILESEXTRAPATHS to extend the search path for
>>> files and patches. This is particularly useful when you want to
>>> add more files or replace existing files using bbappend.
>>>
>>> Bring in support for FILESEXTRAPATHS from OE-core.
>>
>> Thanks. What are the possibilities of adding a test case for this?
>
>
> Baurzhan,
>
> Sorry for the late reply. I was on vacation.
>
> Is there some test suite in ISAR to test features like these? Just wondering
> where would the test cases go.

- add a use case of the new feature a some recipe in meta-isar
- make sure that recipe is built as part of scripts/ci_build.sh, ideally
the fast variant if the feature is not taking too much extra time

Jan

[Jan Kiszka]

On 14.10.19 09:49, [ext] Jan Kiszka wrote:
> On 14.10.19 08:40, Vijai Kumar K wrote:
>> On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
>>> Hello Vijai Kumar,
>>>
>>> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar....@gmail.com wrote:
>>>> OE-core provides FILESEXTRAPATHS to extend the search path for
>>>> files and patches. This is particularly useful when you want to
>>>> add more files or replace existing files using bbappend.
>>>>
>>>> Bring in support for FILESEXTRAPATHS from OE-core.
>>>
>>> Thanks. What are the possibilities of adding a test case for this?
>>
>>
>> Baurzhan,
>>
>> Sorry for the late reply. I was on vacation.
>>
>> Is there some test suite in ISAR to test features like these? Just wondering
>> where would the test cases go.
>
> - add a use case of the new feature a some recipe in meta-isar

... to some recipe ...

[Vijai Kumar K]

On Mon, Oct 14, 2019 at 09:49:47AM +0200, Jan Kiszka wrote:
> On 14.10.19 08:40, Vijai Kumar K wrote:
> > On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
> >> Hello Vijai Kumar,
> >>
> >> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar....@gmail.com wrote:
> >>> OE-core provides FILESEXTRAPATHS to extend the search path for
> >>> files and patches. This is particularly useful when you want to
> >>> add more files or replace existing files using bbappend.
> >>>
> >>> Bring in support for FILESEXTRAPATHS from OE-core.
> >>
> >> Thanks. What are the possibilities of adding a test case for this?
> >
> >
> > Baurzhan,
> >
> > Sorry for the late reply. I was on vacation.
> >
> > Is there some test suite in ISAR to test features like these? Just wondering
> > where would the test cases go.
>
> - add a use case of the new feature a some recipe in meta-isar
> - make sure that recipe is built as part of scripts/ci_build.sh, ideally
> the fast variant if the feature is not taking too much extra time

I believe we could replace FILESPATH with FILESEXTRAPATHS in the below files.
That should give a test case also would make it easier to override the files
using bbappends.

meta/recipes-bsp/u-boot/u-boot-custom.inc
meta/recipes-kernel/linux-module/module.inc
meta/recipes-kernel/linux/linux-custom.inc

-FILESPATH =. "${LAYERDIR_core}/recipes-kernel/linux/files:"
+FILESEXTRAPATHS_prepend := "${FILE_DIRNAME}/files:"

Will test and send a v2 with an additional patch for review.

Thanks,
Vijai Kumar K