[PATCH] opkg-build: Add acls and xattrs support

54 views
Skip to first unread message

Piotr Łobacz

unread,
Jul 10, 2023, 5:03:28 AM7/10/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLS and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..8d9bcfa 100755
--- a/opkg-build
+++ b/opkg-build
@@ -145,6 +145,7 @@ You probably want to chown these to a system user: " >&2
###
# opkg-build "main"
###
+attributesargs=""
ogargs=""
outer=ar
noclean=0
@@ -166,7 +167,7 @@ compressorargs=""
tarformat=""
if tar --help 2>&1 | grep -- "--format" > /dev/null;
then
- tarformat="--format=gnu"
+ tarformat="--format=posix"
fi

compressor_ext() {
@@ -197,13 +198,17 @@ compressor_ext() {
: <<=cut
=head1 SYNOPSIS

-B<opkg-build> [B<-c>] [B<-C>] [B<-Z> I<compressor>] [B<-a>] [B<-O>] [B<-o> I<owner>] [B<-g> I<group>] I<pkg_directory> [I<destination_directory>]
+B<opkg-build> [B<-A>] [B<-X>] [B<-c>] [B<-C>] [B<-Z> I<compressor>] [B<-a>] [B<-O>] [B<-o> I<owner>] [B<-g> I<group>] I<pkg_directory> [I<destination_directory>]

=cut

-usage="Usage: $0 [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] <pkg_directory> [<destination_directory>]"
-while getopts "a:cCg:ho:vOZ:" opt; do
+usage="Usage: $0 [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] <pkg_directory> [<destination_directory>]"
+while getopts "Aa:cCg:ho:vOXZ:" opt; do
case $opt in
+ A ) attributesargs="--acls"
+ ;;
+ X ) attributesargs="$attributesargs --xattrs"
+ ;;
o ) owner=$OPTARG
ogargs="--owner=$owner"
;;
@@ -314,7 +319,7 @@ export LANG=C
export LC_ALL=C
( cd $pkg_dir/$CONTROL && find . -type f | sort > $tmp_dir/control_list )
( cd $pkg_dir && find . -path ./$CONTROL -prune -o -path . -o -print | sort > $tmp_dir/file_list )
-( cd $pkg_dir && tar $ogargs $tsortargs --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )
+( cd $pkg_dir && tar $attributesargs $ogargs $tsortargs --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )
( cd $pkg_dir/$CONTROL && tar $ogargs $tsortargs --no-recursion --mtime=@$build_date -c $tarformat -T $tmp_dir/control_list | gzip $zipargs > $tmp_dir/control.tar.gz )
rm $tmp_dir/file_list
rm $tmp_dir/control_list
--
2.34.1

Alex Stewart

unread,
Jul 12, 2023, 3:50:39 PM7/12/23
to Piotr Łobacz, opkg-...@googlegroups.com
Pulled this PR to opkg-utils as commit
dc50e85ff1ceb2ffae2634a001f524a84a1b40e7 [1].

[1]
https://git.yoctoproject.org/opkg-utils/commit/?id=dc50e85ff1ceb2ffae2634a001f524a84a1b40e7

Thanks!
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

alex.s...@ni.com

Piotr Łobacz

unread,
Jul 22, 2023, 9:41:12 AM7/22/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLs and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..74035f0 100755
+( cd $pkg_dir && tar $attributesargs $ogargs $tsortargs --numeric-owner --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )
( cd $pkg_dir/$CONTROL && tar $ogargs $tsortargs --no-recursion --mtime=@$build_date -c $tarformat -T $tmp_dir/control_list | gzip $zipargs > $tmp_dir/control.tar.gz )
rm $tmp_dir/file_list
rm $tmp_dir/control_list
--
2.34.1

Piotr Łobacz

unread,
Jul 22, 2023, 9:45:27 AM7/22/23
to opkg-...@googlegroups.com
I'm sorry Alex, but for some reason I haven't noticed that somehow, I missed that `--numeric-owner` extra param...

BR
Piotr

Od: Piotr Łobacz <p.lo...@welotec.com>
Wysłane: sobota, 22 lipca 2023 15:40
Do: opkg-...@googlegroups.com <opkg-...@googlegroups.com>
DW: Piotr Łobacz <p.lo...@welotec.com>
Temat: [PATCH v2] opkg-build: Add acls and xattrs support
 

Piotr Łobacz

unread,
Jul 24, 2023, 12:17:15 PM7/24/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLs and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 54 ++++++++++++++++++++++++++++++++++--------------------
1 file changed, 34 insertions(+), 20 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..8e327f5 100755
--- a/opkg-build
+++ b/opkg-build
@@ -145,6 +145,7 @@ You probably want to chown these to a system user: " >&2
###
# opkg-build "main"
###
+attributesargs=""
ogargs=""
outer=ar
noclean=0
@@ -153,22 +154,6 @@ compressor=gzip
zipargs="-9n"
compressorargs=""

-# Determine if tar supports the --format argument by checking the help output.
-#
-# This is needed because:
-# - Busybox tar doesn't support '--format'
-# - On some Linux distros, tar now defaults to posix format if '--format'
-# isn't explicitly specified
-# - Opkg doesn't currently support posix format archives
-#
-# It's easier to check for mention of the '--format' option than to detect the
-# tar implementation and maintain a list of which support '--format'.
-tarformat=""
-if tar --help 2>&1 | grep -- "--format" > /dev/null;
-then
- tarformat="--format=gnu"
-fi
-
compressor_ext() {
case $1 in
gzip|pigz)
@@ -197,13 +182,17 @@ compressor_ext() {
: <<=cut
=head1 SYNOPSIS

-B<opkg-build> [B<-c>] [B<-C>] [B<-Z> I<compressor>] [B<-a>] [B<-O>] [B<-o> I<owner>] [B<-g> I<group>] I<pkg_directory> [I<destination_directory>]
+B<opkg-build> [B<-A>] [B<-X>] [B<-c>] [B<-C>] [B<-Z> I<compressor>] [B<-a>] [B<-O>] [B<-o> I<owner>] [B<-g> I<group>] I<pkg_directory> [I<destination_directory>]

=cut

-usage="Usage: $0 [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] <pkg_directory> [<destination_directory>]"
-while getopts "a:cCg:ho:vOZ:" opt; do
+usage="Usage: $0 [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] <pkg_directory> [<destination_directory>]"
+while getopts "Aa:cCg:ho:vOXZ:" opt; do
case $opt in
+ A ) attributesargs="--acls"
+ ;;
+ X ) attributesargs="$attributesargs --xattrs"
+ ;;
o ) owner=$OPTARG
ogargs="--owner=$owner"
;;
@@ -232,6 +221,31 @@ while getopts "a:cCg:ho:vOZ:" opt; do
esac
done

+# Determine if tar supports the --format argument by checking the help output.
+#
+# This is needed because:
+# - Busybox tar doesn't support '--format'
+# - On some Linux distros, tar now defaults to posix format if '--format'
+# isn't explicitly specified
+# - Opkg doesn't currently support posix format archives
+#
+# It's easier to check for mention of the '--format' option than to detect the
+# tar implementation and maintain a list of which support '--format'.
+tarformat=""
+if tar --help 2>&1 | grep -- "--format" > /dev/null;
+then
+ # For ACLs or xattr support, gnu format will not work
+ # we need to set posix format instead
+ if [ ! -z "$attributesargs" ] ; then
+ tarformat="--format=posix"
+ else
+ tarformat="--format=gnu"
+ fi
+elif [ ! -z "$attributesargs" ] ; then
+ echo "*** Error: Attributes: $attributesargs, doesn't' work, without posix format, which is not supported by tar command." >&2
+ exit 1
+fi
+
cext=$(compressor_ext $compressor)

# pgzip requires -T to avoid timestamps on the gzip archive
@@ -314,7 +328,7 @@ export LANG=C

Piotr Łobacz

unread,
Jul 24, 2023, 12:27:37 PM7/24/23
to opkg-...@googlegroups.com
Hi Alex,
I have reverted/modified some changes to have/use actually still the gnu format in case acl/xattr is not being used.

I need to figure out what is causing tar to actually modify these atime/ctime parameters for the archive packages...

BR
Piotr

Od: Piotr Łobacz <p.lo...@welotec.com>
Wysłane: poniedziałek, 24 lipca 2023 18:07
Temat: [PATCH v3] opkg-build: Add acls and xattrs support
 

Piotr Łobacz

unread,
Jul 24, 2023, 4:43:10 PM7/24/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLs and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 76 +++++++++++++++++++++++++++++++++++-------------------
1 file changed, 50 insertions(+), 26 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..b3127e0 100755
@@ -301,21 +315,31 @@ fi
tmp_dir=$dest_dir/IPKG_BUILD.$$
mkdir $tmp_dir

-build_date="${SOURCE_DATE_EPOCH:-$(date +%s)}"
-
-mtime_args=""
+mtime_args="--mtime=@${SOURCE_DATE_EPOCH:-$(date +%s)}"
# --clamp-mtime requires tar > 1.28. Only use it if SOURCE_DATE_EPOCH is set, to avoid having a generic case dependency on tar > 1.28.
# this setting will make sure files generated at build time have consistent mtimes, for reproducible builds.
if [ ! -z "$SOURCE_DATE_EPOCH" ]; then
- mtime_args="--mtime=@$build_date --clamp-mtime"
+ mtime_args="$mtime_args --clamp-mtime"
+fi
+
+# Notice, that if you create an archive in POSIX format (see section GNU tar and POSIX tar) and the environment variable POSIXLY_CORRECT is set,
+# then the two archives created using the same options on the same set of files will not be byte-to-byte equivalent even with the above option.
+# This is because the posix default for extended header names includes the PID of the tar process, which is different at each run. To produce
+# byte-to-byte equivalent archives in this case, either unset POSIXLY_CORRECT, or use the following option:
+#
+# --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0
+#
+# https://www.gnu.org/software/tar/manual/html_node/PAX-keywords.html
+if [ $tarformat == "--format=posix" ]; then
+ mtime_args="$mtime_args --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0"
fi

export LANG=C
export LC_ALL=C
( cd $pkg_dir/$CONTROL && find . -type f | sort > $tmp_dir/control_list )
( cd $pkg_dir && find . -path ./$CONTROL -prune -o -path . -o -print | sort > $tmp_dir/file_list )
-( cd $pkg_dir && tar $ogargs $tsortargs --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )
-( cd $pkg_dir/$CONTROL && tar $ogargs $tsortargs --no-recursion --mtime=@$build_date -c $tarformat -T $tmp_dir/control_list | gzip $zipargs > $tmp_dir/control.tar.gz )
+( cd $pkg_dir && tar $attributesargs $ogargs $tsortargs --numeric-owner --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )
+( cd $pkg_dir/$CONTROL && tar $ogargs $tsortargs --no-recursion $mtime_args -c $tarformat -T $tmp_dir/control_list | gzip $zipargs > $tmp_dir/control.tar.gz )
rm $tmp_dir/file_list
rm $tmp_dir/control_list

@@ -331,7 +355,7 @@ rm -f $pkg_file
if [ "$outer" = "ar" ] ; then
( cd $tmp_dir && ar -crfD $pkg_file ./debian-binary ./control.tar.gz ./data.tar.$cext )
else
- ( cd $tmp_dir && tar -c $tsortargs --mtime=@$build_date $tarformat ./debian-binary ./control.tar.gz ./data.tar.$cext | gzip $zipargs > $pkg_file )
+ ( cd $tmp_dir && tar -c $tsortargs $mtime_args $tarformat ./debian-binary ./control.tar.gz ./data.tar.$cext | gzip $zipargs > $pkg_file )
fi

rm $tmp_dir/debian-binary $tmp_dir/data.tar.$cext $tmp_dir/control.tar.gz
--
2.34.1

Piotr Łobacz

unread,
Jul 26, 2023, 5:17:45 AM7/26/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLs and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 76 +++++++++++++++++++++++++++++++++++-------------------
1 file changed, 50 insertions(+), 26 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..47ac1a8 100755

Piotr Łobacz

unread,
Aug 4, 2023, 4:55:08 AM8/4/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLs and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 77 ++++++++++++++++++++++++++++++++++++------------------
1 file changed, 51 insertions(+), 26 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..2a2e607 100755
@@ -232,6 +221,32 @@ while getopts "a:cCg:ho:vOZ:" opt; do
esac
done

+# Determine if tar supports the --format argument by checking the help output.
+#
+# This is needed because:
+# - Busybox tar doesn't support '--format'
+# - On some Linux distros, tar now defaults to posix format if '--format'
+# isn't explicitly specified
+# - Opkg currently supports posix format archives, but gnu format is left
+# here intentionally for backward compatibility
+#
+# It's easier to check for mention of the '--format' option than to detect the
+# tar implementation and maintain a list of which support '--format'.
+tarformat=""
+if tar --help 2>&1 | grep -- "--format" > /dev/null;
+then
+ # For ACLs or xattr support, gnu format will not work
+ # we need to set posix format instead
+ if [ ! -z "$attributesargs" ] ; then
+ tarformat="--format=posix"
+ else
+ tarformat="--format=gnu"
+ fi
+elif [ ! -z "$attributesargs" ] ; then
+ echo "*** Error: Attributes: $attributesargs, don't work, without posix format, which is not supported by host's tar command." >&2
+ exit 1
+fi
+
cext=$(compressor_ext $compressor)

# pgzip requires -T to avoid timestamps on the gzip archive
@@ -301,21 +316,31 @@ fi
@@ -331,7 +356,7 @@ rm -f $pkg_file

Alex Stewart

unread,
Aug 15, 2023, 6:00:07 PM8/15/23
to Piotr Łobacz, opkg-...@googlegroups.com
Hey Piotr,

Sorry for the delay. :(

The patch looks good to me in concept. I have a couple concerns inline
below.

Once we have an agreement on this new patch version, I'll put in a
revert for the Patch V1 and apply this new version atop it.


On 8/4/23 04:53, Piotr Łobacz wrote:
> Add support for tar archives created with --acls and/or --xattrs options,
> PAX header format.
>
> GNU tar and libarchive already supports ACLs and extended attributes.
> We can now add this support as well to opkg-build script in order to use
> fsetattr or setcap inside do_install command and end up with a file in
> an image with the relevant ACLs and xattrs.
>
> Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
> ---
> opkg-build | 77 ++++++++++++++++++++++++++++++++++++------------------
> 1 file changed, 51 insertions(+), 26 deletions(-)
>
> diff --git a/opkg-build b/opkg-build
> index a9e45d4..2a2e607 100755
> --- a/opkg-build
> +++ b/opkg-build

The opkg-build script mostly uses tabs for indentation. But this patch
uses spaces. Could you convert your spaces to tabs?
If a user calls `opkg-build -X -A`, then `attributesargs` only gets the
value `--acls`. I think you just need to make the variable assignment in
the A case match the X case.

Piotr Łobacz

unread,
Aug 16, 2023, 8:59:57 AM8/16/23
to opkg-...@googlegroups.com, Piotr Łobacz
Add support for tar archives created with --acls and/or --xattrs options,
PAX header format.

GNU tar and libarchive already supports ACLs and extended attributes.
We can now add this support as well to opkg-build script in order to use
fsetattr or setcap inside do_install command and end up with a file in
an image with the relevant ACLs and xattrs.

Signed-off-by: Piotr Łobacz <p.lo...@welotec.com>
---
opkg-build | 81 +++++++++++++++++++++++++++++++++++-------------------
1 file changed, 53 insertions(+), 28 deletions(-)

diff --git a/opkg-build b/opkg-build
index a9e45d4..fb0873d 100755
--- a/opkg-build
+++ b/opkg-build
@@ -197,20 +182,24 @@ compressor_ext() {
: <<=cut
=head1 SYNOPSIS

-B<opkg-build> [B<-c>] [B<-C>] [B<-Z> I<compressor>] [B<-a>] [B<-O>] [B<-o> I<owner>] [B<-g> I<group>] I<pkg_directory> [I<destination_directory>]
+B<opkg-build> [B<-A>] [B<-X>] [B<-c>] [B<-C>] [B<-Z> I<compressor>] [B<-a>] [B<-O>] [B<-o> I<owner>] [B<-g> I<group>] I<pkg_directory> [I<destination_directory>]

=cut

-usage="Usage: $0 [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] <pkg_directory> [<destination_directory>]"
-while getopts "a:cCg:ho:vOZ:" opt; do
+usage="Usage: $0 [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] <pkg_directory> [<destination_directory>]"
+while getopts "Aa:cCg:ho:vOXZ:" opt; do
case $opt in
+ A ) attributesargs="${attributesargs:+$attributesargs }--acls"
+ ;;
+ X ) attributesargs="${attributesargs:+$attributesargs }--xattrs"
+ ;;
o ) owner=$OPTARG
- ogargs="--owner=$owner"
+ ogargs="${ogargs:+$ogargs }--owner=$owner"
;;
O ) opkext=1
;;
g ) group=$OPTARG
- ogargs="$ogargs --group=$group"
+ ogargs="${ogargs:+$ogargs }--group=$group"
;;
c ) outer=tar
2.34.1

Piotr Łobacz

unread,
Aug 16, 2023, 9:01:13 AM8/16/23
to opkg-...@googlegroups.com, Alex Stewart

Alex Stewart

unread,
Aug 16, 2023, 11:35:58 AM8/16/23
to opkg-...@googlegroups.com, Piotr Łobacz
Merged this patchset as commit 8d9953dd8d589e9b740307976cbe474e0ce292a0.

https://git.yoctoproject.org/opkg-utils/commit/?id=8d9953dd8d589e9b740307976cbe474e0ce292a0

This version of the patch actually still had a couple instances of using
spaces for indentation. But I was already intending to put in a
whitespace fixup commit after this patch anyway. So I've also pushed
3b6b9ff223223b791c2046d0b2253290e197eb66 to fixup the styling in this
file more broadly.

Thanks!

Piotr Łobacz

unread,
Aug 16, 2023, 1:05:58 PM8/16/23
to Alex Stewart, opkg-...@googlegroups.com
I dunno why these problems with spaces happened, maybe different configuration between editors, because I used tab in xed...

Nevertheless I'm glad it's merged:D

BTW. I still need to investigate the issue regarding yocto sstate cache and acls/xattrs because it has fallen for Alexandre after some time...

Wysyłane z aplikacji Outlook dla systemu iOS

Od: Alex Stewart <alex.s...@ni.com>
Wysłane: Wednesday, August 16, 2023 5:35:47 PM
Do: opkg-...@googlegroups.com <opkg-...@googlegroups.com>; Piotr Łobacz <p.lo...@welotec.com>
Temat: Re: [opkg-devel] [PATCH v7] opkg-build: Add acls and xattrs support
 
Merged this patchset as commit 8d9953dd8d589e9b740307976cbe474e0ce292a0.

Reply all
Reply to author
Forward
0 new messages