Isar fork
31 views
Skip to first unread message
Ben Brenson
Oct 18, 2017, 5:23:36 AM10/18/17
to isar-users
Hi,
I am doing further development on Isar for several months now. This work was done on behalf of Siemens for creating reproducible images.
There are several new features, I have implemented and maybe you are interested in a few of them.
The main new features are:
* Running chroot tasks and define them as simple bitbake shell tasks:
Defining bitbake tasks, which should run as chroot tasks, offers the advantage of -append and -prepend those tasks in different layers, and therefore
offers much more flexibility and modularity.
* Debianizing of non debian-compatible source code repositories:
Since Isar follows the strategy, to only produce deb packages, source code which is not debian compatible but should also be able to build, must first be debianized.
The main work is done within the debianize.bbclass.
After inheriting the debianize class you can define (by overwriting the tasks defined in debianize.bbclass) simple makefile tasks within your recipe.
Those tasks are parsed and bitbake will then create the required debian rules makefile.
This also gives the possibility of layering/overwriting those tasks within other layers.
* Cross compile support:
This feature is not completely finished yet and the implementation might change in the future.
The idea is very similar to the isar's *-dpkg-cross branch. But there are some problems we discovered, so there has to be a more generic and proper way of
cross compile packages.
A third buildchroot (cross-buildchroot) is created, and source code is build with the cross compiler within.
I think this feature is very important, since qemu user emulation suffers under very poor performance and slows down development progress a lot.
The following repositories are available on github now:
https://github.com/benbrenson/isar.git
https://github.com/benbrenson/meta-amazon.git
https://github.com/benbrenson/meta-sunxi.git
https://github.com/benbrenson/meta-swupdate.git
https://github.com/benbrenson/meta-unittest.git (refers to company internal repositories, which are not open source yet)
For now, only the isar repository itself is better documented. The user_manual.md was therefore modified.
TODOS:
* Add support for incremental builds
* Cache debian binaries for creating reproducible images, with persistent versions of all packages (I saw, there is already discussion about that on this forum).
* Add support for more hardware (rpi, beagle bone, i.MX6 platforms, odroid, allwinner platforms etc.)
* Add support for qemu images (not tested on isar fork, yet)
The main cause, why I'm writing so late, is just because we have been waiting until the customer gave the permissions for releasing this project on github.
Regards
Benedikt
I am doing further development on Isar for several months now. This work was done on behalf of Siemens for creating reproducible images.
There are several new features, I have implemented and maybe you are interested in a few of them.
The main new features are:
* Running chroot tasks and define them as simple bitbake shell tasks:
Defining bitbake tasks, which should run as chroot tasks, offers the advantage of -append and -prepend those tasks in different layers, and therefore
offers much more flexibility and modularity.
* Debianizing of non debian-compatible source code repositories:
Since Isar follows the strategy, to only produce deb packages, source code which is not debian compatible but should also be able to build, must first be debianized.
The main work is done within the debianize.bbclass.
After inheriting the debianize class you can define (by overwriting the tasks defined in debianize.bbclass) simple makefile tasks within your recipe.
Those tasks are parsed and bitbake will then create the required debian rules makefile.
This also gives the possibility of layering/overwriting those tasks within other layers.
* Cross compile support:
This feature is not completely finished yet and the implementation might change in the future.
The idea is very similar to the isar's *-dpkg-cross branch. But there are some problems we discovered, so there has to be a more generic and proper way of
cross compile packages.
A third buildchroot (cross-buildchroot) is created, and source code is build with the cross compiler within.
I think this feature is very important, since qemu user emulation suffers under very poor performance and slows down development progress a lot.
The following repositories are available on github now:
https://github.com/benbrenson/isar.git
https://github.com/benbrenson/meta-amazon.git
https://github.com/benbrenson/meta-sunxi.git
https://github.com/benbrenson/meta-swupdate.git
https://github.com/benbrenson/meta-unittest.git (refers to company internal repositories, which are not open source yet)
For now, only the isar repository itself is better documented. The user_manual.md was therefore modified.
TODOS:
* Add support for incremental builds
* Cache debian binaries for creating reproducible images, with persistent versions of all packages (I saw, there is already discussion about that on this forum).
* Add support for more hardware (rpi, beagle bone, i.MX6 platforms, odroid, allwinner platforms etc.)
* Add support for qemu images (not tested on isar fork, yet)
The main cause, why I'm writing so late, is just because we have been waiting until the customer gave the permissions for releasing this project on github.
Regards
Benedikt
Jan Kiszka
Oct 18, 2017, 7:16:18 AM10/18/17
to Ben Brenson, isar-users
factored out patch series against upstream, eventually posted here. Are
you looking for suggestion what to present first, and how? Or what are
your plans to consolidate the code bases?
Given that your baseline of Isar is rather old (25 May?), how do the
latest feature addition to upstream relate to what you addressed? Can
you imagine rebasing on top of those features, reusing them, or are
there fundamental differences in the approaches?
Thanks,
Jan
--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
Claudius Heine
Oct 18, 2017, 7:56:08 AM10/18/17
to Ben Brenson, isar-users
Hi Ben,
On 10/18/2017 11:23 AM, 'Ben Brenson' via isar-users wrote:
> Hi,
>
> I am doing further development on Isar for several months now. This work
> was done on behalf of Siemens for creating reproducible images.
> There are several new features, I have implemented and maybe you are
> interested in a few of them.
I see that you are using schroot. How is your progress on 'sudo' removal?
I haven't heard of schroot before, but from what I gather it needs a
privileged service to run in the background.
What are your experience with it? Could it be used for all parts that
currently require root privileges? Have you tried it inside a docker
container?
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de
On 10/18/2017 11:23 AM, 'Ben Brenson' via isar-users wrote:
> Hi,
>
> I am doing further development on Isar for several months now. This work
> was done on behalf of Siemens for creating reproducible images.
> There are several new features, I have implemented and maybe you are
> interested in a few of them.
I haven't heard of schroot before, but from what I gather it needs a
privileged service to run in the background.
What are your experience with it? Could it be used for all parts that
currently require root privileges? Have you tried it inside a docker
container?
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de
Ben Brenson
Oct 18, 2017, 7:56:12 AM10/18/17
to isar-users
Hi Jan,
yes you are right, the baseline of the isar fork is quiet old.
We had to implement some features quiet fast to get a first image build working.
Combined with the open source agreement, we forked this repository and started developing.
Since a lot of the codebase has changed, related to 25 May, a little bit of effort is required to work out some patches, but I will try do handle this.
The first feature I intent to post, would be the chroot related stuff, because the most changes are related to the bitbake core itself, which should be at the same state as
my repository.
So adding this feature won't force Isar to make use of them, but the possibility of using them may be available after.
So I think I will start with that feature, by submitting patch series.
The other features will dive very deep into isar, and the codebase differs a lot now.
This fact never offered the possibility of frequently rebasing against upstream.
So rebasing now against the main repository will be a time consuming job, but is required :-(
May be rebasing the current Isar features against my repo will be much easier to handle,
but doesn't that miss the goal of getting my features mainlined into current mainline Isar?
Regards,
Benedikt
yes you are right, the baseline of the isar fork is quiet old.
We had to implement some features quiet fast to get a first image build working.
Combined with the open source agreement, we forked this repository and started developing.
Since a lot of the codebase has changed, related to 25 May, a little bit of effort is required to work out some patches, but I will try do handle this.
The first feature I intent to post, would be the chroot related stuff, because the most changes are related to the bitbake core itself, which should be at the same state as
my repository.
So adding this feature won't force Isar to make use of them, but the possibility of using them may be available after.
So I think I will start with that feature, by submitting patch series.
The other features will dive very deep into isar, and the codebase differs a lot now.
This fact never offered the possibility of frequently rebasing against upstream.
So rebasing now against the main repository will be a time consuming job, but is required :-(
May be rebasing the current Isar features against my repo will be much easier to handle,
but doesn't that miss the goal of getting my features mainlined into current mainline Isar?
Regards,
Benedikt
Jan Kiszka
Oct 18, 2017, 8:09:18 AM10/18/17
to Ben Brenson, isar-users
On 2017-10-18 13:56, 'Ben Brenson' via isar-users wrote:
> Hi Jan,
> yes you are right, the baseline of the isar fork is quiet old.
> We had to implement some features quiet fast to get a first image build
> working.
> Combined with the open source agreement, we forked this repository and
> started developing.
>
> Since a lot of the codebase has changed, related to 25 May, a little bit
> of effort is required to work out some patches, but I will try do handle
> this.
> The first feature I intent to post, would be the chroot related stuff,
> because the most changes are related to the bitbake core itself, which
> should be at the same state as
> my repository.
>
> So adding this feature won't force Isar to make use of them, but the
> possibility of using them may be available after.
> So I think I will start with that feature, by submitting patch series.
Yes, start with the simplest, best isolated feature first. That will
> Hi Jan,
> yes you are right, the baseline of the isar fork is quiet old.
> We had to implement some features quiet fast to get a first image build
> working.
> Combined with the open source agreement, we forked this repository and
> started developing.
>
> Since a lot of the codebase has changed, related to 25 May, a little bit
> of effort is required to work out some patches, but I will try do handle
> this.
> The first feature I intent to post, would be the chroot related stuff,
> because the most changes are related to the bitbake core itself, which
> should be at the same state as
> my repository.
>
> So adding this feature won't force Isar to make use of them, but the
> possibility of using them may be available after.
> So I think I will start with that feature, by submitting patch series.
help get the ball rolling.
>
> The other features will dive very deep into isar, and the codebase
> differs a lot now.
> This fact never offered the possibility of frequently rebasing against
> upstream.
> So rebasing now against the main repository will be a time consuming
> job, but is required :-(
project pressure comes in, you can easily be forced off-track and then
have to pay the price later on.
However, it would have been valuable to participate in the requirement
discussions early. That might have enabled us to prioritize features and
sharpening their scope to address your needs as well.
>
> May be rebasing the current Isar features against my repo will be much
> easier to handle,
> but doesn't that miss the goal of getting my features mainlined into
> current mainline Isar?
working on a number of products that build upon upstream Isar, are
mostly happy, but will surely further improve the core to improve the
usability. Eventually, you will miss those features in your fork, I'm sure.
Ben Brenson
Oct 18, 2017, 8:11:00 AM10/18/17
to isar-users
Hi Claudius,
I see that you are using schroot. How is your progress on 'sudo' removal?
schroot itself, doesn't solve the 'sudo' problem, but it's very useful for running common tasks when setting up a chroot, like mounting filesystems, setting up binfmt etc.
So basically it extends the chroot command itself.
So basically it extends the chroot command itself.
I haven't heard of schroot before, but from what I gather it needs a
privileged service to run in the background.
I've never heard about or noticed that schroot needs a privileged service running in the background. Maybe I have to check this.
The main cause for introducing the schroot feature, was to have a already implemented framework, when setting up chroots (mostly related to setting up mounts).
Since a lot of chroot tasks running in parallel, if searched for a reliable chroot extension.
The main cause for introducing the schroot feature, was to have a already implemented framework, when setting up chroots (mostly related to setting up mounts).
Since a lot of chroot tasks running in parallel, if searched for a reliable chroot extension.
What are your experience with it? Could it be used for all parts that
currently require root privileges? Have you tried it inside a docker
container?
That is what I think the docker container is for solving the 'sudo' problem.
Running schroot inside a docker container is now problems and behaves exactly the same, like running it without schroot.
Benedikt
Claudius Heine
Oct 18, 2017, 8:52:47 AM10/18/17
to Ben Brenson, isar-users
Hi,
On 10/18/2017 02:11 PM, 'Ben Brenson' via isar-users wrote:
> Hi Claudius,
>
> I see that you are using schroot. How is your progress on 'sudo' removal?
>>
>
> schroot itself, doesn't solve the 'sudo' problem, but it's very useful for
> running common tasks when setting up a chroot, like mounting filesystems,
> setting up binfmt etc.
> So basically it extends the chroot command itself.
I don't now much about schroot, just assumed some stuff when quickly
investigating it.
From the manpage:
A chroot may be used directly as root by running chroot(8), but
normal users are
not able to use this command. schroot allows access to chroots for
normal users
using the same mechanism, but with several additional features.
From that I assumed that it takes care about normal users getting root
users inside the new root path.
>
> I haven't heard of schroot before, but from what I gather it needs a
>> privileged service to run in the background.
>>
>
> I've never heard about or noticed that schroot needs a privileged service
> running in the background. Maybe I have to check this.
I am not sure about it, but when I installed it, this popped up:
Setting up schroot (1.6.10-4) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/schroot.service →
/lib/systemd/system/schroot.service.
So I assumed is uses this service for privileged stuff.
> The main cause for introducing the schroot feature, was to have a already
> implemented framework, when setting up chroots (mostly related to setting
> up mounts).
> Since a lot of chroot tasks running in parallel, if searched for a reliable
> chroot extension.
I only had some experience using proot which uses the ptrace syscall to
put itself between the running application and the kernel. While its a
pretty nice an isolated solutions, it also has some downsides to it.
>
> What are your experience with it? Could it be used for all parts that
>> currently require root privileges? Have you tried it inside a docker
>> container?
>>
>
> That is what I think the docker container is for solving the 'sudo' problem.
> Running schroot inside a docker container is now problems and behaves
> exactly the same, like running it without schroot.
Root privileges inside a Docker container are sadly not a good enough
security mechanism, because you would have to grant the container the
sys_admin capabilities for loop mount and now its able to potentially
overwrite disk content or access the complete host memory.
And since you might use layers from not completely trusted developers it
should be made reasonable secure.
On 10/18/2017 02:11 PM, 'Ben Brenson' via isar-users wrote:
> Hi Claudius,
>
> I see that you are using schroot. How is your progress on 'sudo' removal?
>>
>
> schroot itself, doesn't solve the 'sudo' problem, but it's very useful for
> running common tasks when setting up a chroot, like mounting filesystems,
> setting up binfmt etc.
> So basically it extends the chroot command itself.
investigating it.
From the manpage:
A chroot may be used directly as root by running chroot(8), but
normal users are
not able to use this command. schroot allows access to chroots for
normal users
using the same mechanism, but with several additional features.
From that I assumed that it takes care about normal users getting root
users inside the new root path.
>
> I haven't heard of schroot before, but from what I gather it needs a
>> privileged service to run in the background.
>>
>
> I've never heard about or noticed that schroot needs a privileged service
> running in the background. Maybe I have to check this.
Setting up schroot (1.6.10-4) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/schroot.service →
/lib/systemd/system/schroot.service.
So I assumed is uses this service for privileged stuff.
> The main cause for introducing the schroot feature, was to have a already
> implemented framework, when setting up chroots (mostly related to setting
> up mounts).
> Since a lot of chroot tasks running in parallel, if searched for a reliable
> chroot extension.
put itself between the running application and the kernel. While its a
pretty nice an isolated solutions, it also has some downsides to it.
>
> What are your experience with it? Could it be used for all parts that
>> currently require root privileges? Have you tried it inside a docker
>> container?
>>
>
> That is what I think the docker container is for solving the 'sudo' problem.
> Running schroot inside a docker container is now problems and behaves
> exactly the same, like running it without schroot.
security mechanism, because you would have to grant the container the
sys_admin capabilities for loop mount and now its able to potentially
overwrite disk content or access the complete host memory.
And since you might use layers from not completely trusted developers it
should be made reasonable secure.
Ben Brenson
Oct 18, 2017, 10:02:12 AM10/18/17
to isar-users
Ah ok,
I am not sure about it, but when I installed it, this popped up:
Setting up schroot (1.6.10-4) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/schroot.service →
/lib/systemd/system/schroot.service.
Indeed a service will be installed, but thats only a one-shot which is executed once after boot. It performs some steps on "/var/lock/schroot".
So there is no daemon running in the background.
So there is no daemon running in the background.
Root privileges inside a Docker container are sadly not a good enough
security mechanism, because you would have to grant the container the
sys_admin capabilities for loop mount and now its able to potentially
overwrite disk content or access the complete host memory.
So the best solution would be to implement a non-root approach. Otherwise there will always persist some security issues.
For now I don't have any solutions, yet.
A time ago I tried to setup debootstrap by using fakechroot, but that wasn't staight forward to solve.
Maybe with multistrap, things would be much easier here?
Regards,
Benedikt
For now I don't have any solutions, yet.
A time ago I tried to setup debootstrap by using fakechroot, but that wasn't staight forward to solve.
Maybe with multistrap, things would be much easier here?
Regards,
Benedikt
Henning Schild
Oct 18, 2017, 10:19:45 AM10/18/17
to 'Ben Brenson' via isar-users, isar-users
On Wed, 18 Oct 2017 07:02:12 -0700
'Ben Brenson' via isar-users <isar-...@googlegroups.com> wrote:
> Ah ok,
>
> I am not sure about it, but when I installed it, this popped up:
> >
> > Setting up schroot (1.6.10-4) ...
> > Created symlink
> > /etc/systemd/system/multi-
> >>
> >> user.target.wants/schroot.service →
> >> /lib/systemd/system/schroot.service.
> >
> > Indeed a service will be installed, but thats only a one-shot which
> > is
> executed once after boot. It performs some steps on
> "/var/lock/schroot". So there is no daemon running in the background.
>
> Root privileges inside a Docker container are sadly not a good enough
> > security mechanism, because you would have to grant the container
> > the sys_admin capabilities for loop mount and now its able to
> > potentially overwrite disk content or access the complete host
> > memory.
That top-posting messed up citation. Please avoid top-posting.
'Ben Brenson' via isar-users <isar-...@googlegroups.com> wrote:
> Ah ok,
>
> I am not sure about it, but when I installed it, this popped up:
> >
> > Setting up schroot (1.6.10-4) ...
> > Created symlink
> > /etc/systemd/system/multi-
> >>
> >> user.target.wants/schroot.service →
> >> /lib/systemd/system/schroot.service.
> >
> > Indeed a service will be installed, but thats only a one-shot which
> > is
> executed once after boot. It performs some steps on
> "/var/lock/schroot". So there is no daemon running in the background.
>
> Root privileges inside a Docker container are sadly not a good enough
> > security mechanism, because you would have to grant the container
> > the sys_admin capabilities for loop mount and now its able to
> > potentially overwrite disk content or access the complete host
> > memory.
> So the best solution would be to implement a non-root approach.
> Otherwise there will always persist some security issues.
> For now I don't have any solutions, yet.
> A time ago I tried to setup debootstrap by using fakechroot, but that
> wasn't staight forward to solve.
> Maybe with multistrap, things would be much easier here?
looking for a solution. All the attempts to work around the
sudo-problem have serious limitations. And all but libpseudo seem to be
dying and unmaintained, because containers and VMs are around and
nobody cares.
Henning
> >
>
Claudius Heine
Oct 18, 2017, 10:22:04 AM10/18/17
to Ben Brenson, isar-users
Hi,
On 10/18/2017 04:02 PM, 'Ben Brenson' via isar-users wrote:
> Root privileges inside a Docker container are sadly not a good enough
>> security mechanism, because you would have to grant the container the
>> sys_admin capabilities for loop mount and now its able to potentially
>> overwrite disk content or access the complete host memory.
>>
>
> So the best solution would be to implement a non-root approach. Otherwise
> there will always persist some security issues.
> For now I don't have any solutions, yet.
> A time ago I tried to setup debootstrap by using fakechroot, but that
> wasn't staight forward to solve.
> Maybe with multistrap, things would be much easier here?
AFAIK that not the case. One problem of fakeroot/fakechroot is that is
works with LD_PRELOAD and therefor fails with static binaries. proot is
better there because is uses the same mechanism as strace (ptrace) to
capture syscalls. proot also allows to pseudo bind mount directories
into the chroot-path, which is useful. The problem with proot is however
that chown/chmod/mknot calls are not stored persistently.
So when I read 'schroot' in your commit history I got exited and hoped
that is the solution we all waited for. Doesn't seem to be the case though.
On 10/18/2017 04:02 PM, 'Ben Brenson' via isar-users wrote:
> Root privileges inside a Docker container are sadly not a good enough
>> security mechanism, because you would have to grant the container the
>> sys_admin capabilities for loop mount and now its able to potentially
>> overwrite disk content or access the complete host memory.
>>
>
> So the best solution would be to implement a non-root approach. Otherwise
> there will always persist some security issues.
> For now I don't have any solutions, yet.
> A time ago I tried to setup debootstrap by using fakechroot, but that
> wasn't staight forward to solve.
> Maybe with multistrap, things would be much easier here?
works with LD_PRELOAD and therefor fails with static binaries. proot is
better there because is uses the same mechanism as strace (ptrace) to
capture syscalls. proot also allows to pseudo bind mount directories
into the chroot-path, which is useful. The problem with proot is however
that chown/chmod/mknot calls are not stored persistently.
So when I read 'schroot' in your commit history I got exited and hoped
that is the solution we all waited for. Doesn't seem to be the case though.
Reply all
Reply to author
Forward
0 new messages