Extending handlers for UBI/UBIFS support
287 views
Skip to first unread message
Sergey Nazaryev
Jan 27, 2022, 3:29:07 PM1/27/22
to swupdate
Hi!
I'm trying to integrate SWUpdate for using it with our devices. Device has a 512M raw NAND storage divided into two 256M UBI partitions for implementing A/B scheme. Each UBI partition has two volumes: boot (ubifs) and rootfs (squashfs without compression).
It looks like layout like this is not popular, so I ran into a lot of problems. Several of them I want to discuss:
1. I wanted to update a kernel image located on UBIFS, but rawfile didn't support mounting UBIFS by UBI volume name. I added this feature (see patch 1), but the problem is that I had to duplicate the code (search_volume_global, search_volume) from ubivol_handler. How can I avoid this duplication? Could you give me some suggestions?
2. I wanted to reduce SWU size by using rdiff patches for my rootfs, but because rootfs (squashfs) is located in UBI I couldn't use rdiff_image handler. So, I forked rdiff_handler and now I have a new ubivol_rdiff_image handler (see patch 2) with 50% code from ubivol_handler and 50% code from rdiff_handler. The same question: how can I avoid the duplication? I don't think that extending rdiff_image with UBI functionality or extending ubivol_image with rdiff support would be a good solution.
I would appreciate any suggestions.
Thank you in advance.
Sergey
Stefano Babic
Jan 27, 2022, 6:12:04 PM1/27/22
to Sergey Nazaryev, swupdate
Hi Sergey,
On 27.01.22 21:29, Sergey Nazaryev wrote:
> Hi!
>
> I'm trying to integrate SWUpdate for using it with our devices. Device
> has a 512M raw NAND storage divided into two 256M UBI partitions for
> implementing A/B scheme. Each UBI partition has two volumes: boot
> (ubifs) and rootfs (squashfs without compression).
>
> It looks like layout like this is not popular, so I ran into a lot of
> problems. Several of them I want to discuss:
>
> 1. I wanted to update a kernel image located on UBIFS, but *rawfile
> *didn't support mounting UBIFS by UBI volume name. I added this feature
not (and could not be) at the same level as in kernel, it could be risky
to let the bootloader to extract the kernel from UBIFS. On the other
side, UBI containers are much more stable, and no issues were found even
with older U-Boot version. Kernel is then stored in a separate [<10MB]
UBI container and SWUpdate updates this with the UBI handler.
> 2. I wanted to reduce SWU size by using rdiff patches for my rootfs, but
> because rootfs (squashfs) is located in UBI I couldn't use *rdiff_image*
> The same question: how
> can I avoid the duplication? I don't think that extending rdiff_image
> with UBI functionality or extending ubivol_image with rdiff support
> would be a good solution.
It is not.
You could try to use UBI block device if you do not change the volume
size (last thing should be done by the partitioner, if any), that is
/dev/ubiblockX_Y.
Best regards,
Stefano Babic
>
> I would appreciate any suggestions.
>
> Thank you in advance.
> Sergey
>
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com
> <https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================
On 27.01.22 21:29, Sergey Nazaryev wrote:
> Hi!
>
> I'm trying to integrate SWUpdate for using it with our devices. Device
> has a 512M raw NAND storage divided into two 256M UBI partitions for
> implementing A/B scheme. Each UBI partition has two volumes: boot
> (ubifs) and rootfs (squashfs without compression).
>
> It looks like layout like this is not popular, so I ran into a lot of
> problems. Several of them I want to discuss:
>
> *didn't support mounting UBIFS by UBI volume name. I added this feature
> (see patch 1), but the problem is that I had to duplicate the code
> (search_volume_global, search_volume) from ubivol_handler/./ How can I
> avoid this duplication? Could you give me some suggestions?
>
The use case is handled differently: because UBIFS state in U-Boot is
>
not (and could not be) at the same level as in kernel, it could be risky
to let the bootloader to extract the kernel from UBIFS. On the other
side, UBI containers are much more stable, and no issues were found even
with older U-Boot version. Kernel is then stored in a separate [<10MB]
UBI container and SWUpdate updates this with the UBI handler.
> 2. I wanted to reduce SWU size by using rdiff patches for my rootfs, but
> handler. So, I forked rdiff_handler and now I have a new
> ubivol_rdiff_image handler (see patch 2) with 50% code from
> ubivol_handler and 50% code from rdiff_handler.
Ok......
> ubivol_rdiff_image handler (see patch 2) with 50% code from
> ubivol_handler and 50% code from rdiff_handler.
> The same question: how
> can I avoid the duplication? I don't think that extending rdiff_image
> with UBI functionality or extending ubivol_image with rdiff support
> would be a good solution.
You could try to use UBI block device if you do not change the volume
size (last thing should be done by the partitioner, if any), that is
/dev/ubiblockX_Y.
Best regards,
Stefano Babic
>
> I would appreciate any suggestions.
>
> Thank you in advance.
> Sergey
>
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com
> <https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================
Sergey Nazaryev
Jan 27, 2022, 8:50:07 PM1/27/22
to Stefano Babic, swupdate
On Fri, Jan 28, 2022 at 12:12:01AM +0100, Stefano Babic wrote:
> Hi Sergey,
>
> On 27.01.22 21:29, Sergey Nazaryev wrote:
> > Hi!
> >
> > I'm trying to integrate SWUpdate for using it with our devices. Device
> > has a 512M raw NAND storage divided into two 256M UBI partitions for
> > implementing A/B scheme. Each UBI partition has two volumes: boot
> > (ubifs) and rootfs (squashfs without compression).
> >
> > It looks like layout like this is not popular, so I ran into a lot of
> > problems. Several of them I want to discuss:
> >
> > 1. I wanted to update a kernel image located on UBIFS, but *rawfile
> > *didn't support mounting UBIFS by UBI volume name. I added this feature
> > (see patch 1), but the problem is that I had to duplicate the code
> > (search_volume_global, search_volume) from ubivol_handler/./ How can I
> > avoid this duplication? Could you give me some suggestions?
> >
>
> The use case is handled differently: because UBIFS state in U-Boot is not
> (and could not be) at the same level as in kernel, it could be risky to let
> the bootloader to extract the kernel from UBIFS. On the other side, UBI
> containers are much more stable, and no issues were found even with older
> U-Boot version. Kernel is then stored in a separate [<10MB] UBI container
> and SWUpdate updates this with the UBI handler.
>
Yep, I read about the risks but I thought that it's not actual
> Hi Sergey,
>
> On 27.01.22 21:29, Sergey Nazaryev wrote:
> > Hi!
> >
> > I'm trying to integrate SWUpdate for using it with our devices. Device
> > has a 512M raw NAND storage divided into two 256M UBI partitions for
> > implementing A/B scheme. Each UBI partition has two volumes: boot
> > (ubifs) and rootfs (squashfs without compression).
> >
> > It looks like layout like this is not popular, so I ran into a lot of
> > problems. Several of them I want to discuss:
> >
> > 1. I wanted to update a kernel image located on UBIFS, but *rawfile
> > *didn't support mounting UBIFS by UBI volume name. I added this feature
> > (see patch 1), but the problem is that I had to duplicate the code
> > (search_volume_global, search_volume) from ubivol_handler/./ How can I
> > avoid this duplication? Could you give me some suggestions?
> >
>
> The use case is handled differently: because UBIFS state in U-Boot is not
> (and could not be) at the same level as in kernel, it could be risky to let
> the bootloader to extract the kernel from UBIFS. On the other side, UBI
> containers are much more stable, and no issues were found even with older
> U-Boot version. Kernel is then stored in a separate [<10MB] UBI container
> and SWUpdate updates this with the UBI handler.
>
problem anymore and the UBIFS support is stable enough in modern
U-boot versions. It's strange that there is support for UBIFS in
U-boot since 2009 but it's still risky to use UBIFS in production
environment, even for read-only access (of course, if I understand
correctly, mounting UBIFS could require modifying the file system and
it could lead to damage). So, your advice is to avoid using UBIFS
from bootloaders at all? (I've found [1] and it's a pity that there
is no answer in this thread)
However, my use case requires an ability to choose between different
device tree blobs at boot time. In case of UBI static volumes (if I
understand correctly what you mean by "UBI containers"), I need to
create N UBI static volumes for each DTB, and it doesn't look like
flexible solution.
> > 2. I wanted to reduce SWU size by using rdiff patches for my rootfs, but
> > because rootfs (squashfs) is located in UBI I couldn't use *rdiff_image*
> > handler. So, I forked rdiff_handler and now I have a new
> > ubivol_rdiff_image handler (see patch 2) with 50% code from
> > ubivol_handler and 50% code from rdiff_handler.
>
> Ok......
>
> > The same question: how can I avoid the duplication? I don't think that
> > extending rdiff_image with UBI functionality or extending ubivol_image
> > with rdiff support would be a good solution.
>
> It is not.
>
> You could try to use UBI block device if you do not change the volume size
> (last thing should be done by the partitioner, if any), that is
> /dev/ubiblockX_Y.
need to hardcode the path and hope that X and Y won't change), but
what about the destination? IIRC, ubiblocks are read-only and I can't
put /dev/ubi node as a destination, thus it requires additional glue
code for using UBI volumes and rdiff handler together without
hardcoding. Because of it I wrote this Frankenstein's monster from
rdiff and ubivol handlers.
[1]: https://linux-mtd.infradead.narkive.com/4gCzE6e5/monster-boot
Stefano Babic
Jan 28, 2022, 5:18:53 AM1/28/22
to Sergey Nazaryev, Stefano Babic, swupdate
Hi Sergey,
I exchanged some mails with Richard more recently. Richard (he's the UBI
maintaioner) added a lot of new features (just the encryption is enough)
and a lot of changes into UBIFS. Discussing with him, he explained that
the status of UBIFS in U-Boot is very old and it has a lot of potential
problems. You can understand why: UBIFS was ported once from kernel to
U-Boot, but due to the differences with kernel (U-Boot is just a
bootloader, no OS), the efforts to improve and to fix what was fixed in
kernel is high. There are still differences with just UBI containers,
but they are small as UBI is much more stable as the filesystem.
So yes, my advice is to use UBI and not UBIFS.
> However, my use case requires an ability to choose between different
> device tree blobs at boot time. In case of UBI static volumes (if I
> understand correctly what you mean by "UBI containers"), I need to
> create N UBI static volumes for each DTB, and it doesn't look like
> flexible solution.
The use case is covered by switching to fitImage and embedded multiple
DT into it, and setting which configuration inside fitImage must be
started at boot time, see for example
doc/uImage.FIT/multi-with-loadables.its in U-Boot sources.
>
>>> 2. I wanted to reduce SWU size by using rdiff patches for my rootfs, but
>>> because rootfs (squashfs) is located in UBI I couldn't use *rdiff_image*
>>> handler. So, I forked rdiff_handler and now I have a new
>>> ubivol_rdiff_image handler (see patch 2) with 50% code from
>>> ubivol_handler and 50% code from rdiff_handler.
>>
>> Ok......
>>
>>> The same question: how can I avoid the duplication? I don't think that
>>> extending rdiff_image with UBI functionality or extending ubivol_image
>>> with rdiff support would be a good solution.
>>
>> It is not.
>>
>> You could try to use UBI block device if you do not change the volume size
>> (last thing should be done by the partitioner, if any), that is
>> /dev/ubiblockX_Y.
>
> Of course, I can use /dev/ubiblockX_Y as rdiffbase (even though I
> need to hardcode the path and hope that X and Y won't change), but
> what about the destination? IIRC, ubiblocks are read-only and I can't
> put /dev/ubi node as a destination, thus it requires additional glue
> code for using UBI volumes and rdiff handler together without
> hardcoding. Because of it I wrote this Frankenstein's monster from
> rdiff and ubivol handlers.
It looks more like that Frankenstein and Dracula had sex and this is
their son...:-D
This use case lets me think about how to chain handlers together.
Currently, this is possible only with Lua handler, If the rdiff handler
could write into a pipe and this is taken as input by the UBI handler,
we can cover this use case, too, and maybe some oter. Anyway, this
requires a detail analyses, also about how to extend this into
sw-description without breaking compatibility with the past.
Best regards,
Stefano Babic
maintaioner) added a lot of new features (just the encryption is enough)
and a lot of changes into UBIFS. Discussing with him, he explained that
the status of UBIFS in U-Boot is very old and it has a lot of potential
problems. You can understand why: UBIFS was ported once from kernel to
U-Boot, but due to the differences with kernel (U-Boot is just a
bootloader, no OS), the efforts to improve and to fix what was fixed in
kernel is high. There are still differences with just UBI containers,
but they are small as UBI is much more stable as the filesystem.
So yes, my advice is to use UBI and not UBIFS.
> However, my use case requires an ability to choose between different
> device tree blobs at boot time. In case of UBI static volumes (if I
> understand correctly what you mean by "UBI containers"), I need to
> create N UBI static volumes for each DTB, and it doesn't look like
> flexible solution.
DT into it, and setting which configuration inside fitImage must be
started at boot time, see for example
doc/uImage.FIT/multi-with-loadables.its in U-Boot sources.
>
>>> 2. I wanted to reduce SWU size by using rdiff patches for my rootfs, but
>>> because rootfs (squashfs) is located in UBI I couldn't use *rdiff_image*
>>> handler. So, I forked rdiff_handler and now I have a new
>>> ubivol_rdiff_image handler (see patch 2) with 50% code from
>>> ubivol_handler and 50% code from rdiff_handler.
>>
>> Ok......
>>
>>> The same question: how can I avoid the duplication? I don't think that
>>> extending rdiff_image with UBI functionality or extending ubivol_image
>>> with rdiff support would be a good solution.
>>
>> It is not.
>>
>> You could try to use UBI block device if you do not change the volume size
>> (last thing should be done by the partitioner, if any), that is
>> /dev/ubiblockX_Y.
>
> Of course, I can use /dev/ubiblockX_Y as rdiffbase (even though I
> need to hardcode the path and hope that X and Y won't change), but
> what about the destination? IIRC, ubiblocks are read-only and I can't
> put /dev/ubi node as a destination, thus it requires additional glue
> code for using UBI volumes and rdiff handler together without
> hardcoding. Because of it I wrote this Frankenstein's monster from
> rdiff and ubivol handlers.
their son...:-D
This use case lets me think about how to chain handlers together.
Currently, this is possible only with Lua handler, If the rdiff handler
could write into a pipe and this is taken as input by the UBI handler,
we can cover this use case, too, and maybe some oter. Anyway, this
requires a detail analyses, also about how to extend this into
sw-description without breaking compatibility with the past.
Best regards,
Stefano Babic
David Escalona Ozaeta
Jul 7, 2023, 7:05:07 AM7/7/23
to swupdate
Hello
Sergey &
Stefano,
What would be the best approach to overcome this issue?
Sorry to bring this topic back, but I think we are in the same scenario and I would like to know if this feature has evolved or there has been any specific implementation around it.
We have a dual boot system running two UBI volumes (primary and secondary) with 'squashfs' file systems. Our goal is to provide software updates based on rdiff image handler so that download size of the SWU package is as small as possible. The problem is that the rdiff image handler cannot write to UBI volumes or UBI blocks, so that we are unable to execute these type updates.
What would be the best approach to overcome this issue?
Thanks in advance.
Stefano Babic
Jul 10, 2023, 4:30:52 AM7/10/23
to David Escalona Ozaeta, swupdate
Hola David,
On 07.07.23 13:05, 'David Escalona Ozaeta' via swupdate wrote:
> Hello Sergey & Stefano,
>
> Sorry to bring this topic back, but I think we are in the same scenario
> and I would like to know if this feature has evolved or there has been
> any specific implementation around it.
A requested feature is implemented by me wehn I get a project with the
same requirement, that is when a company is sponsoring. Else it can just
be added to the roadmap.
Of course, someone else could have implemented the feature and sent for
review, but this did not happened.
>
> We have a dual boot system running two UBI volumes (primary and
> secondary) with 'squashfs' file systems. Our goal is to provide software
> updates based on rdiff image handler so that download size of the SWU
> package is as small as possible. The problem is that the rdiff image
> handler cannot write to UBI volumes or UBI blocks, so that we are unable
> to execute these type updates.
>
> What would be the best approach to overcome this issue?
The handler must be reqorked so that it is possible to have a a generic
consumer for the result of a rdiff handler, while it works today only as
raw image.
Best regards,
Stefano Babic
> https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com <https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com> <https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/swupdate/0f7113e1-dc21-477b-90e7-77049b553a9cn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> +49-8142-66989-80 <tel:+49%208142%206698980> Email: sba...@denx.de
> +49-8142-66989-80 <tel:+49%208142%206698980> Email: sba...@denx.de
> =====================================================================
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, 82194 Groebenzell, Germany
On 07.07.23 13:05, 'David Escalona Ozaeta' via swupdate wrote:
> Hello Sergey & Stefano,
>
> Sorry to bring this topic back, but I think we are in the same scenario
> and I would like to know if this feature has evolved or there has been
> any specific implementation around it.
same requirement, that is when a company is sponsoring. Else it can just
be added to the roadmap.
Of course, someone else could have implemented the feature and sent for
review, but this did not happened.
>
> We have a dual boot system running two UBI volumes (primary and
> secondary) with 'squashfs' file systems. Our goal is to provide software
> updates based on rdiff image handler so that download size of the SWU
> package is as small as possible. The problem is that the rdiff image
> handler cannot write to UBI volumes or UBI blocks, so that we are unable
> to execute these type updates.
>
> What would be the best approach to overcome this issue?
consumer for the result of a rdiff handler, while it works today only as
raw image.
Best regards,
Stefano Babic
> >>
> >> --
> >>
> =====================================================================
> >> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> >> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> Germany
> >> Phone: +49-8142-66989-53 <tel:+49%208142%206698953> Fax:
> >> --
> >>
> =====================================================================
> >> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> >> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> Germany
> +49-8142-66989-80 <tel:+49%208142%206698980> Email: sba...@denx.de
> >>
> =====================================================================
> >
>
>
> --
> =====================================================================
> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-53 <tel:+49%208142%206698953> Fax:
> =====================================================================
> >
>
>
> --
> =====================================================================
> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> +49-8142-66989-80 <tel:+49%208142%206698980> Email: sba...@denx.de
> =====================================================================
>
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/f462e6e0-7c86-4b7d-a934-7a2cf6033913n%40googlegroups.com <https://groups.google.com/d/msgid/swupdate/f462e6e0-7c86-4b7d-a934-7a2cf6033913n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, 82194 Groebenzell, Germany
Reply all
Reply to author
Forward
0 new messages