Issue while deploying files to the persistent data partition

[Jean-Baptiste MARIE]

Hello,

I have noticed something very strange with the persistent file put into the data partition:

• I successfully added some files to the data partition by installing them to ${D}/data through my Yocto recipe
• After flashing the uefiimg, they are properly seen into the data partition
  
• But I have also observed that the files are included into the .mender file while they should not (according to the documentation).
  
• Furthermore if I perform a successful update with this mender file, and unmount /data partition after booting, I see the persistent file under /data folder of my rootfs (confirming they are indeed present into the .mender file)
  

I am wondering if that is an issue of the mender file generation or a mistake on my side. Do you have any idea?

Should the recipe creating data persistent file be added to the IMAGE_INSTALL variable?

Is it a proper solution to remove /data folder from the .mender file while it has been generated?

Thanks in advance for your feedback.

Regards,

[Kristian Amlie]

On 23/01/2019 18:24, Jean-Baptiste MARIE wrote:
> Hello,
>
> I have noticed something very strange with the persistent file put into
> the data partition:
>

> * I successfully added some files to the data partition by installing

> them to ${D}/data through my Yocto recipe

> * After flashing the uefiimg, they are properly seen into the data
> partition
> * But I have also observed that the files are included into the

> .mender file while they should not (according to the documentation).

> * Furthermore if I perform a successful update with this mender file,

> and unmount /data partition after booting, I see the persistent file
> under /data folder of my rootfs (confirming they are indeed present
> into the .mender file)
>
> I am wondering if that is an issue of the mender file generation or a
> mistake on my side. Do you have any idea?
> Should the recipe creating data persistent file be added to the
> IMAGE_INSTALL variable?
> Is it a proper solution to remove /data folder from the .mender file
> while it has been generated?

This is already fixed, at least on thud:
https://github.com/mendersoftware/meta-mender/commit/5bd06d73cbf1124765abe40bc299fc8b3bf9b98e

--
Kristian

[Jean-Baptiste MARIE]

Hi Kristian,

Thanks for your answer. I have applied that patch to sumo and it seems to work almost properly:

• When I mount my rootfs from uefiimg data folder is empty (which is good)
• BUT data folder is not empty in .ext4 file and so it is neither empty in the .mender file

Do you have any idea how data folder can be empty in uefiimg and not in the .mender file?

Regards,

[Kristian Amlie]

On 24/01/2019 10:28, Jean-Baptiste MARIE wrote:
> Hi Kristian,
>
> Thanks for your answer. I have applied that patch to sumo and it seems
> to work almost properly:
>

> * When I mount my rootfs from uefiimg data folder is empty (which is good)
> * BUT data folder is not empty in .ext4 file and so it is neither

> empty in the .mender file
>
> Do you have any idea how data folder can be empty in uefiimg and not in
> the .mender file?

They use different mechanisms. One was implemented in OpenEmbedded in
the wic tool, and seems to work for uefiimg. The other, used for ext4,
is implemented in mender-part-images.bbclass (look for
IMAGE_ROOTFS_EXCLUDE_PATH). I don't know why it would not work though,
AFAIK not much has changed there since it was first implemented.

Note also that sumo is community supported now, so it's not being
regularly tested by us anymore. Thud is our current stable branch.

--
Kristian

[Jean-Baptiste MARIE]

Hello Kristian,

Following your answer I had a deeper look into mechanism for generating the mender artifact.

• Mender artifact is a tar of the partition image generated by Yocto (for instance by do_image_ext4).
• That image is generated directly from the rootfs which contains the persistent files (in /data).
• There is no way to exclude /data when generating the .ext4 file (the IMAGE_ROOTFS_EXCLUDE_PATH that you mentioned in your previous email is used for the UEFIIMG creation, not for the ext4)

So I think that the current implementation of mender artifact generation would always put the /data into the .mender file (which is, of course, unwanted). It is neither possible to remove data from rootfs (as it is used for uefiimg creation) nor from the ext4 as the tar will be bigger anyway (due to the blocks used by /data). That issue is not so bad when you put only small files into the /data during image creation but as soon as you generate big files that becomes a critical issue as it strongly increase the size of the artifact.

Do you have any suggestions of how that can be solved? I am a bit surprised that nobody else is facing the same problem and I would be very interested to know how people deals with that in their own projects.

I am willing to make the related modifications and push them to the repo if a convenient mechanism is proposed by others.

Thanks

[Dell Green]

I don't know if this helps, but:

I have had a similar requirement very recently, whereby i worked around it by creating a custom image bbclass recipe to produce a 2nd rootfs ext4 with some things removed.

I recently supplied a pull-request upstream which will be merged soon, that allows you to set the name of the ext4 image that mender-artifact uses to create the mender file via a bitbake variable in local.conf.

https://github.com/mendersoftware/meta-mender/pull/688

combining these two things i was able to have a factory image and .mender image created from different ext4 files

:)

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

[Kristian Amlie]

On 28/01/2019 11:54, Jean-Baptiste MARIE wrote:
> Hello Kristian,
>
> Following your answer I had a deeper look into mechanism for generating
> the mender artifact.
>

> * Mender artifact is a tar of the partition image generated by Yocto
> (for instance by do_image_ext4).
> * That image is generated directly from the rootfs which contains the
> persistent files (in /data).
> * There is no way to exclude /data when generating the .ext4 file (the

> IMAGE_ROOTFS_EXCLUDE_PATH that you mentioned in your previous email
> is used for the UEFIIMG creation, not for the ext4)
>
> So I think that the current implementation of mender artifact generation
> would always put the /data into the .mender file (which is, of course,
> unwanted). It is neither possible to remove data from rootfs (as it is
> used for uefiimg creation) nor from the ext4 as the tar will be bigger
> anyway (due to the blocks used by /data). That issue is not so bad when
> you put only small files into the /data during image creation but as
> soon as you generate big files that becomes a critical issue as it
> strongly increase the size of the artifact.
>
> Do you have any suggestions of how that can be solved? I am a bit
> surprised that nobody else is facing the same problem and I would be
> very interested to know how people deals with that in their own projects.
>
> I am willing to make the related modifications and push them to the repo
> if a convenient mechanism is proposed by others.

TBH I'm not sure why this doesn't work on sumo. However, this definitely
works on thud and later, since we are explicitly testing for this:
https://github.com/mendersoftware/mender-image-tests/blob/d3f7b6235a9c51f2061d99b0cbac5849de2ffda7/tests/test_rootfs.py#L78-L81

--
Kristian