Partitioning for DataLogging

[Matthijs ter Woord]

Hi!

I'm looking into making my device setup more sustainable. My software will do data-logging, and I want to prevent my devices to be bricked due to sudden power losses.

One way I have thought of, is having 2 data partitions, one static (ie, containing the mender keys etc) and one my data. Then my software could take care of a corrupt data partition (my data) by checkdisking, or even reformatting.

Any thoughts on how to achieve this? (I prefer situations where my devices don't contain batteries, etc)

With kind regards,

Matthijs ter Woord

[Harry ten Berge]

Just make sure you use a proper file system on the /data partition (ext4 or f2fs), fsck is enabled (on boot) and that you make sure the system is not aggressively caching before writing out (vm.swappiness=1).

If you then also call 'sync' on regular base when writing data your pretty safe. 

It would then also be relatively easy to detect on boot if fsck failed, in case you can create the filesystem again.

Regards Harry

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

Harry ten Berge +31 6 25 06 83 41 ha...@baseflow.com

[Matthijs ter Woord]

Creating the FS again, means I'd lose the mender key etc, effectively losing the connection to my main server.

Or am I missing something there?

Op di 20 nov. 2018 om 10:07 schreef Harry ten Berge <ha...@baseflow.com>:

[Harry ten Berge]

Well, If you're planning on having 2 partitions then you won't I guess.

My story was in general about /data, but this can also be related to /data2 or whatever you fancy ;-)

[Matthijs ter Woord]

But my assumption, that the mender keys are at risk still holds?

Op di 20 nov. 2018 om 10:16 schreef Harry ten Berge <ha...@baseflow.com>:

[Donal Morrissey]

Hi Matthijs,

I've recently been putting together a platform with similar requirements, it needed to be robust to power-loss, so:

• We required an application data partition in addition to the mender data partition.
• The rootfs needed to be read-only (IMAGE_FEATURES_append=" read-only-rootfs").
  

To define the new application data partition, you have to modify the generated wks file (input to the wic tool). This is generated in mender-part-images.bbclass.  I forked mender and added a new 'part' to the mender_part_image() function. The reason I did this was because I was struggling to get it to work any other way. I need to revisit this, as our preference is to not fork mender. 

To manage the application data partition, I have a systemd service which runs at start-up. This will run fsck on the unmounted application data partition, and then mount it.

Hope that helps,

Donal

[Kristian Amlie]

On 20/11/2018 10:21, Matthijs ter Woord wrote:
> But my assumption, that the mender keys are at risk still holds?

Yes, in general. But you can point the /var/lib/mender symlink to any
persistent location you like (/data/mender is just the default), so you
can create two data partitions: One tiny one for the mender keys and
database, and another, bigger one for your data, which you can reset
whenever you want.

--
Kristian

>
>
> Op di 20 nov. 2018 om 10:16 schreef Harry ten Berge <ha...@baseflow.com

> <mailto:ha...@baseflow.com>>:

>
> Well, If you're planning on having 2 partitions then you won't I guess.
> My story was in general about /data, but this can also be related to
> /data2 or whatever you fancy ;-)
>
> On Tue, 20 Nov 2018 at 10:10, Matthijs ter Woord
> <matthijs...@gmail.com <mailto:matthijs...@gmail.com>> wrote:
>
> Creating the FS again, means I'd lose the mender key etc,
> effectively losing the connection to my main server.
> Or am I missing something there?
>
>
>
>
> Op di 20 nov. 2018 om 10:07 schreef Harry ten Berge

> <ha...@baseflow.com <mailto:ha...@baseflow.com>>:

>
> Just make sure you use a proper file system on the /data
> partition (ext4 or f2fs), fsck is enabled (on boot) and that
> you make sure the system is not aggressively caching before
> writing out (vm.swappiness=1).
> If you then also call 'sync' on regular base when writing
> data your pretty safe. 
>
> It would then also be relatively easy to detect on boot if
> fsck failed, in case you can create the filesystem again.
>
> Regards Harry
>
> On Tue, 20 Nov 2018 at 09:59, Matthijs ter Woord
> <matthijs...@gmail.com

> <mailto:matthijs...@gmail.com>> wrote:
>
> Hi!
>
> I'm looking into making my device setup more
> sustainable. My software will do data-logging, and I
> want to prevent my devices to be bricked due to sudden
> power losses.
>
> One way I have thought of, is having 2 data partitions,
> one static (ie, containing the mender keys etc) and one
> my data. Then my software could take care of a corrupt
> data partition (my data) by checkdisking, or even
> reformatting.
>
> Any thoughts on how to achieve this? (I prefer
> situations where my devices don't contain batteries, etc)
>
> With kind regards,
> Matthijs ter Woord
>
> --
> You received this message because you are subscribed to
> the Google Groups "Mender List mender.io

> <http://mender.io>" group.

> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> mender+un...@lists.mender.io

> <mailto:mender+un...@lists.mender.io>.

> To post to this group, send email to

> men...@lists.mender.io <mailto:men...@lists.mender.io>.

> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.
>
>
>
> --
>

> *Harry ten Berge*

> +31 6 25 06 83 41

> ha...@baseflow.com <mailto:ha...@baseflow.com>
>
> <https://baseflow.com/>

>
> --
> You received this message because you are subscribed to the

> Google Groups "Mender List mender.io <http://mender.io>" group.

> To unsubscribe from this group and stop receiving emails
> from it, send an email to mender+un...@lists.mender.io

> <mailto:mender+un...@lists.mender.io>.

> To post to this group, send email to men...@lists.mender.io

> <mailto:men...@lists.mender.io>.

> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.
>
> --
> You received this message because you are subscribed to the

> Google Groups "Mender List mender.io <http://mender.io>" group.

> To unsubscribe from this group and stop receiving emails from
> it, send an email to mender+un...@lists.mender.io

> <mailto:mender+un...@lists.mender.io>.

> To post to this group, send email to men...@lists.mender.io

> <mailto:men...@lists.mender.io>.

> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.
>
>
>
> --
>

> *Harry ten Berge*

> +31 6 25 06 83 41

> ha...@baseflow.com <mailto:ha...@baseflow.com>
>
> <https://baseflow.com/>

>
> --
> You received this message because you are subscribed to the Google

> Groups "Mender List mender.io <http://mender.io>" group.

> To unsubscribe from this group and stop receiving emails from it,
> send an email to mender+un...@lists.mender.io

> <mailto:mender+un...@lists.mender.io>.

> To post to this group, send email to men...@lists.mender.io

> <mailto:men...@lists.mender.io>.

> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Mender List mender.io" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mender+un...@lists.mender.io

> <mailto:mender+un...@lists.mender.io>.

> To post to this group, send email to men...@lists.mender.io

> <mailto:men...@lists.mender.io>.

[Matthijs ter Woord]

Hi Kristian,

Thanks for your input. Is there an easy way to get a second data partition (or in general, adjust the partitioning of the image)? Donal mentioned forking the mender scripts, but I'd rather not have to do that.

For me, if the data partition were a logical partition, I would be saved!

With kind regards,

Matthijs ter Woord

Op do 22 nov. 2018 om 09:27 schreef Kristian Amlie <kristia...@northern.tech>:

[Kristian Amlie]

On 22/11/2018 11:36, Matthijs ter Woord wrote:
> Hi Kristian,
>
> Thanks for your input. Is there an easy way to get a second data
> partition (or in general, adjust the partitioning of the image)? Donal
> mentioned forking the mender scripts, but I'd rather not have to do that.
> For me, if the data partition were a logical partition, I would be saved!

With the Mender image creator, there unfortunately is not, so a fork is
necessary. :-(

I believe the two things you need are the mender_part_image() function
and the IMAGE_CMD_<some-image-name> function, just renamed so they don't
clash with the originals.

--
Kristian

[Matthijs ter Woord]

Donal,

Any chance you could share the scripts you made for that, maybe via the new community area of Mender?

Op do 22 nov. 2018 om 11:42 schreef Kristian Amlie <kristia...@northern.tech>:

[Donal P Morrissey]

Hi Matthijis,

Here is the diff of the changes made to mender:

diff --git a/meta-mender-core/classes/mender-part-images.bbclass b/meta-mender-core/classes/mender-part-images.bbclass

index 23e9cd5..694799b 100644

--- a/meta-mender-core/classes/mender-part-images.bbclass

+++ b/meta-mender-core/classes/mender-part-images.bbclass

@@ -116,6 +116,8 @@ EOF

 part --source rootfs --ondisk "$ondisk_dev" --fstype=${ARTIFACTIMG_FSTYPE} --label primary --align $alignment_kb --fixed-size ${MENDER_CALC_ROOTFS_SIZE}k $exclude_dirs_options

 part --source rootfs --ondisk "$ondisk_dev" --fstype=${ARTIFACTIMG_FSTYPE} --label secondary --align $alignment_kb --fixed-size ${MENDER_CALC_ROOTFS_SIZE}k $exclude_dirs_options

 part --source rootfs --rootfs-dir ${IMAGE_ROOTFS}/data --ondisk "$ondisk_dev" --fstype=${ARTIFACTIMG_FSTYPE} --label data --align $alignment_kb --fixed-size ${MENDER_DATA_PART_SIZE_MB}

+part --ondisk "$ondisk_dev" --fstype=${ARTIFACTIMG_FSTYPE} --label appdata1 --align $alignment_kb --fixed-size 512

+part --ondisk "$ondisk_dev" --fstype=${ARTIFACTIMG_FSTYPE} --label appdata2 --align $alignment_kb --fixed-size 512

 bootloader --ptable $part_type

 EOF

Note here that I'm actually adding in two partitions, and appdata1 and appdata2, where appdata2 is a backup partition of appdata1. As I mentioned previously, I would like to remove the fork and implement the changes without modifying mender. This will involve somehow adding another step in the yocto build, but I haven't put much thought into it yet, it's down the list of priorities.

To mount these partitions I have added a systemd service which calls a script to fsck and then mount the partitions. You can find a few examples online of how to do this, I based what I've done on this example here: https://stackoverflow.com/questions/45614578/enable-systemd-services-using-yocto

Hope that helps.

BR,

Donal

[Matthijs ter Woord]

I finally gotten around to making this. I add a single extra partition (instead of the 2 you do). It does work, but my raspberry refuses to boot. Doing some deep digging, I find, that, when using fdisk to make an extended partition, it shows me partition type id is 5 ("Extended"). The WIC script uses type f ("W95 Ext'd (LBA)").

Anything I can do here?

Op do 22 nov. 2018 om 16:35 schreef Donal P Morrissey <donal.m...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.

[Kristian Amlie]

On 02/02/2019 19:38, Matthijs ter Woord wrote:
> I finally gotten around to making this. I add a single extra partition
> (instead of the 2 you do). It does work, but my raspberry refuses to
> boot. Doing some deep digging, I find, that, when using fdisk to make an
> extended partition, it shows me partition type id is 5 ("Extended"). The
> WIC script uses type f ("W95 Ext'd (LBA)").
>
> Anything I can do here?

This sounds like a bug in wic itself. One place to start digging would
be scripts/lib/wic/partition.py in the poky repository. Or one of the
nearby files.

--
Kristian

>
>
> Op do 22 nov. 2018 om 16:35 schreef Donal P Morrissey

> <donal.m...@gmail.com <mailto:donal.m...@gmail.com>>:

> Groups "Mender List mender.io <http://mender.io>" group.

> To unsubscribe from this group and stop receiving emails from it,
> send an email to mender+un...@lists.mender.io

> <mailto:mender+un...@lists.mender.io>.

> To post to this group, send email to men...@lists.mender.io

> <mailto:men...@lists.mender.io>.

[Matthijs ter Woord]

What I ended up doing, is adjust the MENDER_DATA_PARTITION to be "LABEL=data"

Then, as part of the on-boarding script, I will be doing repartitioning. (Found a set of steps I can use to do that safely)

If anyone's interested, I can share. The steps aren't fully automated yet, but still might be interesting..

Op ma 4 feb. 2019 om 08:21 schreef Kristian Amlie <kristia...@northern.tech>: