A few more questions about non-Yocto Mender

[Don Cross]

I have a few more questions in my ongoing project of getting Mender to work in a non-Yocto Raspberry Pi 3 environment.

1. I have created an account for hosted Mender. I notice there is an organization token under the account. The documentation mentions putting this token in a file called local.conf. I'm guessing this file is for Yocto and bitbake does something with it, not that there is a file local.conf on the device.  Is this guess correct?  If so, in a non-Yocto build, where on the device should the organization token be stored?

2. Without Yocto, how should I configure a device to connect to Mender hosting, authenticate, and download updates?  I see something about MENDER_SERVER_URL = "https://docker.mender.io".  Is this really right to have "docker" in the URL instead of "hosted"?  In general, I'm a bit lost here because all the documentation I find refers to Yocto, and the non-Yocto documentation I can find so far doesn't include anything about certificates, credentials, URLs, etc.

3. Just a note about an apparent documentation error. In the following blog post, it mentions a file called /etc/mender/artifact-info. Based on looking at the Mender client source code, I believe the file should be "artifact_info" with an underscore rather than a hyphen.

https://mender.io/blog/porting-mender-to-a-non-yocto-build-system

See for example:

.../go/src/github.com/mendersoftware/mender/mender.go

defaultArtifactInfoFile  = path.Join(getConfDirPath(), "artifact_info")

Thanks again for the ongoing help. When I get all this stuff working, I want to publish my own notes online with all the tiny details that are needed to make this work, so others can use it too.

- Don

[Mirza Krak]

On Thu, Jul 12, 2018 at 6:40 PM, Don Cross <cosin...@gmail.com> wrote:
> I have a few more questions in my ongoing project of getting Mender to work
> in a non-Yocto Raspberry Pi 3 environment.
>
> 1. I have created an account for hosted Mender. I notice there is an
> organization token under the account. The documentation mentions putting
> this token in a file called local.conf. I'm guessing this file is for Yocto
> and bitbake does something with it, not that there is a file local.conf on
> the device. Is this guess correct? If so, in a non-Yocto build, where on
> the device should the organization token be stored?

The token should go in the /etc/mender/mender.conf file on the device.
Documented here [1].

Agree that this part is very Yocto focused in the docs. If you have
suggestions on how to improve this, we appreciate pull-requests :)

>
> 2. Without Yocto, how should I configure a device to connect to Mender
> hosting, authenticate, and download updates? I see something about
> MENDER_SERVER_URL = "https://docker.mender.io". Is this really right to
> have "docker" in the URL instead of "hosted"? In general, I'm a bit lost
> here because all the documentation I find refers to Yocto, and the non-Yocto
> documentation I can find so far doesn't include anything about certificates,
> credentials, URLs, etc.

This is configured in /etc/mender/mender.conf, ServerURL [2]. And yeah
it should be "https://hosted.mender.io if you want to connect to
Hosted Mender. When connecting to Hosted Mender the token is all you
need.

The docker.mender.io is for the "demo server" setup [3].

>
> 3. Just a note about an apparent documentation error. In the following blog
> post, it mentions a file called /etc/mender/artifact-info. Based on looking
> at the Mender client source code, I believe the file should be
> "artifact_info" with an underscore rather than a hyphen.
> https://mender.io/blog/porting-mender-to-a-non-yocto-build-system
> See for example:
> .../go/src/github.com/mendersoftware/mender/mender.go
> defaultArtifactInfoFile = path.Join(getConfDirPath(), "artifact_info")

Thanks for reporting. It is indeed a documentation error.

Will make sure to update.

>
> Thanks again for the ongoing help. When I get all this stuff working, I want
> to publish my own notes online with all the tiny details that are needed to
> make this work, so others can use it too.

Thanks. Looking forward to reading your notes and it is highly
appreciated that you share your work.

[1]. https://docs.mender.io/1.5/client-configuration/configuration-file/configuration-options#tenanttoken
[2]. https://docs.mender.io/1.5/client-configuration/configuration-file/configuration-options#serverurl
[3]. https://docs.mender.io/1.5/getting-started/create-a-test-environment

--
Mirza Krak | Embedded Solutions Architect | https://mender.io

Northern.tech AS | @northerntechHQ

[Kristian Amlie]

On 12/07/18 22:53, Mirza Krak wrote:
> On Thu, Jul 12, 2018 at 6:40 PM, Don Cross <cosin...@gmail.com> wrote:
>> 3. Just a note about an apparent documentation error. In the following blog
>> post, it mentions a file called /etc/mender/artifact-info. Based on looking
>> at the Mender client source code, I believe the file should be
>> "artifact_info" with an underscore rather than a hyphen.
>> https://mender.io/blog/porting-mender-to-a-non-yocto-build-system
>> See for example:
>> .../go/src/github.com/mendersoftware/mender/mender.go
>> defaultArtifactInfoFile = path.Join(getConfDirPath(), "artifact_info")
>
> Thanks for reporting. It is indeed a documentation error.
>
> Will make sure to update.

Fixed! Thanks for the notice!

--
Kristian

[Harry ten Berge]

Hi,

I'm running a non-Yocto setup as well, and I've got it working too.

So maybe we can combine the 'non-Yocto' efforts, because right now it's a challenge to get stuff up and running this way.

Regards,

Harry

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

[Mirza Krak]

On Fri, Jul 13, 2018 at 8:32 AM, Harry ten Berge <ha...@baseflow.com> wrote:

Hi,

I'm running a non-Yocto setup as well, and I've got it working too.

So maybe we can combine the 'non-Yocto' efforts, because right now it's a challenge to get stuff up and running this way.

It is really great that you are willing to share.

I also want you to check this out: https://tracker.mender.io/browse/MEN-1812

That is work being done to actually automate integration with non-Yocto images. Since you have done this on your own, your experience is valuable to get the tooling accurate with a good end result.

-- 

Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Harry ten Berge]

oh this looks interesting!

I've build stuff based on ArchLinux and also build an ArchLinux package for the mender client.

TBH the most 'painful' part of the whole exercise was/is u-boot. 

What would already be very helpful if there's just a simple single patch file for u-boot. That's still on my todo list, as I've got the stuff lying around.

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

[Mirza Krak]

On Fri, Jul 13, 2018 at 9:43 AM, Harry ten Berge <ha...@baseflow.com> wrote:

oh this looks interesting!

I've build stuff based on ArchLinux and also build an ArchLinux package for the mender client.

TBH the most 'painful' part of the whole exercise was/is u-boot. 

Interesting with the ArchLinux package. Is that available publicly?

Most people share your opinion with U-boot patching, that is why there is a bit different approach to this for the converted images. Check this out https://tracker.mender.io/browse/MEN-1595. The idea is basically to make sure that U-boot loads a provided GRUB binary in which we do all the "Mender stuff", this approach does not require any patches to U-boot.

 

What would already be very helpful if there's just a simple single patch file for u-boot. That's still on my todo list, as I've got the stuff lying around.

Could you elaborate?

[Harry ten Berge]

I can make the ArchLinux package public, no problem. Rather busy today, so I'll do that in the weekend.

In our case we're using Mender on a Raspberry Pi environment (embedded hardware), so using GRUB is not going to work I think.

What I mean with the 'single patch' file is that right now I needed to get the different patches from the Yocto stuff in the Mender repo and apply that to u-boot.

For people not using Yocto it would be very easy to have a single patch file that they need to apply to the latest u-boot source, build it and go. 

I'll keep you updated on the ArchLinux package.

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

[Matthijs ter Woord]

Harry, Curious, why not use the yocto stuff?

To unsubscribe from this group and stop receiving emails from it, send an email to mender+unsubscribe@lists.mender.io.

To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

--

You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+unsubscribe@lists.mender.io.

[Harry ten Berge]

Hehe... tricky question ;-)

Basically it's personal preference. I tried Yocto, struggled with it and threw it away.

I very much dislike the sense of not 'being in control'. To me it felt like this magic sauce with lots and lots of configuration files all over the place.

But let me be clear: I'm not saying that Yocto is bad or something, it's just not working for me. 

On Fri, 13 Jul 2018 at 09:59, Matthijs ter Woord <matthijs...@gmail.com> wrote:

Harry, Curious, why not use the yocto stuff?

2018-07-13 9:57 GMT+02:00 Harry ten Berge <ha...@baseflow.com>:

I can make the ArchLinux package public, no problem. Rather busy today, so I'll do that in the weekend.

In our case we're using Mender on a Raspberry Pi environment (embedded hardware), so using GRUB is not going to work I think.

What I mean with the 'single patch' file is that right now I needed to get the different patches from the Yocto stuff in the Mender repo and apply that to u-boot.

For people not using Yocto it would be very easy to have a single patch file that they need to apply to the latest u-boot source, build it and go. 

I'll keep you updated on the ArchLinux package.

On Fri, 13 Jul 2018 at 09:54, Mirza Krak <mirza...@northern.tech> wrote:

On Fri, Jul 13, 2018 at 9:43 AM, Harry ten Berge <ha...@baseflow.com> wrote:

oh this looks interesting!

I've build stuff based on ArchLinux and also build an ArchLinux package for the mender client.

TBH the most 'painful' part of the whole exercise was/is u-boot. 

Interesting with the ArchLinux package. Is that available publicly?

Most people share your opinion with U-boot patching, that is why there is a bit different approach to this for the converted images. Check this out https://tracker.mender.io/browse/MEN-1595. The idea is basically to make sure that U-boot loads a provided GRUB binary in which we do all the "Mender stuff", this approach does not require any patches to U-boot.

 

What would already be very helpful if there's just a simple single patch file for u-boot. That's still on my todo list, as I've got the stuff lying around.

Could you elaborate?

--

Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.

To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.

To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.

To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

[Mirza Krak]

On Fri, Jul 13, 2018 at 9:57 AM, Harry ten Berge <ha...@baseflow.com> wrote:

I can make the ArchLinux package public, no problem. Rather busy today, so I'll do that in the weekend.

Cool!

 

In our case we're using Mender on a Raspberry Pi environment (embedded hardware), so using GRUB is not going to work I think.

What I mean with the 'single patch' file is that right now I needed to get the different patches from the Yocto stuff in the Mender repo and apply that to u-boot.

For people not using Yocto it would be very easy to have a single patch file that they need to apply to the latest u-boot source, build it and go. 

Understood. 

 

I'll keep you updated on the ArchLinux package.

Thanks.

[Kristian Amlie]

On 13/07/18 09:57, Harry ten Berge wrote:
> In our case we're using Mender on a Raspberry Pi environment (embedded
> hardware), so using GRUB is not going to work I think.

GRUB should work just fine under U-Boot's UEFI loader mode (it already
does for several other platforms), and using this chain loading, you get
the best of both worlds: U-Boot is the 1st stage boot loader, and GRUB
is the 2nd stage which handles all the update stuff without requiring
any patches. This is what the converted images will likely use.

For certain platforms (and I suspect RPi might be one of them), it may
require a slight tweaking of U-Boot's boot process, but this patch
should be much smaller than the patch that is currently required.

All this is not completely finished, but it's getting close.

--
Kristian

[Don Cross]

On Thu, Jul 12, 2018 at 4:54 PM Mirza Krak <mirza...@northern.tech> wrote:

On Thu, Jul 12, 2018 at 6:40 PM, Don Cross <cosin...@gmail.com> wrote:
> I have a few more questions in my ongoing project of getting Mender to work
> in a non-Yocto Raspberry Pi 3 environment.
>
> 1. I have created an account for hosted Mender. I notice there is an
> organization token under the account. The documentation mentions putting
> this token in a file called local.conf. I'm guessing this file is for Yocto
> and bitbake does something with it, not that there is a file local.conf on
> the device.  Is this guess correct?  If so, in a non-Yocto build, where on
> the device should the organization token be stored?

The token should go in the /etc/mender/mender.conf file on the device.
Documented here [1].

Thank you, Mirza. I think it's getting a little further now. Here is an error I see in my mender.log. I don't know how to fix this. (I don't know what "system roots" are.)

ERRO[0603] Failure occured while executing authorization request: &url.Error{Op:"Post", URL:"https://hosted.mender.io/api/devices/v1/authentication/auth_requests", Err:x509.SystemRootsError{Err:error(nil)}}  module=client_auth

ERRO[0603] authorization request error: x509: failed to load system roots and no roots provided  module=client_auth

ERRO[0603] authorize failed: transient error: authorization request failed: generic error occured while executing authorization request: Post https://hosted.mender.io/api/devices/v1/authentication/auth_requests: x509: failed to load system roots and no roots provided  module=state

Here is the contents of my /etc/mender/mender.conf file, with the tenant token redacted for security (I'm assuming I should not let this leak out to the public!)  The tenant token really is there... the very long string that looks like base64 encoded stuff.

{

  "ClientProtocol": "https",

  "HttpsClient": {

    "Certificate": "",

    "Key": ""

  },

  "RootfsPartA": "/dev/mmcblk0p3",

  "RootfsPartB": "/dev/mmcblk0p4",

  "UpdatePollIntervalSeconds": 1800,

  "InventoryPollIntervalSeconds": 1800,

  "RetryPollIntervalSeconds": 300,

  "ServerURL": "https://hosted.mender.io/",

  "ServerCertificate": "",

  "TenantToken": "...<redacted>..."

}

Maybe I need a Certificate and Key provided under HttpsClient?  If so, how do I obtain them?

 

Agree that this part is very Yocto focused in the docs. If you have
suggestions on how to improve this, we appreciate pull-requests :)

Oh cool, I see now that Mender documentation is on Github as Markdown files.  Once I get something working, I will circle back and look at doing some pull requests there!

- Don

 

[Kristian Amlie]

On 13/07/18 21:16, Don Cross wrote:
> Thank you, Mirza. I think it's getting a little further now. Here is an
> error I see in my mender.log. I don't know how to fix this. (I don't
> know what "system roots" are.)
>
> ERRO[0603] Failure occured while executing authorization request:
> &url.Error{Op:"Post",
> URL:"https://hosted.mender.io/api/devices/v1/authentication/auth_requests",
> Err:x509.SystemRootsError{Err:error(nil)}}  module=client_auth
> ERRO[0603] authorization request error: x509: failed to load system
> roots and no roots provided  module=client_auth
> ERRO[0603] authorize failed: transient error: authorization request
> failed: generic error occured while executing authorization request:
> Post
> https://hosted.mender.io/api/devices/v1/authentication/auth_requests:
> x509: failed to load system roots and no roots provided  module=state

On Yocto this is fixed by installing the ca-certificates package, which
contains the certificates of all the certificate authorities (CAs).

If you can't install that, see below.

> Here is the contents of my /etc/mender/mender.conf file, with the tenant
> token redacted for security (I'm assuming I should not let this leak out
> to the public!)  The tenant token really is there... the very long
> string that looks like base64 encoded stuff.

Indeed!

>
> {

>   "ClientProtocol": "https",
>   "HttpsClient": {
>     "Certificate": "",
>     "Key": ""
>   },
>   "RootfsPartA": "/dev/mmcblk0p3",
>   "RootfsPartB": "/dev/mmcblk0p4",
>   "UpdatePollIntervalSeconds": 1800,
>   "InventoryPollIntervalSeconds": 1800,
>   "RetryPollIntervalSeconds": 300,
>   "ServerURL": "https://hosted.mender.io/",
>   "ServerCertificate": "",

>   "TenantToken": "...</redacted/>..."

> }
>
> Maybe I need a Certificate and Key provided under HttpsClient?  If so,
> how do I obtain them?

You can simply export them using a browser. See this page:
https://superuser.com/questions/97201/how-to-save-a-remote-server-ssl-certificate-locally-as-a-file

Then you store this file on the device and point ServerCertificate to it.

--
Kristian

[Kristian Amlie]

On 16/07/18 19:20, Don Cross wrote:
> On Mon, Jul 16, 2018 at 4:09 AM Kristian Amlie
> <kristia...@northern.tech> wrote:
> I could probably get ca-certificates installed if I had to, although it
> would be a chore. So I tried downloading the certificate for
> hosted.mender.io <http://hosted.mender.io>. I still have problems.
> Please see below.

>  
>
> > Maybe I need a Certificate and Key provided under HttpsClient?  If so,
> > how do I obtain them?
>
> You can simply export them using a browser. See this page:
> https://superuser.com/questions/97201/how-to-save-a-remote-server-ssl-certificate-locally-as-a-file
>
> Then you store this file on the device and point ServerCertificate
> to it.
>
> --
> Kristian
>
>

> I used the following command to download certificates. I saved them to
> the file /etc/mender/server.crt and pointed ServerCertificate to that
> file as you recommended. I ended up getting 2 certificates in that file.
>
> openssl s_client -connect hosted.mender.io:443
> <http://hosted.mender.io:443> -servername hosted.mender.io
> <http://hosted.mender.io> -showcerts < /dev/null | 
>         sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > server.crt
>
> Also, here is the current form of my /etc/mender/mender.conf
>
> {

>   "RootfsPartA": "/dev/mmcblk0p3",
>   "RootfsPartB": "/dev/mmcblk0p4",
>   "UpdatePollIntervalSeconds": 1800,
>   "InventoryPollIntervalSeconds": 1800,
>   "RetryPollIntervalSeconds": 300,

>   "ServerURL": "https://hosted.mender.io",
>   "ServerCertificate": "/etc/mender/server.crt",
>   "TenantToken": "<redacted>"
> }
>
> Below is the latest output in my mender.log. Note that the 2
> certificates show up in the log output. [As before, I have redacted my
> tenant token.]  So the error has changed to "401 Unauthorized". Do my
> certificates look correct? Should there be only one instead of two? Or
> am I doing something else wrong? 

401 Unauthorized is correct. Now you need to accept the devices in the
UI, and then they won't be Unauthorized anymore.

--
Kristian

[Don Cross]

Hi Kristian,

I can tell I'm getting closer now. I did authorize my device and that got me past the 401 error. Thank you!

I created my first artifact and uploaded it. Then Mender on my prototype device started generating certificate errors when it tried to download the artifact. Note that I have redacted possibly sensitive contents from the URL and replaced with "...":

ERRO[9191] Can not fetch update image: Get https://s3.amazonaws.com/hosted-mender-artifacts/.../...?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=...&X-Amz-Date=20180718T212811Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=...: x509: certificate signed by unknown authority module=client_update

So I searched the Mender archives for that "certificate signed by unknown authority" error and found that it meant I don't have ca-certificates installed (which I already know), and I don't have the right certificate in /etc/mender/server.crt.  It occurred to me that my command for generating server.crt only gets the certificate for hosted.mender.io, but this download appears to be coming from s3.amazonaws.com.  So I tried adding both pairs of certificates to server.crt: two from hosted.mender.io, and two from s3.amazonaws.com. (I really have no idea what I'm doing here. You will see all these certificates in the debug log below.)

While Mender was still retrying every 1, 2, 4, etc minutes to download, I rebooted the machine with manually updated server.crt, went away for a while, and came back. Now my prototype Raspberry Pi machine keeps rebooting over and over. I pulled the power plug and put the SD card in another computer so I could look at the log files. Maybe I should not have rebooted manually in this state?

I can tell that the partition I expected to be downloaded to (/dev/mmcblk0p4) has not changed: it still is empty except for a little text file I left there. The existing rootfs partition (/dev/mmcblk0p3) still has the original contents also. So I don't see any evidence that anything on the SD card has been updated.

Below is the entire contents of my mender.log.  I'm including the entire set of 4 certificates to see if they are right or wrong (can anyone tell me)?  I'm pretty sure the last 2 lines are the most important for diagnosing why the reboot keeps happening in a loop. It indicates I'm hitting line 383 in the following source, expecting the state "reboot" but it found "update-fetch".

https://github.com/mendersoftware/mender/blob/master/state.go

Any clues for what I should look for next?  Please let me know if I can provide any other information to help diagnose this. Thanks again... you guys have been very helpful.

- Don

DEBU[0000] Reading Mender configuration from file /etc/mender/mender.conf  module=config

DEBU[0000] block type: RSA PRIVATE KEY                   module=keystore

INFO[0000] API Gateway certificate (in PEM format): 

-----BEGIN CERTIFICATE-----

MIIFyzCCBLOgAwIBAgISESH5uYWqmPBivLjIt6QCEhGWMA0GCSqGSIb3DQEBCwUA

MEwxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSIwIAYD

VQQDExlBbHBoYVNTTCBDQSAtIFNIQTI1NiAtIEcyMB4XDTE1MTAyNzE5MTgwNloX

DTE4MTAyNzE5MTgwNlowOTEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRh

dGVkMRQwEgYDVQQDDAsqLm1lbmRlci5pbzCCAiIwDQYJKoZIhvcNAQEBBQADggIP

ADCCAgoCggIBAJ47otKGkgHmy2OtxneT8uFUS8II65VSeQG3jZFczLWoB5gfWSeP

Yx7jEwqnD1CSvHxsDmrfHWefuF/CH0/U+noVGNGsnYWQpNrilqaHk5Iw117bST26

9r/OQZC7HcabrPawhGJVASgOE6I5YAqdkQ8/5bN5y7D1KWJar8vFgNvaLGMrrYDG

ZQ4pXE4DY+ISGcE6+X+kYrMtEKuGQ4415PQ/Uff9E8+SDCXsX8/C/Lu8NDJeihlX

a41xSkWHKI9Ko5XyrLs14eWvFJQcya7wO9pbJw98flcATJ6FfznmINm8rQ3Vl203

VArKCto5hbJJmqr3NSBYGlAnmfkKHwY5pH3VFZy0GjHSxiNOfGj53jPh4RZ6U6KF

AyBBiI7XXsquKH8jARj4rjsYVsWtUd6woI3OSBtxArHMTfTKx/fuSoAydo7Z9iWE

KDjAa79TU0/ygho3H6wO73bVkX1EZObaenAz/WgCAtKmyO4HvWQU2KqnADAKSWcl

BtmaHMnFwMd4fhCp0UuaOC88ck4MCzhsfNFXlukeButqX9WtT+EIt+0X0L2Y/Xh6

/IDe3c+UzLhfQJlRILn2+s+cee4yD879TSh1xZoq+z68T4mz1cly/xTvlMG4Sj55

6b0OBIY5HeZz7vcUlkl5whMsRAvjmEiBVe4s3Lj4Zv/QhpZa/IABpDOdAgMBAAGj

ggG4MIIBtDAOBgNVHQ8BAf8EBAMCBaAwSQYDVR0gBEIwQDA+BgZngQwBAgEwNDAy

BggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9y

eS8wIQYDVR0RBBowGIILKi5tZW5kZXIuaW+CCW1lbmRlci5pbzAJBgNVHRMEAjAA

MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjA+BgNVHR8ENzA1MDOgMaAv

hi1odHRwOi8vY3JsMi5hbHBoYXNzbC5jb20vZ3MvZ3NhbHBoYXNoYTJnMi5jcmww

gYkGCCsGAQUFBwEBBH0wezBCBggrBgEFBQcwAoY2aHR0cDovL3NlY3VyZTIuYWxw

aGFzc2wuY29tL2NhY2VydC9nc2FscGhhc2hhMmcycjEuY3J0MDUGCCsGAQUFBzAB

hilodHRwOi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20vZ3NhbHBoYXNoYTJnMjAdBgNV

HQ4EFgQUfs7xDA1nZuQzz9W+cu4ZPilVKuUwHwYDVR0jBBgwFoAU9c3VPAhQ+WpP

OreX2laD5mnSaPcwDQYJKoZIhvcNAQELBQADggEBADFEvbPJE0I39sHDXOTM9z/w

izIiMCIN5sgZ9G8HXBDkdEqQ0dFUfALlofuPWTVCf/wF/o9tOiQq/y0WjZjRYHHw

FZKQaHjd8uUulfOt1+JSow5ObELNfegDH7NIHtHqPMsw5Svy+mihGHKj/bwMUAW4

et1D3i43PCq34RQXn34tBadLMWl6qBQz6X6UnUzJl+fXDSYJJGZ1b+5htR9Zv3un

nKTtSBXOuZHtxsb1VkB9EGuz8NKzH5OaC+M0eHwfOfUsGbNAagUX0nBH0bjqVHRB

Fw1cfyzRFmy8gDTERE+JAk7r0S23l8tbNeIRVMxbHW6KS1R9QAWdy/Zl6ap/lv0=

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIETTCCAzWgAwIBAgILBAAAAAABRE7wNjEwDQYJKoZIhvcNAQELBQAwVzELMAkG

A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv

b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw0xNDAyMjAxMDAw

MDBaFw0yNDAyMjAxMDAwMDBaMEwxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i

YWxTaWduIG52LXNhMSIwIAYDVQQDExlBbHBoYVNTTCBDQSAtIFNIQTI1NiAtIEcy

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2gHs5OxzYPt+j2q3xhfj

kmQy1KwA2aIPue3ua4qGypJn2XTXXUcCPI9A1p5tFM3D2ik5pw8FCmiiZhoexLKL

dljlq10dj0CzOYvvHoN9ItDjqQAu7FPPYhmFRChMwCfLew7sEGQAEKQFzKByvkFs

MVtI5LHsuSPrVU3QfWJKpbSlpFmFxSWRpv6mCZ8GEG2PgQxkQF5zAJrgLmWYVBAA

cJjI4e00X9icxw3A1iNZRfz+VXqG7pRgIvGu0eZVRvaZxRsIdF+ssGSEj4k4HKGn

kCFPAm694GFn1PhChw8K98kEbSqpL+9Cpd/do1PbmB6B+Zpye1reTz5/olig4het

ZwIDAQABo4IBIzCCAR8wDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8C

AQAwHQYDVR0OBBYEFPXN1TwIUPlqTzq3l9pWg+Zp0mj3MEUGA1UdIAQ+MDwwOgYE

VR0gADAyMDAGCCsGAQUFBwIBFiRodHRwczovL3d3dy5hbHBoYXNzbC5jb20vcmVw

b3NpdG9yeS8wMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5nbG9iYWxzaWdu

Lm5ldC9yb290LmNybDA9BggrBgEFBQcBAQQxMC8wLQYIKwYBBQUHMAGGIWh0dHA6

Ly9vY3NwLmdsb2JhbHNpZ24uY29tL3Jvb3RyMTAfBgNVHSMEGDAWgBRge2YaRQ2X

yolQL30EzTSo//z9SzANBgkqhkiG9w0BAQsFAAOCAQEAYEBoFkfnFo3bXKFWKsv0

XJuwHqJL9csCP/gLofKnQtS3TOvjZoDzJUN4LhsXVgdSGMvRqOzm+3M+pGKMgLTS

xRJzo9P6Aji+Yz2EuJnB8br3n8NA0VgYU8Fi3a8YQn80TsVD1XGwMADH45CuP1eG

l87qDBKOInDjZqdUfy4oy9RU0LMeYmcI+Sfhy+NmuCQbiWqJRGXy2UzSWByMTsCV

odTvZy84IOgu/5ZR8LrYPZJwR2UcnnNytGAMXOLRc3bgr07i5TelRS+KIz6HxzDm

MTh89N1SyvNTBCVXVmaU6Avu5gMUTu79bZRknl7OedSyps9AsUSoPocZXun4IRZZ

Uw==

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIF4zCCBMugAwIBAgIQAu3/FrPiMkFH6tkTD74KxzANBgkqhkiG9w0BAQsFADBk

MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3

d3cuZGlnaWNlcnQuY29tMSMwIQYDVQQDExpEaWdpQ2VydCBCYWx0aW1vcmUgQ0Et

MiBHMjAeFw0xNzA5MjYwMDAwMDBaFw0xODA5MjAxMjAwMDBaMGkxCzAJBgNVBAYT

AlVTMRMwEQYDVQQIEwpXYXNoaW5ndG9uMRAwDgYDVQQHEwdTZWF0dGxlMRgwFgYD

VQQKEw9BbWF6b24uY29tIEluYy4xGTAXBgNVBAMTEHMzLmFtYXpvbmF3cy5jb20w

ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2hJx2qwDXD3rXPjP1f5C9

s8+3EEnXFS6IqAhyAMdoNUhOfQkc79psuNwIFMOb4PNk85nPrqYMTYrJ0J3h44cz

obJxaEy1/lmU26q4BoN7Y7ZSOMQCglZYZowEwrQTD+k8id3pK1AGwdGobUcQhncm

Y37KbaONDsCck20XM8vf475tfOc4em+xVcgUK2g5ZHVF4WZ+ldC75sRo5AVuwomI

fdXM3hU3wnQdRomKgEUJvAReIn5zWbMx/cUD0fD0hw/mwUWsRMAx0zfZ3qaJmK3r

/5zrc+m845tjKG2s8Ya68mpixua+yZ78+2xWch/siYrwDdShTsjxFC3uzT5l3PLt

AgMBAAGjggKKMIIChjAfBgNVHSMEGDAWgBTAErIodGhGZ+lwJXQaAEVbBn1cRDAd

BgNVHQ4EFgQUiXgMmXDLuBs0YVNK2Z7mWiDp5rwwgbkGA1UdEQSBsTCBroIQczMu

YW1hem9uYXdzLmNvbYISKi5zMy5hbWF6b25hd3MuY29tgiYqLnMzLmR1YWxzdGFj

ay51cy1lYXN0LTEuYW1hem9uYXdzLmNvbYIkczMuZHVhbHN0YWNrLnVzLWVhc3Qt

MS5hbWF6b25hd3MuY29tghwqLnMzLnVzLWVhc3QtMS5hbWF6b25hd3MuY29tghpz

My51cy1lYXN0LTEuYW1hem9uYXdzLmNvbTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0l

BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMIGBBgNVHR8EejB4MDqgOKA2hjRodHRw

Oi8vY3JsMy5kaWdpY2VydC5jb20vRGlnaUNlcnRCYWx0aW1vcmVDQS0yRzIuY3Js

MDqgOKA2hjRodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vRGlnaUNlcnRCYWx0aW1v

cmVDQS0yRzIuY3JsMEwGA1UdIARFMEMwNwYJYIZIAYb9bAEBMCowKAYIKwYBBQUH

AgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwCAYGZ4EMAQICMHkGCCsG

AQUFBwEBBG0wazAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQuY29t

MEMGCCsGAQUFBzAChjdodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGlnaUNl

cnRCYWx0aW1vcmVDQS0yRzIuY3J0MAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEL

BQADggEBAEp7l1HT/nU+7SunFmjACmbJjI8rmQzHF1lkMy+n9g1O83TE6JxEiNxw

y7tGGc3M3u4pZfmPBK0n1kvv1j8vob1XYisSYjJ1T0Tqf+HzFMv3TgW1kn+xC8xW

bIPAmU6pv9awW3cs+rEQIinyMgxEojM+beax7urOuyfJBYeNiV77Gs1eU1P/UxNl

wDClsVrwmUKzX1CWmaiuweTGuukiJoQ+jVbrc/8v8ttlOWpt9AKJrRLMjQ2XnjUo

TaESxsRm+yORTy1czZIssjGklDeHrSoKn5tsbxF5IyxwBXlybr2VCJCptORvu9ab

ghbESAGtRuK3NWoPOR17vpHyY8TGP8U=

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIEYzCCA0ugAwIBAgIQAYL4CY6i5ia5GjsnhB+5rzANBgkqhkiG9w0BAQsFADBa

MQswCQYDVQQGEwJJRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJl

clRydXN0MSIwIAYDVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTE1

MTIwODEyMDUwN1oXDTI1MDUxMDEyMDAwMFowZDELMAkGA1UEBhMCVVMxFTATBgNV

BAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRpZ2ljZXJ0LmNvbTEjMCEG

A1UEAxMaRGlnaUNlcnQgQmFsdGltb3JlIENBLTIgRzIwggEiMA0GCSqGSIb3DQEB

AQUAA4IBDwAwggEKAoIBAQC75wD+AAFz75uI8FwIdfBccHMf/7V6H40II/3HwRM/

sSEGvU3M2y24hxkx3tprDcFd0lHVsF5y1PBm1ITykRhBtQkmsgOWBGmVU/oHTz6+

hjpDK7JZtavRuvRZQHJaZ7bN5lX8CSukmLK/zKkf1L+Hj4Il/UWAqeydjPl0kM8c

+GVQr834RavIL42ONh3e6onNslLZ5QnNNnEr2sbQm8b2pFtbObYfAB8ZpPvTvgzm

+4/dDoDmpOdaxMAvcu6R84Nnyc3KzkqwIIH95HKvCRjnT0LsTSdCTQeg3dUNdfc2

YMwmVJihiDfwg/etKVkgz7sl4dWe5vOuwQHrtQaJ4gqPAgMBAAGjggEZMIIBFTAd

BgNVHQ4EFgQUwBKyKHRoRmfpcCV0GgBFWwZ9XEQwHwYDVR0jBBgwFoAU5Z1ZMIJH

WMys+ghUNoZ7OrUETfAwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMC

AYYwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdp

Y2VydC5jb20wOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDovL2NybDMuZGlnaWNlcnQu

Y29tL09tbmlyb290MjAyNS5jcmwwPQYDVR0gBDYwNDAyBgRVHSAAMCowKAYIKwYB

BQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwDQYJKoZIhvcNAQEL

BQADggEBAC/iN2bDGs+RVe4pFPpQEL6ZjeIo8XQWB2k7RDA99blJ9Wg2/rcwjang

B0lCY0ZStWnGm0nyGg9Xxva3vqt1jQ2iqzPkYoVDVKtjlAyjU6DqHeSmpqyVDmV4

7DOMvpQ+2HCr6sfheM4zlbv7LFjgikCmbUHY2Nmz+S8CxRtwa+I6hXsdGLDRS5rB

bxcQKegOw+FUllSlkZUIII1pLJ4vP1C0LuVXH6+kc9KhJLsNkP5FEx2noSnYZgvD

0WyzT7QrhExHkOyL4kGJE7YHRndC/bseF/r/JUuOUFfrjsxOFT+xJd1BDKCcYm1v

upcHi9nzBhDFKdT3uhaQqNBU4UtJx5g=

-----END CERTIFICATE-----

  module=client

INFO[0000] Issuer: [GlobalSign nv-sa], Valid from: 2015-10-27 19:18:06 +0000 UTC, Valid to: 2018-10-27 19:18:06 +0000 UTC  module=client

WARN[0000] No system certificates found.                 module=client

INFO[0000] State transition: init [none] -> init [none]  module=mender

INFO[0000] handling loaded state: update-fetch           module=state

INFO[0000] Entered the uncommitted partition with invalid state-data stored. Rebooting.  module=state

 

[Kristian Amlie]

On 19/07/18 03:29, Don Cross wrote:
> I can tell I'm getting closer now. I did authorize my device and that
> got me past the 401 error. Thank you!
>
> I created my first artifact and uploaded it. Then Mender on my prototype
> device started generating certificate errors when it tried to download
> the artifact. Note that I have redacted possibly sensitive contents from
> the URL and replaced with "...":
>
> ERRO[9191] Can not fetch update image: Get
> https://s3.amazonaws.com/hosted-mender-artifacts/.../...?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=...&X-Amz-Date=20180718T212811Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=...:
> x509: certificate signed by unknown authority module=client_update
>
> So I searched the Mender archives for that "certificate signed by
> unknown authority" error and found that it meant I don't have
> ca-certificates installed (which I already know), and I don't have the
> right certificate in /etc/mender/server.crt.  It occurred to me that my
> command for generating server.crt only gets the certificate for

> hosted.mender.io <http://hosted.mender.io>, but this download appears to
> be coming from s3.amazonaws.com <http://s3.amazonaws.com>.  So I tried

> adding both pairs of certificates to server.crt: two from

> hosted.mender.io <http://hosted.mender.io>, and two from
> s3.amazonaws.com <http://s3.amazonaws.com>. (I really have no idea what

> I'm doing here. You will see all these certificates in the debug log below.)
>
> While Mender was still retrying every 1, 2, 4, etc minutes to download,
> I rebooted the machine with manually updated server.crt, went away for a
> while, and came back. Now my prototype Raspberry Pi machine keeps
> rebooting over and over. I pulled the power plug and put the SD card in
> another computer so I could look at the log files. Maybe I should not
> have rebooted manually in this state?
>
> I can tell that the partition I expected to be downloaded to
> (/dev/mmcblk0p4) has not changed: it still is empty except for a little
> text file I left there. The existing rootfs partition (/dev/mmcblk0p3)
> still has the original contents also. So I don't see any evidence that
> anything on the SD card has been updated.
>
> Below is the entire contents of my mender.log.  I'm including the entire
> set of 4 certificates to see if they are right or wrong (can anyone tell
> me)?

Yes, that should be correct.

>  I'm pretty sure the last 2 lines are the most important for
> diagnosing why the reboot keeps happening in a loop. It indicates I'm
> hitting line 383 in the following source, expecting the state "reboot"
> but it found "update-fetch".
>
> https://github.com/mendersoftware/mender/blob/master/state.go
>
> Any clues for what I should look for next?  Please let me know if I can
> provide any other information to help diagnose this. Thanks again... you
> guys have been very helpful.
>
> - Don
>
> DEBU[0000] Reading Mender configuration from file
> /etc/mender/mender.conf  module=config
> DEBU[0000] block type: RSA PRIVATE KEY                   module=keystore
> INFO[0000] API Gateway certificate (in PEM format): 
> -----BEGIN CERTIFICATE-----

> ...

> -----END CERTIFICATE-----
>   module=client
> INFO[0000] Issuer: [GlobalSign nv-sa], Valid from: 2015-10-27 19:18:06
> +0000 UTC, Valid to: 2018-10-27 19:18:06 +0000 UTC  module=client
> WARN[0000] No system certificates found.                 module=client
> INFO[0000] State transition: init [none] -> init [none]  module=mender
> INFO[0000] handling loaded state: update-fetch           module=state
> INFO[0000] Entered the uncommitted partition with invalid state-data
> stored. Rebooting.  module=state

If Mender thinks that it's on the uncommitted partition, this is an
indication that your U-Boot user space tools are not working correctly.
The first thing I would do is run through the Integration Checklist [1]
if you haven't already. You will obviously have to reflash the device if
it's already stuck in the loop. You can just skip the steps in the
checklist that ask you to recompile with bitbake, it just means you
won't be able to verify that particular part, but I don't think that
this is the problem here anyway.

Failing that, or if you have already done it and the problem keeps
happening, please post the contents of /etc/mender/mender.conf and the
output from running "fw_printenv".

[1]
https://docs.mender.io/1.5/devices/integrating-with-u-boot/integration-checklist

--
Kristian

[Don Cross]

[I just realized I sent this to Kristian directly instead of the Mender list, as I had intended. Sorry about that.]

On Thu, Jul 19, 2018 at 1:56 AM Kristian Amlie <kristia...@northern.tech> wrote:

If Mender thinks that it's on the uncommitted partition, this is an
indication that your U-Boot user space tools are not working correctly.
The first thing I would do is run through the Integration Checklist [1]
if you haven't already. You will obviously have to reflash the device if
it's already stuck in the loop. You can just skip the steps in the
checklist that ask you to recompile with bitbake, it just means you
won't be able to verify that particular part, but I don't think that
this is the problem here anyway.

Failing that, or if you have already done it and the problem keeps
happening, please post the contents of /etc/mender/mender.conf and the
output from running "fw_printenv".

[1]
https://docs.mender.io/1.5/devices/integrating-with-u-boot/integration-checklist

--
Kristian

I reflashed the device and this time I saw a crucial clue in the mender.log that I missed last time because of my unhelpful manual reboot:

ERRO[0002] update image installation failed: No match between boot and root partitions.  module=installer

I found this error message in the source code:

https://github.com/mendersoftware/mender/blob/master/partitions.go

This led me to discover that fw_printenv isn't working, as I would have noticed if I had gone through the integration checklist like you also suggested.  Here is what I see:

root@camserver:/home/casa $ fw_printenv

Read error on /uboot/uboot.env: Success

root@camserver:/uboot $ hexdump uboot.env

0000000 0000 0000 0000 0000 0000 0000 0000 0000

*

0004000

root@camserver:/uboot $ fw_setenv

## Error: variable name missing

My interpretation is that /uboot/uboot.env is exactly 16K but is completely filled with zero bytes. Which partly makes sense because that's how I created it in the first place. I'm guessing it should have all the u-boot environment variables in it, but it doesn't.

Here is where I'm confused: I thought the environment variables were supposed to be outside the filesystem entirely.  As a matter of fact, there should be two copies of the environment on the raw SD card both 16K in size, one starting at offset 16K and the other at 32K.  This is based on what I have in my config_mender_defines.h when I build u-boot. See [1] below.

But this is not correct either because when I dump the first megabyte of my SD card, all I see is the boot sector:

root@camserver:/home/casa $ dd if=/dev/mmcblk0 bs=1M count=1 | hexdump -C

00000000  fa b8 00 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |................|

00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|

00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|

00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |.........|...t..|

00000040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L.....|.........|

00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*

000001b0  00 00 00 00 00 00 00 00  74 f5 d2 87 00 00 00 46  |........t......F|

000001c0  06 01 0c 6e 25 09 00 50  00 00 00 00 02 00 00 8f  |...n%..P........|

000001d0  07 09 83 d4 0a 4a 00 58  02 00 00 00 10 00 00 f4  |.....J.X........|

000001e0  2b 4a 83 81 b9 08 00 60  12 00 00 38 6d 00 00 a2  |+J.....`...8m...|

000001f0  9b 08 83 2f e9 c6 00 a0  7f 00 00 38 6d 00 55 aa  |.../.......8m.U.|

00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*

1+0 records in

1+0 records out

1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648465 s, 16.2 MB/s

00100000

So I'm left both confused about two things:

(a) How is the environment supposed to be stored: raw blocks on the SD card outside the partitions, or in the file uboot.env?

(b) Why is the environment not being stored in either place?

I have included the U-Boot environment, extracted from u-boot.bin using the Linux "strings" command [2] and my mender.conf with my tenant token redacted [3].

I hope this is enough diagnostic info. Thanks!

- Don

[1]

/*

 *  config_mender_defines.h

 */

#ifndef HEADER_CONFIG_MENDER_DEFINES_H

#define HEADER_CONFIG_MENDER_DEFINES_H

/* Shell variables */

#define MENDER_BOOT_PART_NUMBER         1

#define MENDER_ROOTFS_PART_A_NUMBER     3

#define MENDER_ROOTFS_PART_B_NUMBER     4

#define MENDER_UBOOT_STORAGE_INTERFACE  "mmc"

#define MENDER_UBOOT_STORAGE_DEVICE     0

/* BB variables. */

#define MENDER_STORAGE_DEVICE_BASE "/dev/mmcblk0p"

/* Match environment size to CONFIG_ENV_SIZE in include/configs/rpi.h */

#define MENDER_BOOTENV_SIZE     SZ_16K

#define MENDER_MTD_UBI_DEVICE_NAME  "ubi"

#define MENDER_BOOT_KERNEL_TYPE     "bootz"

#define MENDER_KERNEL_NAME          "zImage"

#define MENDER_DTB_NAME             "bcm2710-rpi-3-b-plus.dtb"

#define MENDER_ROOTFS_PART_A_NAME   "/dev/mmcblk0p3"

#define MENDER_ROOTFS_PART_B_NAME   "/dev/mmcblk0p4"

#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1    SZ_16K

#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2    (MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 + MENDER_BOOTENV_SIZE)

#endif /* HEADER_CONFIG_MENDER_DEFINES_H */

[2]

altbootcmd=run mender_altbootcmd; run bootcmd

bootlimit=1

bootcount=0

upgrade_available=0

mender_mtd_ubi_dev_name=ubi

mender_boot_part=3

mender_uboot_boot=mmc 0:1

mender_uboot_if=mmc

mender_uboot_dev=0

mender_boot_kernel_type=bootz

mender_kernel_name=zImage

mender_dtb_name=bcm2710-rpi-3-b-plus.dtb

mender_setup=setenv mender_kernel_root /dev/mmcblk0p${mender_boot_part}; if test ${mender_boot_part} = 3; then setenv mender_boot_part_name /dev/mmcblk0p3; else setenv mender_boot_part_name /dev/mmcblk0p4; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs

mender_altbootcmd=if test ${mender_boot_part} = 3; then setenv mender_boot_part 4; else setenv mender_boot_part 3; fi; setenv upgrade_available 0; saveenv; run mender_setup

mender_try_to_recover=if test ${upgrade_available} = 1; then reset; fi

bootcmd=run mender_setup; setenv bootargs root=${mender_kernel_root} ${bootargs}; fdt addr ${fdt_addr} && fdt get value bootargs /chosen bootargs; load ${mender_uboot_root} ${kernel_addr_r} /boot/${mender_kernel_name}; bootz ${kernel_addr_r} - ${fdt_addr}; run mender_try_to_recover; 

bootdelay=2

baudrate=115200

preboot=usb start

loadaddr=0x00200000

arch=arm

cpu=armv7

board=rpi

board_name=rpi

vendor=raspberrypi

soc=bcm283x

dhcpuboot=usb start; dhcp u-boot.uimg; bootm

stdin=serial,usbkbd

stdout=serial,vidconsole

stderr=serial,vidconsole

fdt_high=ffffffff

initrd_high=ffffffff

fdt_addr_r=0x00000100

pxefile_addr_r=0x00100000

kernel_addr_r=0x01000000

scriptaddr=0x02000000

ramdisk_addr_r=0x02100000

mmc_boot=if mmc dev ${devnum}; then setenv devtype mmc; run scan_dev_for_boot_part; fi

boot_net_usb_start=usb start

usb_boot=usb start; if usb dev ${devnum}; then setenv devtype usb; run scan_dev_for_boot_part; fi

boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootarm.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi

load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}

efi_dtb_prefixes=/ /dtb/ /dtb/current/

scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; then setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootarm.efi; then echo Found EFI removable media binary efi/boot/bootarm.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile

boot_prefixes=/ /boot/

boot_scripts=boot.scr.uimg boot.scr

boot_script_dhcp=boot.scr.uimg

boot_targets=mmc0 usb0 pxe dhcp 

boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}extlinux/extlinux.conf

scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}extlinux/extlinux.conf; then echo Found ${prefix}extlinux/extlinux.conf; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi

boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}

scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done

scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;

scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done

bootcmd_mmc0=setenv devnum 0; run mmc_boot

bootcmd_usb0=setenv devnum 0; run usb_boot

bootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fi

bootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; then setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00010:UNDI:003000;setenv bootp_arch 0xa;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;

distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done

[3]

{

  "RootfsPartA": "/dev/mmcblk0p3",

  "RootfsPartB": "/dev/mmcblk0p4",

  "UpdatePollIntervalSeconds": 1800,

  "InventoryPollIntervalSeconds": 1800,

  "RetryPollIntervalSeconds": 300,

  "ServerURL": "https://hosted.mender.io",

  "ServerCertificate": "/etc/mender/server.crt",

  "TenantToken": "..."

}

[Kristian Amlie]

Please keep discussions on the list. Replies below.

On 19/07/18 21:50, Don Cross wrote:

> On Thu, Jul 19, 2018 at 1:56 AM Kristian Amlie
> <kristia...@northern.tech> wrote:
>
> If Mender thinks that it's on the uncommitted partition, this is an
> indication that your U-Boot user space tools are not working correctly.
> The first thing I would do is run through the Integration Checklist [1]
> if you haven't already. You will obviously have to reflash the device if
> it's already stuck in the loop. You can just skip the steps in the
> checklist that ask you to recompile with bitbake, it just means you
> won't be able to verify that particular part, but I don't think that
> this is the problem here anyway.
>
> Failing that, or if you have already done it and the problem keeps
> happening, please post the contents of /etc/mender/mender.conf and the
> output from running "fw_printenv".
>
> [1]
> https://docs.mender.io/1.5/devices/integrating-with-u-boot/integration-checklist
>
> --
> Kristian
>
>

> I reflashed the device and this time I saw a crucial clue in the
> mender.log that I missed last time because of my unhelpful manual reboot:
>
> ERRO[0002] update image installation failed: No match between boot and
> root partitions.  module=installer
>
> I found this error message in the source code:
> https://github.com/mendersoftware/mender/blob/master/partitions.go
>
> This led me to discover that fw_printenv isn't working, as I would have
> noticed if I had gone through the integration checklist like you also
> suggested.  Here is what I see:
>
> root@camserver:/home/casa $ fw_printenv
> Read error on /uboot/uboot.env: Success

This (extremely confusing) error message means that the configuration
file is incorrect, and probably points to some location that is too
small, and certainly wrong.

It should point to /dev/mmcblk0 (assuming you are using memory card),
and some offset and size which are dependent on your exact setup. In
your case, based on your values below it should be something like:

/dev/mmcblk0 0x4000 0x4000
/dev/mmcblk0 0x8000 0x4000

The former.

> (b) Why is the environment not being stored in either place?

It is not stored unless changed, since there is a default built in
environment which is used otherwise. So you will only see nulls until
you use either "saveenv" in the U-Boot prompt or the fw_setenv tool.

--
Kristian

>   "RootfsPartA": "/dev/mmcblk0p3",
>   "RootfsPartB": "/dev/mmcblk0p4",
>   "UpdatePollIntervalSeconds": 1800,
>   "InventoryPollIntervalSeconds": 1800,
>   "RetryPollIntervalSeconds": 300,
>   "ServerURL": "https://hosted.mender.io",
>   "ServerCertificate": "/etc/mender/server.crt",

>   "TenantToken": "..."
> }
>
>

[Don Cross]

> root@camserver:/home/casa $ fw_printenv
> Read error on /uboot/uboot.env: Success

This (extremely confusing) error message means that the configuration
file is incorrect, and probably points to some location that is too
small, and certainly wrong.

It should point to /dev/mmcblk0 (assuming you are using memory card),
and some offset and size which are dependent on your exact setup. In
your case, based on your values below it should be something like:

/dev/mmcblk0 0x4000 0x4000
/dev/mmcblk0 0x8000 0x4000

Thank you Kristian!  That was my indeed my problem.  My fw_env.config file was referring to /uboot/uboot.env, when it needed to have the two lines exactly like you wrote above.  I had another issue that was Pi-specific, where I needed to fix the command line sent to the Raspbian kernel so that it would pick the correct partition to mount as root. Thankfully I was able to figure that out on my own.

So I'm very happy to report that I had my first Hosted Mender upgrade success this morning!  It downloaded the deployment, and rebooted into the alternate partition. I can see the prototype system now has /dev/mmcblk0p4 mounted as root (instead of /dev/mmcblk0p3), and that it is running the updated version of my software. I even have a heartwarming green check mark in the Mender Hosted web UI saying the deployment was successful.

Thank you guys so much for all your help and patience. Over the next few weeks I want to write up notes for this mailing list regarding all the stuff I learned. I'm thinking about also creating an example project that illustrates all the atypical things I had to get working:

• Raspberry Pi (the GPU-based boot process is very different from other Linux systems).
• Non-Yocto integration of Mender and U-Boot.
• Using sysvinit instead of systemd: had to create init.d scripts for Mender.
• Using PiLFS (Linux From Scratch customized for the Pi) to build the OS.
• Tricks for trimming the OS size down so that my Mender artifacts are 122 MB. (Can probably make it even smaller.)

I know you guys are working on tools for converting an existing root partition into an artifact. I will definitely want to beta test that stuff and provide feedback when it is ready.

- Don

[Ole Petter]

That is great Don!

Have a quick look at https://github.com/mendersoftware/mender/blob/master/CONTRIBUTING.md as you're preparing your contribution :)