Client fails to download artifact

421 views

Skip to first unread message

Richard Collins

unread,

Sep 21, 2018, 8:03:53 AM9/21/18

to Mender List mender.io

Client:

I have a custom Linux distro build with Yocto 2.5 (Sumo) for our embedded IoT device.

The mender integration went well.

Server:

I have built our own Mender server as per the instructions here: https://docs.mender.io/1.6/administration/production-installation

This is running on AWS EC2 instance. It all looks to be running fine and I can not see any errors in the logs although there is so mush info it's hard for me to understand the output.

The server is behind a load balancer and the certificates were generated by Amazon. It is pointed to by a sub domain. This works and I can log in using https and get the green padlock.

I am able to upload new Artifacts and no errors are reported.

The IoT device is able to connect to the server and registered and in a device group.

I can create a deployment which starts.

The IoT device sees this deployment and the update transitions to being in progress.

This is when the problems begin.

The update fails almost instantly, the IoT device sees the update and then it's game over.

P.s I have removed some keys from this logs as I am not sure if it's safe to post them.

With the following logs from the mender interface:

2018-09-21 11:50:45 +0000 UTC debug: handle update fetch state

2018-09-21 11:50:45 +0000 UTC debug: status reported, response &{204 No Content %!s(int=204) HTTP/2.0 %!s(int=2) %!s(int=0) map[X-Frame-Options:[DENY] X-Content-Type-Options:[nosniff] X-Xss-Protection:[1; mode=block] Cache-Control:[no-cache, no-store] Pragma:[no-cache] Date:[Fri, 21 Sep 2018 11:50:45 GMT] Server:[openresty/1.13.6.2] Content-Encoding:[gzip] Vary:[Accept-Encoding] X-Deployments-Version:[unknown] X-Men-Requestid:[ 2018-09-21 11:50:45 +0000 UTC debug: Received fetch update response &{404 Not Found 404 HTTP/2.0 2 0 map[Date:[Fri, 21 Sep 2018 11:50:45 GMT] Content-Type:[text/html] Content-Length:[175] Server:[openresty/1.13.6.2]] 0x1249e220 175 [] false false map[] 0x12298500 0x12384480}+ 2018-09-21 11:50:45 +0000 UTC error: Error fetching shcheduled update info: code (404) 2018-09-21 11:50:45 +0000 UTC error: update fetch failed: (request_id: ): error receiving scheduled update information server error message: failed to parse server response: http2: response body closed 2018-09-21 11:50:45 +0000 UTC info: State transition: update-fetch [Download] -> fetch-install-retry-wait [Download] 2018-09-21 11:50:45 +0000 UTC debug: handle fetch install retry state 2018-09-21 11:50:46 +0000 UTC info: State transition: fetch-install-retry-wait [Download] -> update-status-report [none] 2018-09-21 11:50:46 +0000 UTC debug: handle update status report state 2018-09-21 11:50:46 +0000 UTC debug: status reported, response &{204 No Content %!s(int=204) HTTP/2.0 %!s(int=2) %!s(int=0) map[X-Deployments-Version:[unknown] X-Men-Requestid:[ 2018-09-21 11:50:46 +0000 UTC debug: attempting to upload deployment logs for failed update

Logs from the device

Sep 21 11:50:45 gk-pod-1 kern.info mender[196]: level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" module=mender Sep 21 11:50:45 gk-pod-1 kern.info mender[196]: level=info msg="Correct request for getting image from: https://s3._DOMAIN_REMOVED_:9000/mender-artifact-storage/_REMOVED_GUID_?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD Sep 21 11:50:45 gk-pod-1 kern.info mender[196]: level=info msg="State transition: update-check [Sync] -> update-fetch [Download]" module=mender Sep 21 11:50:45 gk-pod-1 kern.err mender[196]: level=error msg="Error fetching shcheduled update info: code (404)" module="client_update" Sep 21 11:50:45 gk-pod-1 kern.err mender[196]: level=error msg="update fetch failed: (request_id: ): error receiving scheduled update information server error message: failed to parse server response: http2: response body closed" module=state Sep 21 11:50:45 gk-pod-1 kern.info mender[196]: level=info msg="State transition: update-fetch [Download] -> fetch-install-retry-wait [Download]" module=mender Sep 21 11:50:46 gk-pod-1 kern.info mender[196]: level=info msg="State transition: fetch-install-retry-wait [Download] -> update-status-report [none]" module=mender Sep 21 11:50:45 gk-pod-1 kern.err mender[196]: level=error msg="Error fetching shcheduled update info: code (404)" module="client_update" Sep 21 11:50:45 gk-pod-1 kern.err mender[196]: level=error msg="update fetch failed: (request_id: ): error receiving scheduled update information server error message: failed to parse server response: http2: response body closed" module=state Sep 21 11:50:46 gk-pod-1 kern.info mender[196]: level=info msg="State transition: update-status-report [Download] -> idle [Idle]" module=mender Sep 21 11:50:46 gk-pod-1 kern.info mender[196]: level=info msg="authorization data present and valid" module=mender Sep 21 11:50:46 gk-pod-1 kern.info mender[196]: level=info msg="State transition: idle [Idle] -> check-wait [Idle]" module=mender

I am sure I have setup the s3 part wrong on the server as I found the documentation for this confusing.

Containers:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

5636949e10cd mendersoftware/api-gateway:1.5.0 "/entrypoint.sh" About an hour ago Up 43 minutes 0.0.0.0:443->443/tcp menderproduction_mender-api-gateway_1

d63506a1e0ff mendersoftware/openresty:1.11.2.2-alpine "/usr/local/openrest…" About an hour ago Up 43 minutes 0.0.0.0:9000->9000/tcp menderproduction_storage-proxy_1

9e1e978acbd0 mendersoftware/deployments:1.5.0 "/entrypoint.sh --co…" About an hour ago Up 43 minutes 8080/tcp menderproduction_mender-deployments_1

87a990989e02 mendersoftware/useradm:1.6.0 "/usr/bin/useradm --…" About an hour ago Up 43 minutes 8080/tcp menderproduction_mender-useradm_1

ec255a840564 mendersoftware/inventory:1.4.1 "/usr/bin/inventory …" About an hour ago Up 43 minutes 8080/tcp menderproduction_mender-inventory_1

21651e662780 mendersoftware/deviceauth:1.6.0 "/usr/bin/deviceauth…" About an hour ago Up 43 minutes 8080/tcp menderproduction_mender-device-auth_1

ea6922145232 mendersoftware/deviceadm:1.4.0 "/usr/bin/deviceadm …" About an hour ago Up 43 minutes 8080/tcp menderproduction_mender-device-adm_1

96ed4f352768 mendersoftware/mender-conductor:1.1.0 "/srv/start_conducto…" About an hour ago Up 43 minutes 8080/tcp menderproduction_mender-conductor_1

9c5569b1b32f mendersoftware/minio:RELEASE.2016-12-13T17-19-42Z "minio server /export" About an hour ago Up 43 minutes 9000/tcp menderproduction_minio_1

a717972d426f mendersoftware/gui:1.6.0 "/entrypoint.sh" About an hour ago Up 43 minutes 80/tcp menderproduction_mender-gui_1

ade109a31531 mongo:3.4 "docker-entrypoint.s…" About an hour ago Up 43 minutes 27017/tcp menderproduction_mender-mongo_1

48e73fc81264 redis:3.2.11-alpine "/redis/entrypoint.sh" About an hour ago Up 43 minutes 6379/tcp menderproduction_mender-redis_1

dfc9387574ed mendersoftware/elasticsearch:2.4 "/docker-entrypoint.…" About an hour ago Up 43 minutes 9200/tcp, 9300/tcp menderproduction_mender-elasticsearch_1

Again, __VERY_LONG_HASH__ is where a big number was an I was not sure if it should be included for security.

Volumes

DRIVER VOLUME NAME local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local __VERY_LONG_HASH__ local mender-artifacts local mender-db local mender-elasticsearch-db local mender-redis-db

This is my prod.yml again __PRIVATE__ is in the place of sensitive data.

# this is a template file for production setup, consult # https://docs.docker.com/compose/compose-file/ for details on syntax and usage # # Notes: # - integration/docker-compose.yml file is assumed to be included # - integration/docker-compose.storage.minio.yml is assumed to be included # - all services are part of `mender` network (service names are unchanged) # - keys and certificates are generated using keygen utility from integration # repository, keys and certificates are stored in ./keys-generated directory # - certificates and key are mounted into containers using volumes # - minio artifacts are stored in a named volume `mender-artifacts`; volume # needs to be created manually using `docker volume create mender-artifacts` # - paths need to be adjusted, ie, replace /production/ with production directory # (see list of compose bugs) # related compose bugs: # - https://github.com/docker/compose/issues/3874 # - https://github.com/docker/compose/issues/3568 # - https://github.com/docker/compose/issues/3219 version: '2' services: mender-useradm: command: server --automigrate volumes: - ./production/keys-generated/keys/useradm/private.key:/etc/useradm/rsa/private.pem:ro logging: options: max-file: "10" max-size: "50m" mender-device-auth: command: server --automigrate volumes: - ./production/keys-generated/keys/deviceauth/private.key:/etc/deviceauth/rsa/private.pem:ro logging: options: max-file: "10" max-size: "50m" environment: DEVICEAUTH_MAX_DEVICES_LIMIT_DEFAULT: 0 mender-inventory: command: server --automigrate logging: options: max-file: "10" max-size: "50m" mender-device-adm: command: server --automigrate logging: options: max-file: "10" max-size: "50m" mender-api-gateway: ports: # list of ports API gateway is made available on - "443:443" networks: - mender volumes: - ./production/keys-generated/certs/api-gateway/cert.crt:/var/www/mendersoftware/cert/cert.crt:ro - ./production/keys-generated/certs/api-gateway/private.key:/var/www/mendersoftware/cert/private.key:ro logging: options: max-file: "10" max-size: "50m" environment: ALLOWED_HOSTS: mender.__PRIVATE__ storage-proxy: ports: # outside port mapping for artifact storage (note that storage-proxy listens on port 9000) - "9000:9000" networks: mender: aliases: # change the alias to DNS name that storage will be # available on, for instance if devices will access storage # using https://s3.acme.org:9000, then set this to # s3.acme.org - s3.__PRIVATE__ environment: # use nginx syntax for rate limiting, see # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate # Examples: # 1m - 1MB/s # 512k - 512kB/s DOWNLOAD_SPEED: 1m MAX_CONNECTIONS: 100 volumes: - ./production/keys-generated/certs/storage-proxy/cert.crt:/var/www/storage-proxy/cert/cert.crt:ro - ./production/keys-generated/certs/storage-proxy/private.key:/var/www/storage-proxy/cert/private.key:ro mender-deployments: command: server --automigrate volumes: - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/storage-proxy.crt:ro environment: STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt # access key, the same value as MINIO_ACCESS_KEY DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments # secret, the same valie as MINIO_SECRET_KEY DEPLOYMENTS_AWS_AUTH_SECRET: __PRIVATE__ # deployments service uses signed URLs, hence it needs to access # storage-proxy using exactly the same name as devices will; if # devices will access storage using https://s3.acme.org:9000, then # set this to https://s3.acme.org:9000 DEPLOYMENTS_AWS_URI: https://s3.__PRIVATE__:9000 logging: options: max-file: "10" max-size: "50m" minio: environment: # access key MINIO_ACCESS_KEY: mender-deployments # secret MINIO_SECRET_KEY: __PRIVATE__ volumes: # mounts a docker volume named `mender-artifacts` as /export directory - mender-artifacts:/export:rw mender-conductor: volumes: - ./conductor/server/config:/app/config environment: - CONFIG_PROP=config.properties mender-mongo: volumes: - mender-db:/data/db:rw mender-elasticsearch: volumes: - mender-elasticsearch-db:/usr/share/elasticsearch/data:rw mender-redis: volumes: - mender-redis-db:/var/lib/redis:rw - ./conductor/redis/redis.conf:/etc/redis/redis.conf - ./conductor/redis/entrypoint.sh:/redis/entrypoint.sh entrypoint: /redis/entrypoint.sh volumes: # mender artifacts storage mender-artifacts: external: # use external volume created manually name: mender-artifacts # mongo service database mender-db: external: # use external volume created manually name: mender-db # elasticsearch database mender-elasticsearch-db: external: # use external volume created manually name: mender-elasticsearch-db # redis database mender-redis-db: external: # use external volume created manually name: mender-redis-db

Thanks for your help. :)

Mirza Krak

unread,

Sep 21, 2018, 8:19:37 AM9/21/18

to Mender List mender. io

On Fri, 21 Sep 2018, 14:03 Richard Collins, <richard...@guidedknowledge.com> wrote:

Client:
I have a custom Linux distro build with Yocto 2.5 (Sumo) for our embedded IoT device.
The mender integration went well.

Server:
I have built our own Mender server as per the instructions here: https://docs.mender.io/1.6/administration/production-installation
This is running on AWS EC2 instance. It all looks to be running fine and I can not see any errors in the logs although there is so mush info it's hard for me to understand the output.
The server is behind a load balancer and the certificates were generated by Amazon. It is pointed to by a sub domain. This works and I can log in using https and get the green padlock.

I am able to upload new Artifacts and no errors are reported.
The IoT device is able to connect to the server and registered and in a device group.
I can create a deployment which starts.
The IoT device sees this deployment and the update transitions to being in progress.

This is when the problems begin.

The update fails almost instantly, the IoT device sees the update and then it's game over.

P.s I have removed some keys from this logs as I am not sure if it's safe to post them.

With the following logs from the mender interface:

2018-09-21 11:50:45 +0000 UTC debug: handle update fetch state
2018-09-21 11:50:45 +0000 UTC debug: status reported, response &{204 No Content %!s(int=204) HTTP/2.0 %!s(int=2) %!s(int=0) map[X-Frame-Options:[DENY] X-Content-Type-Options:[nosniff] X-Xss-Protection:[1; mode=block] Cache-Control:[no-cache, no-store] Pragma:[no-cache] Date:[Fri, 21 Sep 2018 11:50:45 GMT] Server:[openresty/1.13.6.2] Content-Encoding:[gzip] Vary:[Accept-Encoding] X-Deployments-Version:[unknown] X-Men-Requestid:[ 2018-09-21 11:50:45 +0000 UTC debug: Received fetch update response &{404 Not Found 404 HTTP/2.0 2 0 map[Date:[Fri, 21 Sep 2018 11:50:45 GMT] Content-Type:[text/html] Content-Length:[175] Server:[openresty/1.13.6.2]] 0x1249e220 175 [] false false map[] 0x12298500 0x12384480}+ 2018-09-21 11:50:45 +0000 UTC error: Error fetching shcheduled update info: code (404) 2018-09-21 11:50:45 +0000 UTC error: update fetch failed: (request_id: ): error receiving scheduled update information server error message: failed to parse server response: http2: response body closed 2018-09-21 11:50:45 +0000 UTC info: State transition: update-fetch [Download] -> fetch-install-retry-wait [Download] 2018-09-21 11:50:45 +0000 UTC debug: handle fetch install retry state 2018-09-21 11:50:46 +0000 UTC info: State transition: fetch-install-retry-wait [Download] -> update-status-report [none] 2018-09-21 11:50:46 +0000 UTC debug: handle update status report state 2018-09-21 11:50:46 +0000 UTC debug: status reported, response &{204 No Content %!s(int=204) HTTP/2.0 %!s(int=2) %!s(int=0) map[X-Deployments-Version:[unknown] X-Men-Requestid:[ 2018-09-21 11:50:46 +0000 UTC debug: attempting to upload deployment logs for failed update

Logs from the device

Sep 21 11:50:45 gk-pod-1 kern.info mender[196]: level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" module=mender Sep 21 11:50:45 gk-pod-1 kern.info mender[196]: level=info msg="Correct request for getting image from: https://s3._DOMAIN_REMOVED_:9000/mender-artifact-storage/_REMOVED_GUID_?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD

I would start looking at this URL if it is valid. It is generated on the server side, or even more correctly it is generated by s3 in your case I guess. It seems that it is not correct as the client receives a 404 error when trying to download.

You should be able to copy/paste the URL in to a browser and it should start download.

/ Mirza

Richard Collins

unread,

Sep 21, 2018, 8:50:24 AM9/21/18

to Mender List mender.io, mirza...@northern.tech

This is where I have got confused with the setup of the server. I am not using an s3 bucket on AWS as I thought minio did that for me. The s3 domain that we have points back to the same server that is running the mender server. I did this as our IoT client seems to always want to connect to s3._OUR_DOMAIN_ and ignore what was set in the prod.yml

If I list this folder /var/lib/docker/volumes/mender-artifacts/_data/mender-artifact-storage I can see the update artifact. So it is stored on the server.

As you say, the client can't download it. I can't either from a web browser.

One question I have, if we're not using an aws s3 bucket does any of that need to be configured? I would prefer to use our server for storage as well.

Many thanks :)

Mirza Krak

unread,

Sep 21, 2018, 8:57:04 AM9/21/18

to Mender List mender.io

On Fri, Sep 21, 2018 at 2:50 PM Richard Collins <richard...@guidedknowledge.com> wrote:

This is where I have got confused with the setup of the server. I am not using an s3 bucket on AWS as I thought minio did that for me. The s3 domain that we have points back to the same server that is running the mender server. I did this as our IoT client seems to always want to connect to s3._OUR_DOMAIN_ and ignore what was set in the prod.yml

If I list this folder /var/lib/docker/volumes/mender-artifacts/_data/mender-artifact-storage I can see the update artifact. So it is stored on the server.

As you say, the client can't download it. I can't either from a web browser.

One question I have, if we're not using an aws s3 bucket does any of that need to be configured? I would prefer to use our server for storage as well.

By default it will use the "minio" service to store artifacts. So you will not need to configure anything if you want to use that.

s3._OUR_DOMAIN must point to the "OUR_DOMAIN":9000, if the minio is to be accessible. Could this be the issue?

Mirza Krak | Embedded Solutions Architect | https://mender.io

Northern.tech AS | @northerntechHQ

Richard Collins

unread,

Sep 21, 2018, 9:43:39 AM9/21/18

to Mender List mender.io, mirza...@northern.tech

It does, I need to look at how our EC2 server is configured again. So close to getting it working.

If I point a browser to our s3 domain without the port 9000 mapping I see the mender UI login page.

Ta :)

Richard Collins

unread,

Sep 21, 2018, 10:32:38 AM9/21/18

to men...@lists.mender.io, mirza...@northern.tech

After some rubber ducking with a work mate we found my load balancer was forwarding port 9000 to pork 443. I added a new target group and it looks to be working now. :)

Thanks for the help :)

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

Reply all

Reply to author

Forward

0 new messages