0.9.0-beta2: driver start failed: failed to decode driver config

[Chris Stevens]

I'm seeing a new driver error with 0.9.0-beta2 that I haven't seen before with all the other system components remaining the same.

Recent Events:

Time                  Type             Description

2019-02-01T15:30:15Z  Killing          Sent interrupt

2019-02-01T15:30:14Z  Alloc Unhealthy  Unhealthy because of failed task

2019-02-01T15:30:14Z  Not Restarting   Error was unrecoverable

2019-02-01T15:30:14Z  Driver Failure   driver start failed: failed to decode driver config: [pos 379]: readContainerLen: Unrecognized descriptor byte: hex: 82, decimal: 130

2019-02-01T15:30:14Z  Task Setup       Building Task Directory

2019-02-01T15:30:13Z  Received         Task received by client

In addition, I noticed that these failures left consul health checks registered until I stopped and restarted the nomad agent.

Finally, this same environment also started to emit consul script check errors:

2019-02-01T15:16:54Z Driver Failure driver start failed: failed to decode driver config: [pos 379]: readContainerLen: Unrecognized descriptor byte: hex: 82, decimal: 130

2019-02-01T15:16:54Z Task hook failed consul_services: driver doesn't support script checks

The script is inside the container and has worked up through nomad 0.8.7.

Nomad version

Nomad v0.9.0-beta2 (29a5d242e69d2db5a36cf58d975f15c279f49a7e)

Operating system and Environment details

CentOS 7.6 1810 VM

Consul v1.4.2

Docker 18.09.1

[msch...@hashicorp.com]

Hi Chris!

Thanks for testing the beta. We knew we were going to have some ugly error messages during our transition from HCL1 -> HCL2, but this one is unacceptably useless!

Any chance you could share your job file? Just the task config stanza should be sufficient if the rest has private details, but the more context the better.

Also can you check the logs for that alloc/task to see if we output anything more useful there (DEBUG level might be required).

Thanks again,

Michael Schurter

[Chris Stevens]

Hi Michael.

No problem at all. I just emailed you a job file and debug logs.

- Chris

[msch...@hashicorp.com]

PR up: https://github.com/hashicorp/nomad/pull/5297

Not sure when our next prerelease will be, but unfortunately Docker's logging stanza will be broken until then. I uploaded a binary to the PR if you want to retest.

Thanks again for the testing Chris!

[Chris Stevens]

Thanks Michael! I'll try it out with the next beta or build a local copy if I have time.