Nomad LXC Driver setup
234 views
Skip to first unread message
Aaron Ruddick
Aug 19, 2019, 12:57:00 PM8/19/19
to Nomad
Hi Again,
I am following the steps of this article to setup LXC Containers with Nomad (like docker).
When I get to the part that lets me run the lxc demo job, my agent is saying "missing drivers".
ANy idea what I am missing? The server sees the plugin (below) adn seems happy?
DO I need to add somethign to the Agetn Client
ubuntu@vnomadlab01:~$ sudo nomad run lxc.nomad
==> Monitoring evaluation "5c221348"
Evaluation triggered by job "example-lxc"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "5c221348" finished with status "complete" but failed to place all allocations:
Task Group "example" (failed to place 1 allocation):
* Constraint "missing drivers" filtered 1 nodes
Evaluation "a76af8c1" waiting for additional capacity to place remainder
Server side:
2019-08-19T16:53:24.948Z [DEBUG] agent.plugin_loader: plugin exited: plugin_dir=/opt/nomad/data/plugins
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=lxc type=driver plugin_version=0.1.0-rc2
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2019-08-19T16:53:24.948Z [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Client config (should add plugin path too?):
# Increase log verbosity
log_level = "DEBUG"
# Setup data dir
data_dir = "/opt/nomad"
# Give the agent a unique name. Defaults to hostname
name = "client1"
# Enable the client
client {
enabled = true
# For demo assume we are talking to server1. For production,
# this should be like "nomad.service.consul:4647" and a system
# like Consul used for service discovery.
servers = ["127.0.0.1:4647"]
}
# Modify our port to avoid a collision with server1
ports {
http = 5656
}
~
Aaron Ruddick
Aug 19, 2019, 1:37:40 PM8/19/19
to Nomad
Hmm, the issue seems to be "API error (500)" from the client:
ubuntu@vnomadlab01:~$ nomad alloc status 8c6d6c82
ID = 8c6d6c82
Eval ID = c7961bf9
Name = example.cache[0]
Node ID = 769aa3b7
Node Name = client1
Job ID = example
Job Version = 824636223616
Client Status = failed
Client Description = Failed tasks
Desired Status = run
Desired Description = <none>
Created = 4m49s ago
Modified = 2m50s ago
Reschedule Eligibility = 28m5s from now
Task "redis" is "dead"
Task Resources
CPU Memory Disk Addresses
500 MHz 256 MiB 300 MiB db: ************.109:25753
Task Events:
Started At = N/A
Finished At = 2019-08-19T17:31:23Z
Total Restarts = 2
Last Restart = 2019-08-19T17:30:59Z
Recent Events:
Time Type Description
2019-08-19T17:31:24Z Killing Sent interrupt. Waiting 5s before force killing
2019-08-19T17:31:23Z Not Restarting Exceeded allowed attempts 2 in interval 30m0s and mode is "fail"
2019-08-19T17:31:23Z Driver Failure Failed to start container 0d6732ed61fb0316a0ff1a18f91d5661620ba360ab07bf03fdc264e3f77848df: API error (500): error while creating mount source path '/opt/nomad/alloc/8c6d6c82-a8dc-e798-902b-4d2762c4018c/alloc': mkdir /opt/nomad: read-only file system
2019-08-19T17:30:59Z Restarting Task restarting in 18.517524359s
2019-08-19T17:30:59Z Driver Failure Failed to start container 8a334ec891392a010030bc813e2aff325cd3791ce9360d8224e1356cb174b2d8: API error (500): error while creating mount source path '/opt/nomad/alloc/8c6d6c82-a8dc-e798-902b-4d2762c4018c/alloc': mkdir /opt/nomad: read-only file system
2019-08-19T17:30:36Z Restarting Task restarting in 16.830114495s
2019-08-19T17:30:36Z Driver Failure Failed to start container 0226728d1682309816e12f41c35f151ac341db6fe49300cb47f77a3c57eca136: API error (500): error while creating mount source path '/opt/nomad/alloc/8c6d6c82-a8dc-e798-902b-4d2762c4018c/alloc': mkdir /opt/nomad: read-only file system
2019-08-19T17:30:30Z Task Setup Building Task Directory
2019-08-19T17:30:30Z Received Task received by client
ubuntu@vnomadlab01:~$
Lowe Schmidt
Aug 19, 2019, 1:56:37 PM8/19/19
to Aaron Ruddick, Nomad
Is your Nomad clients running as root?
--
Lowe Schmidt | +46 723 867 157
--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/26fd8be3-6356-4c81-940b-4560379944a9%40googlegroups.com.
Shantanu Gadgil
Aug 19, 2019, 2:56:45 PM8/19/19
to Nomad
I see another thing, you use the 0.1.0 version and not the rc2 (release candidate= version
Shantanu Gadgil
Aug 19, 2019, 3:00:04 PM8/19/19
to Nomad
Sorry, hit send too soon. Typically a "driver" error means the appropriate runtime hasn't been detected by the Nomad agent. If the correct lxc packages are installed, there could be something else which is preventing the Nomad agent from detecting LXC suport
Aaron Ruddick
Aug 20, 2019, 2:13:29 PM8/20/19
to Nomad
Thank you! the tutorial had me place the plugin in a path that was a little different then default:
ubuntu@vnomadlab01:/opt/nomad/data/plugins$ ls
nomad-driver-lxc
>> I see another thing, you use the 0.1.0 version and not the rc2
But I am still interested in this. Are you using the RC2 for LXC? is it better then 0.1.0?
Jobs seems to be running now, but the lxc is still not appearing, so maybe I need to go with the latest?
I just assumed it would be unstable (testing only), but "1.0" feels scary too! haha.
Aaron Ruddick
Aug 20, 2019, 2:22:30 PM8/20/19
to Nomad
Yes, it is running now as a service as root.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages