Docker failed to start contain with "invalid argument", how to debug it?

[Xing Zhao]

Hi All,

I am trying to start containers with my custom-built docker, but docker always failed to start the container with "invalid argument". I have enabled debug mode, but cannot figure out why it failed. Who knows how to debug it inside or what is the error reason? Thanks in advance.

Here is the detail info:

root@rk3288_box:/var/run # docker info

Containers: 4

 Running: 0

 Paused: 0

 Stopped: 4

Images: 2

Server Version: 1.12.0-dev

Storage Driver: vfs

Logging Driver: json-file

Cgroup Driver: cgroupfs

Plugins: 

 Volume: local

 Network: bridge null host

Kernel Version: 3.10.0

Operating System: <unknown>

OSType: linux

Architecture: armv7l

CPUs: 4

Total Memory: 1.966 GiB

Name: localhost

ID: KHZ7:BIXU:OX3I:ECGV:JQER:3LUG:HSCJ:YIUW:YO4F:V7X3:5OOA:4AJD

Docker Root Dir: /var/lib/docker

Debug Mode (client): false

Debug Mode (server): true

 File Descriptors: 15

 Goroutines: 30

 System Time: 2016-04-26T13:35:40.202902765Z

 EventsListeners: 0

Registry: https://index.docker.io/v1/

Insecure registries:

 127.0.0.0/8

root@rk3288_box:/var/run # docker version

Client:

 Version:      1.12.0-dev

 API version:  1.24

 Go version:   go1.5.4

 Git commit:   7fd53f7-unsupported

 Built:        Tue Apr 26 10:30:02 2016

 OS/Arch:      linux/arm

Server:

 Version:      1.12.0-dev

 API version:  1.24

 Go version:   go1.5.4

 Git commit:   7fd53f7-unsupported

 Built:        Tue Apr 26 10:30:02 2016

 OS/Arch:      linux/arm

root@rk3288_box:/var/run # 

Here is the debug message:

DEBU[1758] Calling POST /v1.24/containers/create        

DEBU[1758] form data: {"AttachStderr":true,"AttachStdin":true,"AttachStdout":true,"Cmd":["/bin/bash"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NetworkMaximumBandwidth":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"StorageOpt":{},"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"armv7/armhf-debian","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":true,"Tty":true,"User":"","Volumes":{},"WorkingDir":""} 

DEBU[1758] Creating dest directory: /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980-init 

DEBU[1758] Calling TarUntar(/var/lib/docker/vfs/dir/716e7d412c692e6b2df4ea41d05acb2c4c7331071bee81137cff3d645b7586c2, /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980-init) 

DEBU[1758] TarUntar(/var/lib/docker/vfs/dir/716e7d412c692e6b2df4ea41d05acb2c4c7331071bee81137cff3d645b7586c2 /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980-init) 

DEBU[1760] Creating dest directory: /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980 

DEBU[1760] Calling TarUntar(/var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980-init, /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980) 

DEBU[1760] TarUntar(/var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980-init /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980) 

DEBU[1762] container mounted via layerStore: /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980 

DEBU[1762] Calling POST /v1.24/containers/0a4a00d8793dc112a196655fa4343625b889efbb2bbdf237576fe90b5b8de60f/attach?stderr=1&stdin=1&stdout=1&stream=1 

DEBU[1762] attach: stdin: begin                         

DEBU[1762] attach: stdout: begin                        

DEBU[1762] attach: stderr: begin                        

DEBU[1762] Calling POST /v1.24/containers/0a4a00d8793dc112a196655fa4343625b889efbb2bbdf237576fe90b5b8de60f/start 

DEBU[1762] container mounted via layerStore: /var/lib/docker/vfs/dir/12d889b363057aa1d4fe3d0a4942b62faeebcb05b5fef385cab590a7c8d27980 

DEBU[1762] Assigning addresses for endpoint hopeful_dubinsky's interface on network bridge 

DEBU[1762] Assigning ipv4 addresses for endpoint hopeful_dubinsky's interface on network bridge 

DEBU[1762] RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[]) 

DEBU[1762] Assigning addresses for endpoint hopeful_dubinsky's interface on network bridge 

DEBU[1763] Programming external connectivity on endpoint hopeful_dubinsky (3a51415d240c7c75b6ed6ce5f46854cfaea7383c9840276f9594044764283e41) 

DEBU[1763] attach: stdout: end                          

DEBU[1763] attach: stdin: end                           

DEBU[1763] attach: stderr: end                          

DEBU[1763] Revoking external connectivity on endpoint hopeful_dubinsky (3a51415d240c7c75b6ed6ce5f46854cfaea7383c9840276f9594044764283e41) 

DEBU[1764] Releasing addresses for endpoint hopeful_dubinsky's interface on network bridge 

DEBU[1764] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2) 

ERRO[1764] Handler for POST /v1.24/containers/0a4a00d8793dc112a196655fa4343625b889efbb2bbdf237576fe90b5b8de60f/start returned error: invalid argument 

DEBU[1764] Closing buffered stdin pipe                  

Thanks,

Felix