Consistent docker_image module failure on ec2 instances

[Ananda Debnath]

Gurus,

I'm trying to create a docker image from a Dockerfile in an ec2 instance. I'm able to create the ec2 instances and copy over the dockerfile fine. However, for *every* ec2 instance I fire up, it *always* fails the first time I try to build the docker image.

- name: Build the docker Nginx image
  docker_image:
    path: "{{ docker_nginx_dir }}"
    state: build
    name: "grid/nginx"

The ansible failure stack trace looks like:
failed: [54.XX.XX.XX] => {"failed": true, "parsed": false}
invalid output was: SUDO-SUCCESS-bdbbnpaupjsbudcxajoycrwmbmlehwux
Traceback (most recent call last):
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1414698075.43-175058707465136/docker_image", line 1455, in <m                                                                    odule>
    main()
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1414698075.43-175058707465136/docker_image", line 220, in mai                                                                    n
    image_id = manager.build()
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1414698075.43-175058707465136/docker_image", line 143, in bui                                                                    ld
    chunk_json = json.loads(chunk)
  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/json/decoder.py", line 322, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 87 - line 1 column 52865 (char 87 - 52865)
OpenSSH_6.2p2 Ubuntu-6ubuntu0.4, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 24845
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
Shared connection to 54.XX.XX.XX closed.

Looking at the docker log file on the ec2 instance, the only obvious error I always see is something like:

[f648bc68] -job log(start, aab655401a74e9e6bb098c1d62d31e44beb1c116eb530296f315392a82c2e3dc, 4d76087678ee) = OK (0)
[error] attach.go:186 attach: stdout: write unix @: broken pipe
[error] attach.go:258 attach: job 1 returned error write unix @: broken pipe, aborting all jobs
write unix @: broken pipe
[f648bc68] -job build() = ERR (1)

I subsequently bumped up the timeout value thinking that the ssh client was timing out:

- name: Build the docker Nginx image
  docker_image:
    timeout: 1200
    path: "{{ docker_nginx_dir }}"
    state: build
    name: "grid/nginx"

But it fails much before this timeout. So I added a retry condition:

- name: Build the docker Nginx image
  docker_image:
    timeout: 1200
    path: "{{ docker_nginx_dir }}"
    state: build
    name: "grid/nginx"
  register: build_image_result
  until: not build_image_result|failed
  retries: 2
  delay: 30

This seems to consistently work - and I see that it always gets called twice - but I'm concerned as to why it would always need to. Any ideas?

Thanks!

More info if anyone needs it:
EC2 setup
- name: Setup AWS EC2 Instances - US-West-1
  ec2:
    instance_type: t2.micro
    image: "ami-41a7ea71"
    monitoring: no
    wait: yes
    group: microserver_security_group
    key_name: mykey
    instance_tags:
      Name: "MicroServer"
    count: 1
    region: "us-west-2"
  register: ec2west2

Docker install tasks:
- name: Update packages on the Amazon AMI
  yum: name=* state=latest
  when: ansible_distribution == 'Amazon'

- name: Install or Update Python on the Amazon AMI
  yum: name=python state=latest
  when: ansible_distribution == 'Amazon'

- name: Install or Update Python-pip on the Amazon AMI
  yum: name=python-pip state=latest
  when: ansible_distribution == 'Amazon'

- name: Install or Update Docker on the Amazon AMI
  yum: name=docker state=latest
  when: ansible_distribution == 'Amazon'

- name: Install or Update docker-py on all docker hosts
  pip: name=docker-py version=0.3.1
  # Had to delete /tmp/pip-build-root because pip would keep installing docker-py 0.5.3 as it was cached in this directory

- name: Start the docker service on the Amazon AMI
  service: name=docker state=restarted
  # Trying restarted because it always seems to fail the first time around
  when: ansible_distribution == 'Amazon'

Cheers.

[Michael DeHaan]

Can you share the versions of Ansible, docker, and docker libraries you are using for starters?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/fda1e638-e0fc-4b3a-9a9e-e36d7cf41c69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ananda Debnath]

Thanks for responding Michael. Here's all the version info:

Ansible 1.66 on an Ubuntu 13.10 EC2 host (below)

Ansible host info:
$ ansible --version
ansible 1.6.6

$ cat /etc/*{release,version}
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=13.10
DISTRIB_CODENAME=saucy
DISTRIB_DESCRIPTION="Ubuntu 13.10"
NAME="Ubuntu"
VERSION="13.10, Saucy Salamander"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 13.10"
VERSION_ID="13.10"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
wheezy/sid
Ubuntu 13.10 (Saucy Salamander)

Docker info
$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.3
Git commit (client): fa7b24f/1.2.0
OS/Arch (client): linux/amd64

Python info:
$ python --version
Python 2.6.9
$ pip --version
pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
$ pip list
argparse (1.2.1)
aws-cfn-bootstrap (1.4)
awscli (1.5.4)
backports.ssl-match-hostname (3.4.0.2)
bcdoc (0.12.2)
boto (2.34.0)
botocore (0.68.0)
chardet (2.0.1)
Cheetah (2.4.1)
cloud-init (0.7.2)
colorama (0.2.5)
configobj (4.7.2)
docker-py (0.3.1)
docutils (0.11)
iniparse (0.3.1)
jmespath (0.4.1)
kitchen (1.1.1)
lockfile (0.8)
Markdown (2.4.1)
ordereddict (1.1)
paramiko (1.7.5)
PIL (1.1.6)
ply (3.4)
pycrypto (2.6.1)
pycurl (7.19.0)
Pygments (1.4)
pygpgme (0.1)
pyliblzma (0.5.3)
pystache (0.5.3)
python-daemon (1.5.2)
python-dateutil (2.1)
pyxattr (0.5.0)
PyYAML (3.10)
requests (2.2.1)
rsa (3.1.2)
simplejson (3.5.3)
six (1.3.0)
urlgrabber (3.9.1)
urllib3 (1.8.2)
websocket-client (0.11.0)
yum-metadata-parser (1.1.4)

[Ananda Debnath]

Any ideas?

[Michael DeHaan]

Please check on the latest version of Ansible -- preferably the development branch since this is Docker and changing extremely quickly - and let us know if you still encounter problems.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/0a295152-7361-4966-b6da-595ab1f65b67%40googlegroups.com.