amazon-choot builder and CoreOS (newbie in Packer)

[Sergey Esin]

Hi,

I am newbie in Packer, would like to use Packer with amazon-chroot builder to customize CoreOS-stable-1122.2.0-hvm (officia public AMI ID in EU Ireland region: ami-e3d6ab90)

Packer config is really simple:

$ cat coreos-1122.2.0-hvm-aws-chroot.json

{

  "variables": {

    "aws_access_key": "",

    "aws_secret_key": ""

  },

  "builders": [{

    "name": "build-aws-hmv",

    "type": "amazon-chroot",

    "access_key": "{{user `aws_access_key`}}",

    "secret_key": "{{user `aws_secret_key`}}",

    "ami_virtualization_type": "hvm",

    "region": "eu-west-1",

    "source_ami": "ami-0092eb73",

    "ami_name": "packer-example {{timestamp}} coreos",

    "tags": {

        "component": "allapps",

        "version": "404"

    }

  }],

  "provisioners": [{

    "type": "shell",

    "inline": [

      "docker pull centos:7"

    ]

  }]

}

$ ./packer validate coreos-1122.2.0-hvm-aws-chroot.json

Template validated successfully.

I am getting this error:

$ ./packer build coreos-1122.2.0-hvm-aws-chroot.json

build-aws-hmv output will be in this color.

==> build-aws-hmv: Prevalidating AMI Name...

==> build-aws-hmv: Gathering information about this EC2 instance...

==> build-aws-hmv: Error retrieving the ID of the instance Packer is running on.

==> build-aws-hmv: Please verify Packer is running on a proper AWS EC2 instance.

Build 'build-aws-hmv' errored: Error retrieving the ID of the instance Packer is running on.

Please verify Packer is running on a proper AWS EC2 instance.

==> Some builds didn't complete successfully and had errors:

--> build-aws-hmv: Error retrieving the ID of the instance Packer is running on.

Please verify Packer is running on a proper AWS EC2 instance.

==> Builds finished but no artifacts were created.

AMI "ami-0092eb73" that I am using in my config above was created the following way:

1. Used original public CoreOS image (ami-e3d6ab90) to run instance.

2. Stopped the instance and took an AMI

3. Run the instance back and left it in running state

What's wrong with my config?

Do I understand right that in order to be able to use amazon-chroot builder with Packer I have to leave my instance running?

p.s. The following example.json works fine for me (so it's all fine with AWS IAM permissions):

$ cat example.json

{

  "variables": {

    "aws_access_key": "",

    "aws_secret_key": ""

  },

  "builders": [{

    "type": "amazon-ebs",

    "access_key": "{{user `aws_access_key`}}",

    "secret_key": "{{user `aws_secret_key`}}",

    "region": "eu-west-1",

    "source_ami": "ami-f95ef58a",

    "instance_type": "t2.micro",

    "ssh_username": "ubuntu",

    "ami_name": "packer-example {{timestamp}}"

  }]

}

[Rickard von Essen]

Quoting the docs

"amazon-chroot - Create EBS-backed AMIs from an existing EC2 instance by mounting the root device and using a Chroot environment to provision that device. This is an advanced builder and should not be used by newcomers."

It seems like you are not running your build from an EC2 instance. I sugest that you use amazon-ebs builder until you have a particular reason for using the chroot builder.

[Sergey Esin]

Thanks, it's getting more and more clear now but even very simple json (see my original message) causes:

...

==> build-aws-hmv: Mounting the root device...

2016/09/16 09:26:02 ui: ==> build-aws-hmv: Mounting the root device...

2016/09/16 09:26:02 ui: ==> build-aws-hmv: Mounting additional paths within the chroot...

==> build-aws-hmv: Mounting additional paths within the chroot...

2016/09/16 09:26:02 ui:     build-aws-hmv: Mounting: /proc

    build-aws-hmv: Mounting: /proc

2016/09/16 09:26:02 ui:     build-aws-hmv: Mounting: /sys

    build-aws-hmv: Mounting: /sys

2016/09/16 09:26:02 ui:     build-aws-hmv: Mounting: /dev

    build-aws-hmv: Mounting: /dev

2016/09/16 09:26:02 ui:     build-aws-hmv: Mounting: /dev/pts

    build-aws-hmv: Mounting: /dev/pts

2016/09/16 09:26:02 ui:     build-aws-hmv: Mounting: /proc/sys/fs/binfmt_misc

    build-aws-hmv: Mounting: /proc/sys/fs/binfmt_misc

2016/09/16 09:26:02 ui: ==> build-aws-hmv: Copying files from host to chroot...

==> build-aws-hmv: Copying files from host to chroot...

    build-aws-hmv: /etc/resolv.conf

2016/09/16 09:26:02 ui:     build-aws-hmv: /etc/resolv.conf

2016/09/16 09:26:02 packer: 2016/09/16 09:26:02 Copying '/etc/resolv.conf' to '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf'

2016/09/16 09:26:02 ui error: ==> build-aws-hmv: Error copying file: exit status 1

==> build-aws-hmv: nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

==> build-aws-hmv:

==> build-aws-hmv: Error copying file: exit status 1

==> build-aws-hmv: nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

==> build-aws-hmv:

2016/09/16 09:26:02 ui: ==> build-aws-hmv: Unmounting the root device...

==> build-aws-hmv: Unmounting the root device...

2016/09/16 09:26:02 ui: ==> build-aws-hmv: Detaching EBS volume...

==> build-aws-hmv: Detaching EBS volume...

2016/09/16 09:26:02 packer: 2016/09/16 09:26:02 Waiting for state to become: detached

2016/09/16 09:26:02 packer: 2016/09/16 09:26:02 Allowing 300s to complete (change with AWS_TIMEOUT_SECONDS)

2016/09/16 09:26:11 ui: ==> build-aws-hmv: Deleting the created EBS volume...

==> build-aws-hmv: Deleting the created EBS volume...

2016/09/16 09:26:11 ui error: Build 'build-aws-hmv' errored: Error copying file: exit status 1

nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

2016/09/16 09:26:11 Builds completed. Waiting on interrupt barrier...

2016/09/16 09:26:11 machine readable: error-count []string{"1"}

2016/09/16 09:26:11 ui error:

==> Some builds didn't complete successfully and had errors:

2016/09/16 09:26:11 machine readable: build-aws-hmv,error []string{"Error copying file: exit status 1\nnStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory\n"}

2016/09/16 09:26:11 ui error: --> build-aws-hmv: Error copying file: exit status 1

nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

2016/09/16 09:26:11 ui:

==> Builds finished but no artifacts were created.

2016/09/16 09:26:11 waiting for all plugin processes to complete...

2016/09/16 09:26:11 /packer/packer: plugin process exited

Build 'build-aws-hmv' errored: Error copying file: exit status 1

nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

==> Some builds didn't complete successfully and had errors:

--> build-aws-hmv: Error copying file: exit status 1

nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

==> Builds finished but no artifacts were created.

2016/09/16 09:26:11 /packer/packer: plugin process exited

 

 

[Rickard von Essen]

The error is there:

cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdf/etc/resolv.conf': No such file or directory

You haven't created the directory. Repeating my advice use the amazon-ebs builder instead, at least until you get comfortable with packer and as log as you don't need a special feature of the amazon-chroot builder.

[Sergey Esin]

пятница, 16 сентября 2016 г., 12:45:37 UTC+3 пользователь Rickard von Essen написал:

You haven't created the directory. Repeating my advice use the amazon-ebs builder instead, at least until you get comfortable with packer and as log as you don't need a special feature of the amazon-chroot builder.

Answer is obvious but I still can not see why I need to create this dir. 

Documentation for amazon-chroot does not help too much.

I need amazon-chroot builder because of the speed. 

[Rickard von Essen]

If the directory doesn't exist you must create it, see https://www.packer.io/docs/provisioners/file.html#destination

If attach your full template plus scripts I can probably point out what is wrong.

If you need fast builds I recommend that you add both amazon-ebs and amazon-chroot to your template and run with packer build -only=amazon-ebs template.json after you have a build that works in the way you expect you switch to -only=amazon-chroot.

[Sergey Esin]

Thanks for the tip!

Here's my full template - https://gist.github.com/sirocode/909f9f90bb7756cf3594d1ea3505d645 ("device_type" has been added)

I just want to do really simple thing (as a first step with Packer and amazon-chroot builder) - to get CoreOS of required version with a docker image downloaded locally.

"source_ami" mentioned in the template was done the following way:

1. Run my own ebs-backed instance (with default setting) using public HVM image of the latest release of CoreOS

2. Stopped the instance and took AMI image from the stopped instance

3. Started the instance, logged in, sudo to root, configured ~/.aws/credentials

4. Run the packer build

# ./packer build coreos-1122.2.0-hvm-aws-chroot.json

build-aws-hmv-ebs output will be in this color.

==> build-aws-hmv-ebs: Prevalidating AMI Name...

==> build-aws-hmv-ebs: Gathering information about this EC2 instance...

==> build-aws-hmv-ebs: Inspecting the source AMI...

==> build-aws-hmv-ebs: Checking the root device on source AMI...

==> build-aws-hmv-ebs: Creating the root volume...

==> build-aws-hmv-ebs: Attaching the root volume to /dev/sdg

==> build-aws-hmv-ebs: Mounting the root device...

==> build-aws-hmv-ebs: Mounting additional paths within the chroot...

    build-aws-hmv-ebs: Mounting: /proc

    build-aws-hmv-ebs: Mounting: /sys

    build-aws-hmv-ebs: Mounting: /dev

    build-aws-hmv-ebs: Mounting: /dev/pts

    build-aws-hmv-ebs: Mounting: /proc/sys/fs/binfmt_misc

==> build-aws-hmv-ebs: Copying files from host to chroot...

    build-aws-hmv-ebs: /etc/resolv.conf

==> build-aws-hmv-ebs: Error copying file: exit status 1

==> build-aws-hmv-ebs: nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdg/etc/resolv.conf': No such file or directory

==> build-aws-hmv-ebs:

==> build-aws-hmv-ebs: Unmounting the root device...

==> build-aws-hmv-ebs: Detaching EBS volume...

==> build-aws-hmv-ebs: Deleting the created EBS volume...

Build 'build-aws-hmv-ebs' errored: Error copying file: exit status 1

nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdg/etc/resolv.conf': No such file or directory

==> Some builds didn't complete successfully and had errors:

--> build-aws-hmv-ebs: Error copying file: exit status 1

nStderr: cp: cannot create regular file '/mnt/packer-amazon-chroot-volumes/xvdg/etc/resolv.conf': No such file or directory

==> Builds finished but no artifacts were created.

What I am missing?

пятница, 16 сентября 2016 г., 13:50:21 UTC+3 пользователь Rickard von Essen написал:

[Rickard von Essen]

It sucks when the default values are not documented, this is what you are hitting:

https://github.com/mitchellh/packer/blob/master/builder/amazon/chroot/builder.go#L97

I guess there aren't any /etc/resolv.conf in a CoreOS AMI. The simple work around is to add "copy_files": [ "somefile that exist" ]

to your template.

Some comments about the workflow:

1. Run my own ebs-backed instance (with default setting) using public HVM image of the latest release of CoreOS
2. Stopped the instance and took AMI image from the stopped instance
3. Started the instance, logged in, sudo to root, configured ~/.aws/credentials
4. Run the packer build

Just use the public HVM image of the latest release of CoreOS as the source_ami and don't use access keys when you are running inside AWS, use IAM instance profile/role.

This would give you:

1) Start an EC2 instance with the correct instance profile

2) Login to it and run the packer build  

[Rickard von Essen]

Improved the docs a bit:

https://github.com/mitchellh/packer/pull/3893

[Sergey Esin]

Thanks a lot for all your help but seems things are not that easy with CoreOS :)

Tried to use "copy_files": "/boot/grub/menu.lst" but no luck.

I've run with "--debug" and stopped after snapshot is mounted:

==> build-aws-hmv: Attaching the root volume to /dev/sdg

==> build-aws-hmv: Pausing after run of step 'StepAttachVolume'. Press enter to continue.

==> build-aws-hmv: Pausing after run of step 'StepEarlyUnflock'. Press enter to continue.

==> build-aws-hmv: Mounting the root device...

==> build-aws-hmv: Pausing after run of step 'StepMountDevice'. Press enter to continue.

/dev/xvdg1 on /mnt/packer-amazon-chroot-volumes/xvdg type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

Then I've had a look on what is actually mounted:

ip-172-31-19-164 / # cd /mnt/packer-amazon-chroot-volumes/xvdg

ip-172-31-19-164 xvdg # ls -lah

total 32K

drwxr-xr-x. 6 root root  16K Jan  1  1970 .

drwxr-xr-x. 3 root root 4.0K Sep 16 12:09 ..

drwxr-xr-x. 3 root root 2.0K Sep  6 15:06 EFI

drwxr-xr-x. 3 root root 2.0K Sep  6 15:06 boot

drwxr-xr-x. 3 root root 2.0K Sep  6 15:06 coreos

drwxr-xr-x. 2 root root 2.0K Sep  6 15:06 xen

[Alvaro Miranda Aguilera]

are you using the right partition? seems its the EFI one with vfat format.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/cba22921-c581-406b-84fb-03e5e348be5c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alvaro

(+31)103400555

[Rickard von Essen]

So I did some further research and what I found is:

1) You need to recreate the official AMI by launching an instance and create an AMI from that since the official AMI dosen't seem to have public access to the associated snapshot.

2) CoreOS have a very different and complex partitioning. This requires "mount_partition": 9, to be set to mount the correct root partition. The following template works:

{

  "builders": [{

    "type": "amazon-chroot",

    "region": "eu-central-1",

    "ami_virtualization_type": "hvm",

    "source_ami": "ami-eab64b85",

    "ami_name": "packer-example {{timestamp}} coreos",

    "root_volume_size": 8,

    "mount_partition": 9,

    "chroot_mounts": [

         [ "proc", "proc", "/proc" ],

         [ "sysfs", "sysfs", "/sys" ],

         [ "bind", "/dev", "/dev" ],

         [ "devpts", "devpts", "/dev/pts" ]

    ]

  }]

}

But adding any provisioner crashes since there is no /bin or /usr/bin in the root partition. There are TWO partitions containing /usr (which /bin links to) USR-A and USR-B

USR-A is /dev/xvdb3 but trying to mount it in the chroot_mounts list fails with:

    amazon-chroot: Mounting: /usr

==> amazon-chroot: Error mounting: exit status 32

==> amazon-chroot: Stderr: mount: wrong fs type, bad option, bad superblock on /dev/xvdb3,

==> amazon-chroot:        missing codepage or helper program, or other error

==> amazon-chroot:

==> amazon-chroot:        In some cases useful info is found in syslog - try

==> amazon-chroot:        dmesg | tail or so.

==> amazon-chroot:

 

I think you should switch to amazon-ebs (it's not slow, you don't need chroot for speed) unless you want to learn every detail about CoreOS disk-layout, partitioning, and submit patches to Packer to get it to work for this very special case.