Packer SCP loop on Amazon CodeBuild

[Chris Bateman]

I wrote a simple web server packer template that uses Amazon Linux.  It previously worked but now it doesn't.  The only thing that has changed is the Amazon Linux AMI as I have it set to use the latest.

This runs through Amazon CodeBuild on Ubuntu 14.04 Base - This is the buildspec file probably not needed but just giving context -

---
version: 0.2

phases:
  pre_build:
    commands:
      - echo "Installing HashiCorp Packer..."
      - curl -o packer.zip https://releases.hashicorp.com/packer/1.3.4/packer_1.3.4_linux_amd64.zip && unzip packer.zip
      - echo "Validating amazon-linux_xx.json"
      - ./packer validate ./amazon-linux_xx.json
  build:
    commands:
      - echo "Building HashiCorp Packer template, amazon-linux_xx.json"
      - PACKER_LOG=1 ./packer build -color=false ./amazon-linux_xx.json | tee build.log
  post_build:
    commands:
      - egrep "${AWS_REGION}\:\sami\-" build.log | cut -d' ' -f2 > ami_id.txt
      # Packer doesn't return non-zero status; we must do that if Packer build failed
      - test -s ami_id.txt || exit 1
      - echo "HashiCorp Packer build completed"
artifacts:
  files:
    # ami_builder_event.json
    - build.log
  discard-paths: yes

This is the json file -

 "variables": {
        "aws_region": "{{env `AWS_REGION`}}",
        "aws_ami_name": "amazon-linux_bw2_{{isotime \"02Jan2006\"}}"
    },

    "builders": [{
        "type": "amazon-ebs",
        "region": "{{user `aws_region`}}",
        "instance_type": "t2.micro",
        "ssh_username": "ec2-user",
        "ami_name": "{{user `aws_ami_name`}}",
        "ami_description": "Customized Amazon Linux",
        "associate_public_ip_address": "true",
        "source_ami_filter": {
            "filters": {
                "virtualization-type": "hvm",
                "name": "amzn-ami*-ebs",
                "root-device-type": "ebs"
            },
            "owners": ["137112412989", "591542846629", "801119661308", "102837901569", "013907871322", "206029621532", "286198878708", "443319210888"],
            "most_recent": true
        }
    }],


    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "echo 'Running sudo yum update -y'",
                "sudo yum update -y",
                "echo 'Installing httpd'",
                "sudo yum install httpd24 -y",
                "echo 'Installing git'",
                "sudo yum install git -y",
                "echo 'Installing PHP'",
                "sudo yum update -y",
                "sudo yum install -y httpd24",
                "sudo yum install -y git",
                "sudo yum install -y php71",
                "sudo yum install -y php71-opcache",
                "sudo yum install -y php71-apcu",
                "sudo yum install -y mysql56",
                "sudo yum install -y php71-mysqlnd",
                "sudo yum install -y php71-fpm",
                "sudo yum install -y php71-mbstring",
                "sudo yum install -y php71-soap",
                "sudo yum install php71-gd ",
                "sudo yum install php71-mcrypt",
                "sudo chkconfig httpd on",
                "echo 'Installing CodeDeploy Agent'",
                "sudo yum install ruby wget -y",
                "cd /home/ec2-user",
                "wget https://aws-codedeploy-ap-southeast-2.s3.amazonaws.com/latest/install",
                "chmod +x ./install",
                "sudo ./install auto",
                "sudo service codedeploy-agent status"
            ]
        }
    ]
}

This is the main error I get -

amazon-ebs: Instance ID: i-00aa3dc02ddc1b6c8
==> amazon-ebs: Waiting for instance (i-00aa3dc02ddc1b6c8) to become ready...
==> amazon-ebs: Using ssh communicator to connect: 13.54.75.137
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with shell script: /tmp/packer-shell021567065  - This line fails then it will timeout
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Retryable error: Error uploading script: SCP failed to start. This usually means that SCP is not
properly installed on the remote system.
 
==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Retryable error: Error uploading script: SCP failed to start. This usually means that SCP is not
properly installed on the remote system.

This is the debug output -

2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [INFO] 821 bytes written for 'uploadData'
2019/02/28 01:02:17 [INFO] 821 bytes written for 'uploadData'
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] Opening new ssh session
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] Starting remote scp process:  scp -vt /tmp
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] Started SCP session, beginning transfers...
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] Copying input data into temporary file so we can read the length
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] scp: Uploading script_8564.sh: perms=C0644 size=821
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] SCP session complete, closing stdin pipe.
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] Waiting for SSH session to complete.
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] non-zero exit status: 127
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 [DEBUG] scp output:
2019/02/28 01:02:17 packer: 2019/02/28 01:02:17 Retryable error: Error uploading script: SCP failed to start. This usually means that SCP is not
2019/02/28 01:02:17 packer: properly installed on the remote system.

That loops for a few minutes before failing the build.

Now installing SCP seems to be the solution here but I just can't get it to install so I would like some advice on that.

But this did work previously so I would like to know how that has occurred.  Maybe amazon removed SCP in their base image?

[Rickard von Essen]

Try setting ssh_file_transfer_method to sftp. 

https://packer.io/docs/templates/communicator.html#ssh_file_transfer_method

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/123133f9-d9a7-496e-87f6-fff6c97497e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Bateman]

That worked :) Awesome.