Using existing EBS volume to create AMI from scratch

[subodh...@druva.com]

Hi,

I need a packer builder/provisioner/post-processor etc. to create AMI from an existing EBS volume containing OS, without providing source-ami. The reason I need to use existing EBS volume is that I want to retain my OS data and want Packer to provision/harden/install-drivers,etc on top of it.

I have gone through amazon-chroot builder, it has a flag 'from_scratch' which can be used to create AMI from scratch. It creates a new EBS volume and creates AMI. But this has no option of providing existing AMI.

I have gone through amazon-ebssurrogate builder, it can be used to create AMI by attaching it to a running instance. But as per the required parameters, it needs source_ami.

• Please let me know why source_ami parameter is needed here, can we not create AMI based on the
  attached EBS volume itself ? Is there a way to skip it ?
• If EBS volume is of Windows, then do we need to attach it the running Windows EC2 Instance only ?

I have gone through packer amazon-import, it creates OVA, uploads to S3 and create AMI from it. I already have my VMDK (from which EBS volume is created) on S3. Can I use the amazon-import post-processor to directly create AMI from this OVA/VMDK ?

Please provide any other possible way using packer.

[Rickard von Essen]

What do you have to start with?

- A EBS Snapshot?

- With Windows, Linux or something else?

This message contains confidential information and is intended only for the individual to whom it is addressed. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and permanently delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, late or incomplete, or could contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version from the sender. Druva, www.druva.com

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/d1acdd05-27b6-4cc2-aedc-d74b2b2f04ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[subodh...@druva.com]

Hi Rickard,

We have VMware VM OS Disk data on S3 in RAW format. 

We have tried 

1. Create EBS volume from VMware VM OS disk by raw write, then create EBS snapshot of this volume. From this EBS snapshot, we created AMI. The AMI does not boot. We get "1/2 checks" error, No system log generated. Also, creating AMI from snapshot will not work for Windows (as recommended by Amazon AWS).

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

[subodh...@druva.com]

VMware VM can be either of Windows or Linux depending upon customer's VM.

[Rickard von Essen]

Then you don't need packer, just follow:

https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html

(After you set up policies etc it's just a oneliner with the cli)

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/3bf14640-20ab-41db-98d2-a73ba89b92c3%40googlegroups.com.

[subodh...@druva.com]

Hi Rickard,

Following are the reasons I need packer.

1. Creating AMI from AWS import-image is very slow but I have tried creating AMI using packer, it is done in minutes.

2. This is a VM replication app. When first time FULL backup is taken and put on S3, then we can AWS import-image to create AMI. But during second time, INCREMENTAL backup is taken and OS disk on S3 is patched with latest block changes. So, now with the latest data, I can AWS import-image to create AMI. The second time as well it takes time equivailent to FULL scan. Ideally, it should take lesser time than first time. So, I am trying to make VMware OS disk bootable in AWS EC2 using some kind provision/patching/installing-driver/etc using Packer. 

[subodh...@druva.com]

Packer's amazon-chroot builder has from_scratch option which can make an empty EBS volume bootable into EC2. I want to make similar thing with an existing EBS volume which has OS data in it. 

[Rickard von Essen]

Still a bit unclear, so the full (first) backup is put on S3, and the incremental patch set is that too on S3? 

You could use the amazon-chroot from_scratch and just dd the full backup to the root disk. With the incremental I guess it depends on how you "apply" it to a previous version (the full backup for the second backup)

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/b86db204-fa3c-4729-b893-5f41a48871a8%40googlegroups.com.

[subodh...@druva.com]

Let me explain.

First backup and create AMI :: 

1.VMware OS disk data copied to S3 into chunks of fixed size

2. Combine all chunks to create single RAW file or VMDK file

3. Create AMI using import-image.

Second backup(INCREMENTAL) and create AMI:

1. Copy Changed chunks to S3 (on different directory) 

2. Patch(i.e. replace) the already present chunks with these new changed chunks.

3. Combine all chunks to create single RAW file or VMDK file. Result will be latest VMware VM OS Disk on S3.

4. Create AMI from this new VMware VM OS Disk.

[subodh...@druva.com]

I did not understand what you are trying to say here "You could use the amazon-chroot from_scratch and just dd the full backup to the root disk." 

AFAIK, using 'from_scratch' option in amazon-chroot, it will create a new EBS volume, install OS onto the new EBS volume, does some provisioning and make it bootable into EC2 environment.

If I copy my full backup to this newly created EBS volume(I am not sure how to do it ...), then packer will install OS onto this volume which can wipeout my own data. So, this AMI will not usable to me as it does not have my VMware OS disk data.  

Or in other way, if Packer has created new EBS volume and installed OS into it, then after that if I copy my full backup this volume, it will overwrite the existing data and other provisioning done by packer. This will make the AMI not bootable.

Please help me understand this. 

[subodh...@druva.com]

Adding one more thing,

When my VMware OS Disk is present on S3, then I can directly use import-image API which is very slow.

So, I created a new EBS volume and copied VMware OS Disk to the EBS volume by raw writing to the correct offsets(there is no data corruption. I have confirmed by mounting the ebs volume and checking the files as well as checking their checksums)

Now, I am trying to find a way to use this EBS volume(which contains VMWare OS disk data) and create bootable AMI using packer.

[subodh...@druva.com]

Hi Rickard, 

Any update on this ?

[Rickard von Essen]

To me there is no obvious solution, and the use case is so weird I don't think it's likely that there is a solution with Packer or that it's any better than writing a bash+cli/python+boto3 script that handle this.

This message contains confidential information and is intended only for the individual to whom it is addressed. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and permanently delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, late or incomplete, or could contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version from the sender. Druva, www.druva.com

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/791dcfa1-5ee9-4f73-b4e6-fff32fc1808e%40googlegroups.com.