[COMMIT osv master] aws: add new script to easily deploy to and run OSv on EC2

[Commit Bot]

From: Waldemar Kozaczuk <jwkoz...@gmail.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master

aws: add new script to easily deploy to and run OSv on EC2

This patch adds new script deploy_to_aws.sh which streamlines
the process of uploading OSv image (usr.img) to AWS as a snapshot,
creating AMI out of it and finally instantiating simple stack
with a single EC2 instance.

To run the scripts you need to have AWS cli installed and configured
to work with your account. In addition, you also need to clone
the snapshot tool https://github.com/awslabs/flexible-snapshot-proxy
and adjust deploy_to_aws.sh accordingly to point to it on your local
filesystem.

The workflow is this:

1. Build your desired OSv image, for example (use fs_size_mb to limit
the image size):
./scripts/build image=golang-pie-httpserver,httpserver-monitoring-api fs_size_mb=72

2. Run the image locally or use imgedit.py to set/change desired boot command
line:
./scripts/imgedit.py setargs build/release/usr.img "<cmdline>"

3. Run deploy_to_aws.sh to upload the image to AWS and create the stack:
./scripts/deploy_to_aws.sh <name>

Behind the scenes, deploy_to_aws.sh converts usr.img to usr.raw, then
uploads usr.raw to AWS as a snapshot using flexible-snapshot-proxy tool
and then creates AMI out of it (this patch adjusts ec2-make-ami.py to
support creating AMI our of pre-created snapshost). Finally, it uses
aws cli to create a stack with single EC2 instance based on the new AMI.

Before you use deploy_to_aws.sh make sure to fill in the values of your
account VPC ID and subnet ID in scripts/aws/instance-parameters.json.
Also change instance type to the one you desire (the t2-* instances use
Xen and t3, t4, etc use Nitro).

The new process is much simpler and faster. You no longer need to run
ec2-make-ami.py on some tool EC2 instance to build OSv ami which also
takes long time. Instead you can build and deploy your OSv image locally
under 30 seconds time.

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>

---
diff --git a/scripts/aws/instance-parameters.json b/scripts/aws/instance-parameters.json
--- a/scripts/aws/instance-parameters.json
+++ b/scripts/aws/instance-parameters.json
@@ -0,0 +1,22 @@
+[
+ {
+ "ParameterKey": "pVpcId",
+ "ParameterValue": "vpc-????????"
+ },
+ {
+ "ParameterKey": "pSubnetId",
+ "ParameterValue": "subnet-????????"
+ },
+ {
+ "ParameterKey": "pInstanceName",
+ "ParameterValue": "INSTANCE_NAME"
+ },
+ {
+ "ParameterKey": "pInstanceType",
+ "ParameterValue": "t2.nano"
+ },
+ {
+ "ParameterKey": "pImageId",
+ "ParameterValue": "AMI_ID"
+ }
+]
diff --git a/scripts/aws/instance.yaml b/scripts/aws/instance.yaml
--- a/scripts/aws/instance.yaml
+++ b/scripts/aws/instance.yaml
@@ -0,0 +1,70 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: Create EC2 Instance
+Parameters:
+ pVpcId:
+ Description: ID of the VPC
+ Type: AWS::EC2::VPC::Id
+ Default: '-'
+ pSubnetId:
+ Description: Subnet ID
+ Type: AWS::EC2::Subnet::Id
+ Default: '-'
+ pInstanceName:
+ Description: Instance Name
+ Type: String
+ Default: '-'
+ pInstanceType:
+ Description: Size of the Instance
+ Type: String
+ AllowedValues:
+ - t2.nano
+ - t2.micro
+ - t2.small
+ - t3.nano
+ - t3.micro
+ - t3.small
+ Default: t2.nano
+ pImageId:
+ Description: AMI for the instances
+ Type: AWS::EC2::Image::Id
+ Default: '-'
+Resources:
+ InstanceSecurityGroup:
+ Type: AWS::EC2::SecurityGroup
+ Properties:
+ GroupDescription: Allow external addresses to access to management console
+ SecurityGroupIngress:
+ - IpProtocol: tcp
+ FromPort: '9000'
+ ToPort: '9000'
+ CidrIp: '0.0.0.0/0'
+ - IpProtocol: tcp
+ FromPort: '8000'
+ ToPort: '8000'
+ CidrIp: '0.0.0.0/0'
+ VpcId:
+ Ref: pVpcId
+ Instance:
+ Type: AWS::EC2::Instance
+ Properties:
+ ImageId:
+ Ref: pImageId
+ InstanceType:
+ Ref: pInstanceType
+ SecurityGroupIds:
+ - Ref: InstanceSecurityGroup
+ SubnetId:
+ Ref: pSubnetId
+ Tags:
+ - Key: Name
+ Value:
+ Ref: pInstanceName
+Outputs:
+ PublicDnsName:
+ Value:
+ Fn::GetAtt:
+ - Instance
+ - PublicDnsName
+
+ InstanceID:
+ Value: !Ref Instance
diff --git a/scripts/deploy_to_aws.sh b/scripts/deploy_to_aws.sh
--- a/scripts/deploy_to_aws.sh
+++ b/scripts/deploy_to_aws.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+
+NAME=$1
+
+qemu-img convert -O raw build/release/usr.img build/release/usr.raw
+echo "Converted to raw image"
+
+snapshot_id=$(python3 ~/projects/flexible-snapshot-proxy/src/main.py upload build/release/usr.raw | tail -n 1)
+echo "Created snapshot: $snapshot_id"
+
+ami_id=$(./scripts/ec2-make-ami.py -n "$NAME" -s "$snapshot_id" | grep '^ami' | tail -n 1)
+echo "Created AMI: $ami_id"
+
+cat scripts/aws/instance-parameters.json | sed -e "s/INSTANCE_NAME/$NAME/" | sed -e "s/AMI_ID/$ami_id/" > /tmp/instance-parameters.json
+aws cloudformation create-stack \
+ --stack-name $NAME \
+ --template-body file://./scripts/aws/instance.yaml \
+ --capabilities CAPABILITY_IAM \
+ --parameters file:///tmp/instance-parameters.json
+
+#To clean
+#aws ec2 deregister-image --image-id <id>
+#aws ec2 delete-snapshot --snapshot-id <id>
diff --git a/scripts/ec2-make-ami.py b/scripts/ec2-make-ami.py
--- a/scripts/ec2-make-ami.py
+++ b/scripts/ec2-make-ami.py
@@ -1,12 +1,13 @@
-#!/usr/bin/python2
+#!/usr/bin/python3

import boto3, argparse, urllib, time, json, subprocess, os.path
import argparse
+from urllib.request import urlopen

class Metadata(object):
base = 'http://169.254.169.254/latest/meta-data/'
def _get(self, what):
- return urllib.urlopen(Metadata.base + what).read()
+ return urlopen(Metadata.base + what).read()
def instance_id(self):
return self._get('instance-id')
def availability_zone(self):
@@ -71,16 +72,24 @@ def make_snapshot(input):
print('Snapshot {} created\n'.format(snap))
return snap.id

-def make_ami_from_snapshot(name,snapshot_id):
- metadata = Metadata()
- print('Connecting')
- ec2 = boto3.resource('ec2',region_name=metadata.region())
- print('STEP 7: Registering image from {}'.format(snapshot_id)) # aws ec2 register-image
+def make_ami_from_snapshot(name,snapshot_id,on_ec2):
+ print('Connecting to make AMI from snapshot')
+ if on_ec2:
+ metadata = Metadata()
+ ec2 = boto3.resource('ec2',region_name=metadata.region())
+ else:
+ ec2 = boto3.resource('ec2')
+ print('STEP 7: Registering image {} from {}'.format(name, snapshot_id)) # aws ec2 register-image
time_point = time.time()
+
+ snapshot = ec2.Snapshot(snapshot_id)
+ snapshot.wait_until_completed()
+
ami = ec2.register_image(Name=name,
Architecture='x86_64',
RootDeviceName='xvda',
VirtualizationType='hvm',
+ EnaSupport=True,
BlockDeviceMappings=[
{
'DeviceName' : 'xvda',
@@ -92,6 +101,7 @@ def make_ami_from_snapshot(name,snapshot_id):
])
print('STEP 7: Took {} seconds to create ami'.format(time.time() - time_point,ami))
print('ami {} created\n'.format(ami))
+ print('{}\n'.format(ami.id))
return ami

if __name__ == "__main__":
@@ -101,7 +111,13 @@ def make_ami_from_snapshot(name,snapshot_id):
help="ami name to be created")
parser.add_argument("-i", "--input", action="store", default="build/release.x64/usr.img",
help="path to the image on local filesysten")
+ parser.add_argument("-s", "--snapshot", action="store",
+ help="snapshot ID")

args = parser.parse_args()
- snapshot_id = make_snapshot(args.input)
- make_ami_from_snapshot(args.name,snapshot_id)
+ if args.snapshot == None:
+ snapshot_id = make_snapshot(args.input)
+ else:
+ snapshot_id = args.snapshot
+
+ make_ami_from_snapshot(args.name, snapshot_id, args.snapshot == None)