Python module for creating Terraform JSON configuration.

[Markus Juenemann]

Hi all,

I am currently working on a Python module that can assist in generating Terraform JSON configuration files:  https://github.com/mjuenema/python-terrascript.

The objective is to effectively add the features of the Python programming language (loops, conditionals, extensions, etc.) to the Terrafrom declarative syntax. 

See below for an example Python script and the JSON it generates (the current version 0.1.0 only supports AWS resources).

I'd like to hear your feedback, ideas, criticism, suggestions, bug reports, ..., anything really. 

Python:

from terrascript import provider, dump

# As of release 0.1.0 only AWS is supported.
from terrascript.aws.r import aws_instance

provider('aws', access_key='ACCESS_KEY_HERE', secret_key='SECRET_KEY_HERE', region='us-east-1')

aws_instance('example', ami='ami-2757f631', instance_type='t2.micro')

# Print the JSON-style configuration to stdout.
print(dump())

Terraform JSON

{
  "provider": {
    "aws": {
      "access_key": "ACCESS_KEY_HERE",
      "region": "us-east-1",
      "secret_key": "SECRET_KEY_HERE"
    }
  },
  "resource": {
    "aws_instance": {
      "example": {
        "ami": "ami-2757f631",
        "instance_type": "t2.micro"
      }
    }
  }
}

As a design decision, terrascript does not perform any error checking at all. The generated output will be valid JSON but is not guaranteed to make sense to Terraform!!!

[Dustin Williams]

Hi Markus, 

I've recently started learning Terraform and have been looking for a way to work with it and python together. Specifically I was thinking about using Boto3 and Terraform together. Do you think your module would help solve my current tasks? I've placed an  *  next to items that I think I should/could use python.  

Use case:
Creating a data-pipeline that will deposit process a few hundred thousand web articles. 
Each publisher will have control over their crawler. They want to set the crawl frequency & schedule.
Each publication could have a custom text extractions for each template
Each crawler has to have a custom Browser ID. 

Crawler Stage:
Scrapy crawler will run in ECS containers.
The container hosts will have a NFS share mounted.
The NFS share will have a folder for each publication's dedicated crawler, this will provide persistence for configurations & crawl history. 

*Use python to loop through a dict of publications, checking to see if there is a folder on the NFS share & create it if it's not. 
* Loop through the publication dict and create a container that is mapped to each publication's sub directory (nfs://nfs_srvc/crawler/nytimes/)

      - Set the container environment variables so the crawler knows where to deposit HTML & Images in S3. 
~Each publication now has a dedicated crawler~

The other issue that I'm trying to solve is how best to create tens of thousands of folders in S3 buckets. As files move through the data pipeline, they will be deposited in different buckets. In some cases the files will be deposited in to one folder and will be deleted as they are processed. In other cases they will remain in the bucket (archival) which will have a folder for each publication. 

Infrastructure build:
Create S3 buckets
* When a new publication crawler is created using python, the s3 bucket folders are created as well. 

My thoughts is I would have a python app that would do the following:
Run a new initial setup where I use Terraform & python to create a new copy of the platform & infrastructure (for enterprise customers). 
Add a new publication: 
     - *new publication yml is loaded in python
     - * create new Terraform files
     - * create folders on S3 & NFS

     - * generate and copy crawler configuration files to NFS

     - run terraform to add the new containers and set them to run on schedule (per publication). 

[Markus Juenemann]

I am certain that my Python package can help you in some way but you have to understand the overall workflow which consists of two stages:

a) Run the Python script to create the Terraform JSON configuration. It is NOT a Terraform replacement.

b) Run Terraform (validate, plan, apply) with the JSON configuration. Obviously in this step you don't run Python anymore so you are "limited" to the features Terraform provides.

Given that for step (a) you write Python code, anything that's possible with Python and its built-in and third-party libraries (e.g. Boto3, Querying databases, Scrapy, ..., anything) can be used to create the Terraform JSON configuration file. Python-terrascript is meant to be run on the management host to generate Terraform JSON configuration files.

I hope that helps a bit to understand the general workflow. So for example you wouldn't run Terraform and Boto3 together. It would be more like Terrascript+Boto3 to generate the JSON configuration file and then run Terraform.

Markus

[Dustin Williams]

I think I understand. I need to learn more about JSON configurations and how Terraform uses them. 

:) 

[Markus Juenemann]

In my view if you create Terraform configurations by hand you should really stick to HCL format (*.tf extension). In contrast, JSON format is best if you intend to create Terraform configurations through a program. Both formats provide they same functionality in regards to Terraform. 

[Ashutosh Kumar]

I am trying to use it but getting an error stating terrascript.aws does not exist. What am I missing here.

Thanks

Ashutosh

[Markus Juenemann]

Did you install it from PyPi, e.g. pip install terrascript or did you clone the Github repo? Installing from PyPi is the recommended way.

The Github repo does not contain the provider specific modules like terrascript.aws as these are actually auto-generated through the makecode.py script. You can run that script yourself but you would have to set the GITHUB_TOKEN environment variable to your Github token. See the makecode.py script for details. 

Let me know how you go. This is the first feedback I ever received :-)

Markus

[Ashutosh Kumar]

How can I create ECS task definition with container definition using Terrascript? 

Thanks

Ashutosh