Terraform directory structure: advice?

[Lars Sommer]

Hello,

   I am trying to create a directory structure for a project that has multiple layers with each layer composed of modules. Example:

• root_dir
• layer_1 (network)
• Code lives at this level:
• instantiate modules
• layer variables
• env
• dev
• corp
• lab
• layer_2 (data)
• Code lives at this level:
• instantiate modules
• layer variables
• env
• dev
• corp
• lab
• layer_3 (application)
• Code lives at this level:
• instantiate modules
• layer variables
• env
• dev
• corp
• lab

The idea here is that you can cd into the layer you want to deploy, say, the "network" layer and then do a terraform init/plan/apply, and be off to the races. I see that an init can pass -backend=true and -backend-config=(path to file) but it doesn't seem to allow the additional parameter of -var-file=(path to file) which seems to be required, as my 'terraform init' is failing on missing variables that I would've passed in.

So, how can I create a directory structure with respect to environmental differences using command line only? I could symlink all over the place but honestly I'd really like to avoid that.

Thank you!

[Brandon O'Connor]

I've done this very similarly but also using a combination of using workspaces (formerly `terraform env`) and symlinking. Having to maintain a quick and dirty bash script that iterates over services and creates the symlinks and establishes state seems unavoidable at this point BUT it's really not that hairy and mine haven't required a lot in the way of change once the architecture design is in place. I'm working to put together a reference repo of how this looks so others can comment on this approach. A word on workspaces first:

Workspaces

For how I've been designing terraform repos, all terraform code is invoked at the leaf directories within terraform/services/. Workspaces can/should be leveraged such that multiple regions and service environments are encoded in the workspaces. For example, the workspaces contained in terraform/services/application might be:

dev:us-west-2

qa:us-west-2

stage:us-west-2

stage:us-east-1

prod:us-west-2

prod:us-east-1

This allows CI/CD to select any environments that match a given branch name (e.g. prod matches both prod:us-west-2 and prod:us-east-1) and execute terraform init/verify/plan/apply against them.

Using maps within the environment specific variables files (e.g. stage.tf), you're able to create variables to encode differences that might be needed from region to region through lookups.

Repo structure

This is generally how the infrastructure repo for a project looks for me:

• repo_root/
• README.MD
• travis.yml # calls packer build steps, invokes terraform init/verify/plan/apply through code in the scripts dir
• .pre-commit.yaml # keeps code clean
• scripts/
• create_tf_symlinks_and_init.sh # based on the branch, cd into each terraform service, and run terraform init providing the s3 backend variables that terraform is unable to set as variables
• tf_apply_all.sh # call create_tf_symlinks, cd into each service, and run terraform verify/plan/apply against workspaces mapping to the current branch name. If prod, require that a <env>_<region>_plan.out exists
• packer_build.sh # takes care of any complexity in invoking packer that packer makes a difficult or impossible since json is limited (e.g. discovering tags to apply to the build artifacts dynamically)
• packer/
• [packer dir contains anything related to image/container build process. see hashicorp/best-practices repo on github for an example]
• terraform/
• scripts/ # directory containing any tf templates to be rendered or code needed for local_exec e.g. userdata
  
• variables/ # contains all variables that might be used across service directories
• variables.tf # common variables across all services, always symlinked
• prod_account.tf # production account variables e.g. s3 bucket name and account ID, conditionally symlinked
• dev_account.tf # dev AWS account variables
• backend.tf # a shared backend file
• environments/ # contains variables (often maps and strings) specific to an environment e.g. mapping of tf workspace names to a canonical environment short name OR a true/false value for scaling down to 0 on a nightly basis which you might want in dev but not in prod/stage
• dev.tf
• qa.tf
• stage.tf
• prod.tf
• services/
• network/
• service_variables.tf # contains variables specific to this service
• main.tf # contains resources and module blocks
• data.tf # contains all data declarations
• outputs.tf # contains all output declarations
• environment_variables.tf # -> ../../environments/dev.tf this is a symlink created by create_tf_symlinks_and_init.sh IF this were the dev environment
• global_variables.tf # -> ../../global_variables.tf this is a symlink created by create_tf_symlinks_and_init.sh
• account_variables.tf # -> ../../global_variables.tf this is a symlink created by create_tf_symlinks_and_init.sh
• backend.tf # -> ../../backend.tf this is a symlink created by create_tf_symlinks_and_init.sh
• data/
• all files present in the network service should also exist for data with a data specific main.tf, service_variables.tf, data.tf, and outputs.tf
• application/
• all files present in the network service should also exist for application with an application specific main.tf, service_variables.tf, data.tf, and outputs.tf

The terraform directory structure used to be much uglier before the addition of workspaces/environments making the symlinking is much less burdensome now.

I'll revisit this thread once I've created the reference repo which should hopefully clear up anything I glossed over above. Hopefully that answers your question from at least one angle. Any thoughts as to how this could be made simpler is welcome! :)

- Brandon

[Lars Sommer]

Whoa hey thanks for the awesome response Brandon!

Here's what I ended up going with:

Directory structure:

New additions:

• I added a run.sh script (attached) that handles all my terraform interaction and moves .terraform files around for storage and use, essentially filing away .terraform directories for next use. 
• The script creates a .terraform_wrapper directory which holds all this information

Using this model, I can ensure that my states remain isolated and I can maintain a clean working environment.

The run.sh file is stored in my root dir and symlinked into each layer that needs to execute terraform commands. 

[Brandon O'Connor]

Are you saying you're checking in your .terraform directories and moving them around to handle state? If so, you should step back to think a bit about how collaborating with other developers on the same project might look, the benefits of remote state sharing on an HA system, and state locking. With local files being central to state, those feel like unsolved problems. If you're using remote state as enforced through automation, there's no need to manage .terraform files at all. Maybe I'm misunderstanding, and if so, please clarify.

[Lars Sommer]

Ah, no. I see your concern however. We use remote state and remote state locking, so all that you end up with in terraform.tfstate is:

{

    "version": 3,

    "serial": 1,

    "lineage": "5894c841-fbaf-40f6-8df7-56f5a8320235",

    "backend": {

        "type": "s3",

        "config": {

            "bucket": "terraform-state-(identifier)",

            "dynamodb_table": "terraform-state-lock-(identifier)",

            "encrypt": "true",

            "key": "(region)/(env)/(project type)/(terraform layer)/terraform.tfstate",

            "profile": "(profilename)",

            "region": "(region)"

        },

        "hash": 16992147291369940518

    },

    "modules": [

        {

            "path": [

                "root"

            ],

            "outputs": {},

            "resources": {},

            "depends_on": []

        }

    ]

}

So in actuality I am just shuffling around files to tell Terraform where to reference the remote state.