Should I use workspaces to separate dev / test / prod configs and components?

[Kohen Chia]

The Workspaces page says:

When Terraform is used to manage larger systems, teams should use multiple separate Terraform configurations that correspond with suitable architectural boundaries within the system so that different components can be managed separately and, if appropriate, by distinct teams. Workspaces alone are not a suitable tool for system decomposition, because each subsystem should have its own separate configuration and backend, and will thus have its own distinct set of workspaces.

In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production) or different internal teams. In this case, the backend used for each deployment often belongs to that deployment, with different credentials and access controls. Named workspaces are not a suitable isolation mechanism for this scenario.

The Recommended Workflow page says:

The best approach is to use one workspace for each environment of a given infrastructure component. Or in other words, Terraform configurations * environments = workspaces.

[...]

Name your workspaces with both their component and their environment. For example, if you have a Terraform configuration for managing an internal billing app and another for your networking infrastructure, you could name the workspaces as follows:

- billing-app-dev

- billing-app-stage

- billing-app-prod

- networking-dev

- networking-stage

- networking-prod

So... which?

I'm probably missing the requisite experience and context to make sense of what appears to be conflicting advice, but that's why I'm here asking for help.

[Jd Daniel]

I've written Makefiles to build states programmatically, but I've also used workspaces and directory separation. I'm sure there's a "right way" to do it, but I just tend to base it around the project I'm on. Here's a Makefile that builds Vault for example (tf excluded)

TF_VAR_pub_key 			:= $(shell cat _keys/vault-key.pub)
ANSIBLE_ROLES_PATH 	:= ./ansible/roles
ANSIBLE_CONFIG 			:= ./ansible/ansible.cfg

export ANSIBLE_CONFIG ANSIBLE_ROLES_PATH
export TF_VAR_aws_profile TF_VAR_aws_account TF_VAR_pub_key
export TF_VAR_aws_prvnet  TF_VAR_aws_subnet
export TF_VARS_aws_ami


# An implicit guard target, used by other targets to ensure
# that environment variables are set before beginning tasks
.assert-%:
	@ if [ "${${*}}" = "" ] ; then 																						\
	    echo "Environment variable $* not set" ; 															\
	    exit 1 ; 																															\
	fi


# Modify our Terraform input vars on the fly, we bake these
# from several envvars and otfs, this is a reiteration of
# how i rewrite nomad templates on demand...
.convert-template-vars:
	@ sed 																								\
		-e "s/<AWS_ACCOUNT_NUMBER>/$(TF_VAR_aws_account)/g" \
    -e "s/<AWS_VPC_ID>/$(TF_VAR_aws_prvnet)/g" 					\
    -e "s/<AWS_AMI_ID>/$(TF_VARS_aws_ami)/g" 						\
	terraform.tmpl.tfvars >| terraform.tfvars



.require-packer: assert-TF_VAR_aws_prvnet assert-TF_VAR_aws_subnet
	@ echo "[info] VAR VPC:     $(TF_VAR_aws_prvnet)" ;
	@ echo "[info] VAR Subnet:  $(TF_VAR_aws_subnet)" ;
	packer --version &> /dev/null

.require-vault:
	aws-vault --version &> /dev/null

.require-ansible:
	ansible --version &> /dev/null

.require-tf: assert-TF_VAR_aws_profile require-vault
	@ echo "[info] VAR Profile:  $(TF_VAR_aws_profile)"
	terraform --version &> /dev/null
	terraform init

.require-jq:
	jq --version &> /dev/null



keypair:
	@ echo "[info] Building keypairs"
	yes y |ssh-keygen -q -N ''  -f _keys/vault-key >/dev/null

ansible-roles:
	@ echo "[info] Installing Galaxy rolers"
	ansible-galaxy install --force -r ansible/requirements.yml


vault: ansible-roles
	@ read -p "Enter AWS Profile Name: " profile ; 																																																							\
	prvnet=`aws --profile "$${profile}" --region us-west-2 ec2 describe-vpcs |jq -r '.[] | first | .VpcId'` ; 																									\
	subnet=`aws --profile "$${profile}" --region us-west-2 ec2 describe-subnets --filters "Name=vpc-id,Values=$${prvnet}" |jq -r '.[] | first | .SubnetId'` ; 	\
																																																																															\
	TF_VAR_aws_profile=$$profile TF_VAR_aws_prvnet=$$prvnet TF_VAR_aws_subnet=$$subnet make keypair && \
	TF_VAR_aws_profile=$$profile TF_VAR_aws_prvnet=$$prvnet TF_VAR_aws_subnet=$$subnet make build   && \
	TF_VAR_aws_profile=$$profile TF_VAR_aws_prvnet=$$prvnet TF_VAR_aws_subnet=$$subnet make plan    && \
	TF_VAR_aws_profile=$$profile TF_VAR_aws_prvnet=$$prvnet TF_VAR_aws_subnet=$$subnet make apply

build: .require-packer
	aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- \
	"/usr/local/bin/packer" "build" 															\
		"-var" "builder_subnet_id=$(TF_VAR_aws_subnet)" 						\
		"-var" "builder_vpc_id=$(TF_VAR_aws_prvnet)" 								\
	"packer/vault.json"



plan: .require-tf
	aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- "/usr/local/bin/terraform" "plan"

apply: .require-tf .require-ansible
	@ if [ -z "$TF_VAR_pub_key" ] ; then 																\
		echo "\$TF_VAR_pub_key is empty; run 'make keypair' first!"	; 		\
		exit 1 ; 																													\
	fi
	aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- "/usr/local/bin/terraform" "apply" "-auto-approve"



plan-destroy: .require-tf
	aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- "/usr/local/bin/terraform" "plan" "-destroy"

destroy: .require-tf
	aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- "/usr/local/bin/terraform" "destroy" "-auto-approve"

clean: destroy
	rm -rf _keys/*.ovpn _keys/ec2-key* .terraform terraform.*



reprovision: .require-tf .require-jq
	ansible-playbook 																										\
	 -i `terraform output -json |jq -r '. |map(.value) |join (",")'`, 	\
	 -v	ansible/openvpn.yml |tee _logs/reprovision.log

ssh: require-tf
	 @ read -p "Enter AWS Region Name: " region  ; 											\
	 ssh 																																\
	 -i _keys/ec2-key 																									\
	 -l ubuntu 																													\
	 `terraform output -json |jq -r --arg region "$$region" ".[$$region].value"`



debug-reprovision: .require-tf .require-jq
	echo >| _logs/debug-reprovision ;
	ANSIBLE_DEBUG=1 ansible-playbook 																		\
	 -i `terraform output -json |jq -r '.[].value' |tail -n1`, 					\
	 -vvvvv	ansible/openvpn.yml |tee _logs/debug-reprovision.log

[Jd Daniel]

Honestly the above is overkill for 100% of situations though, and not fun to maintain.

[Alexi Karuna]

This article helped me a lot thinking things through: https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstate-file-per-env/

I've recently picked up TF on a brownfield project, where we have 9 AWS regions and basically I've followed the structure outlined in this blog post and so far so good. I have a state file for each environment + aws region combination. 

It comes down to what the size of your infra/project is / might be. And how you want to separate things out and also keep things safe.

On Thu, May 2, 2019 at 3:16 PM Jd Daniel <dodo...@gmail.com> wrote:

Honestly the above is overkill for 100% of situations though, and not fun to maintain.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/2b897dd8-e720-4aac-b432-ab1872e669a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.