Terraform and Serverless AWS ?

[Dave]

I come from a php developer background and have recently done some scripting with Terraform for EC2 and RDS stacks 

but I would like to use a serverless approach and I am having difficulties ...

I have a simple example to try and code, I have taken this example

because it is one that was given somewhere as a "real life" challenge.

I can write this in php easily enough - but I want to use a Serverless approach,

So I would like to use AWS API Gateway - Lambda - DynamoDB.

Even though in this little example we only store 4 items, as a principal, I'd like to use DynamoDB.

• create a REST API which displays ISO codes for a given country, for example:
  • curl --header "Accept: application/json" http://127.0.0.1:5000/api/v1/countries?name=Spain - this would return “ES”
  • curl --header "Accept: application/json" http://127.0.0.1:5000/api/v1/countries?name=UK  - this would return “GB”
  • Repeat for four countries.
    • UK returns GB
    • Spain returns ES
    • Ireland returns IE
    • France returns FR
• Once the application has been written, package this up into a re-deployable app. This could be a container and run on ECS, or it could just run on an EC2 instance and installed via user data.
• Use Terraform to deploy the infrastructure for which this application relies on, also bootstrap the application onto the resource. Doesn’t matter if it’s ECS or EC2 or even Lambda. What I’m looking for is a way to check out your code and run terraform plan/terraform apply. This will deploy everything I require to have a functional application.
• I should be able to test the application is up and running via curl as shown in the example above. Ideally using the private/public IP address of the instance/container service.

Can you help me out with this, or recommend someone who can ?

Many, many thanks,

David Bird

London, UK

[Stephen Eilert]

Do it manually first, then look into Terraform. Terraform will let you have a repeatable deployment, but first you need to understand what it is that you want it to do.

Thanks,

– Stephen

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/788b677e-41a1-4882-b592-be7cfe016dbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brandon O'Connor]

If you haven't come across this doc yet, there's a guide. There's also a couple suitable modules in the registry, but still, it's not the route I'd go...

I'm of the opinion that terraform isn't the best tool for lambda function management today. I think the serverless framework is a better solution overall in this problem space despite being cloudformation under the hood, and from what I know, unable to produce a state that terraform could work with (perhaps through ingesting the serverless cfn template ala this data source). The serverless framework provides a simple CLI to do the things you'd need to do in serverless development: local invocation, wrapping a set of deploy steps into a single command, invocation through a single command, tidy configuration of integrated events, IAM handling, a log checking utility, and several useful plugins to extend functionality.

On this topic, if anyone has integrated an Apex (terraform under the hood) or serverless applications into live terraform states, I've been hoping to find examples of this.

- Brandon

[Stephen Eilert]

This is ironic, because I’m at this very moment trying to convince Serverless to deploy an API across multiple AWS accounts, each of which has different DynamoDB permissions. The solution I found uses variables (and environment variables) in serverless.yml, so I am now writing Jenkinsfiles to do this so that other people can use it (and installing Serverless dependencies and the like). Feels like I’m repairing my car to change a lightbulb.

This flexibility you are talking about is great during development and I benefited greatly. It is so convenient. But now things are becoming serious and so we need to build tools to manage all those functions across Dev and Production enviroments. I certainly do not want anyone (including myself) running SLS from a random laptop and would rather have a CI/CD pipeline. This changes the equation a bit, but I haven’t yet bit the bullet to scrap Serverless and do it on Terraform.

I did not know that Apex was Terraform under the hood, that’s useful information. 

Thanks,

– Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/ac5cb372-cb34-4188-b7dc-a3fce2ca35b7%40googlegroups.com.

[Clint Shryock]

Hello –

I'm curious if anyone could expand on this:

>Terraform isn't the best tool for lambda function management today

Other than local invocation / development. I'm curious if Terraform is missing something(s), or not handling something ideally, and if they could be added into Terraform while staying true to it's nature (declarative, iterative provisioning, et. al).

It's possible not of course, and you should use the best tool for the job, but I'd love to hear if there's something that could be added here!

Thanks :D 

Clint

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/ac5cb372-cb34-4188-b7dc-a3fce2ca35b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/edcd5b0d-bb95-4ec4-8a7a-bae727780f3c%40Spark.

[Brandon O'Connor]

Hey Clint -

Sure! Happy to elaborate.

Local invocation is very important and I'm glad you highlighted as much. To touch quickly on something Stephen said:

"...flexibility you are talking about is great during development..."

I may be misinterpreting but this to me is making a slight mistake in thinking there's a distinct development phase that's different from production operation and ongoing dev maintenance and feature development. There are differences between non-prod and prod-ops but the development toolchain and frameworks really need to be able to sustain through a project from design to sunset. I think it's healthy if we all think of development as a core part of basically all steps in application lifecycle.

What else to consider:

1. workflow is captured in one tool - sls gives the developer a one stop shop for serverless development. Check logs, deploy to a certain environment, roll back a deployment, check function stats, etc. I think the 3 step dance of packaging an application and its dependencies, uploading that to S3, and only then deploying to lambda for a certain stage/alias, makes it difficult to fit this in the terraform mold of a single step. I can imagine potentially a CI/CD pipeline operating on the sls application repo having a final step of S3 upload and have terraform handle the rest... this is not something I've implemented but would likely be worth exploring.

2. configuration captured in one place - with any of the serverless frameworks, your project (which can be a set of lambdas or a singleton) is by convention contained in a single repo. A serverless.yml (while likely split by environment, section, etc) captures all permissions, extra resources, and events. Most developers will find this more comfortable than a separation of potentially 3 or 4 repos that make up a well architected lambda app in terraform (a repo for each of the lambda application, a lambda module, potentially a lambda event module, and a terraform live repo to instantiate). IAM of course needs to be squeezed within one of those modules too.

3. out of box experience - probably the biggest reason to chose a framework is getting up and running with serverless development in minutes. With terraform, that's definitely not the case. Following the guide I linked to in my last post, while fine for someone with terraform experience, would not be such a simple task for someone uninitiated in this space.

4. The plugin ecosystem is not always high quality but it exists and covers tons of uses- If you want to extend how sls works, search github for a plugin or drop into node and hack away. My milage has varied with some of these plugins (hello orphaned resources), but others are solid (local mock dynamo, ).

To me it's obvious why sls (and friends) does a better job in this space: it's the only thing sls is designed to do whereas terraform handles the world (quite well!). I think as a developer using heroku, app engine, or elastic beanstalk you'd probably find a similar experience when comparing to terraform. I've shepherded an EB app to prod using terraform at ${last_job}. Though we did it, but it never really felt like a clean fit.

I want to stress that I'm not a serverless framework (zappa, apex, SAM, sls) fanboy. Quite the opposite: I'd MUCH rather be using terraform for everything and have often considered cobbling together the modules to bridge as much of the above as possible. I just don't think tf has the right ingredients today to be the best for working with FaaS. A subset of the problems above, terraform doesn't have an answer for because it wasn't built with AWS lambda at the center of its universe. I'd love to be shown otherwise!

Stephen - To your problem, at a previous gig, I built CI/CD around the sls tool and give limited IAM policies (not for the faint of heart) to the core developers such that they could run sls for local feature branches deploying to live (dev) functions but needed to invoke it through the normal git workflow to hit dev/test/stage/prod via the pipeline. This was adequate for us as the team owned nothing else that was terraform defined so it honestly didn't matter that we had a different tool under the hood. Obviously, every project has its own requirements and I doubt this will be a perfect fit for you but hopefully it's at least a jumping off point.

- Brandon

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/ac5cb372-cb34-4188-b7dc-a3fce2ca35b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

[Stephen Eilert]

I may be misinterpreting but this to me is making a slight mistake in thinking there's a distinct development phase that's different from production operation and ongoing dev maintenance and feature development. 

I’ll try to be brief not to derail the thread. There are different phases. One is, someone has just started a new feature or a bugfix. Usually this is done on someone’s laptop(this model breaks down for lambda functions, so make it a AWS dev account with proper IAM permissions). In any case, It will take several iterations until it is deemed functional, at which time it will be sent to the CI/CD pipeline (after a pull request or similar). Then the process will be the same to deploy in every enviroment (QA/Stage/Production, etc).

What I tried to say about Serverless is that I have found it great for experimentation(before the code is fully tested, as described above). Want a new function? Add a small snippet to the config, code the function, sls deploy and off you go. Same for setting up supporting infra like DynamoDB tables, or API Gateway, by adding a couple of lines. On the other hand, if one wanted to do the same on Terraform, there would be lots more to write before even getting a function deployed, friction is way higher.

What you would gain is: Terraform has the excellent ‘plan’ phase, which tells you if it is about to give someone a bad day(like dropping a DynamoDB table). I have seen no equivalent for Serverless. Terraform is also great at cleaning up after itself.

They really work at different abstraction levels. On Terraform, you have to specify every last detail of whatever it is you want deployed. Serverless lets you not care about it, until you do. For instance, the Terraform guide for lambda and api gateway: https://www.terraform.io/docs/providers/aws/guides/serverless-with-aws-lambda-and-api-gateway.html 

That is several pages long. 

By comparison: https://serverless.com/framework/docs/providers/aws/events/apigateway/

This is what you do:

functions: index: handler: handler.hello events: - http: GET hello

Serverless will create and deploy the lambda function and setup API Gateway for you (provided you have paid the one-time cost of setting up proper IAM permissions).

Not sure what Terraform itself could do better in this scenario. I feel like it is the LLVM of the infrastructure world, and we need something on top of it :)

Thanks,

– Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/36ee6300-47ce-42ac-835c-0c4e59cc4fd2%40googlegroups.com.

[Brandon O'Connor]

Thanks for following up, and whatever you come up with Stephen, I can't be the only one curious to hear about it.

I don't want to follow this tangent too much further but as it happens, an sls plugin exists to create sls cloudformation changesets and diff with the live cfn state.

- Brandon