Serverless Applications w/ Ben Miner

[Nick Chouard]

Ben and I jumped on an impromptu Penny Chat to talk about building serverless applications. I had done a bit of reading on building serverless applications, and thought that it would be a great solution to a project I just started on at work. Ben has lots of experience putting together AWS services for serverless applications, and was able to help me work out the architecture for mine.

Here is a simplified version of what I'm trying to accomplish:

• I have a batch of jobs for various clients that require hitting some API endpoints with a payload. If I send too large of a payload with these requests, they will be incredibly slow, and it is better to break them up. Up to now, this was being done manually.
• I only want to have one job running at a time, so I have to wait for a job to finish before starting another.
• The only place I can see that jobs have completed is in the logs, which are kept in Papertrail. Papertrail can send notifications about matching logs to AWS SNS.

Ben helped me to figure out how to I could combine AWS services to accomplish this. Here is the basic architecture we arrived at:

• Send a POST request representing the job to an endpoint that stores the job in DynamoDB. You can set up INSERT listeners in Dynamo to send messages and kick off other Lambdas. If no job is currently running, this job will be processed, otherwise it will be queued to process.
• Another Lambda (kicked off from the INSERT listener) will chunk the job's payload if necessary and send it to be processed.
• When when there is a log in Papertrail, an SNS notification will be sent. This will kick off another Lambda to start a new job.

Here are some resources on the technologies we discussed.

• https://serverless.com/framework/docs/providers/aws/events/sqs/
• https://serverless.com/framework/docs/providers/aws/events/sns/
  
• https://serverless.com/framework/docs/providers/aws/events/streams/
  

I'm really excited to start on this project and work with serverless applications. They seem incredibly powerful. I plan to write about my process on dev.to which I just joined. You can follow along here: https://dev.to/nchouard

Thanks Ben!