AWS Lambda Deployment Recipe

[appj...@gmail.com]

I've created a first pass at an AWS Lambda deployment recipe which uses an open-source tool called Zappa.

I welcome any comments or suggested improvements.  In particular, step 3 requires modifying gluon/main.py which I would like to avoid.

Deployment Recipe:

https://docs.google.com/document/d/1ff_JxiZPKy1Mdck8UQbUybBtP3bRK48U1Uw-iHgr4wc/edit?usp=sharing

[Anthony]

On Tuesday, November 27, 2018 at 6:51:54 PM UTC-5, appj...@gmail.com wrote:

I've created a first pass at an AWS Lambda deployment recipe which uses an open-source tool called Zappa.

I welcome any comments or suggested improvements.  In particular, step 3 requires modifying gluon/main.py which I would like to avoid.

I wouldn't say that is modifying gluon/main.py -- it is simply providing a wrapper around gluon.main.wsgibase. In fact, the web2py gaehandler.py handler does something very similar: https://github.com/web2py/web2py/blob/master/handlers/gaehandler.py#L83.

Note, according to PEP 3333, strings passed to the server should not be unicode: https://www.python.org/dev/peps/pep-3333/#unicode-issues.

Actually, it looks like Zappa is already handling this here: https://github.com/Miserlou/Zappa/blob/master/zappa/wsgi.py#L110 (see also, https://github.com/Miserlou/Zappa/blob/master/zappa/wsgi.py#L110). Perhaps there is still some unicode getting through from Zappa, so maybe you can open an issue with Zappa.

Anthony

[appj...@gmail.com]

Thank you Anthony, I added the wsgibase wrapper after my initial post.  I was modifying gluon/main.py but I added the wrapper (lambda_handler.py) to avoid that.

I confirmed which environ variables are coming through as unicode and submitted an issue to zappa.

https://github.com/Miserlou/Zappa/issues/1713

I'll also modify the recipe to make it more clear that this is likely a zappa and not a web2py issue.

[Massimo Di Pierro]

Good job! If you change that to a markdown or html page we can post on the web2py web site. I would also be happy to take the code from your example and turn it into a script that shops with web2py. Want to propose one?

[Jaime Sempere]

That's receipe is great, thanks for the work.

One thing that I have not seen is which database are you using? I think with Zappa you need to use RDS, at least that's what I have seen with Django apps.

So, I would like to ask you two questions:

1) Can you confirm about the database (is a RDS)?

2) After this time using Zappa+web2py for Lambda... is it a robust and mantainable environment? Have you confronted any important issues?

I am asking #2, because for me lambda + web2py would be a win-win, but I find it somekind of 'mixing too much environments/framework'... I smell danger xD So, I would like to know if I can use it as a 'robust/final' solution for my projects.

Thanks in advance

[App Jar]

Hi Jaime,

I now have 4 production web2py apps that have been up and running on Lambda for many months now with no issues. It is easily maintainable and I can update/deploy with a single command line: zappa update

As for the databases, the DB doesn't have to be AWS. You can connect to any database you like with the URL in the connection string in the DAL constructor. Out of my 4 apps:

2 of them use MySQL RDS on AWS

1 of them uses MySQL Aurora on AWS

1 connects to a non-AWS in-house database on a dedicated server at a non-AWS data center.

The only issue with lambda, and it is a minor one, is that each idle lambda instance will go to sleep after about 10 minutes and then take about 10 seconds to spin up again. So when your number of concurrent requests hits a new high water mark for the last 10 minutes or so, that last user will have to wait up to 10 seconds for his request to get served.  This is easily fixed by using https://www.lambdawarmer.com -- It's cheap and I use this to keep 8 instances warm.  Zappa out of the box only lets you keep 1 instance warm.

[Jaime Sempere]

Hi Mike,

I've been following your recipe, but I have an 'read-only file system issue' (I think is due that I did not set up a database and web2py is trying to create the datatables files)

So, can you give me some clues about it? Also, how should i set up and configure my rds to work with web2py? I have not noticed anything about it on the recipe.

Check my log error from Zappa:

[1568721920013] [ERROR] 2019-09-17T12:05:20.12Z 75fc21f1-e92e-4c80-ada1-a3e042112bf3 Traceback (most recent call last):

  File "/var/task/gluon/restricted.py", line 219, in restricted

  exec(ccode, environment)

  File "/var/task/applications/zerca_maps/models/db.py", line 179, in <module>

  Field('registered', 'boolean', default=False),

  File "/var/task/gluon/packages/dal/pydal/base.py", line 590, in define_table

  table = self.lazy_define_table(tablename, *fields, **kwargs)

  File "/var/task/gluon/packages/dal/pydal/base.py", line 624, in lazy_define_table

  polymodel=polymodel)

  File "/var/task/gluon/packages/dal/pydal/adapters/base.py", line 798, in create_table

  return self.migrator.create_table(*args, **kwargs)

  File "/var/task/gluon/packages/dal/pydal/migrator.py", line 318, in create_table

  fake_migrate=fake_migrate

  File "/var/task/gluon/packages/dal/pydal/migrator.py", line 463, in migrate_table

  self.save_dbt(table, sql_fields_current)

  File "/var/task/gluon/packages/dal/pydal/migrator.py", line 472, in save_dbt

  tfile = self.file_open(table._dbt, 'wb')

  File "/var/task/gluon/packages/dal/pydal/migrator.py", line 495, in file_open

  fileobj = portalocker.LockedFile(filename, mode)

  File "/var/task/gluon/packages/dal/pydal/contrib/portalocker.py", line 185, in __init__

  self.file = open_file(filename, mode.replace('w', 'a'))

  File "/var/task/gluon/packages/dal/pydal/contrib/portalocker.py", line 170, in open_file

  f = open(filename, mode)

IOError: [Errno 30] Read-only file system: '/var/task/applications/zerca_maps/databases/c8b669d15150d7109e5f7ab36744a5b7_retailers.table'

[App Jar]

Here is how I set up my db to work with RDS.

Note: res-combined-ca-bundle.pem is the public key for RDS found here: https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem.

driver_args = { 'ssl':{ 'ca': 'applications/ads/private/ssl/rds-combined-ca-bundle.pem'} }
db_auth = 'mysql://username:pass...@myinst-1.cluster-xxxx.us-east-1.rds.amazonaws.com:3306/mydbname'


if settings.aws_lambda:
 from gluon.dal import InDBMigrator
 adapter_args = dict(migrator=InDBMigrator)
 pool_size = 0
else:
 adapter_args = dict()
 pool_size = 8


db = DAL(db_auth,
         pool_size=pool_size,
         migrate_enabled=False,
         migrate = False,
         adapter_args=adapter_args,
         lazy_tables=True,
         driver_args = driver_args,
         fake_migrate=True )