Decouple All Persistent Filesystem State From Application

[nmacneil]

Hi all,


I'm looking for a mechanism to decouple all persistent filesystem state from my py4web application, similar to what is mentioned for Avoiding the filesystem in web2py.

Context:
    -    I'm running py4web in a docker container on render.com but when you incorporate
        persistent filesystem state it prevents horizontal auto-scaling of the application.

Issues:

1.     Persistent databases folder is required if migrate = True.
   • If you use fake_migrate_all=True instead then how do you manage table structure changes/migrations in a automated approach?
     
     
   • Not sure if this is natively possible, but is there a mechanism to connect the DAL to a centralized nfs server to store all persistent state external to the application but still on a filesystem?
     
     
2. How would I setup the "Uploads" folder to upload all attachments into the database by default rather than field by field with the "uploadfield" as is done for GAE?
   
   
3. Any other persistent folders I may have missed and how these could be decoupled from the application?

   
   
Thank you,
Nathan

[Luca de Alfaro]

My reply is imperfect, but while we wait for better replies: 

I put BOTH fake_migrate and migrate to False to avoid having files there. 

For the upload, I have descriptions of how to upload on Google Cloud Storage here: https://learn-py4web.github.io/unit19.html .  You could modify that to upload to Amazon S3.  I think in general uploading to the db is a bad idea for scalable apps, because it's better to put big objects in a blobstore than in a db, but it could be ok for a smaller app. 

You can find here instructions for uploading a project to Google Appengine, which include removing the state: https://learn-py4web.github.io/unit20.html

I do not know how to change the behavior of the 'upload' field to upload to a database, but I think it would be a better default than uploading to the file system. 

More generally, I am a big fan of de-stating web servers, as all their state should be in db + blobstore + etc.  I would be very interested in documenting this better in a Unit in that series of units, and changing (optionally maybe) the behavior of upload to upload to db (or perhaps have a new type called 'dbupload' ?).  

Only problem is, I cannot directly work on it before September, due to other work I need to finish first, but I am certainly interested, so if you like to collaborate / think at it / ... it would be great. 

Luca

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/0174e4a9-96ed-4653-9799-27fd79582f2bn%40googlegroups.com.

[nmacneil]

Hi Luca, 

Thank you for the information and reference material, it looks very helpful. I'll probably work through what you've set up in the two repos as a start, I like your idea of storing the files in a blobstore. 

As you're not migrating, how do you manage your table structure changes, i.e. creating and updating tables, as well as generating the script(s) based on your py4web models?

Thank you,
Nathan

[nmacneil]

After digging through the DAL, this functionality seems to work out the box if you're using one of these three database types  ("mysql", "postgres", "sqlite",)  and if you import the InDBMigrator and update your DAL declaration to pass the adapter_args below. There was a couple issues with getting it running initially for PostgreSQL but I've corrected the issues and submitted a pydal issue/pull request (web2py_filesystem Updates/Corrections #696). This allows decoupling of database structure metadata as well as auto-uploads all "upload" field types into the database rather than the filesystem.

# common.py

from pydal.migrator import InDBMigrator

# Added adapter_args to store blobs and database table meta data in the database directly.

db = DAL(settings.DB_URI,    

                  pool_size=settings.DB_POOL_SIZE,

                  migrate=settings.DB_MIGRATE,

                 fake_migrate=settings.DB_FAKE_MIGRATE,

                 adapter_args=dict(migrator = InDBMigrator,

                                                   uploads_in_blob = True),    

)

[Massimo]

Thank you. This was just merged and will deploy a new version later today.

I am thinking about making this simpler db=DAL(...., in_db_migrations=True)

Thoughts? any suggestions for a better parameter name?

[nmacneil]

I've setup an example py4web stateless docker application that can be cloned and used as required. 

It's contains (full information contained in the repo README):

1. Table structures and uploads stored directly in the database,
2. New database upload functionality to ensure database uploads do not impact general performance for the primary table containing the upload field.
3. Ready to run py4web docker application and corresponding database in just a few steps.
   • The docker container is also pre-setup/ready for debugging.

Hopefully this helps others avoid the headaches I had working through setting up a stateless docker application https://github.com/macneiln/docker-py4web-stateless-example.

- Nathan