Ferris3 viability and app-engine modules vs python modules

[Dewey Gaedcke]

A few questions:

Is Ferris3 being actively developed/maintained?  In other words, should I be using it?

I like the ease of setting things up, but with Parse closing down, I would have expected a lot more activity on this forum and it's been pretty silent.

I'm wondering if someone knows something that I don't?

Second:

In the docs, it says:

Endpoint services must be defined in app/[module]/[module]_service.py. All services within the file will be discovered and wired

Does the term "module" above refer to a Python module, or an app-engine module?

And how do I manually wire-up my own service classes if I deviate from this folder structure?

I'm asking because with app.yaml at the project root, it seems like "app" must be an app-engine module whereas the folders inside it appear to be Python modules.

I may be missing something but I want more than one app-engine module and I don't want it called "app".

Assuming I wanted multiple app-engine modules in my application (not all being endpoints), I would prefer a structure like this:

projectName/

   endpoint_module/

      endpoints.yaml

      endpoints_config.py

      endpoints_service.py

   task_module/

   webui_module/

   admin_module/

   models/      # all ndb models here

   rpc_messages/      # all rpc message types here

   queue.yaml

   index.yaml

   cron.yaml

Would that be a mistake?

[Jonathan Parrott]

Myself and Ben (from Cloud Sherpas) are working on getting Ferris back on a consistent release schedule and merging pull requests.

In terms of your GAE module and Python module confusion - let's disconnect the terms. For app engine modules - call them microservices. You should structure your project like this:

• project
• default_service
• app.yaml (module: default)
• backend_service
• app.yaml (module: backend)
• endpoints.yaml
• task_service
  
• app.yaml (module: task)
• queue.yaml
• index.yaml
• cron.yaml

Sharing python modules between services is tricky. You could try symlinks:

• project
• default_service
• shared/protos -> ../protos
• backend_service
• shared/protos -> ../protos
• protos
• __init__.py
• ...

Or you can make it where every microservice contains all the code for the entire project, but just uses a particular wsgi app. You can do this by moving the app.yamls for each service up into the project level:

• project
• default_service
• backend_service
• protos
• service-default.yaml -> module: default, script: default_service.app
• service-backend.yaml -> module: backend, script: backend_service.app

Or you can look into tooling (something like buildout or bazel) to build dependencies all together as an exportable package, although I'm not overly familiar with these tools.

[Dewey Gaedcke]

This is really helpful!!  Thank you.

Your recommended folder structure is what I'm after, but it deviates from the docs/tutorial  (ie  app/[module]/[module]_service.py )

How do I make Ferris connect my service endpoints in this new structure since it appears (from the docs) it won't be automatic?

And I'm assuming your "protos" directory is the same as my "messages" (the rpc msg models) directory?

Can you say more about "sharing Python modules between microservices" being difficult??

I'm assuming you mean it would be difficult to keep a common "models" or "messages" directories at the top level??   

.... Which implies that app-engine uploads ONLY the microservice directory and its related config info up to it's runtime container?

If that is true, then directories like "lib", "tests" would not be there either?

So if I'm tracking all that, it seems like the best option is to put the app.yaml files at the top level so every microservice is using the same ndb-models and messages -- your third suggestion

any links or reading material welcome......I'll do my own homework but have found it difficult finding sufficient docs on modules and what app-engine does with them....

[ey...@breezometer.com]

If you're going with the top-level yaml files option, you might want to try adding other modules to the skip_files section [1] for each of your modules.

For example:

In module_one.yaml:

skip_files:

- ^module_two$

- ^module_three$

In module_two.yaml:

skip_files:

- ^module_one$

- ^module_three$

etc.

Cheers,

Eyal

[1] https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Skipping_files

[Jonathan Parrott]

As far as manual wiring, ferris' discovery just gathers webapp2 routes / api services and you're responsible for constructing the actual WSGIApplication or APIServer. Instead of using the discovery module, just explicitly import and use your routes/services.

See the docs here: http://ferris-framework.appspot.com/docs3alpha/users_guide/discovery.html

And the source for discovery here: https://github.com/jonparrott/ferris3/blob/master/ferris3/discovery.py

[Dewey Gaedcke]

I can tell you guys are really trying to help me, and perhaps I'm trying to bite off too much at once, but I'm really confused about the issue with importing Python modules from a directory ABOVE the micro-service.....I don't think I understand what happens upon deployment to app-engine so I don't know how to think about organizing the code.....

I think if I understood the problem, I'd be able to understand your various proposed solutions....or the architectural risks I'd be taking with each distinct approach

Here's my goal:

I have 2 different class of endpoints ( mobile-app, admin) and 2 different class of handlers (webUI, task/batch processing)

And I want ALL FOUR of these different code-bases to share Python Modules for:   Models, OAuth, Utils, Tests, etc

And I want the 2 endpoint subsystems to share the same proto-message-models

Is the "top-level yaml files" option the right choice???

What will be the consequence if I don't tell each YAML file to "skip_files" on the other 3 microservice directories?

Not that I intend to ignore your advice, but reading about "skip_files" says that those skipped files are never even sent to app-engine.....again, I'm confused about deployment.....

How do independent modules land inside of app-engine.....is each pushed to it's own distinct container??

Thanks

Dewey

[Jonathan Parrott]

App engine uploads all files in the same directory as the module's .yaml file. So, if your .yaml files are in a subdirectory, you can't import from the directory above. Each module is uploaded individually, so yes, the files are "separate". However, you can make the modules all share a common codebase, but just run different parts of it.

If I were you, I would structure my project like this:

• common
• __init__.py
• models.py
• oauth2.py
• utils.py
• service_one
• __init__.py
• main.py
• service_two
• __init__.py
• main.py
• service_three
• service_one.yaml
• service_two.yaml
• service_three.yaml
• queue.yaml
• index.yaml

The service_{x}.yaml files would look like this:

module: service_one
runtime: python
threadsafe: 1

handlers:
- uri: .*
script: service_one.main.app

# This is optional. If you don't mind the other module's code being deployed with

# this module, then don't worry about it.

skip_files:
- ^module_two$
- ^module_three$

Now, from any module you can do `from common import utils` or whatever.

[Dewey Gaedcke]

That is exactly the clarity I was looking for!!

Thanks so much!!!

[Dewey Gaedcke]

I also found this from Google support:

Modules documentations may not be explicitly stated, but the folder 'Module1', 'Module2' as well as the default module actually run inside separate Python virtual environments on separate instances and need to be self contained. They cannot 'see' any directories above them which exist on the local filesystem, and 'default.py' can't see anything in each of the module directories. The whole folder tree isn't copied to each module instance.