Distributing a REST API Pyramid project

[Jimmy Thrasibule]

Hi,

I'm working on a Pyramid project composed of different tiers:

- Pyramid REST API
- Asynchronous Celery tasks

Since my REST API and asynchronous tasks targets the same database, I
set my data models in their own package. My project is therefore
structured this way:


/
├─ myproj/
│ ├─ common/
│ │ ├─ utils/
│ │ ├─ vendor/
│ │ └─ __init__.py
│ ├─ data/
│ │ ├─ models/
│ │ ├─ queries/
│ │ ├─ schemas/
│ │ └─ __init__.py
│ ├─ server/
│ │ ├─ views/
│ │ ├─ ...
│ │ └─ __init__.py
│ ├─ tasks/
│ │ ├─ common/
│ │ │ ├─ __init__.py
│ │ │ └─ celery.py
│ │ ├─ task1/
│ │ │ ├─ ...
│ │ │ └─ __init__.py
│ │ ├─ task2/
│ │ │ ├─ ...
│ │ │ └─ __init__.py
│ │ └─ __init__.py
│ └─ __init__.py
└─ setup.py


``myproj`` and ``myproj.tasks`` are two `namespace packages
<https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages>`_
with the ``__init__.py`` file only containing the following:


__path__ = __import__('pkgutil').extend_path(__path__, __name__)


This way, I can build a clear dependency graph for example:

* ``myproj.server`` depends on ``myproj.common`` and ``myproj.data``.
* Any tasks will depend on ``myproj.tasks.common`` and optionally on
``myproj.common`` or ``myproj.data``.

Now come the hard part. Our team is too small to get the project
divided in multiple sub-projects and I would also like to keep
distribution as simple as possible meaning keeping only one
``setup.py`` so that with one ``./setup.py bdist``, I can create
**all** the packages I need to distribute the project.

And indeed I'd like to have multiple packages for this one project so
I can install only the required dependencies for each subsystem. In
the end, I'd like to get a distribution package for the following:

* myproj.common
* myproj.data
* myproj.server
* myproj.tasks.common
* myproj.tasks.task1
* myproj.tasks.task2

Any idea how I can get this far keeping the overall project management
as simple as possible?

Regards,
Jimmy

[Tres Seaver]

On 09/29/2017 03:34 AM, Jimmy Thrasibule wrote:
> Now come the hard part. Our team is too small to get the project
> divided in multiple sub-projects and I would also like to keep
> distribution as simple as possible meaning keeping only one
> ``setup.py`` so that with one ``./setup.py bdist``, I can create
> **all** the packages I need to distribute the project.
>
> And indeed I'd like to have multiple packages for this one project so
> I can install only the required dependencies for each subsystem. In
> the end, I'd like to get a distribution package for the following:
>
> * myproj.common
> * myproj.data
> * myproj.server
> * myproj.tasks.common
> * myproj.tasks.task1
> * myproj.tasks.task2
>
> Any idea how I can get this far keeping the overall project management
> as simple as possible?

The simplest solution: assuming this is a purely "bespoke" project (only
your team runs the app), I would just ditch the "install only required
dependencies" bit: the size of the software load should be minimal on a
modern system, and keeping the hosts / VMS interchangeable makes for a more
flexible deployment.

If you really cannot do that, then 'setup.py' can contain multiple
invocations of the 'setuptools.setup()' function, once for each separate
sdist / wheel you want to build ('bdist' support is already deprecated in
modern setuptools, and pip has never been able to install those artifacts).


Tres.
--
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

[Jonathan Vanasco]

originally we did your approach, but now we use multiple discrete packages, and manage the virtualenv with fabric

|python_packages/
|     |----app_pyramid/

|     |          |---setup.py

|     |          |---app_pyramid/

|     |----app_core/

|     |          |---setup.py

|     |          |---app_core/

|     |----app_celery/

|     |          |---setup.py

|     |          |---app_celery/

|     |----app_model/

|     |          |---setup.py

|     |          |---app_model/

|fabfile.py

the fabric file is used to manage everything and proxy commands to setup.py/pip/etc. it can build packages as needed, install from source, or even deploy into select environments.