running pytest and pyramid within a docker container, resolving distribution not found error and import mismatch errors

[dcs3spp]

Hi,

Has anyone had any experience with testing pyramid from within a docker container so that changes to source code are recognised within the container. I am experiencing issues with the container failing to recognise python package. 

My pytests and custom library, pyramid_core, are running within a docker container via docker-compose. The get_appsettings line in the code listing below raises a pkg_resources.DistributionNotFound exception.

from pyramid_core import load_environment

def test_loadenvironment(self):
    data_file = Path(THIS_DIR, 'testdata', 'development.ini')

    appsettings = get_appsettings(str(data_file), name="main")
    ...

An extract from the docker-compose file is included in the listing below. I am using volumes so that changes to src and tests on my host machine are recognised within the docker container:

version: "3.7"

networks:
  localnet:
    ipam:
      config:
        - subnet: 172.22.0.0/16

services:

  ...

  core-library:

    container_name:          service-core-library

    image:                   core-dev
    env_file:
      - ./config/.core.dev.env
    volumes:
      - ../src/pyramid_core:/server/src/pyramid_core
      - ../tests:/server/tests
    networks:
      - localnet

The Dockerfile installs dependencies via pip install -r requirements.txt, e.g.

...
pluggy==0.8.0
psycopg2-binary==2.7.6.1
py==1.7.0
pyramid==1.10.1
#-e git+https://oauth2:${TOKEN}@<git url>...@master#egg=pyramid_core

If I uncomment the last line of the requirements.txt file to install my custom library from VCS as an editable package I then receive the following error when running the pytests within the container....

py._path.local.LocalPath.ImportMismatchError: ('tests.conftest', '/server/tests/conftest.py', local('/server/src/pyramid-core/tests/conftest.py'))

Has anyone managed to successful run pyramid within a docker container so that changes to the source code on local development machine is reflected in the container?

Kind Regards

dcs3spp

[Lele Gaifax]

"'dcs3spp' via pylons-discuss"

<pylons-...@googlegroups.com> writes:

> Has anyone had any experience with testing pyramid from within a docker
> container so that changes to source code are recognised within the
> container.

Yes, I do that very often: basically I install all project modules in
edit-mode inside the image, and override their sources with explicit volumes
when I run the tests.

What works for me is more or less:

- my Dockerfile.tests does something like::

WORKDIR /srv

ENTRYPOINT ["python3", "-m", "pytest", "-p", "no:cacheprovider"]

COPY constraints.txt /tmp/constraints.txt
COPY app/requirements.txt /tmp/app-reqs.txt
COPY model/requirements.txt /tmp/model-reqs.txt
COPY tests/requirements.txt /tmp/tests-reqs.txt

RUN pip install --no-cache \
-c /tmp/constraints.txt \
-r /tmp/app-reqs.txt \
-r /tmp/model-reqs.txt \
-r /tmp/tests-reqs.txt \
&& rm -f /tmp/*-reqs.txt /tmp/constraints.txt

COPY app /usr/src/app
COPY model /usr/src/model

RUN pip install --no-cache \
-e /usr/src/app \
-e /usr/src/model

- my docker-compose.yml has a "tests" service like the following::

tests:
build:
context: .
dockerfile: Dockerfile.tests
volumes:
- "./app/tests:/srv/app"
- "./app:/usr/src/app"
- "./lib/model/tests:/srv/model"
- "./lib/model:/usr/src/model"

Hope this helps,
ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
le...@metapensiero.it | -- Fortunato Depero, 1929.

[dcs3spp]

Thanks Lele, much appreciated.

Yes, your response has helped a lot and confirms it is possible. I like the use of no:cacheprovider and will add that to my Dockerfile to save a lot of headaches, see below...

Based on your response it seems that I am almost there...

I have been installing my source library dependencies in edit mode by including a link to repositories in requirements.txt, e.g. -e git+https://<url>.....

When I do this, it causes the following error:

py._path.local.LocalPath.ImportMismatchError: ('tests.conftest', '/server/tests/conftest.py', local('/server/src/pyramid-core/tests/conftest.py'))

My package has the name pyramid_core. It looks as though pip is automatically converting the underscore character to a hyphen.

After some googling, this error can be caused by pycache files....

Will try and experiment with installing editable packages with pip install -e .....

Apologies for another question, but does your requirements.txt files contain the same dependencies as your setup.py files?

Again, much appreciated, this has been really helpful :)

Kind Regards

dcs3spp

[Lele Gaifax]

"'dcs3spp' via pylons-discuss"
<pylons-...@googlegroups.com> writes:

> Thanks Lele, much appreciated.

You're welcome!

> My package has the name pyramid_core. It looks as though pip is
> automatically converting the underscore character to a hyphen.
> After some googling, this error can be caused by pycache files....

Yes, that origins to a "normalization" step applied by pip and warehouse (see
https://www.python.org/dev/peps/pep-0503/#normalized-names).

> Apologies for another question, but does your requirements.txt files
> contain the same dependencies as your setup.py files?

It's a superset: I usually put only strict dependencies in my setup.py, then
in the requirements.txt I put all common ones, so that I can split the
installation in two steps, to better use the cache used by docker when
building the images.

[dcs3spp]

Cheers Lele

ok thanks for the tip re requirements.txt and setup.py dependencies.

At the moment I am battling with changing over to pip 19. Once sorted will then try the ideas out .... 

Again thanks or your help, much appreciated :)

Kind regards

dcs3spp

[dcs3spp]

Got it working :) Thanks again to Lele. Much appreciated :)