Pytests fixtures AttributeError: 'ProfileManager' object has no attribute 'root_dir'

[Jonathan Chico]

Dear all

I have been building some CI/CD test for some local workflows and I have been running into some issues when trying to use the pytest fixtures. I was wondering if you perhaps could point out what is the issue.

I have setup my conftest.py to include both the aiida and aiida-vasp fixtures 

pytest_plugins = ('aiida.manage.tests.pytest_fixtures', 'aiida_vasp.utils.fixtures')

When I submit the tests I follow the procedure I see in the aiida-core source code, i.e. configure a test profile and submit the test in that profile.

- source aiida_env/bin/activate

- reentry scan

- sed -i "s#PROFILE_PLACEHOLDER#test_relax#g" tests/config/test_config.yml

- verdi setup --config tests/config/test_config.yml

- AIIDA_TEST_PROFILE=test_relax LC_COLLATE='en_US.UTF-8' LC_CTYPE='en_US.UTF-8' pytest -q tests/relaxation/test_relaxation_vasp.py

My test makes use of the aiida-vasp fixture fresh_aiida_env which should yield the aiida_profile instance which is in turn used to create a mockup of the VASP code so that a test of the workflow can be performed. However, when the fixture is called, I run into the following error

==================================== ERRORS ====================================
_______________________ ERROR at setup of test_relax_wc ________________________
aiida_profile = <aiida.manage.tests.TestManager object at 0x7f7495476af0>
@pytest.fixture()
def fresh_aiida_env(aiida_profile):
"""Reset the database before and after the test function."""
> print('The root directory of the fixture manager is: {}'.format(aiida_profile._manager.root_dir)) # pylint: disable=protected-access
E AttributeError: 'ProfileManager' object has no attribute 'root_dir'

It seems to be that instead of using the `TestProfileManager` (which does have root_dir as a property) it is calling the `ProfileManager` which does not.

However, the tests in aiida-vasp which uses this same fixture, seem to work (in Github, in running in Gitlab) so I'm wondering what could be the issue behind this.

Thanks!

Cheers

[Dominik Gresch (Experis AG)]

Hi Jonathan,

 

This is a compatibility issue between pytest versions – you’ll need version 6.0 or newer.

 

In aiida-core, we keep track of the testing dependencies through the “tests” extra requirement: https://github.com/aiidateam/aiida-core/blob/d35a9d7905d8d94c35b6e6eebaae2e5440fec975/setup.json#L104

However, that also includes dependencies that are specific to the aiida-core tests themselves, and not required to use the fixtures that aiida-core provides. Maybe that is something to be improved.

 

Best,

Dominik

 

--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/612aaddf-99f3-47a8-9664-bbdf64ca8f92n%40googlegroups.com.

[Jonathan Chico]

Hi Dominik

Thanks for the suggestion. I have made sure to update pytest to version 6.2.2, but the same error keeps showing up, I put here the error message as well as the results from `pip list` that I get when running inside the gitlab container.

$ pip list

Package Version

--------------------- ---------

aiida-core 1.6.0

aiida-pseudo 0.5.0

aiida-quantumespresso 3.4.0

aiida-vasp 2.1.0

aio-pika 6.8.0

aiormq 3.3.1

aldjemy 0.9.1

alembic 1.5.8

appdirs 1.4.4

archive-path 0.2.1

ase 3.21.1

asn1crypto 1.4.0

astroid 2.5

async-generator 1.10

attrs 20.3.0

backcall 0.2.0

bcrypt 3.2.0

certifi 2020.12.5

cffi 1.14.5

cfgv 3.2.0

chardet 4.0.0

circus 0.17.1

click 7.1.2

click-completion 0.5.2

click-config-file 0.6.0

click-spinner 0.1.10

configobj 5.0.6

coverage 5.5

cryptography 3.4.7

cycler 0.10.0

decorator 4.4.2

deprecation 2.1.0

distlib 0.3.1

Django 2.2.19

elementpath 2.2.1

ete3 3.1.2

filelock 3.0.12

future 0.18.2

graphviz 0.16

icet 1.3

identify 2.2.2

idna 2.10

iniconfig 1.1.1

ipython 7.22.0

ipython-genutils 0.2.0

isort 4.3.21

jedi 0.18.0

Jinja2 2.11.3

joblib 1.0.1

jsonschema 3.2.0

kiwipy 0.7.4

kiwisolver 1.3.1

lazy-object-proxy 1.6.0

lxml 4.6.3

Mako 1.1.4

MarkupSafe 1.1.1

matplotlib 3.4.0

mccabe 0.6.1

monty 2021.3.3

mpmath 1.2.1

multidict 5.1.0

mypy 0.790

mypy-extensions 0.4.3

nest-asyncio 1.4.3

networkx 2.5

nodeenv 1.5.0

numpy 1.20.2

packaging 20.3

palettable 3.3.0

pamqp 2.3.0

pandas 1.2.3

paramiko 2.7.2

parsevasp 2.0.1

parso 0.8.1

pexpect 4.8.0

pg8000 1.19.0

pgsu 0.2.0

pgtest 1.3.2

pickleshare 0.7.5

Pillow 8.1.2

pip 21.0.1

plotly 4.14.3

pluggy 0.13.1

plumpy 0.19.0

pre-commit 2.11.1

prompt-toolkit 3.0.18

psutil 5.8.0

psycopg2-binary 2.8.6

ptyprocess 0.7.0

py 1.10.0

py-cpuinfo 7.0.0

pybind11 2.6.2

PyCifRW 4.4.2

pycparser 2.20

Pygments 2.8.1

pylint 2.5.3

pylint-django 2.3.0

pylint-plugin-utils 0.6

pymatgen 2020.12.3

Pympler 0.9

PyMySQL 0.9.3

PyNaCl 1.4.0

pyparsing 2.4.7

pyrsistent 0.17.3

pytest 6.2.2

pytest-asyncio 0.14.0

pytest-benchmark 3.2.3

pytest-cov 2.11.1

pytest-rerunfailures 9.1.1

pytest-timeout 1.4.2

python-dateutil 2.8.1

python-editor 1.0.4

pytray 0.3.1

pytz 2019.3

PyYAML 5.1.2

pyzmq 22.0.3

qe-tools 2.0.0rc2

reentry 1.3.2

requests 2.25.1

retrying 1.3.3

ruamel.yaml 0.17.2

ruamel.yaml.clib 0.2.2

scikit-learn 0.24.1

scipy 1.6.2

scramp 1.3.0

seekpath 1.9.7

setuptools 54.1.2

shellingham 1.4.0

shortuuid 1.0.1

simplejson 3.17.2

six 1.15.0

spglib 1.16.1

SQLAlchemy 1.3.23

SQLAlchemy-Utils 0.36.8

sqlparse 0.4.1

subprocess32 3.5.4

sympy 1.7.1

tabulate 0.8.9

threadpoolctl 2.1.0

toml 0.10.2

tomlkit 0.7.0

tornado 6.1

tqdm 4.59.0

traitlets 5.0.5

typed-ast 1.4.2

typing-extensions 3.7.4.3

tzlocal 2.1

uncertainties 3.1.5

upf-to-json 0.9.3

urllib3 1.26.4

virtualenv 20.4.3

voluptuous 0.12.1

wcwidth 0.2.5

wheel 0.36.2

wrapt 1.11.2

xmlschema 1.5.3

yapf 0.30.0

yarl 1.6.3

$ AIIDA_TEST_PROFILE=test_relax LC_COLLATE='en_US.UTF-8' LC_CTYPE='en_US.UTF-8' pytest -q tests/relaxation/test_relaxation_vasp.py

E [100%]

==================================== ERRORS ====================================

_______________________ ERROR at setup of test_relax_wc ________________________

aiida_profile = <aiida.manage.tests.TestManager object at 0x7fc656e84040>

@pytest.fixture()

def fresh_aiida_env(aiida_profile):

"""Reset the database before and after the test function."""

> print('The root directory of the fixture manager is: {}'.format(aiida_profile._manager.root_dir)) # pylint: disable=protected-access

E AttributeError: 'ProfileManager' object has no attribute 'root_dir'

aiida_env/lib/python3.8/site-packages/aiida_vasp/utils/fixtures/environment.py:19: AttributeError

=============================== warnings summary ===============================

aiida_env/lib/python3.8/site-packages/past/builtins/misc.py:45

/builds/cdpm/aiida-coro/aiida_env/lib/python3.8/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

from imp import reload

aiida_env/lib/python3.8/site-packages/monty/serialization.py:81: 17 warnings

/builds/cdpm/aiida-coro/aiida_env/lib/python3.8/site-packages/monty/serialization.py:81: PendingDeprecationWarning:

load will be removed, use

yaml=YAML(typ='unsafe', pure=True)

yaml.load(...)

instead

return yaml.load(fp, *args, **kwargs)




As you see, I guess due to the change in version of some packages, some new warnings show up, but everything else is basically the same.





Cheers


To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DBAPR83MB03908860744DF2C6BD0AAD93857D9%40DBAPR83MB0390.EURPRD83.prod.outlook.com.

--

Jonathan Chico

[Dominik Gresch (Experis AG)]

Hi Jonathan,

 

I had confused your error message with a very similar one -- indeed it seems the pytest version is not involved.

 

From my testing, it seems this error occurs only when an AIIDA_TEST_PROFILE is set, not when using a temporary test profile. That may explain why the aiida-vasp CI is working.

 

In principle the aiida_profile._manager is an internal variable, so we shouldn’t rely on its behavior. I’d still suggest opening an issue with aiida-core to discuss if the missing root_dir is intentional or a bug, though.

 

Maybe the fresh_aiida_env fixture should also be updated to either not rely on the _manager, or fail gracefully when root_dir is not present.

 

Best,

Dominik

 

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CAO8p0kUAacxPmvEAFp%3D1Xv2hHBdJbN_remv9PMbcZKwTBYjo4Q%40mail.gmail.com.

[Jonathan Chico]

Hi Dominik

Thanks for your reply! I thought it could be that, I have tried running the test without defining the AIIDA_TEST_PROFILE but it sadly fails with another error

==================================== ERRORS ====================================

_______________________ ERROR at setup of test_relax_wc ________________________

@pytest.fixture(scope='session', autouse=True)

def aiida_profile():

"""Set up AiiDA test profile for the duration of the tests.

Note: scope='session' limits this fixture to run once per session. Thanks to ``autouse=True``, you don't actually

need to depend on it explicitly - it will activate as soon as you import it in your ``conftest.py``.

"""

> with test_manager(backend=get_test_backend_name(), profile_name=get_test_profile_name()) as manager:

aiida_env/lib/python3.8/site-packages/aiida/manage/tests/pytest_fixtures.py:45:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

/usr/local/lib/python3.8/contextlib.py:113: in __enter__

return next(self.gen)

aiida_env/lib/python3.8/site-packages/aiida/manage/tests/__init__.py:460: in test_manager

_GLOBAL_TEST_MANAGER.use_temporary_profile(backend=backend, pgtest=pgtest)

aiida_env/lib/python3.8/site-packages/aiida/manage/tests/__init__.py:92: in use_temporary_profile

mngr.create_profile()

aiida_env/lib/python3.8/site-packages/aiida/manage/tests/__init__.py:330: in create_profile

self.create_aiida_db()

aiida_env/lib/python3.8/site-packages/aiida/manage/tests/__init__.py:311: in create_aiida_db

self.create_db_cluster()

aiida_env/lib/python3.8/site-packages/aiida/manage/tests/__init__.py:301: in create_db_cluster

self.pg_cluster = PGTest(**self._pgtest)

aiida_env/lib/python3.8/site-packages/pgtest/pgtest.py:341: in __init__

self._init_base_dir()

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = PGTest(database='postgres', username='postgres', port=41707, log_file='/tmp/tmpg4o0x_ct/pgtest_log.txt', no_cleanup=False, copy_cluster=None, cluster='/tmp/tmpg4o0x_ct', pg_ctl='/usr/lib/postgresql/11/bin/pg_ctl')

def _init_base_dir(self):

"""Initiates the base directory and creates a cluster, either brand new

or by copying the cluster defined by the user

"""

try:

if self._copy_cluster:

shutil.rmtree(self._cluster)

shutil.copytree(self._copy_cluster, self._cluster)

else:

cmd = ('"{pg_ctl}" initdb -D "{cluster}" -o "-U {username} -A '

'trust"').format(pg_ctl=self._pg_ctl_exe,

cluster=self._cluster,

username=self._username)

proc = subprocess.Popen(cmd, shell=True,

stdout=subprocess.PIPE,

stderr=subprocess.PIPE)

_, err = proc.communicate()

if err:

> raise IOError(err)

E OSError: b'pg_ctl: cannot be run as root\nPlease log in (using, e.g., "su") as the (unprivileged) user that will\nown the server process.\n'

aiida_env/lib/python3.8/site-packages/pgtest/pgtest.py:513: OSError




This seems to be an error related to how one sets up postgres in Gitlab, it seems to be that due to the differences between Gitlab and Github, at least as far as I can tell.





I'll make the bug report in aiida-core and perhaps I'll link it in aiida-vasp.




Thanks




Cheers





To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DBAPR83MB039073200AC68E2887A047FD857D9%40DBAPR83MB0390.EURPRD83.prod.outlook.com.

--

Jonathan Chico

[Dominik Gresch (Experis AG)]

Hi Jonathan,

 

Yes, I think you’re right. The way I interpret this error message is that at the time the tests are executed, you’re logged into the instance as root. That could well be different on GitHub vs. Gitlab. Not sure if there’s a way to change it.

 

Just to give some context on what’s happening when you don’t define an AIIDA_TEST_PROFILE: In order to not affect the “normal” database, the tests create a new temporary “database cluster”. The pgtest library takes care of this process.

 

Best,

Dominik

 

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CAO8p0kVxkx6gxkT93nuk%2B0mC1Kd5XrjeqZv_hrEYgMu6sKatrQ%40mail.gmail.com.

[Jonathan Chico]

Hi Dominik

Yes this seems to be an issue in how the Gitlab container is setup, it seems to be that (at least in how it has been setup in our system) that you are always root. Which means that if you try to run the pgtest process it will fail because of this limitation. I tried to get around this, by changing to the postgres user in my `before_script` part of the script, but it seems to be that the actual Gitlab container works a bit differently, since when I try to find out which user is actually using the container (simple echo -e $USER) it turns out it returns an empty string.

Thanks for your answer!

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DBAPR83MB0390F431E2B79F036F3C5C04857D9%40DBAPR83MB0390.EURPRD83.prod.outlook.com.

--

Jonathan Chico