Problems with pytests fixtures from aiida-vasp

[Jonathan Chico]

Dear all

I have been working in a series of workflows and I have made a series of tests to be run in a CI/CD pipeline.

I'm making use of the aiida-core and aiida-vasp fixtures and running my pipeline in gitlab, I have noticed a couple of things that I wonder if somebody else has noticed. I can define computers, codes and potential families by making use of the verdi CLI. However, if I try to use pytest fixtures they do not seem to work, for example potential families uploaded inside pytest fixtures are not recognized.

This happens when I even use fixtures that work in github actions (such as fixtures in aiida-vasp).

I noticed that the verdi daemon is not working, and when I try to start it inside the gitlab container it always results in failure, and if I try to use the --foreground option the process just stalls forever.

So I was wondering if this is a gitlab exclusive issue, or if someone else has seen similar problems.

Thanks!

Cheers

[Leopold Talirz]

Hi Jonathan,

it might be useful to provide a couple more details here, including error messages etc. (and perhaps some aiida-vasp developer will chime in).

When you say "aiida-vasp fixtures" are you talking of pytest fixtures, i.e. functions that are being run in order to set up the test environment during a test run?

And when you say they work on github actions but not on gitlab, what is different in the two environments?

Cheers,

Leo

[Leopold Talirz]

P.S. If the problem is just the tests "stalling", it might be that either rabbitmq or postgresql are not accessible. 

You might want to have a look at how these are set up in the aiida-core repo:

https://github.com/aiidateam/aiida-core/blob/403f7e7d896f8408d0cacee9fe41c030a1072eaf/.github/workflows/ci-code.yml#L55-L72 

[Espen Flage-Larsen]

Hi there Jonathan,

We are using GitLab regularly and I am happy to try to bring the AiiDA-
VASP tests in one of our environments. In fact, I have not yet done
this so anyway good to check and maybe add a snapshot of the config to
the documentation.

Can you share your current GitLab config? Also, can you open an issue
with the plugin? That would be awesome. We can continue discussing
there.

Cheers,
Espen

[Jonathan Chico]

Dear all

That is good to know, it is probably a configuration issue. I'll open an issue in the aiida-vasp github to see if I can figure out the issue. Mostly what I observe is that assertions fail

fresh_aiida_env = <aiida.manage.tests.TestManager object at 0x7f053e4a4be0>

temp_pot_folder = PosixPath('/tmp/pytest-of-root/pytest-0/test_relax_wc0/potentials')

@pytest.fixture

def coro_potcar_family(fresh_aiida_env, temp_pot_folder):

"""Create a POTCAR family."""

potcar_ga = Path(data_path('potcar')) / 'Ga'

family_name = POTCAR_FAMILY_NAME

family_desc = 'A POTCAR family used as a test fixture. Contains only unusable POTCAR files.'

potcar_cls = get_data_class('vasp.potcar')

temp = potcar_cls.upload_potcar_family(str(temp_pot_folder), family_name, family_desc, stop_if_existing=False)

print(temp)

print(str(temp_pot_folder))

print(potcar_cls.find(full_name='In_d'))

if len(potcar_cls.find(full_name='In_d')) == 1:

family_group = potcar_cls.get_potcar_group(POTCAR_FAMILY_NAME)

print(POTCAR_FAMILY_NAME)

print(str(temp_pot_folder))

print(family_group)

in_d = potcar_cls.find(full_name='In_d')[0]

in_d_double = duplicate_potcar_data(in_d)

> family_group.add_nodes(in_d_double)

E AttributeError: 'NoneType' object has no attribute 'add_nodes'




I can confirm that the POTCAR family is uploaded properly, that the potential path has the potential folders, etc. As you can see in the stdouts here

Using existing profile: test_relax

/tmp/pytest-of-root/pytest-0/test_relax_wc0/potentials

['Zn', 'In_sv', 'In_d', 'S', 'export.aiida', 'Si', 'Ga.tar', 'As', 'H_AE', 'P']

(9, 9, 9)

/tmp/pytest-of-root/pytest-0/test_relax_wc0/potentials

[<PotcarData: uuid: 7d96b873-7826-425b-b67c-635ca1429103 (pk: 7)>]

test_family

/tmp/pytest-of-root/pytest-0/test_relax_wc0/potentials

None

As you can see the potentials are uploaded, the family is recognized, however, when trying to get the family_group making use of the get_potcar_group the system cannot seem to find the potcar family.

If I instead define the potcar family via the verdi CLI the test continues without problem. It however fails when `mock-vasp` is running, I do not think mock-vasp runs at all, since the output does not contain anything.

Cheers

--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).

Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/fe6e2ccaa5b094c0d6638666c5201df813fa8ad3.camel%40sintef.no.

--

Jonathan Chico

[Jonathan Chico]

Dear all

So I looked a bit into detail, and yes in the gitlab container rabbitmq is indeed not working, when I try to start the service there is an issue

service rabbitmq-server start

Starting RabbitMQ Messaging Server: rabbitmq-serverFAILED




I do think that whatever issue is causing the pytest fixtures to fail must be related to this, as if I run the exact same test locally in my machine and define the potcar family, code and computer in a given test profile (basically the steps of my CI/CD) the calculation works without issue.





--

Jonathan Chico

[Jonathan Chico]

Dear all

So I have been looking into this, and it seems to be a more general issue than just the pytest fixtures from aiida-vasp. When running aiida-core inside the gitlab container several things seem to not function as expected. For example, one is supposed to use rabbitmq by configuring a service in gitlab, this seems to fail, causing the rabbitmq service to never start. However, if one installs rabbitmq-server directly into the container and then manually starts it rabbitmq reports that it works without problems. However, the daemon is never able to start, even if according to verdi status every other system is properly working

$ verdi status
✔ version: AiiDA v1.6.2
✔ config: /root/.aiida
✔ profile: test_relax
✔ repository: Connected to Repository<DiskObjectStoreRepository: baf00984fed54bcdbf56b9dba629f50e | /tmp/test_repository_test_relax/container>
✔ postgres: Connected as postgres@localhost:5432
✔ rabbitmq: Connected as amqp://guest:gu...@127.0.0.1:5672?heartbeat=600
⏺ daemon: The daemon is not running

For what I can see this results in certain nodes not being reachable, for example if I upload a vasp-potcar family verdi informs that the potential files are properly uploaded (they are dummy files)

$ verdi -p test_relax data vasp-potcar uploadfamily -p aiida_coro/test_data/potcar -n coro_test_family -D 'A POTCAR family used as a test fixture. Contains only unusable POTCAR files.'

POTCAR files found: 8. New files uploaded: 8, Added to Family: 8

However, if I try to check on those files they do not appear according to the CLI

$ verdi -p test_relax data vasp-potcar listfamilies

No POTCAR family available.

This seems to be the reason behind the failures, as the pytest fixtures try to create several nodes that are used during the tests (such as the potentials).

Trying to start the daemon results in failure. 

This seems to be an issue on how the daemon interacts with the gitlab container, it seems that it is not possible to start it there.

Cheers