Unable to connect to rabbitmq

199 views
Skip to first unread message

Alexandra Dávila

unread,
Dec 17, 2020, 3:22:10 AM12/17/20
to aiida...@googlegroups.com
Dear aiida-users,

I am facing problems with rabbitmq connection. As I perform verdi status, I obtained the following error message:

rabbitmq:    Unable to connect to rabbitmq with URL: amqp://guest:gu...@127.0.0.1:5672?heartbeat=600
Error: AttributeError: module 'tornado.gen' has no attribute '_make_coroutine_wrapper'

This happened since yesterday, before it was working without problems. I tried to stop, start and restart the rabbitmq-server by: sudo service rabbitmq-server ...,  it worked fine but without changes on the verdi status.

I am currently using: aiida-core-1.4.3 with aiida-quantumespresso-3.2.1 in a Virtual Machine with Ubuntu-Mate 20.04.1 LTS.

I would really appreciate any help!

-- 
Kind regards,
Alexandra Dávila
Fritz Haber Institute, Theory Department


Sebastiaan Huber

unread,
Dec 17, 2020, 3:31:56 AM12/17/20
to aiida...@googlegroups.com
Dear Alexandra,

This looks to me like a potential mismatch in some of the underlying libraries used by AiiDA
Could you please run the following command and report the output

    pip freeze | grep -E 'tornado|kiwipy|plumpy|pika'

HTH,

SPH
--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CAHBXNC%3DTR-A39SXachvEAkkVX5i3CHXRVLxaM52vA__5VQ7d%3DQ%40mail.gmail.com.

bonan zhu

unread,
Dec 17, 2020, 3:35:51 AM12/17/20
to aiida...@googlegroups.com
Can you check your tornado versions with: pip list | grep tornado?

AiiDA requires a older version (for now) and it looks like you have an incompatible one. So it has nothing to do with rabbitmq. You probably installed something that automatically updated tornado in your environment such as jupyter.

The solution is to create an dedicated virtual environment (with conda or virtualenv) for AiiDA for maximum compatibility. If you need jupyter you can install it inside a separate environment, which is in general the best practice anyway. It will be able to connect to kernels within other environments.

Best wishes,
Bonan Zhu

From: aiida...@googlegroups.com <aiida...@googlegroups.com> on behalf of Alexandra Dávila <a.celi...@gmail.com>
Sent: Thursday, December 17, 2020 8:27:02 AM
To: aiidausers <aiida...@googlegroups.com>
Subject: [aiidausers] Unable to connect to rabbitmq
 
Dear aiida-users,

I am facing problems with rabbitmq connection. As I perform verdi status, I obtained the following error message:

rabbitmq:    Unable to connect to rabbitmq with URL: amqp://guest

Error: AttributeError: module 'tornado.gen' has no attribute '_make_coroutine_wrapper'

This happened since yesterday, before it was working without problems. I tried to stop, start and restart the rabbitmq-server by: sudo service rabbitmq-server ...,  it worked fine but without changes on the verdi status.

I am currently using: aiida-core-1.4.3 with aiida-quantumespresso-3.2.1 in a Virtual Machine with Ubuntu-Mate 20.04.1 LTS.

I would really appreciate any help!
--
Kind regards,
Alexandra Dávila
Fritz Haber Institute, Theory Department

--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.

Alexandra Dávila

unread,
Dec 17, 2020, 3:52:02 AM12/17/20
to aiidausers
Thanks for answers!

pip freeze | grep -E 'tornado|kiwipy|plumpy|pika'
kiwipy==0.5.5
pika==1.1.0
plumpy==0.15.1
topika==0.2.2
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1604105045397/work

pip list| grep tornado
tornado               6.1

Do I need to create another environment? or unistall jupiter?

Sebastiaan Huber

unread,
Dec 17, 2020, 3:54:21 AM12/17/20
to aiida...@googlegroups.com
Dear Alexandra,

The problem is the version of tornado.
AiiDA v1.4.3 requires <5.
So you can fix it by running

    pip install tornado==4.5.3

Note that we are planning to release AiiDA v1.6 in the near future that no longer depends on tornado, which will therefore alleviate this upper limit version requirement.
Until then, you will have to limit it tornado==4.5.3 unfortunately.

HTH,

SPH

Alexandra Dávila

unread,
Dec 17, 2020, 3:54:55 AM12/17/20
to aiidausers
Sorry! But I checked again and in this environment I didnt install jupyter!

Thanks for your help!

Alexandra Dávila

unread,
Dec 17, 2020, 3:57:50 AM12/17/20
to aiidausers
Dear Sebastian!
Dear Bonan!

Thanks a lot!  That solve the problem!!!

Best regards!
Alexandra

Alexandra Dávila

unread,
Jan 7, 2021, 10:12:41 AM1/7/21
to aiidausers
Dear Sebastian,
Dear All,

I got this problem again . However, this time the version of tornado is 4.5.3

a)pip freeze | grep -E 'tornado|kiwipy|plumpy|pika'

kiwipy==0.5.5
pika==1.1.0
plumpy==0.15.1
topika==0.2.2
tornado==4.5.3

I tried to restart the rabbitmq, but I got an error message:

b) sudo service rabbitmq-server restart
Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

c) systemctl status rabbitmq-server.service
rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2021-01-07 16:06:37 CET; 7s ago
Process: 22496 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
 Main PID: 22496 (code=exited, status=1/FAILURE)

I don't understand why is this error happening again, would you tell me please what I am doing wrong?
By the way I didn't perform any update in my VM, ubuntu 20.04.1 LTS, aiida-core-1.4.3, aiida-quantumespresso-3.2.1.

I hope you can help me!

Best regards,
Alexandra
FHI
El jueves, 17 de diciembre de 2020 a las 9:54:21 UTC+1, sebastia...@epfl.ch escribió:

Alexandra Dávila

unread,
Jan 7, 2021, 10:35:11 AM1/7/21
to aiidausers
I forgot, the error message  I got is :

verdi status
✘ rabbitmq:    Unable to connect to rabbitmq with URL: amqp://guest:gu...@127.0.0.1:5672?heartbeat=600
Error: ConnectionWrongStateError: Channel allocation requires an open connection: <TornadoConnection CLOSED transport=None params=<ConnectionParameters host=127.0.0.1 port=5672 virtual_host=/ ssl=False>>

sorry!
Thanks in advance!

Giovanni Pizzi

unread,
Jan 7, 2021, 11:25:34 AM1/7/21
to AiiDA users mailing list
Hi,
I think the problem is with the bold part here:

Active: activating (auto-restart) (Result: exit-code) since Thu 2021-01-07 16:06:37 CET; 7s ago
Process: 22496 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
 Main PID: 22496 (code=exited, status=1/FAILURE)

I think this means that the RabbitMQ server died, and the service manager is starting to restart it every few seconds (last time it tried was 7 seconds before you run the command).

I suggest that you use systemctl to stop the rabbitmq.service, make sure the service was indeed stopped (e.g. check with `ps auxw` that there are no more processes (note that the process might have weird names, like beam.smp - on Ubuntu you can check e.g. all processes run by the `rabbitmq` user), and then restart it. Maybe, in between, reboot the computer (and also check that there are no obvious issues, e.g. a full hard drive).
Hopefully this will work?

Otherwise one would need to check the logs of rabbitmq to see why it fails restarting.

Best,
Giovanni


Alexandra Dávila

unread,
Jan 7, 2021, 1:08:31 PM1/7/21
to aiidausers
Dear Giovanni,

thanks a lot for your response!

The version I am using:
rabbitmqctl --version 3.8.2
erlang 22.2.7

I tried your suggestion :

a) systemctl stop rabbitmq-server.service
(no message error)

b) Then I start but I got the same message error again

Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

I attached the log file. There I observed that when I tried to start rabbitmq-server, I observed the following error message:

Line 37598
2021-01-07 18:56:10.865 [info] <0.301.0> ra: meta data store initialised. 0 record(s) recovered
2021-01-07 18:56:10.866 [info] <0.312.0> WAL: recovering ["/var/lib/rabbitmq/mnesia/rabbit@alexandra-VirtualBox/quorum/rabbit@alexandra-VirtualBox/00000055.wal"]
2021-01-07 18:56:10.867 [error] <0.312.0> CRASH REPORT Process <0.312.0> with 0 neighbours exited with reason: no case clause matching {ok,<<>>} in ra_log_wal:open_existing/1 line 646 in gen_batch_server:init_it/6 line 165

The first error appeared in line 605
2021-01-07 15:12:07.080 [info] <0.306.0> ra: meta data store initialised. 0 record(s) recovered
2021-01-07 15:12:07.081 [info] <0.318.0> WAL: recovering ["/var/lib/rabbitmq/mnesia/rabbit@alexandra-VirtualBox/quorum/rabbit@alexandra-VirtualBox/00000055.wal"]
2021-01-07 15:12:07.083 [error] <0.318.0> CRASH REPORT Process <0.318.0> with 0 neighbours exited with reason: no case clause matching {ok,<<>>} in ra_log_wal:open_existing/1 line 646 in gen_batch_server:init_it/6 line 165
2021-01-07 15:12:07.083 [error] <0.317.0> Supervisor ra_log_wal_sup had child ra_log_wal started with ra_log_wal:start_link(#{compute_checksums => true,dir => "/var/lib/rabbitmq/mnesia/rabbit@alexandra-VirtualBox/quorum...",...}, [{max_batch_size,32768}]) at undefined exit with reason no case clause matching {ok,<<>>} in ra_log_wal:open_existing/1 line 646 in context start_error

I hope you can help me with this!

Many thanks in advance,
Alexandra
error_rabbit.log

Giovanni Pizzi

unread,
Jan 8, 2021, 4:30:43 AM1/8/21
to AiiDA users mailing list
Hi Alexandra,
this seems a problem of RabbitMQ, somehow it got stopped without a clean shutdown, and now it's crashing when trying to restart as it has issues reading the data.

I have no good suggestion, except to check in the rabbitmq mailing list if there are similar error reports, and maybe post a question there:

Hope they can help!

Best,
Giovanni

simeon....@gmail.com

unread,
Jan 8, 2021, 5:13:04 AM1/8/21
to aiidausers

Hi Alexandra, dear *!

Sorry, I think my first email did not arrive...

I am currently not using Aiida, but I have the same setup installed as Alexandra. In the morning I was wondering why my Computer is running high in cpu usage every ~10sec, and figured out that is indeed also related to rabbitmq-server, i.e. the erlang-installation (whatever this is).  Even though verdi was not running, just from a fresh restart this issue showed up.

[System: Ubuntu Mate 20.04 in VirtualBox 6.1.16, Aiida 1.4.3]

It seems as if I was able to fix it (at least everything was still at its place afterwards, but you should definitely make a full backup beforhand, no guarantee that this will not break things) by following the next steps:

0) Backup the whole System!!!!

1) be sure verdi and nothing else using rabbit is not running

2) sudo systemctl stop rabbitmq-server [it shouldn't evern be started up]

3) sudo apt purge rabbitmq-server [caution: this will delete all system-level configuration files of rabbit, so be sure to have made a backup if you want to use the old files again.]

4) sudo apt update

5) sudo apt upgrade

6) sudo apt autoremove

7) sudo reboot

8) sudo apt install rabbitmq-server

9) check that everything is still there by running some things like verdi process list -a [this should list all processes ever run] verdi daemon start.

10) Be sure that all your data is still there and that everything else is still working!

I hope this helps and I even more hope it works fine for everyone.


Have a nice Friday and a nice weekend, stay healty,

Cheers, Simeon

-------------------------------------------------------------------
Simeon Beinlich

PhD Student
Theory Department
Fritz-Haber-Institut der Max-Planck-Gesellschaft
Faradayweg 4-6
D-14195 Berlin-Dahlem (Germany)
www.fhi.mpg.de/th-department

Department of Chemistry
Technical University of Munich
Lichtenbergstraße 4
D-85747 Garching (Germany)
www.department.ch.tum.de/th4

-------------------------------------------------------------------

Alexandra Dávila

unread,
Jan 8, 2021, 5:52:35 AM1/8/21
to aiidausers
Dear Simeon and Giovanni,
Dear aiida-users,

I tried the steps suggested by Simeon, and now it is working! All data is still there!

Thanks a lot for your suggestions!

Best regards,

Alexandra Dávila
FHI-Theory

Muhammad Maikudi ISAH

unread,
Feb 3, 2021, 10:47:11 AM2/3/21
to aiidausers
Dear Giovanni and Alexandra,

I have similar issues on RabbitMQ you highlighted above and all the packages were installed on conda environment[https://aiida-core.readthedocs.io/en/latest/intro/install_conda.html#intro-get-started-conda-install)and]
and the version of Tornado was 4.5.3. I have been encountering the problem for a while now and it works 
after a re-start with 
rabbitmq-server  -detached
At the moment  I can no longer get to start the rabbitmq working properly the error message persist

 ✘ rabbitmq:    Unable to connect to rabbitmq with URL: amqp://guest:gu...@127.0.0.1:5672?heartbeat=600
Error: ConnectionWrongStateError: Channel allocation requires an open connection: <TornadoConnection CLOSED transport=None params=<ConnectionParameters host=127.0.0.1 port=5672 virtual_host=/ ssl=False>>

I am using aiida-core=1.5.2
version of 
kiwipy==0.5.5
pika==1.1.0
topika==0.2.2
plumpy==0.15.1
tornado==4.5.3

I don't have a sudo privilege to attempt the solutions pointed by Simeon and co.
I am attaching the RabbitMQ error message.

Hope you can help me.

Best regard,
Muhammad
error.log
Reply all
Reply to author
Forward
0 new messages