Connecting to MongoDB atlas

[Zoe Zhu]

Hi, 

I have been trying to set up the MongoDB atlas database for fireworks. 

My launchpad looks like the following: 

host: mongodb+srv://cluster0-b05rv.mongodb.net/test

port: 27017

name: test

username: admin

password: <<PASSWORD>>

logdir: null

ssl: false 

strm_lvl: INFO

user_indices: []

wf_user_indices: []

However, I have been getting the following error when I run lpad reset: 

ValueError: FireWorks was not able to connect to MongoDB. Is the server running? The database file specified was /n/home02/zzhu/atomate/config/my_launchpad.yaml.

It only connects if I change the fire /fireworks/core/.launchpad.py around line 149 to the following:

# get connection

        #self.connection = MongoClient(host, port, ssl=self.ssl,

        #    username=username,password=password,

        #    ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,

        #    ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,

        #    socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False)

        #self.db = self.connection[name]

        #if username:

        #     self.db.authenticate(username, password, source=self.authsource)

        self.connection = MongoClient(host, port, username=username, password=password)

        self.db = self.connection[name]

  

The db.authenticate function always gives me trouble. 

I wonder if this is a known issue and if there's any solution to this?

Thanks,

Zoe Zhu

[Zoe Zhu]

Hi Anubhav, 

Thank you so much for your help. 

I am still getting an error (but a different one): 

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

____________________________________

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz...@g.harvard.edu, zzy9...@gmail.com

(310)-210-0580

On Tue, Feb 12, 2019 at 4:37 PM Anubhav Jain <AJ...@lbl.gov> wrote:

Hi Zoe,

Based on talking to some colleagues, it seems the db.authenticate() function in pymongo is deprecated. In FWS v1.8.5 (just released now) we changed the authentication code. Let us know if it works for you.

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
For more options, visit https://groups.google.com/d/optout.

--

Best,
Anubhav

[Anubhav Jain]

Hi Zoe

Do you happen to know what part of the connection code is causing the error? I'm happy to push a fix, just not sure which part of the code is causing problems on your end.

Best,

Anubhav

[Anubhav Jain]

Hi Zoe,

Actually, we found an issue in the previous release and pushed a fix in v1.8.6. Let me know if that one works or if you still have problems.

--

Best,
Anubhav

[Zoe Zhu]

Hi Anubhav, 

Sorry I didn't include all the error messages! See below: 

Traceback (most recent call last):

  File "/n/home02/zzhu/.conda/envs/atomate_env/bin/lpad", line 11, in <module>

    load_entry_point('FireWorks==1.8.5', 'console_scripts', 'lpad')()

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py", line 1147, in lpad

    args.func(args)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py", line 148, in reset

    lp.workflows.count()))[0].upper() == 'Y':

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py", line 1766, in count

    return self._count(cmd, collation, session)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py", line 1563, in _count

    with self._socket_for_reads(session) as (sock_info, slave_ok):

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/contextlib.py", line 81, in __enter__

    return next(self.gen)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1099, in _socket_for_reads

    server = topology.select_server(read_preference)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py", line 224, in select_server

    address))

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py", line 183, in select_servers

    selector, server_timeout, address)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py", line 199, in _select_servers_loop

    self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

Thanks, 

Zoe 

____________________________________

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz...@g.harvard.edu, zzy9...@gmail.com

(310)-210-0580

[Zoe Zhu]

And I got this error when I did lpad reset. 

____________________________________

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz...@g.harvard.edu, zzy9...@gmail.com

(310)-210-0580

[Anubhav Jain]

Hi Zoe,

Based on talking to some colleagues, it seems the db.authenticate() function in pymongo is deprecated. In FWS v1.8.5 (just released now) we changed the authentication code. Let us know if it works for you.

On Tue, Feb 12, 2019 at 12:38 PM Zoe Zhu <Zzy9...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
For more options, visit https://groups.google.com/d/optout.

--

Best,
Anubhav

[Zoe Zhu]

Hi Anubhav, 

Thanks for your response. However, I am still getting the same error message when I do lpad reset: 

Traceback (most recent call last):

  File "/n/home02/zzhu/.conda/envs/atomate_env/bin/lpad", line 11, in <module>

    load_entry_point('FireWorks==1.8.6', 'console_scripts', 'lpad')()

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py", line 1147, in lpad

    args.func(args)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py", line 148, in reset

    lp.workflows.count()))[0].upper() == 'Y':

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py", line 1766, in count

    return self._count(cmd, collation, session)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py", line 1563, in _count

    with self._socket_for_reads(session) as (sock_info, slave_ok):

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/contextlib.py", line 81, in __enter__

    return next(self.gen)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1099, in _socket_for_reads

    server = topology.select_server(read_preference)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py", line 224, in select_server

    address))

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py", line 183, in select_servers

    selector, server_timeout, address)

  File "/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py", line 199, in _select_servers_loop

    self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

____________________________________

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz...@g.harvard.edu, zzy9...@gmail.com

(310)-210-0580

[Anubhav Jain]

Hi all,

Due to Google groups withholding these messages, I can't tell if these problems are with FWS v1.8.5 or v1.8.6.

Can you clarify if the problem is still present in FWS v1.8.6?

On Fri, Feb 15, 2019 at 6:55 AM <marco.di...@external.toyota-europe.com> wrote:

Hello everyone,

I am testing the connection to a remote server as explained in the tutorial and I get the a similar error.

I have tried either the "my_launchpad_testing.yaml" as given at the bottom of https://materialsproject.github.io/fireworks/installation.html,

as well as to change username and password to my system my_username and my_password, and finally to create a user in my local mongodb.

In all cases I get a "pymongo.errors.ServerSelectionTimeoutError: ds049170.mongolab.com:49170: timed out"

followed by a  "ValueError: FireWorks was not able to connect to MongoDB. Is the server running? The database file specified was my_launchpad_test ing.yaml. "

My mongo looks like this for the moment (with a couple of Warning message which I am trying to figure out):

$mongo
MongoDB shell version v4.0.3
connecting to: mongodb://127.0.0.1:27017
WARNING: No implicit session: Logical Sessions are only supported on server versions 3.6 and greater.
Implicit session: dummy session
MongoDB server version: 2.6.10
WARNING: shell and server versions do not match
>
> show dbs
admin      0.078GB
fireworks  0.078GB
test       (empty)
> use fireworks
switched to db fireworks
> show collections
fireworks
fw_id_assigner
launches
system.indexes
workflows
> show users

{
    "_id" : "fireworks.mdi0316",
    "user" : "mdi0316",
    "db" : "fireworks",
    "roles" : [
        {
            "role" : "readWrite",
            "db" : "accounts"
        }
    ]
}
>
>

$

I attach the whole error message as a text file.

Thank you in advance.

Best Regards

Marco Di Gennaro

--

Best,
Anubhav

[marco.di...@external.toyota-europe.com]

I realized I had v 1.8.4 installed

Upgrading did not work nonetheless, and I still have a pymongo.error with version 1.8.6 (see below).

I am actually behind a firewall, how can I make sure this is not the problem?

Thank you

$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)

>>> import fireworks
>>> print(fireworks.__version__)
1.8.6

>>> import pymongo
>>> print(pymongo.__version__)
3.7.2

$ which lpad
/home/mdi0316/anaconda3/bin/lpad

$ lpad -l my_launchpad_testing.yaml get_wflows

Traceback (most recent call last):

  File "/home/mdi0316/anaconda3/bin/lpad", line 10, in <module>
    sys.exit(lpad())
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py", line 1147, in lpad
    args.func(args)
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py", line 297, in get_wfs
    ids = lp.get_wf_ids(query, sort, args.max, count_only=args.display_format == 'count')
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/fireworks/core/launchpad.py", line 667, in get_wf_ids
    for fw in self.workflows.find(criteria, {"nodes": True}, sort=sort).limit(limit):
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/cursor.py", line 1189, in next
    if len(self.__data) or self._refresh():
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/cursor.py", line 1087, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1563, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1516, in __start_session
    server_session = self._get_server_session()
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1549, in _get_server_session
    return self._topology.get_server_session()
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/topology.py", line 427, in get_server_session
    None)
  File "/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/topology.py", line 199, in _select_servers_loop
    self._error_message(selector))

[Anubhav Jain]

Hi Marco

To make the firewall is not the problem, I would first try connecting to your database using the standard mongo shell (from MongoDB) rather than FireWorks. If you can connect with the mongo shell, then you should (in theory) be able to connect with FWS - so let us know if you can connect with mongo shell but not FWS. If you cannot connect with the mongo shell, FireWorks will also not work. In that case, you will need to work with a system administrator to get that aspect worked out before moving on to FWS..

Best,

Anubhav

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
For more options, visit https://groups.google.com/d/optout.

--

Best,
Anubhav

[marco.di...@external.toyota-europe.com]

Dear Anubhav,

as I stated in my original question, yes, I can connect to port 27017

Nevertheless I get two wanings and I am not sure I am really connected

$ mongo

MongoDB shell version v4.0.3
connecting to: mongodb://127.0.0.1:27017
WARNING: No implicit session: Logical Sessions are only supported on server versions 3.6 and greater.
Implicit session: dummy session
MongoDB server version: 2.6.10
WARNING: shell and server versions do not match
>

Please let me know if you have any suggestion on this

Kind regards

[Anubhav Jain]

Hi Marco,

I can connect to the testing ds049170.mongolab.com:49170 server using the my_launchpad_testing.yaml document specified in the documentation using the latest FireWorks. So, my guess is that there is a firewall issue that is preventing you from accessing that test database hosted online.

In terms of testing using the "mongo" shell, I meant testing that you can connect to the remote server using the mongo shell. That is, testing to see if you can connect to the ds049170.mongolab.com:49170 server using the mongo shell.

There are some details here:

https://docs.mongodb.com/manual/mongo/

But you can just type the following shell commands:

> mongo --username test_user --password testing123 --authenticationDatabase fireworks --host ds049170.mongolab.com --port 49170

MongoDB shell version v3.4.4

connecting to: mongodb://ds049170.mongolab.com:49170/

MongoDB server version: 3.6.6

WARNING: shell and server versions do not match

> use fireworks

switched to db fireworks

> db.fireworks.count()

2

If those commands do not work on your system (they do on mine), it is almost certainly a firewall problem as  those commands just use the mongo shell and not FWS

--

You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
For more options, visit https://groups.google.com/d/optout.

--

Best,
Anubhav

[Anubhav Jain]

Note that another way to test the firewall issue is to try the same commands on home network vs a company network.

--

Best,
Anubhav

[Athreya Ramesh]

Hello Anubhav,

I am trying to connect to MongoDB atlas through fireworks. The error is below. 

My Fireworks is version 1.8.7.

Please let me know if there is any other information that I can provide that would help.

C:\Users\athre\Miniconda3\Lib\site-packages\fw_tutorials\introduction>lpad reset

Traceback (most recent call last):

  File "c:\users\athre\miniconda3\lib\runpy.py", line 193, in _run_module_as_main

    "__main__", mod_spec)

  File "c:\users\athre\miniconda3\lib\runpy.py", line 85, in _run_code

    exec(code, run_globals)

  File "C:\Users\athre\Miniconda3\Scripts\lpad.exe\__main__.py", line 9, in <module>

  File "c:\users\athre\miniconda3\lib\site-packages\fireworks\scripts\lpad_run.py", line 1146, in lpad

    args.func(args)

  File "c:\users\athre\miniconda3\lib\site-packages\fireworks\scripts\lpad_run.py", line 148, in reset

    lp.workflows.count()))[0].upper() == 'Y':

  File "c:\users\athre\miniconda3\lib\site-packages\pymongo\collection.py", line 1766, in count

    return self._count(cmd, collation, session)

  File "c:\users\athre\miniconda3\lib\site-packages\pymongo\collection.py", line 1563, in _count

    with self._socket_for_reads(session) as (sock_info, slave_ok):

  File "c:\users\athre\miniconda3\lib\contextlib.py", line 112, in __enter__

    return next(self.gen)

  File "c:\users\athre\miniconda3\lib\site-packages\pymongo\mongo_client.py", line 1099, in _socket_for_reads

    server = topology.select_server(read_preference)

  File "c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py", line 224, in select_server

    address))

  File "c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py", line 183, in select_servers

    selector, server_timeout, address)

  File "c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py", line 199, in _select_servers_loop

    self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

Thanks,

Athreya Ramesh

[Anubhav Jain]

Hi Athreya

Sorry but this is not enough information to debug.

Can you first try connecting to Atlas using the pymongo tool directly (and not using FireWorks)? Once you have that working, you can let us know what settings you used to get it to work with direct pymongo connection that are not working when you try it with FireWorks.