Setting up with MongoDB Atlas

[mrck...@gmail.com]

Hi! For some reason, I'm having some real trouble connecting FireWorks to my MongoDB Altas cluster.

Here's what I've done:

I successfully completed the 'Testing connection to a remote server' section on the fireworks installation page.

Next, I ran

 lpad init

which created the file 

my_launchpad.yaml

In this file, I have the following:

authsource: research
host: mongodb+srv://chenge:<password>@research-nwvup.mongodb.net/test?retryWrites=true&w=majority
logdir: null
name: fireworks
password: <password>
port: 27017
ssl: false
ssl_ca_certs: null
ssl_certfile: null
ssl_keyfile: null
ssl_pem_passphrase: null
strm_lvl: INFO
uri_mode: false
user_indices: []
username: chenge
wf_user_indices: []

My Mongo cluster is called "research", and my username is chenge. 

When I run the suggested command

lpad add_scripts 'echo "hello"' 'echo "goodbye"' -n hello goodbye -w test_workflow

I get the following error:

login1(1044)$ lpad add_scripts 'echo "hello"' 'echo "goodbye"' -n hello goodbye -w test_workflow
Traceback (most recent call last):
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/fireworks/core/launchpad.py", line 1377, in get_new_fw_id
    return self.fw_id_assigner.find_one_and_update({}, {'$inc': {'next_fw_id': quantity}})['next_fw_id']
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/collection.py", line 3136, in find_one_and_update
    array_filters, session=session, **kwargs)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/collection.py", line 2885, in __find_and_modify
    write_concern.acknowledged, _find_and_modify, session)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1491, in _retryable_write
    with self._tmp_session(session) as s:
  File "/opt/apps/intel18/python3/3.7.0/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1820, in _tmp_session
    s = self._ensure_session(session)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1807, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1760, in __start_session
    server_session = self._get_server_session()
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1793, in _get_server_session
    return self._topology.get_server_session()
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/topology.py", line 482, in get_server_session
    None)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/pymongo/topology.py", line 205, in _select_servers_loop
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/home1/05778/chenge/.local/bin/lpad", line 11, in <module>
    load_entry_point('FireWorks==1.9.4', 'console_scripts', 'lpad')()
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/fireworks/scripts/lpad_run.py", line 1176, in lpad
    args.func(args)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/fireworks/scripts/lpad_run.py", line 579, in add_scripts
    lp.add_wf(Workflow(fws, links, args.wf_name))
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/fireworks/core/launchpad.py", line 349, in add_wf
    old_new = self._upsert_fws(list(wf.id_fw.values()), reassign_all=reassign_all)
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/fireworks/core/launchpad.py", line 1412, in _upsert_fws
    first_new_id = self.get_new_fw_id(quantity=len(fws))
  File "/home1/05778/chenge/.local/lib/python3.7/site-packages/fireworks/core/launchpad.py", line 1379, in get_new_fw_id
    raise ValueError("Could not get next FW id! If you have not yet initialized the database,"
ValueError: Could not get next FW id! If you have not yet initialized the database, please do so by performing a database reset (e.g., lpad reset)

Google searches have not gotten me anywhere, as the two results are 1) source code for FireWorks, and 2) some other package for which I have no idea what their setup is. 

Could someone help me out on this? Thanks in advance!

[agrav]

Hi,

I have had a similar problem. The ValueError is raised due to an underlying pymongo error. In my case my MongoDB database did not allow 'retryWrites=true' in the MongoDB URI. Setting 'retryWrites=false' fixed it for me.

Audun

[Erik Cheng]

Hi Audun,

Thanks, I just tried that. However, it still doesn't work. The exact same error seems to appear for me. I have also checked my Atlas cluster and confirmed I have the IP address 0.0.0.0/0 whitelisted, so that shouldn't be the problem either (as I've seen on some stack overflow posts regarding similar issues). Do you have further thoughts on why this issue seems to be persisting?

Thanks in advance

[agrav]

Hi,

You may try to simplify your my_launchpad.yaml to:

host: '<URI string>'
uri_mode: true

where <URI string> is the connection URI which should be available somewhere in your Atlas web interface (I have only used mlab, so not familiar with the Atlas interface).

I typically looks like ''mongodb://<username>:<password>@<url>:<port>/<db name>?retryWrites=false''

The 'retryWrites=false' may still be needed in the URI string.

Audun

[Anubhav Jain]

There is a similar thread on the "atomate" forum:

https://groups.google.com/forum/#!topic/atomate/FxaoHkHrcZI

Let me know if that helps or if there's still a problem!

[Erik Cheng]

Hi Audun,

Your suggestion actually prompted me to take a look at uri_mode in my original file- and I noticed it was false there, for some reason. Changing it to true appears to have made it work. Thanks so much!

-Erik