Issues with the latest version on both Hopper and Carver

76 views
Skip to first unread message

Hark Lee

unread,
Apr 11, 2015, 7:42:01 AM4/11/15
to matproj...@googlegroups.com
Dear Anubhav, 

After the maintenance, I was getting occasional errors and fizzled runs, mostly with compounds 
with higher number of atoms. So I decided to reinstall a fresh copy on Carver, which appears to have 
some issues. The installation worked fine and I could successfully activate the environment, but 
submitting jobs doesn't go through with the following errors.

------------------------------------------------------------------------------------------

(virtenv_hark_vasp)-bash-3.2$ go_testing --clear
hark_vasp environment

2015-04-11 03:51:32,899 DEBUG RESTARTED fw_id, launch_id to (1, 1)
2015-04-11 03:51:32,899 INFO Performing db tune-up
2015-04-11 03:51:32,900 DEBUG Updating indices...
2015-04-11 03:51:32,911 INFO LaunchPad was RESET.

Traceback (most recent call last):
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/bin/go_testing", line 7, in <module>
    execfile(__file__)
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/scripts/go_testing", line 4, in <module>
    go_testing()
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/scripts/submit_canonical_run.py", line 41, in go_testing
    clear_and_submit(args.clear, names, params)
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/processors/submit_canonical.py", line 77, in clear_and_submit
    clear_env()
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/processors/submit_canonical.py", line 33, in clear_env
    conn = MongoClient(db_creds['host'], db_creds['port'])
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/pymongo-3.0-py2.7-linux-x86_64.egg/pymongo/mongo_client.py", line 364, in __init__
    executor.open()
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/pymongo-3.0-py2.7-linux-x86_64.egg/pymongo/periodic_executor.py", line 64, in open
    thread.start()
  File "/usr/common/usg/python/2.7.3/lib/python2.7/threading.py", line 494, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/common/usg/python/2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/pymongo-3.0-py2.7-linux-x86_64.egg/pymongo/periodic_executor.py", line 139, in _shutdown_executors
    executor.join(10)
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/pymongo-3.0-py2.7-linux-x86_64.egg/pymongo/periodic_executor.py", line 80, in join
    self._thread.join(timeout)
  File "/usr/common/usg/python/2.7.3/lib/python2.7/threading.py", line 652, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/common/usg/python/2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/pymongo-3.0-py2.7-linux-x86_64.egg/pymongo/periodic_executor.py", line 139, in _shutdown_executors
    executor.join(10)
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/pymongo-3.0-py2.7-linux-x86_64.egg/pymongo/periodic_executor.py", line 80, in join
    self._thread.join(timeout)
  File "/usr/common/usg/python/2.7.3/lib/python2.7/threading.py", line 652, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started

------------------------------------------------------------------------------------------





I also tried installing on Hopper, and it also seems to have issues.

------------------------------------------------------------------------------------------

(virtenv_hark_vasp)harklee@hopper06:~> go_testing --clear
hark_vasp environment

2015-04-11 04:33:34,467 DEBUG RESTARTED fw_id, launch_id to (1, 1)
2015-04-11 04:33:34,468 INFO Performing db tune-up
2015-04-11 04:33:34,469 DEBUG Updating indices...
2015-04-11 04:33:34,501 INFO LaunchPad was RESET.

Traceback (most recent call last):
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/bin/go_testing", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/scripts/go_testing", line 4, in <module>
    go_testing()
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/scripts/submit_canonical_run.py", line 41, in go_testing
    clear_and_submit(args.clear, names, params)
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/processors/submit_canonical.py", line 78, in clear_and_submit
    submit_tests(names=names, params=params)
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/processors/submit_canonical.py", line 64, in submit_tests
    sid = mpr.get_materials_id_from_task_id("mp-{}".format(sid))
  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/matproj/rest.py", line 158, in get_materials_id_from_task_id
    return self._make_request("/materials/mid_from_tid/%s" % task_id)
  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/matproj/rest.py", line 145, in _make_request
    raise MPRestError(msg)
pymatgen.matproj.rest.MPRestError: [SSL: TLSV1_UNRECOGNIZED_NAME] unknown error (_ssl.c:581)

------------------------------------------------------------------------------------------

(virtenv_hark_vasp)harklee@hopper06:~/vw/bi> go_submissions
hark_vasp environment

Traceback (most recent call last):
  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/bin/go_submissions", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/scripts/go_submissions", line 1, in <module>
    from mpworks.scripts.submissions_run import go_submissions
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/scripts/submissions_run.py", line 1, in <module>
    from mpworks.processors.process_submissions import SubmissionProcessor
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/processors/process_submissions.py", line 6, in <module>
    from mpworks.workflows.snl_to_wf import snl_to_wf
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/workflows/snl_to_wf.py", line 5, in <module>
    from mpworks.firetasks.controller_tasks import AddEStructureTask
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/firetasks/controller_tasks.py", line 6, in <module>
    from mpworks.firetasks.boltztrap_tasks import BoltztrapRunTask
  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/firetasks/boltztrap_tasks.py", line 16, in <module>
    from mpcollab.thermoelectrics.boltztrap_TE import BoltztrapAnalyzerTE, BoltzSPB
ImportError: No module named mpcollab.thermoelectrics.boltztrap_TE

------------------------------------------------------------------------------------------


I am not sure if there is anything I am missing. I just hope that this is some silly mistake I made 
and specific to my account.  

Any suggestions what I could try next?


Thank you. 

Best, 

Hark 
Message has been deleted

Hark Lee

unread,
Apr 15, 2015, 4:53:10 AM4/15/15
to matproj...@googlegroups.com
So here is what I figured out so far, 

(1) The Botztrap problem is due to the recent updates to MPWorks, which can be resolved 
by reverting MPWorks back to a previous, more stable commit. I tried head 7ccf927 
and it seems to be working (git reset --hard 7ccf927). 
(I also reverted MPenv back a bit, but I don't think it is essential, although minor changes
to .bashrc.ext needed to reflect the changes in available modules)

(2) The pymongo / threading issue can be mitigated by downgrading Pymongo to 2.8.

(3) After installing from scratch on Carver, the 'requests' dependency, which is installed on Carver,
doesn't seem to be present on Hopper. And pip couldn't download from source, so I tried 
easy_install it, which worked. This didn't screw up the setup on Carver either. 


Thanks!

Hark

Anubhav Jain

unread,
Apr 16, 2015, 11:23:00 PM4/16/15
to Hark Lee, matproj...@googlegroups.com
Great, thanks! I will try to look into the MPWorks issue soon

--
You received this message because you are subscribed to the Google Groups "Materials Project Development Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to matproj-devel...@googlegroups.com.
To post to this group, send email to matproj...@googlegroups.com.
Visit this group at http://groups.google.com/group/matproj-develop.

Hark Lee

unread,
Apr 17, 2015, 9:01:43 PM4/17/15
to Anubhav Jain, matproj...@googlegroups.com
Anubhav, 

So I got to run jobs until they FIZZLED at the second step (C-RES -> C-RUN -> C-F) 
Do you have a clue on what might be causing the following errors? Please don't bother
answering them if it's too much of trouble, thanks!



(1) POTCAR parsing

So the codes appeared to work fine until it 'rlaunch'ed the actual vasp runs.
Looks like pymatgen is having difficulty parsing POTCAR files, it's the line 1293 of 
/io/vaspio/vasp_input.py.


------------------------------------------------------------------------------------------------------
Line 1293 of /io/vaspio/vasp_input.py
------------------------------------------------------------------------------------------------------

search_lines = re.search(r"(?s)(parameters from PSCTR are:"r".*?END of PSCTR-controll parameters)",data).group(1)



It's a pretty recent update too, and I am not entirely sure if this is a bug or has 
something to do with the version difference of python (2.7.9)  or virtualenv (12.0.2)
on Hopper. I will try reverting pymatgen and pymatgen-db to see if it makes any difference.

(The codes are compiled on Carver, running on Hopper, with only MPworks reverted back, 
all others are current)

------------------------------------------------------------------------------------------------------
Associated Git Commit (6381734)
------------------------------------------------------------------------------------------------------


Head # : 6381734 (new storage and hashing for potcar), 01/23/2015

------------------------------------------------------------------------------------------------------
Error ('Group' Attribute Doesn't Exist)
------------------------------------------------------------------------------------------------------

(admin_env)-bash-3.2$ cat FW_job.error 

INFO:custodian.custodian:Run started at 2015-04-17 15:41:38.085128 in /global/scratch2/sd/harklee/vw/bi/base/block_2015-04-16-22-18-26-231809/launcher_2015-04-17-05-18-52-145180.

INFO:custodian.custodian:Custodian running on Python version 2.7.9 (default, Mar  3 2015, 11:05:52)  [GCC 4.9.2 20141030 (Cray Inc.)]

INFO:custodian.custodian:Starting job no. 1 (VaspJob) attempt no. 1. Errors thus far = 0.

INFO:custodian.custodian:Logging to custodian.json...

INFO:custodian.custodian:Run ended at 2015-04-17 15:58:04.547083.

INFO:custodian.custodian:Run completed. Total time taken = 0:16:26.461955.

Traceback (most recent call last):

  File "/global/u1/h/harklee/hark_vasp/codes/fireworks/fireworks/core/rocket.py", line 202, in run

    m_action = t.run_task(my_spec)

  File "/global/u1/h/harklee/hark_vasp/codes/MPWorks/mpworks/firetasks/custodian_task.py", line 105, in run_task

    custodian_out = c.run()

  File "/global/u1/h/harklee/hark_vasp/codes/custodian/custodian/custodian.py", line 212, in run

    self._run_job(job_n, job)

  File "/global/u1/h/harklee/hark_vasp/codes/custodian/custodian/custodian.py", line 260, in _run_job

    p = job.run()

  File "/global/u1/h/harklee/hark_vasp/codes/custodian/custodian/vasp/jobs.py", line 183, in run

    vi = VaspInput.from_directory(".")

  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/io/vaspio/vasp_input.py", line 1659, in from_directory

    sub_d[fname.lower()] = ftype.from_file(fullzpath)

  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/io/vaspio/vasp_input.py", line 1513, in from_file

    single = PotcarSingle(p)

  File "/global/u1/h/harklee/hark_vasp/virtenv_hark_vasp/lib/python2.7/site-packages/monty-0.6.4-py2.7.egg/monty/design_patterns.py", line 94, in __new__

    inst = klass(*args, **kwargs)

  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/io/vaspio/vasp_input.py", line 1293, in __init__

    data).group(1)

AttributeError: 'NoneType' object has no attribute 'group'

INFO:rocket.launcher:Rocket finished


(2) SSL error  [SSL: TLSV1_UNRECOGNIZED_NAME] 

This is still unsolved, but less of a priority at this point.

I contacted Shreyas about this, he asked me to submit an official ticket, 
which I will do over the weekend or sometime next week. I get the error when 
I install everything from scratch on Hopper and run on Hopper. This happens
when running a python script to load structures through rest API.

------------------------------------------------------------------------------------------------------
SSL TLSV1 Error
------------------------------------------------------------------------------------------------------
hark_vasp environment
Traceback (most recent call last):
  File "mg_211_full.py", line 17, in <module>
    primitive = mpr.get_structure_by_material_id(i)
  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/matproj/rest.py", line 341, in get_structure_by_material_id
    data = self.get_data(material_id, prop=prop)
  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/matproj/rest.py", line 194, in get_data
    return self._make_request(sub_url)
  File "/global/u1/h/harklee/hark_vasp/codes/pymatgen/pymatgen/matproj/rest.py", line 145, in _make_request
    raise MPRestError(msg)
pymatgen.matproj.rest.MPRestError: [SSL: TLSV1_UNRECOGNIZED_NAME] unknown error (_ssl.c:581)



Thank you so much for all the help!


Hark

Anubhav Jain

unread,
May 12, 2015, 12:05:29 PM5/12/15
to matproj...@googlegroups.com, hark...@gmail.com
Hi Hark

I moved the import inside the function using it - hopefully this fixes the issue
Reply all
Reply to author
Forward
0 new messages