IndexError: list index out of range

[Sarah Ur]

Hi Mike,

I've recently started setting up cloud computing using your software and tutorial, but I'm having a bit of an issue with sending my jobs to the cloud. I'm currently trying to run a gctf job there, but get the following message:

Traceback (most recent call last):

  File "relion/qsub_aws", line 4584, in <module>

  File "relion/qsub_aws", line 2817, in relion_run_ctffind

  File "relion/qsub_aws", line 134, in rclone_to_s3_mics

IndexError: list index out of range

Failed to execute script qsub_aws

In the relion GUI, the job gets stuck in the running jobs panel and I see this output:

"Starting relion job in the cloud...

Started micrograph upload on Wed Jan 17 "

What have I done wrong, do you have any suggestions for how I can fix it?

In my bumbling around, I've tried switching keypairs/regions in the aws_init.sh file and also used two different computers in our lab (both Mac OSX) but I still get the same error.

Thanks for your help!!

-Sarah

[Michael Cianfrocco]

Hi Sarah, 

Hmm - interesting error! 

What happens if you type: 

$ awsls 

[Sarah Ur]

I get this output:

AWS EC2 information for user Sarah_Ohio in region us-east-2

/bin/sh: aws: command not found

----------------------------------------------------------------------------------------------------

InstanceType Avail. Zone InstanceID Status User Login info

----------------------------------------------------------------------------------------------------

No instances found

/bin/sh: aws: command not found

----------------------------------------------------------------------------------------------------------------------------------------------------

SpotInstanceType Avail. Zone SpotInstanceID SpotStatus InstanceID Status Price User Login info

----------------------------------------------------------------------------------------------------------------------------------------------------

No spot instances found

/bin/sh: aws: command not found

----------------------------------------------------------------------------------------------------------

Volume ID Description Avail. Zone Size User Status Instance

----------------------------------------------------------------------------------------------------------

No volumes found

/bin/sh: aws: command not found

------------------------------------------------------------------------------------------------------------------------------------

Snapshot ID Description User Status Volume ID Size Start time

------------------------------------------------------------------------------------------------------------------------------------

No snapshots found

[Michael Cianfrocco]

Hi Sarah,

This explains it - you are missing AWS CLI from your environment. 

Did you install this software? 

$ pip install awscli 

Mike

[sarahn...@gmail.com]

Hey Mike,

Yes I installed it using the pip command. But I just uninstalled and reinstalled. And now see a bunch of errors. It's possible I just blew past these the first time. That was unfortunate of me, as they are literally in red. 

This is the output when I install now. Do you have any insight for how I can get around this? Thanks again for your help troubleshooting!!

-Sarah

Collecting awscli

  Downloading awscli-1.14.29-py2.py3-none-any.whl (1.2MB)

    100% |████████████████████████████████| 1.2MB 632kB/s 

Requirement already satisfied: docutils>=0.10 in /Library/Python/2.7/site-packages (from awscli)

Requirement already satisfied: colorama<=0.3.7,>=0.2.5 in ./Library/Python/2.7/lib/python/site-packages (from awscli)

Requirement already satisfied: s3transfer<0.2.0,>=0.1.12 in ./Library/Python/2.7/lib/python/site-packages (from awscli)

Collecting botocore==1.8.33 (from awscli)

  Downloading botocore-1.8.33-py2.py3-none-any.whl (4.0MB)

    100% |████████████████████████████████| 4.1MB 302kB/s 

Requirement already satisfied: PyYAML<=3.12,>=3.10 in ./Library/Python/2.7/lib/python/site-packages (from awscli)

Requirement already satisfied: rsa<=3.5.0,>=3.1.2 in ./Library/Python/2.7/lib/python/site-packages (from awscli)

Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in ./Library/Python/2.7/lib/python/site-packages (from s3transfer<0.2.0,>=0.1.12->awscli)

Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.8.33->awscli)

  Using cached python_dateutil-2.6.1-py2.py3-none-any.whl

Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in ./Library/Python/2.7/lib/python/site-packages (from botocore==1.8.33->awscli)

Requirement already satisfied: pyasn1>=0.1.3 in ./Library/Python/2.7/lib/python/site-packages (from rsa<=3.5.0,>=3.1.2->awscli)

Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.8.33->awscli)

  Using cached six-1.11.0-py2.py3-none-any.whl

Installing collected packages: six, python-dateutil, botocore, awscli

  Found existing installation: six 1.4.1

    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

    Uninstalling six-1.4.1:

Exception:

Traceback (most recent call last):

  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main

    status = self.run(options, args)

  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run

    prefix=options.prefix_path,

  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 778, in install

    requirement.uninstall(auto_confirm=True)

  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 754, in uninstall

    paths_to_remove.remove(auto_confirm)

  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove

    renames(path, new_path)

  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 267, in renames

    shutil.move(old, new)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move

    copy2(src, real_dst)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2

    copystat(src, dst)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat

    os.chflags(dst, st.st_flags)

OSError: [Errno 1] Operation not permitted: '/var/folders/0m/54gzn9314k983bbn7jzgwdh40000gq/T/pip-l1CRDD-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

[Michael Cianfrocco]

Hi Sarah, 

This looks like a permissions issue - do you have administrative rights? If so, try typing: 

$ sudo pip install awscli

Otherwise, you can also try: 

$ pip install awscli --user

[Sarah Ur]

Hi Mike,

My user setting is admin.. But after $ pip install awscli --user. It installed successfully. I still get the same error when running qsub_aws and for the awsls readout (see below). Do you have any other ideas as to what it might be? In the meantime I think I'll just start the process again with my own laptop.

Traceback (most recent call last):

  File "relion/qsub_aws", line 4584, in <module>

  File "relion/qsub_aws", line 2817, in relion_run_ctffind

  File "relion/qsub_aws", line 134, in rclone_to_s3_mics

IndexError: list index out of range

Failed to execute script qsub_aws

[Michael Cianfrocco]

Hi Sarah,

It looks like that, while you have installed AWS CLI, it's still not in your environment shell. 

For instance, it's suppose to install the software into your /usr/local/bin directory, such as /usr/local/bin/aws. 

If you type 'aws', what do you see?

[sarahn...@gmail.com]

Hi Mike,

macintosh:~ Sarah$ aws

-bash: aws: command not found

And it is not in my usr/local/bin directory.

[Michael Cianfrocco]

Thanks for sending that - it shows that it still can't find the AWS CLI software. 

Could send along the output for: 

$ pip show awscli

[Sarah Ur]

The output is:

Name: awscli

Version: 1.14.29

Summary: Universal Command Line Environment for AWS.

Home-page: http://aws.amazon.com/cli/

Author: Amazon Web Services

Author-email: UNKNOWN

License: Apache License 2.0

Location: /Users/Sarah/Library/Python/2.7/lib/python/site-packages

Requires: docutils, colorama, s3transfer, botocore, PyYAML, rsa

[Michael Cianfrocco]

Hi Sarah,

I'm sorry but it's hard for me to figure out what is happening on your shell. Clearly you have awscli installed, but, at the same time, you do not have the software in your path. This means that you cannot use any of the awscli commands that are used by my software.

Whenever I install awscli, it automatically puts the aws commands into my $PATH, but yours has not done that and it's hard for me to troubleshoot. I'll keep looking around to see how to address this.

Mike