Jug throwing error on PBS based HPC cluster

[Jaideep Reddy Gedi]

Here is the error, I've just run the prime numbers example, can somebody look into this.

jug execute "/home/jaideep.gedi.17cse/qgis/test.py" | qsub -t 1-100

qsub: invalid option -- 't'

usage: qsub [-a date_time] [-A account_string] [-c interval]

        [-C directive_prefix] [-e path] [-f ] [-h ] [-I [-X]] [-j oe|eo] [-J X-Y[:Z]]

        [-k o|e|oe] [-l resource_list] [-m mail_options] [-M user_list]

        [-N jobname] [-o path] [-p priority] [-P project] [-q queue] [-r y|n]

        [-S path] [-u user_list] [-W otherattributes=value...]

        [-v variable_list] [-V ] [-z] [script | -- command [arg1 ...]]

       qsub --version

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>

BrokenPipeError: [Errno 32] Broken pipe

Thanks in Advance :)

[Luis Pedro Coelho]

Dear Jaideep,

> Here is the error, I've just run the prime numbers example, can somebody look into this.
>
> jug execute "/home/jaideep.gedi.17cse/qgis/test.py" | qsub -t 1-100

1. This is not how you should run jug, in general.

You can either look into https://pypi.org/project/gridjug/ or https://pypi.org/project/jug-schedule/ which directly interact with the grid engine or do it old-school:

create a script

echo jug execute "/home/jaideep.gedi.17cse/qgis/test.py" > run-jug.sh
qsub ./run-jug.sh

2. Apparently, your version of qsub does not have the `-t` option. This is something for your sysadmin, or you can do it old-school, again:

for i in $(seq 100); do
qsub ./run-jug.sh
done

HTH
Luis

PS: This is a semi-moderated list: every email that comes from an account that has not sent email before is held until I approve the account. It does introduce a bit of a delay for first time posters, but it also means effectively zero spam unlike in a fully un-moderated list. Now, you should be able to post directly (assuming you use the same email address).



Luis Pedro Coelho | Fudan University | http://luispedro.org
https://orcid.org/0000-0002-9280-7885


>
>
> Thanks in Advance :)
>

> --
> You received this message because you are subscribed to the Google Groups "jug-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jug-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jug-users/77674566-6a97-4f11-ada5-2f97dacd723co%40googlegroups.com <https://groups.google.com/d/msgid/jug-users/77674566-6a97-4f11-ada5-2f97dacd723co%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Jaideep Reddy Gedi]

 Thank you for response!

Can you also let me know how I can run the same thing through python console?
I have to use Jug in Qgis.

so I'm currently executing jug like this:

from subprocess import Popen, PIPE
import shlex
process = Popen(shlex.split("jug execute '/home/jaideep.gedi.17cse/qgis/test.py'"), shell=False, universal_newlines=True,stdin=PIPE, stdout=PIPE, stderr=PIPE )
output, error = process.communicate();
print (output);

As qgis only provide its console, to handle the qgis project via python. So this is the only option I have which is running through the console.

Please suggest some way!

[Luis Pedro Coelho]

Can you also let me know how I can run the same thing through python console?

I have to use Jug in Qgis.

so I'm currently executing jug like this:

from subprocess import Popen, PIPE
import shlex
process = Popen(shlex.split("jug execute '/home/jaideep.gedi.17cse/qgis/test.py'"), shell=False, universal_newlines=True,stdin=PIPE, stdout=PIPE, stderr=PIPE )
output, error = process.communicate();
print (output);

That should work, but so should the following:

> import jug

> jug.jug.main(['execute', '/home/jaideep.gedi.17cse/qgis/test.py'])

I say should because in v2.0.0, it actually didn't work, but it does in v2.0.1 (released 7 seconds ago!)

HTH,

Luis

[Luis Pedro Coelho]

Ok, v2.0.1 introduced a silly bug. v2.0.2 fixed it and there are now regression tests.

Best

Luis

Luis Pedro Coelho | Fudan University | http://luispedro.org

https://orcid.org/0000-0002-9280-7885

--

You received this message because you are subscribed to the Google Groups "jug-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jug-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jug-users/ffd4d832-1876-4b64-820a-7ae62b070d30%40www.fastmail.com.

[Jaideep Reddy Gedi]

That should work, but so should the following:

import jug

jug.jug.main(['execute', '/home/jaideep.gedi.17cse/qgis/test.py'])

Does the above snippet also, creates job array and uses all nodes in the cluster?

I have to use all nodes in cluster, but i have only option which is python console. 

[Luis Pedro Coelho]

This will be equivalent to running "jug execute" on the command line. Nothing more.

I am not sure how your cluster is set up. I think you need to discuss this with your cluster admins.

Best

Luis

Luis Pedro Coelho | Fudan University | http://luispedro.org

https://orcid.org/0000-0002-9280-7885

--

You received this message because you are subscribed to the Google Groups "jug-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jug-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jug-users/2c820be0-803d-401a-9641-9e33c89232c1o%40googlegroups.com.