Parallel Processing

41 views
Skip to first unread message

Nathan Thomas

unread,
Jun 19, 2018, 4:36:08 PM6/19/18
to RSGISLib Support
Hi Dan,

I have two directories of data (same number of files in each) that I would like to use with image math. However, I have over 1500 of them so would like to parallel the tasks. I have a server with up to 48 CPUs I can use. I read your blog post on GNU parallel, but noticed that this seemed to be for the command line only (which I used and it worked brilliantly - thanks!). Do you have any advice on parallel processing using rsgislib and python?

Cheers, Nathan

Daniel Clewley

unread,
Jun 19, 2018, 4:50:34 PM6/19/18
to Nathan Thomas, RSGISLib Support
Hi Nathan,


Unless the image maths you are applying is really complicated you’re probably going to hit the I/O limit of your disks before you max our all your cores so probably best not to use all 48. If you are on Linux the iotop command is a useful one (probably need to install and has to run as root).

Thanks,

Dan

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

Nathan Thomas

unread,
Jun 19, 2018, 5:42:35 PM6/19/18
to Daniel Clewley, RSGISLib Support

Thanks Dan,

I checked the blog, but obviously not well enough.

I think I'll easily hit the I/O limit before I Max the cores. I'm not sure it would be appreciated if I used all 48 anyway. I could probably manage this locally, but doing it in parallel will be beneficial.

Cheers, Nathan

On Jun 19, 2018 4:50 PM, "Daniel Clewley" <daniel....@gmail.com> wrote:
Hi Nathan,


Unless the image maths you are applying is really complicated you’re probably going to hit the I/O limit of your disks before you max our all your cores so probably best not to use all 48. If you are on Linux the iotop command is a useful one (probably need to install and has to run as root).

Thanks,

Dan

On 19 Jun 2018, at 21:36, Nathan Thomas <nmtho...@gmail.com> wrote:

Hi Dan,

I have two directories of data (same number of files in each) that I would like to use with image math. However, I have over 1500 of them so would like to parallel the tasks. I have a server with up to 48 CPUs I can use. I read your blog post on GNU parallel, but noticed that this seemed to be for the command line only (which I used and it worked brilliantly - thanks!). Do you have any advice on parallel processing using rsgislib and python?

Cheers, Nathan

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsubscribe@googlegroups.com.
To post to this group, send email to rsgislib-support@googlegroups.com.

Pete Bunting [pfb]

unread,
Jun 25, 2018, 6:34:31 AM6/25/18
to Nathan Thomas, Dan Clewley, RSGISLib Support

Hi Nathan,

 

This is info on this in the RSGISLib intro tutorial as well: https://www.rsgislib.org/documentation.html#rsgislib-introduction-training

 

Cheer,s Pete

 

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: pete.b...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148

* Website: https://www.remotesensing.info
****************************************************

"Please consider the environment before printing this email or any documents attached”

 

 

From: <rsgislib...@googlegroups.com> on behalf of Nathan Thomas <nmtho...@gmail.com>
Date: Tuesday, 19 June 2018 at 22:42
To: Dan Clewley <daniel....@gmail.com>
Cc: RSGISLib Support <rsgislib...@googlegroups.com>
Subject: Re: [rsgislib-support] Parallel Processing

 

Thanks Dan,

I checked the blog, but obviously not well enough.

I think I'll easily hit the I/O limit before I Max the cores. I'm not sure it would be appreciated if I used all 48 anyway. I could probably manage this locally, but doing it in parallel will be beneficial.

Cheers, Nathan

On Jun 19, 2018 4:50 PM, "Daniel Clewley" <daniel....@gmail.com> wrote:

Hi Nathan,

 

 

Unless the image maths you are applying is really complicated you’re probably going to hit the I/O limit of your disks before you max our all your cores so probably best not to use all 48. If you are on Linux the iotop command is a useful one (probably need to install and has to run as root).

 

Thanks,

 

Dan



On 19 Jun 2018, at 21:36, Nathan Thomas <nmtho...@gmail.com> wrote:

 

Hi Dan,

 

I have two directories of data (same number of files in each) that I would like to use with image math. However, I have over 1500 of them so would like to parallel the tasks. I have a server with up to 48 CPUs I can use. I read your blog post on GNU parallel, but noticed that this seemed to be for the command line only (which I used and it worked brilliantly - thanks!). Do you have any advice on parallel processing using rsgislib and python?

 

Cheers, Nathan

 

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.

 

--

You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.



--------------------------------------------------------------------
Prifysgol Aberystwyth www.aber.ac.uk
Prifysgol y Flwyddyn ar gyfer Ansawdd Dysgu - The Times & The Sunday Times 2018.

Aberystwyth University www.aber.ac.uk
University of the Year for Teaching Quality - The Times & The Sunday Times 2018.
Reply all
Reply to author
Forward
0 new messages