mulitprocessing

The BigB

unread,

Nov 27, 2012, 4:14:10 PM11/27/12

to geop...@googlegroups.com

Just was trying out multiprocessing for exporting subsets of many featureclasses. Does anyone have experience with this? Something I would like to do is multiprocess a function with multiple variables. multiprocessing.Pool.map does not allow for this. It would be cleaner to have the clipFeature, and output directory as variables for clpFc2Shp. Here is my script example where the Feature dataset has 20 featureclasses for export. Yes I know this will spawn as many processes as there are processors :<|

import arcpy, os, multiprocessing, re

def clpFc2Shp(fc):

#export fc to output directory

print "Clipping: " + fc

print "\tto " + fc + ".shp"

inFD = r"T:\Base_Data.gdb\Base_Map_20K"

arcpy.env.workspace = inFD

clpFeature = r"T:\KB_REGION.shp"

outFolder = r"T:\shpOut"

outShp = os.path.join(outFolder,fc + ".shp")

try:

if not arcpy.Exists(outShp):

arcpy.Clip_analysis(fc,clpFeature,outShp)

print "\t...done"

except:

print "\t...failed clp"

def main():

#Creates a pool class and runs the # of clips equal to the # of featureclasses.

inFD = r"T:\Base_Data.gdb\Base_Map_20K"

arcpy.env.workspace = inFD

fcList = arcpy.ListFeatureClasses()

pool = multiprocessing.Pool()

pool.map(clpFc2Shp,fcList)

pool.close()

pool.join()

if __name__ =='__main__':

main()

The BigB

unread,

Nov 27, 2012, 4:15:32 PM11/27/12

to geop...@googlegroups.com

Also I noticed that each process sticks around until all processes are done. There must be a way to clean these up to free up memory.

Thanks.

tarkenton

unread,

Dec 7, 2012, 12:07:22 PM12/7/12

to geop...@googlegroups.com

Sorry Will, I've looked at the multiprocessing module a while back but never had time get it working. Keep me posted if you come up with anything. Maybe put a little page together on how to if you do figure it out!

Cheers

Kevin

Will

unread,

Dec 7, 2012, 10:58:52 PM12/7/12

to geop...@googlegroups.com

Thanks Kevin. This works really quickly but it is just not right having to hard code the parameters into the function. I have seen a couple methods to fix this just have not had time to implement. I think this would work really well whenever processing by units or parts. Best done when you won't kill a shared machine :)

WB