Multiprocessing Returns New Maya Instance

[Ruchit Bhatt]

Hi,

I tried multiprocessing module in maya python with basic example and every time it returns new maya instance plus

object of multiprocessing.Queue() returns nothing on object.get()
is python interpreter is not a separate process in Maya ?
please share efficient example on how to deal with multiprocessing in Maya.
Thank you

[Marcus Ottosson]

The Python interpreter and Maya run in the same process, so this is expected.

If your multiprocessing doesn't require access to your current session, then you may consider instead launching mayapy explicitly (either from or outside of Maya), and from there use the multiprocessing module. That would create additional instances of mayapy instead which may be what you want.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/ca05daed-3c24-43de-829c-9ca8aa2f7725%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Justin Israel]

On Tue, Jul 17, 2018 at 7:03 PM Marcus Ottosson <konstr...@gmail.com> wrote:

The Python interpreter and Maya run in the same process, so this is expected.

If your multiprocessing doesn't require access to your current session, then you may consider instead launching mayapy explicitly (either from or outside of Maya), and from there use the multiprocessing module. That would create additional instances of mayapy instead which may be what you want.

On 17 July 2018 at 07:43, Ruchit Bhatt <ruchitinn...@gmail.com> wrote:

Hi,

I tried multiprocessing module in maya python with basic example and every time it returns new maya instance plus

object of multiprocessing.Queue() returns nothing on object.get()
is python interpreter is not a separate process in Maya ?
please share efficient example on how to deal with multiprocessing in Maya.
Thank you

You should be able to follow this example:
https://pymotw.com/2/multiprocessing/communication.html#passing-messages-to-processes

The queue has to be passed to the child process as an arg, because it will be a completely separate process with its own memory.

One other thing to note... be careful when choosing to use multiprocessing within Maya. Starting a Process() results in a fork of the current Maya process, which means if you have a huge scene open, you are duplicating that memory in your child process. It can also result in deadlocks because of the heavily threaded environment in Maya, and locks that were acquired may not be released in the new child process (at least this is true on osx/linux).

If you are set on using multiprocessing, try to create a pool very early before any memory is allocated in your Maya session. Or better yet, go the route Marcus suggested and launch a mayapy in a subprocess.

Justin

 

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/ca05daed-3c24-43de-829c-9ca8aa2f7725%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCwVF05XD4aYq9Bmr94-jpHmPB526AXLxwb%3D5HwXRgvnQ%40mail.gmail.com.

[Ruchit Bhatt]

This way ?

import platform

import multiprocessing

if platform.system() == 'Windows':
    multiprocessing.set_executable("C:/Program    Files/Autodesk/Maya201x/bin/mayapy.exe")
elif platform.system() == 'Linux':
    multiprocessing.set_executable('/usr/Autodesk/maya201x/bin/mayapy')

[Justin Israel]

Well no. The suggestion was to use the subprocess module to avoid forking the potentially large Maya process and possibly hitting deadlocks. 

I haven't had to use set_executable so I am not sure if that is necessary or not. 

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/cacf4c80-cacc-4593-857a-509782ba7a87%40googlegroups.com.

[Ruchit Bhatt]

I tried your example & its working fine. Thanks

In script Editor

import subprocess
import cPickle

mayapy_path = r'C:/Program Files/Autodesk/Maya201x/bin/mayapy.exe'
script_path = r'E:/multi_test.py'

proc_obj = subprocess.Popen(mayapy_path + ' ' + script_path + ' -po', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result = cPickle.load(proc_obj.stdout)
print result

multi_test.py

import multiprocessing
from time import time
import cPickle
import sys

## Simple func to eat up cpu power.
def whileFunc(z):
    while z < 100000:
        z += 1
    return z

if __name__ == "__main__":
    ## Get current time
    currtime = time()

    ## How often to run (just a test value)
    N = 10000
    ## Just a list with 1s
    myList = [1]*N

    nrOfProcessors = multiprocessing.cpu_count()

    ## Set our pool of processors
    po = multiprocessing.Pool(nrOfProcessors)
   
    ## create the threads
    res = po.map_async(whileFunc, myList)
   
    ## If we pass a -po flag, pickle the output and write it out    
    if '-po' in sys.argv[1:]:
        results = len(res.get())
        cPickle.dump(results, sys.stdout, -1)
        sys.stdout.flush()
        sys.exit(0)

    print 'This value below should be a 1000:'
    print len(res.get())
    print 'time elapsed:', time() - currtime

[Justin Israel]

Awesome. Glad that it worked with subprocess. One more suggestion. You might want to try calling it like this :

proc_obj = subprocess.Popen([mayapy_path, script_path, ' -po'] , stdout=subprocess.PIPE, stderr=subprocess.PIPE)

out, err = proc_obj.communicate()

result = cPickle.load(out)

If you don't need to send stdin to your process, no need to create a pipe. And you should use communicate to avoid a deadlock of your child process produces lots of stdout and stderr. 

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/2f4122d0-ff64-4ba9-8bfa-71e7bad7f0f7%40googlegroups.com.

[Ruchit Bhatt]

On Wednesday, July 18, 2018 at 1:45:39 PM UTC+5:30, Justin Israel wrote:

If you don't need to send stdin to your process, no need to create a pipe. And you should use communicate to avoid a deadlock of your child process produces lots of stdout and stderr. 

Thanks for this tips, i was not aware of it.