[Maya-Python] Subprocess and deadlock

[Marcus Ottosson]

Hi all,

I’m struggling to understand why my Maya session deadlocks when leaving a subprocess open whilst listening to its stdout in a thread.

The thread is deamonised, and without the thread then closing down Maya works well.

I’ve narrowed down the problem into this.

import sys
import time
import subprocess
import threading

child = """
import sys

while True:
    line = sys.stdin.readline()
    sys.stdout.write("child: Input: %s" % line)

"""

popen = subprocess.Popen(
    ["c:/python27/python.exe", "-u", "-c", child],
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE
)

def listen():
    def _listen(out):
        time.sleep(1)  # Give subprocess a second to launch
        print("parent: Listening..")
        for line in iter(out.readline, b""):
            sys.stdout.write(line)
        print("parent: Closing..")
        out.close()

    thread = threading.Thread(target=_listen, args=[popen.stdout])
    thread.daemon = True
    thread.start()

def send(text):
    popen.stdin.write(text + "\n")
    popen.stdin.flush()

listen()
send("hello")

This is on Windows, I haven’t tested other OSs yet, but would expect different results there. I suspect the issue is an open handle to stdout, but would still have expected the thread to forcefully exit and not make a fuzz.

Any ideas?

​

--

Marcus Ottosson
konstr...@gmail.com

[Marcus Ottosson]

Just to be clear, the problem is leaving this subprocess open (you'll get a new terminal window appear) when trying to close down Maya. Doing so causes Maya to hang.

--

Marcus Ottosson
konstr...@gmail.com

[Alok Gandhi]

Hi Marcus,

I am not sure if this will surely help, but you can set `close_fds=True` in subprocess.Popen(). This ensures the subprocess is spawned without inheriting any handles from the parent process. Again, I am not sure if this is the correct solution as I am not on windows and also do not have access to maya at this moment, but I think it is worth investigating. Here is a StackOverflow answer dealing with a similar problem:

http://stackoverflow.com/a/13593715

-Alok

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOA_eDnLSUr_v89jxS6Qa%3DUNEGc0eF%2B0ygHbqTo-E03Sgw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Alok Gandhi]

And for completeness also pass  creationflags=DETACHED_PROCESS in Popen().

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOA_eDnLSUr_v89jxS6Qa%3DUNEGc0eF%2B0ygHbqTo-E03Sgw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

--

[Marcus Ottosson]

Thanks Alok, I did experiment with both of those. Will try this again next.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPaTLMR37jwOsJSW9MenrX2i4KsrGbEtdyZBCdW1zKdWewhRng%40mail.gmail.com.

[Marcus Ottosson]

close_fds isn’t supported on Windows apparently.

# ValueError: close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr #

DETACHED_PROCESS didn’t work either unfortunately, but it was worth a shot. I feel it must be related to this somehow. I’ve tried CREATE_NO_WINDOW and CREATE_NEW_PROCESS_GROUP as well without avail.

Here’s.aspx) all possible creation flags for reference.

Any other ideas?

Best,
Marcus

​

[Christopher Crouzet]

Disclaimer: this might be totally unrelated to your problem! :)

A common source of deadlocks when using the `multiprocessing` module (or anything related to IPC) comes from calling receiving/sending functions that are blocking. Maybe `readline()` is one of these, meaning that it won't return until it receives something, which might never happen in your case after exiting Maya?

A common solution is to wrap the body of the process function (`_listen()` in your case) into an infinite loop and then receive any incoming message only if the inter-process connection is not empty. The infinite loop is exited when the process receives a sentinel/stop message (usually sent from the parent process) to tell it to quit.

Not sure if this can be applied to your code though.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOATTheY4DsCQbrLugUQxJgq7OSyNdHTUY%3Db%3DWvU1kZ_HQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Christopher Crouzet
https://christophercrouzet.com

[Marcus Ottosson]

Thanks Christopher.

Maybe readline() is one of these, meaning that it won’t return until it receives something, which might never happen in your case after exiting Maya?

That’s right, readline() blocks until something is passed, via send() in this case. The for line in iter(out.readline, b""): is an infinite loop in this case, but as with any infinite loop in a thread I would have expected it to break when the thread is shutdown.

For example, this works fine.

import sys
import time
import subprocess
import threading

def _listen():
    while True:
        print("Still running..")
        time.sleep(1)

thread = threading.Thread(target=_listen)
thread.daemon = True
thread.start()

Both cases also work well in Houdini, and standalone in a vanilla Python process. This symptoms seem highly Maya specific, though I suspect it’s only more sensitive to something I’m doing wrong. Such as shutting down a thread that is waiting to hear back from the OS about a read from a file.

What’s more, in Maya 2016+ Maya unfreezes as soon as I close the external terminal that appears. With 2015 it does not.

Mystery.

​

On 22 February 2017 at 17:00, Christopher Crouzet <christoph...@gmail.com> wrote:

Disclaimer: this might be totally unrelated to your problem! :)

A common source of deadlocks when using the `multiprocessing` module (or anything related to IPC) comes from calling receiving/sending functions that are blocking. Maybe `readline()` is one of these, meaning that it won't return until it receives something, which might never happen in your case after exiting Maya?

A common solution is to wrap the body of the process function (`_listen()` in your case) into an infinite loop and then receive any incoming message only if the inter-process connection is not empty. The infinite loop is exited when the process receives a sentinel/stop message (usually sent from the parent process) to tell it to quit.

Not sure if this can be applied to your code though.

On 22 February 2017 at 23:22, Marcus Ottosson <konstr...@gmail.com> wrote:

close_fds isn’t supported on Windows apparently.

# ValueError: close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr #

DETACHED_PROCESS didn’t work either unfortunately, but it was worth a shot. I feel it must be related to this somehow. I’ve tried CREATE_NO_WINDOW and CREATE_NEW_PROCESS_GROUP as well without avail.

Here’s.aspx) all possible creation flags for reference.

Any other ideas?

Best,
Marcus

​

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOATTheY4DsCQbrLugUQxJgq7OSyNdHTUY%3Db%3DWvU1kZ_HQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Christopher Crouzet
https://christophercrouzet.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CANuKW523MnX4g3AOATrFuqajJb6w4S1p2y5K2ATqd89Qwpfoaw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Marcus Ottosson]

Ok, I might have found a way to break this down further.

This also freezes Maya on exit, until I close the new console window.

import subprocess
import threading

child = 

"""\
import time

while True:
    time.sleep(1)
    print("child: Still running..")

"""

popen = subprocess.Popen(
    ["c:/python27/python.exe", "-u", "-c", child],
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE
)

What’s more, if I remove the stdout=PIPE and stdin=PIPE, all is well and nothing hangs.

Whyyy? :(

​

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CANuKW523MnX4g3AOATrFuqajJb6w4S1p2y5K2ATqd89Qwpfoaw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

--

Marcus Ottosson
konstr...@gmail.com

[Christopher Crouzet]

If I am not mistaken, when a parent process is trying to exit normally, that is with no exception or such forced termination, then it needs to wait for all of its child processes to be joined. If one of them is blocked, then it won't join.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CANuKW523MnX4g3AOATrFuqajJb6w4S1p2y5K2ATqd89Qwpfoaw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCzbHgea%3DDTpXB4Haf0q0z%3DDHUNAjAimZMHjae_KTh90A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

That sounds logical, but the theory doesn't hold when considering it works with Python standalone and with Houdini. :(​

[Christopher Crouzet]

It's common when doing multiprocessing to have things working when they shouldn't in theory (I remember stumbling on a few such examples on StackOverflow, and writing many myself). This can be due to the order in which operations are being executed on either processes, the size of the inter-process messages, and whatnot. This also makes debugging a pain because reproducing some issues can be based on luck. So if I were you, I wouldn't put too much weight on different behaviours coming from other softwares.

On 23 February 2017 at 00:27, Marcus Ottosson <konstr...@gmail.com> wrote:

That sounds logical, but the theory doesn't hold when considering it works with Python standalone and with Houdini. :(​

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBX--vbvpOzUDP%2BsiX04MpT-yBMaj6DXwao2sk%3DFz6nPw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

Here’s another one I fully expected to work; replacing PIPE with a regular file.

import subprocess
import threading
import tempfile

child = """\
import time

while True:
    time.sleep(1)
    print("child: Still running..")

"""

pipe = tempfile.TemporaryFile()
popen = subprocess.Popen(
    ["c:/python27/python.exe", "-u", "-c", child],
    stdout=pipe
)

My theory was that perhaps Maya has mutilated PIPE for it’s own benefit. But alas, still hangs..

​

On 22 February 2017 at 17:34, Christopher Crouzet <christoph...@gmail.com> wrote:

It's common when doing multiprocessing to have things working when they shouldn't in theory (I remember stumbling on a few such examples on StackOverflow, and writing many myself). This can be due to the order in which operations are being executed on either processes, the size of the inter-process messages, and whatnot. This also makes debugging a pain because reproducing some issues can be based on luck. So if I were you, I wouldn't put too much weight on different behaviours coming from other softwares.

On 23 February 2017 at 00:27, Marcus Ottosson <konstr...@gmail.com> wrote:

That sounds logical, but the theory doesn't hold when considering it works with Python standalone and with Houdini. :(​

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBX--vbvpOzUDP%2BsiX04MpT-yBMaj6DXwao2sk%3DFz6nPw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Christopher Crouzet
https://christophercrouzet.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CANuKW51e42hnVhzWTYGMuBy49pRxLQWyU6VSK_nQtno1vzUq2Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

I am not 100% sure on windows, but I will make a guess. It might be because when you are using a pipe you are having the child inherit the file descriptor (regardless of whether close_fds is used or works on windows). The child process is looping and writing on stdout (the inherited fd). When the parent process (Maya) closes it may want to wait for the child to terminate, but the child won't terminate because it has an open pipe and nothing has closed it. This would be completely independent from how you are using a daemon thread. It just means your main process wont wait for that thread to end, but processes are a different story. 

You may want to experimental with trying to ensure you can close the pipe. Does this have any different behaviour for you on Windows? It may not.

pipe = tempfile.TemporaryFile()

popen = subprocess.Popen(

    ["c:/python27/python.exe", "-u", "-c", child],

    stdout=pipe

)

pipe.close()

If not, it could also be a combination of that and how windows handles process groups (detach?). Maybe this SO answer might help on Windows in order to manage the child process when the parent terminates: http://stackoverflow.com/a/12942797/496445

As a workaround if it becomes to difficult to resolve, you could try avoiding the inheriting of file descriptors altogether and see if it works for you by using something like a named pipe, and passing the string name argument to the child so it can open for reading or create for writing. 

Justin

On Thu, Feb 23, 2017 at 6:37 AM Marcus Ottosson <konstr...@gmail.com> wrote:

Here’s another one I fully expected to work; replacing PIPE with a regular file.

import subprocess
import threading
import tempfile

child = """\
import time

while True:
    time.sleep(1)
    print("child: Still running..")

"""

pipe = tempfile.TemporaryFile()
popen = subprocess.Popen(
    ["c:/python27/python.exe", "-u", "-c", child],
    stdout=pipe
)

My theory was that perhaps Maya has mutilated PIPE for it’s own benefit. But alas, still hangs..

​

On 22 February 2017 at 17:34, Christopher Crouzet <christoph...@gmail.com> wrote:

It's common when doing multiprocessing to have things working when they shouldn't in theory (I remember stumbling on a few such examples on StackOverflow, and writing many myself). This can be due to the order in which operations are being executed on either processes, the size of the inter-process messages, and whatnot. This also makes debugging a pain because reproducing some issues can be based on luck. So if I were you, I wouldn't put too much weight on different behaviours coming from other softwares.

On 23 February 2017 at 00:27, Marcus Ottosson <konstr...@gmail.com> wrote:

That sounds logical, but the theory doesn't hold when considering it works with Python standalone and with Houdini. :(​

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBX--vbvpOzUDP%2BsiX04MpT-yBMaj6DXwao2sk%3DFz6nPw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Christopher Crouzet
https://christophercrouzet.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CANuKW51e42hnVhzWTYGMuBy49pRxLQWyU6VSK_nQtno1vzUq2Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODENpMQFw9U7Q7fugFmPsaFWR3jKjMfZas-pXf1saAw3g%40mail.gmail.com.

[Marcus Ottosson]

Thanks for this Justin.

I think this is far too deep of a rabbit hole for a problem this minor but take it there is no other way. Named pipes seem rather unheard of on Windows, so the other option of starting a subprocess via the Win32 API directly seems the better approach but this particular StackOverflow example requires an external and compiled library, namely pywin32 which is too much to bear.

The plan then is to ignore the problem for now and hope a solution makes itself known in the future.

Thanks everyone for your help!

​

[Marcus Ottosson]

I stopped looking for a global cross-platform solution and found this to solve the problem.

def on_application_quit():
    try:
        popen.kill()
    except OSError:
        # Already dead
        pass

cmds.scriptJob(
    event=["quitApplication", on_application_quit],
    protected=True
)

For the record, the API route did not.

def on_application_quit():
    try:
        popen.kill()
    except OSError:
        # Already dead
        pass

OpenMaya.MSceneMessage.addCallback(OpenMaya.MSceneMessage.kMayaExiting, on_application_quit)

Assuming this isn’t called until after whatever is deadlocking Maya.

​

--

Marcus Ottosson
konstr...@gmail.com