mayapy questions

235 views
Skip to first unread message

Christopher Evans

unread,
Jan 18, 2017, 11:29:45 PM1/18/17
to python_in...@googlegroups.com
I am making a freely available maya task server. If you'd like to help lemme know. Basically you send it maya files and tasks to run on them, like batching animations to FBX, validating files, publishing files, etc.

Anyway, I have been having a difficult time getting the stdout of a mayapy launched through an mp pool of apply_async processes. Anyone have experience with getting feedback out of mayapy?

--
CE

Justin Israel

unread,
Jan 19, 2017, 5:37:52 AM1/19/17
to python_in...@googlegroups.com

What kind of difficulty are you having? Could you be more specific about the problem? How are you attempting to capture stdout currently?

Justin


--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gwjiLCSJEb%3DS2-WdO9CyGMm8Ou8bZynqiza0F5kDEJ0cw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marcus Ottosson

unread,
Jan 19, 2017, 8:34:18 AM1/19/17
to python_in...@googlegroups.com

Anyone have experience with getting feedback out of mayapy?

Something like this?

$ mayapy -c "import sys;sys.stdout.write('Hello world!')"
Hello world!
​

Tony Barbieri

unread,
Jan 19, 2017, 12:33:30 PM1/19/17
to python_in...@googlegroups.com
Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBAp5CqQ-YTgw9nN97DmyWyyVBuY-D6HaPpJakssW8Fig%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Tony

Justin Israel

unread,
Jan 19, 2017, 1:22:30 PM1/19/17
to python_in...@googlegroups.com


On Fri, Jan 20, 2017, 6:33 AM Tony Barbieri <grea...@gmail.com> wrote:
Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes.


I've done probably one or both of those in the past as well, without an issue in collecting stdout for display. 

@Christopher, we probably need to see a small example of what you are doing, that isn't working as expected. 


On Thu, Jan 19, 2017 at 5:34 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

Anyone have experience with getting feedback out of mayapy?

Something like this?

$ mayapy -c "import sys;sys.stdout.write('Hello world!')"
Hello world!
​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBAp5CqQ-YTgw9nN97DmyWyyVBuY-D6HaPpJakssW8Fig%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Tony

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsSGvt2gMT-4sh4-RUo1Uo6iiJ02mUrAqQzjpD-f-zt5mQ%40mail.gmail.com.

Christopher Evans

unread,
Jan 19, 2017, 4:34:26 PM1/19/17
to python_in...@googlegroups.com
https://github.com/chrisevans3d/mayaTaskServer

It's on github here, I am no master of python, I have written one of these at a previous company, but I used RPYC for distributed computing and a 'task finished' signal coupled with a timeout to know when something crashed. I would now like to parse the stdout in realtime and also print it to each worker tab in the UI in realtime to give someone an idea why it crashed.

This is something every company needs, so I thought that this would be useful for people. If anyone wants access, let me know how to give people edit access on github, I am a github newb.

On Thu, Jan 19, 2017 at 1:22 PM, Justin Israel <justin...@gmail.com> wrote:


On Fri, Jan 20, 2017, 6:33 AM Tony Barbieri <grea...@gmail.com> wrote:
Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes.


I've done probably one or both of those in the past as well, without an issue in collecting stdout for display. 

@Christopher, we probably need to see a small example of what you are doing, that isn't working as expected. 
On Thu, Jan 19, 2017 at 5:34 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

Anyone have experience with getting feedback out of mayapy?

Something like this?

$ mayapy -c "import sys;sys.stdout.write('Hello world!')"
Hello world!
​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBAp5CqQ-YTgw9nN97DmyWyyVBuY-D6HaPpJakssW8Fig%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--
Tony

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsSGvt2gMT-4sh4-RUo1Uo6iiJ02mUrAqQzjpD-f-zt5mQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2idOnzBMfb6mF6V_7mDQdDexD7jD_f%2BRa_pLk8qO9oDw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
CE

Justin Israel

unread,
Jan 19, 2017, 7:17:40 PM1/19/17
to python_in...@googlegroups.com
On Fri, Jan 20, 2017 at 10:34 AM Christopher Evans <chris...@gmail.com> wrote:
https://github.com/chrisevans3d/mayaTaskServer

It's on github here, I am no master of python, I have written one of these at a previous company, but I used RPYC for distributed computing and a 'task finished' signal coupled with a timeout to know when something crashed. I would now like to parse the stdout in realtime and also print it to each worker tab in the UI in realtime to give someone an idea why it crashed.

Are you talking about this bit of code?

Currently you are calling Popen without piping the stdout/stderr anywhere so it would go to the console or default output location. I see you have commented out an attempt to pipe stdout, but you don't have any code that actually uses it. If you are looking to capture stdout as the application is running (that is, without waiting for mayapy to exit before getting the output) then something needs to be constantly reading from that pipe. Usually you can accomplish this in a reader thread. You can read from the stdout pipe in a thread until the process ends and the pipe closes.
 

This is something every company needs, so I thought that this would be useful for people.

From a glance at the readme, it looks like a very specific implementation of a render queue solution (Rush, Dr Queue, Qube, Smedge, ...). Is the difference here that the Maya process stays running instead of loading for each batch of tasks? I know Qube uses this approach for their custom support for Maya Batch tasks. When they schedule a task to a node they start a maya session and feed it work over the commandPort until the job has no more tasks.
 
If anyone wants access, let me know how to give people edit access on github, I am a github newb.

On Thu, Jan 19, 2017 at 1:22 PM, Justin Israel <justin...@gmail.com> wrote:


On Fri, Jan 20, 2017, 6:33 AM Tony Barbieri <grea...@gmail.com> wrote:
Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes.


I've done probably one or both of those in the past as well, without an issue in collecting stdout for display. 

@Christopher, we probably need to see a small example of what you are doing, that isn't working as expected. 
On Thu, Jan 19, 2017 at 5:34 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

Anyone have experience with getting feedback out of mayapy?

Something like this?

$ mayapy -c "import sys;sys.stdout.write('Hello world!')"
Hello world!
​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
--
Tony

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gyawRvEpVMzxzScJ6fXE%3DFZw9sfyFVkGG6BNrjppkEnjw%40mail.gmail.com.

Christopher Evans

unread,
Jan 19, 2017, 11:15:39 PM1/19/17
to python_in...@googlegroups.com
The idea wasn't really to render anything, but to send multiple headless mayas 'tasks', do many people use these render management solutions to do something like batch export 4000 fbx files from 3000 maya files? I thought they were always very specific to rendering.

I did try using pipe before, but it wasn't really working. I believe the issue is that I am running this popen from here:

On Thu, Jan 19, 2017 at 7:17 PM, Justin Israel <justin...@gmail.com> wrote:
On Fri, Jan 20, 2017 at 10:34 AM Christopher Evans <chris...@gmail.com> wrote:
https://github.com/chrisevans3d/mayaTaskServer

It's on github here, I am no master of python, I have written one of these at a previous company, but I used RPYC for distributed computing and a 'task finished' signal coupled with a timeout to know when something crashed. I would now like to parse the stdout in realtime and also print it to each worker tab in the UI in realtime to give someone an idea why it crashed.

Are you talking about this bit of code?

Currently you are calling Popen without piping the stdout/stderr anywhere so it would go to the console or default output location. I see you have commented out an attempt to pipe stdout, but you don't have any code that actually uses it. If you are looking to capture stdout as the application is running (that is, without waiting for mayapy to exit before getting the output) then something needs to be constantly reading from that pipe. Usually you can accomplish this in a reader thread. You can read from the stdout pipe in a thread until the process ends and the pipe closes.
 

This is something every company needs, so I thought that this would be useful for people.

From a glance at the readme, it looks like a very specific implementation of a render queue solution (Rush, Dr Queue, Qube, Smedge, ...). Is the difference here that the Maya process stays running instead of loading for each batch of tasks? I know Qube uses this approach for their custom support for Maya Batch tasks. When they schedule a task to a node they start a maya session and feed it work over the commandPort until the job has no more tasks.
 
If anyone wants access, let me know how to give people edit access on github, I am a github newb.

On Thu, Jan 19, 2017 at 1:22 PM, Justin Israel <justin...@gmail.com> wrote:


On Fri, Jan 20, 2017, 6:33 AM Tony Barbieri <grea...@gmail.com> wrote:
Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes.


I've done probably one or both of those in the past as well, without an issue in collecting stdout for display. 

@Christopher, we probably need to see a small example of what you are doing, that isn't working as expected. 
On Thu, Jan 19, 2017 at 5:34 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

Anyone have experience with getting feedback out of mayapy?

Something like this?

$ mayapy -c "import sys;sys.stdout.write('Hello world!')"
Hello world!
​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
--
Tony

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2idOnzBMfb6mF6V_7mDQdDexD7jD_f%2BRa_pLk8qO9oDw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
CE

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2_eX3SzcjVJpbYO6N%3D0MhkP4j-aSvggvWCzAhZcetamA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
CE

Justin Israel

unread,
Jan 20, 2017, 2:31:09 AM1/20/17
to python_in...@googlegroups.com


On Fri, Jan 20, 2017, 5:15 PM Christopher Evans <chris...@gmail.com> wrote:
The idea wasn't really to render anything, but to send multiple headless mayas 'tasks', do many people use these render management solutions to do something like batch export 4000 fbx files from 3000 maya files? I thought they were always very specific to rendering.

Ya, "render" is used loosely here as a term they usual call these VFX targeted distributed task queues. Obviously rendering is a key task, but really they are just systems designed to schedule and dispatch work to available nodes on a farm. We use them to do every imaginable type of pipeline operation that we don't want taking up resources on our local machines. Disk migrations, image conversions, publish processes, review system processing. So yea, running match operations on a Maya file would be a reasonable task to submit to a "render" farm. Keeping a long running Maya instance on a booked node is a special optimisation to avoid the overhead of starting it for each batch. We don't do it in our own render farm since its more complicated and tends to not always net a significant benefit over just standard task batching. 


I did try using pipe before, but it wasn't really working. I believe the issue is that I am running this popen from here:

What I think is happening that is giving you trouble is that your maya work happens in a forked process. Within that process you have no direct access back to what is happening in your main process. You can't consider any state really useful outside the arguments you have been passed to your target function. So when you run you maya process you need some way to ship the stdout/stderr back to the main process. You can look at multiprocessing.Pipe to accomplish this. You create it on the main process side and pass the write side of the pipe into the child process of the pool. Then in that child process function you can probably again pass that write pipe as the stdout for your subprocess mayapy. It should then write the stdout over the pipe and you can collect it somehow in your main process. Hope this gives you some direction?
 


On Thu, Jan 19, 2017 at 7:17 PM, Justin Israel <justin...@gmail.com> wrote:
On Fri, Jan 20, 2017 at 10:34 AM Christopher Evans <chris...@gmail.com> wrote:
https://github.com/chrisevans3d/mayaTaskServer

It's on github here, I am no master of python, I have written one of these at a previous company, but I used RPYC for distributed computing and a 'task finished' signal coupled with a timeout to know when something crashed. I would now like to parse the stdout in realtime and also print it to each worker tab in the UI in realtime to give someone an idea why it crashed.

Are you talking about this bit of code?

Currently you are calling Popen without piping the stdout/stderr anywhere so it would go to the console or default output location. I see you have commented out an attempt to pipe stdout, but you don't have any code that actually uses it. If you are looking to capture stdout as the application is running (that is, without waiting for mayapy to exit before getting the output) then something needs to be constantly reading from that pipe. Usually you can accomplish this in a reader thread. You can read from the stdout pipe in a thread until the process ends and the pipe closes.
 

This is something every company needs, so I thought that this would be useful for people.

From a glance at the readme, it looks like a very specific implementation of a render queue solution (Rush, Dr Queue, Qube, Smedge, ...). Is the difference here that the Maya process stays running instead of loading for each batch of tasks? I know Qube uses this approach for their custom support for Maya Batch tasks. When they schedule a task to a node they start a maya session and feed it work over the commandPort until the job has no more tasks.
 
If anyone wants access, let me know how to give people edit access on github, I am a github newb.

On Thu, Jan 19, 2017 at 1:22 PM, Justin Israel <justin...@gmail.com> wrote:


On Fri, Jan 20, 2017, 6:33 AM Tony Barbieri <grea...@gmail.com> wrote:
Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes.


I've done probably one or both of those in the past as well, without an issue in collecting stdout for display. 

@Christopher, we probably need to see a small example of what you are doing, that isn't working as expected. 
On Thu, Jan 19, 2017 at 5:34 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

Anyone have experience with getting feedback out of mayapy?

Something like this?

$ mayapy -c "import sys;sys.stdout.write('Hello world!')"
Hello world!
​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
--
Tony

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2idOnzBMfb6mF6V_7mDQdDexD7jD_f%2BRa_pLk8qO9oDw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
CE

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gwon2wX1j6p2TBhoTeZvggnCRfHY9_uwa-rjRzvG0NQ1Q%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages