[Maya-Python] Export scene to memory

[Marcus Ottosson]

Hi all,

I’m looking for a way to export similar to cmds.file(exportSelected=True) but have the result of that export remain in memory, so that I can parse the output.

I’d like to go from this.

1. Export ascii to disk
2. Read ascii from disk
3. Parse ascii

To this.

1. Export ascii
2. Parse ascii

It isn’t solving any burning issue at the moment, it’s more of a curious enquiry. :) Any techniques are ok; no limits.

Any ideas?

​

--

Marcus Ottosson
konstr...@gmail.com

[Joe Weidenbach]

Well, I don't know that it helps much for your particular situation, but you might use the tempfile system in Python to get a file handle and save to that instead.  It still goes to a file, but that file is a temporary file that will get deleted later.

As far as I can tell, there's no way to intercept the raw string of the cmds.file operation before a save occurs.  There is the MFileIO class in the API, but that only gives you access to a limited subset of what's available in cmds.file, so I don't see that as being a real help.

Now, depending on what you're trying to do, you *could* simply parse the scene data itself, and store that in memory, and then you could do whatever you want with it.  At that point you're basically writing a custom exporter that could output to a string, or whatever you want to do with it.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODq-yy0Dho8x-uUomgvRJv7epsXcYskqg7yGq0G6DdewA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

---

This email is free from viruses and malware because avast! Antivirus protection is active.

[Marcus Ottosson]

Thanks Joe!​

Now, depending on what you’re trying to do, you could simply parse the scene data itself, and store that in memory, and then you could do whatever you want with it

Yes, this is essentially what I’m looking to do, but to rely on the export mechanism as opposed to making calls to cmds.ls, cmds.listAttr and cmds.getAttr for performance reasons, as they take quite the amount of time to run/query, whereas an export is (exponentially?) faster.

So the reason for looking to bypass the file-system in this case was simply to avoid the cost of de-touring the hard-disk and go directly from serialising to parsing.

I had a quick look at writing a custom exporter/translator. After just a quick read, it did look rather hard-coded that the end result should be a file so I’m not sure that’s the tree I should be barking up on.

Would anyone have enough experience writing a file translator to know whether it’s possible to translate to memory instead?

Alternatively, I also had an idea of perhaps mapping a portion of memory into a file-path, and then simply exporting to that. Basically, the inverse of a memory-mapped file.

• http://en.wikipedia.org/wiki/Memory-mapped_file

In which the file-path would actually be pointing to a location in memory, as opposed to on disk, if that makes sense?

I should also note that not all data is relevant; I’m only looking to serialise names of nodes, along with their attributes and values that are of type string and number.

Here’s a performance test for a cmds.getAttr approach, along with some results.
https://gist.github.com/mottosso/e2d33de7f22f5370aa02

Test A
    Number of nodes: 288
    Number of attributes: 48371
    Time taken to generate instance: 1.64s
    Size of instance: 2.536 kb
    Size of compressed instance: 0.73 kb

Test B
    Number of nodes: 574
    Number of attributes: 87591
    Time taken to generate instance: 3.04s
    Size of instance: 4.848 kb
    Size of compressed instance: 1.346 kb

Whereas the time it takes to export, which includes polygonal data, meaning it could potentially be optimised further.

Test A
    Time taken, including writing to disk (SSD): 0.078s
    Size of instance: 223.00 kb

Test B
    Time taken, including writing to disk (SSD): 0.295s
    Size of instance: 1940.00 kb

Note that my use of instance in this case is analogous to output.

Best,
Marcus

​

[Justin Israel]

Ya that would be an actual memory mapped file (not an inverse of it). A memory mapped file is exactly how you described it. On posix filesystem there is usually also some kind of a tmpfs virtual filesystem available, which acts the same way, on top of mmap. You make a file using the same standard file system commands but it isn't a block device, rather a memory backed file. The closest thing on Windows is called ram disks but I think you have to explicitly mount those yourself as opposed to one usually being available.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAgzwxNWeT-zrxcXZqDbxmr1GWEsaQhjFaDT7jO%3D9Z8UA%40mail.gmail.com.

[Marcus Ottosson]

Aah, I see. I read it the other way around, basically this..

Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.

..sounded to me like virtual memory; in which memory is dumped to disk yet treated as memory, only slower.

Yeah, I’d imagine it being tricky to set-up, but if it does it’s thing “as advertised” then it could possibly be a way forward.

​

[Justin Israel]

Also, I did a quick test using a named pipe, but it doesn't work since I think Maya writes out its file to a temp area and the moves it into place,  as opposed to opening the filepath you give it and writing to it. That would end up preventing you from using pipes or memory mapped files (I think).
I've never done a file translator plugin, but if it give you control of the actual writting process and not just telling it a path that Maya handles, then maybe it would work correctly since you can open the existing path.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

[Marcus Ottosson]

Ah, shoot. That is not surprising but would make it a dead end I suppose.

I had a quick glance at this which looked promising.
http://pymotw.com/2/mmap/

But yeah, if that’s what it does then it probably won’t work any better.

Maybe this temporary path can be overridden?

​

[Justin Israel]

Right ya that's what I wondered about the file translator, having never written one.
Like you found, the mmap functionality is all there in python, but it may be a blocked solution by however Maya's file write process is limited.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODDcKppPCWfBwiD8HPPxTL2Lnu8PL3ukVwS%3D%2BjtugjH0w%40mail.gmail.com.

[Marcus Ottosson]

Also, I did a quick test using a named pipe, but it doesn’t work since I think Maya writes out its file to a temp area and the moves it into place, as opposed to opening the filepath you give it and writing to it

I had a go with this, just monitoring which files were being accessed during saving a big file, and on my end it does look like it is directly writing to the given path.

Are you able to figure out where your temporary area was? Could it have been a Linux-thing, or perhaps something to do with the way you were writing it?

​

[Justin Israel]

My original named pipe test was on OSX, but I did an strace, on my linux workstation, on the Maya process to see what is happening when it saves the scene. This seems to explain the behaviour:

# Scene saved to:  /tmp/test.ma
stat("/tmp", ...)
...
lstat("tempUIFilezofrPP", ...)
access("/tmp/justin/tempUIFilezofrPP", ...)
open("/tmp/justin/tempUIFilezofrPP", O_WRONLY|O_CREAT|O_TRUNC, 0666)
stat("/tmp/justin/tempUIFilezofrPP", ...)
chmod("/tmp/justin/tempUIFilezofrPP", 0664)
open("/tmp/justin/tempUIFilezofrPP", O_RDONLY)
unlink("/tmp/justin/tempUIFilezofrPP")
...
access("/tmp/test.ma", F_OK)
access("/tmp/test.ma", W_OK)
access("/tmp/test.ma", F_OK)
lstat("/tmp/test.ma.matu7GqL", ...)
rename("/tmp/test.ma", "/tmp/test.ma.matu7GqL")
open("/tmp/test.ma", O_WRONLY|O_CREAT|O_TRUNC, 0666)
stat("/tmp/test.ma.matu7GqL", ...)
chmod("/tmp/test.ma", 0664)
unlink("/tmp/test.ma.matu7GqL")

​

So what Maya does is it moves the given path to a temp name if it already exists, then writes the new scene to your given path, and if all succeeds, it removes the previously existing one. I don't know what it is doing with that first temp file that it uses. Not sure what gets written there, but the location is based on your tmp location. The rename all happens in the same location of the specified filepath. So while it won't work for solutions where you expect the file to be created ahead of time and directly written to, this does seem to work... On my linux box, by default, I have "/dev/shm" which is the shared memory tmpfs device. I can save a file to "/dev/shm/test.ma", and all of that takes place on the tmpfs device, as opposed to needing to work with a single file. Although this isn't really a portable solution.

All this academic stuff aside, are you pretty confident that your bottleneck is disk IO on the scene file? I always thought Maya spent more time on its actual export/import logic than the time it really takes to read/write a sizeable scene file. 

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCO2BWdSu7S12y1%3DLRgzZ-DGWTBpfMO0i4SGXNG7dzv2A%40mail.gmail.com.

[Marcus Ottosson]

Thanks for trying this out Justin. I’ll address this one first in case it affects your interest in the other answers.

All this academic stuff aside, are you pretty confident that your bottleneck is disk IO on the scene file?

No, it certainly isn’t the bottleneck, I though that much was clear from the first tests I posted in which exporting was many times faster, even with writing to disk. This is out of plain curiosity only.

So what Maya does is it moves the given path to a temp name if it already exists, then writes the new scene to your given path, and if all succeeds, it removes the previously existing one.

Would it be possible that it does things differently, when writing to a non-existing file? In my case, it’s safe to assume the path will always be different and non-existing. Or were you referring to the fact that an mmap file must be written to disk before being manipulated?

Here’s my limited understanding.

import mmap
import contextlib

cmds.file(rename="temp.ma")  # Does not exist
path = cmds.file(sceneName=True, q=True)

# This has to happen, which makes it exist, which may cause the behaviour you described.
with open(path, "wb") as f:
    f.write("empty")

with open(path, "r+b") as f:
    with contextlib.closing(mmap.mmap(f.fileno(), 0)) as mm:    
        cmds.file(exportAll=True, force=True)

In which case, cmds.file is attempting to open a new file handle via the path given; but it won’t work (obviously), and I’m getting a “Can’t write to …” error.

I also had a look at an example file translator, and it looks like this is it’s given arguments.

def writer(self, fileObject, optionString, accessMode):

Reference

In which case fileObject is a MFileObject from which a fullName() is taken and given to a Python open() command. If that’s the case, then you couldn’t give it a file-handle directly which makes mmap possibly impossible (?) without writing a translator yourself and making one in there.

On my linux box, by default, I have “/dev/shm” which is the shared memory tmpfs device.

That could work; the solution can be studio-specific and doesn’t need to be cross-platform. The question then is whether it’s simpler to write your own translator, or setup and work with a tempfs device.

​

[Justin Israel]

On Mon, Mar 16, 2015 at 8:15 PM Marcus Ottosson <konstr...@gmail.com> wrote:

Thanks for trying this out Justin. I’ll address this one first in case it affects your interest in the other answers.

All this academic stuff aside, are you pretty confident that your bottleneck is disk IO on the scene file?

No, it certainly isn’t the bottleneck, I though that much was clear from the first tests I posted in which exporting was many times faster, even with writing to disk. This is out of plain curiosity only.

Ya no worries. It is exactly the same reason I am interesting in this conversation as well. Just had to ask.

So what Maya does is it moves the given path to a temp name if it already exists, then writes the new scene to your given path, and if all succeeds, it removes the previously existing one.

Would it be possible that it does things differently, when writing to a non-existing file? In my case, it’s safe to assume the path will always be different and non-existing. Or were you referring to the fact that an mmap file must be written to disk before being manipulated?

Here’s my limited understanding.

import mmap
import contextlib

cmds.file(rename="temp.ma")  # Does not exist
path = cmds.file(sceneName=True, q=True)

# This has to happen, which makes it exist, which may cause the behaviour you described.
with open(path, "wb") as f:
    f.write("empty")

with open(path, "r+b") as f:
    with contextlib.closing(mmap.mmap(f.fileno(), 0)) as mm:    
        cmds.file(exportAll=True, force=True)

In which case, cmds.file is attempting to open a new file handle via the path given; but it won’t work (obviously), and I’m getting a “Can’t write to …” error.

That previous strace was on my linux workstation, at work. A dtruss on my OSX laptop shows the exact same behavior. In your example, I think you are overlooking the fact that you still create a file ahead of time, when you establish the file and then memory map it. What Maya does is the same, where it sees the existing file and renames it to a temporary one, and then proceeds to open the original name freshly, and writes the scene to it. So the memory mapped file and the one Maya writes are independent. 

The only thing that is different when the named file absolutely doesn't exist and you issue a save from Maya is that there is no rename of a previously existing file.

If you are curious, you can watch the dtruss output:

$ sudo dtruss -p <maya pid>

or on linux:

$ strace -p <maya pid> -e trace=file

 

I also had a look at an example file translator, and it looks like this is it’s given arguments.

def writer(self, fileObject, optionString, accessMode):

Reference

In which case fileObject is a MFileObject from which a fullName() is taken and given to a Python open() command. If that’s the case, then you couldn’t give it a file-handle directly which makes mmap possibly impossible (?) without writing a translator yourself and making one in there.

Since I haven't written a file translator, I would hate to speak falsely. But my impression of the interface is that you are handed a target path and then it is up to you to perform the complete write. At that point, you are basically free to do it however you want, be it a memory mapped file, a pipe, tempfs, or writing nothing at all and updating your Facebook status. I suppose really it is just a hook into the standard IO mechanisms for file handling in Maya, and kind of feels similar to if you just had a function called do_stuff() that did your same custom export into memory. It seemed like the real thing you were after was to perform the native mayaAscii scene export into memory instead of a disk-backed file, but apparently all of these public routes are limited to just giving your expected file path. Whereas what you probably want is to be able to provide an open file descriptor. 

It is completely possible that I am overlooking another way, but this is all I can seem to see so far. 

 

On my linux box, by default, I have “/dev/shm” which is the shared memory tmpfs device.

That could work; the solution can be studio-specific and doesn’t need to be cross-platform. The question then is whether it’s simpler to write your own translator, or setup and work with a tempfs device.

Well if it is indeed Linux, then /dev/shm probably already exists depending on your distro. So that would be really easy since it is just like writing a normal file. You have to explicitly mount and unmount one on OSX, and I don't know how to do it on Windows. But if what I assumed about the translator option is true, and your goal is to use the internal mayaAscii export, then it doesn't sound like you have much of a choice.

 

​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODoaYPfPBWdpuS_weNnD3cPy-rCtac0K28scAJN4YDivA%40mail.gmail.com.

[Marcus Ottosson]

Well if it is indeed Linux, then /dev/shm probably already exists depending on your distro.

Ah, yes there it is, useful!

Just so I got this right, because I’m not in a position to test Maya on Linux at the moment, you’re saying that even if you export to /dev/shm via cmds.file it will still not work? Because it writes to this temporary directory first? Or are you saying it did work, in which case the file never enters the hard-disk and can still be accessed like a normal file?

Sorry for the confusion, I’m half-way lost.

At that point, you are basically free to do it however you want, be it a memory mapped file, a pipe, tempfs, or writing nothing at all and updating your Facebook status.

Can you imagine?

Win!

​

[Justin Israel]

On Tue, 17 Mar 2015 5:31 AM Marcus Ottosson <konstr...@gmail.com> wrote:

Well if it is indeed Linux, then /dev/shm probably already exists depending on your distro.

Ah, yes there it is, useful!

Just so I got this right, because I’m not in a position to test Maya on Linux at the moment, you’re saying that even if you export to /dev/shm via cmds.file it will still not work? Because it writes to this temporary directory first? Or are you saying it did work, in which case the file never enters the hard-disk and can still be accessed like a normal file?

Sorry for the confusion, I’m half-way lost.

It did work as you wanted, because the operations still take place in a virtual file system, and it doesn't matter that Maya wants to write a new file. As long as you give it a path to that tmpfs file system, all of the scene save stuff should be in memory.
The file will be written and is accessible like a normal file, but is backed by shared memory.
There is still some other business that I see taking place in the system temp location, but it is small and doesn't seem to be the scene file. Maybe some kind of small scratch work or a lock file. It happens before any of the scene file writes and is removed before that as well.

At that point, you are basically free to do it however you want, be it a memory mapped file, a pipe, tempfs, or writing nothing at all and updating your Facebook status.

Can you imagine?

<img src="https://ci5.googleusercontent.com/proxy/oYx7MqnRqf1rCMVTGaJFnEiCe0BgJ8j_E0SPyprNZI7QFurrbNP373GOMH0SpARjh7qY2xJMdnwfRzLt-7hU9TWazxvZxfh0rDGGO1E9OcNXmvr_86i_0nBcYEZyI4YmNcs6R-FmjyrJ9lfcYIvoJM8oa-HwdcF_dQ=s0-d-e1-ft#https://cloud.githubusercontent.com/assets/2152766/6665185/5ae6c4c4-cbce-11e4-953a-b2ada5cfb538.png">

Win!

​

Makes sense. We pretty much post everything from our lives to social networks now. Why not Maya scene file status? It's like the next Foursquare. "Marcus has just exported a Maya scene file at 123 Happy Street"

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBepramdkFEgh4P4BgFi0-qAg7d%3DMXzojivk7RHiYHxVg%40mail.gmail.com.

[Marcus Ottosson]

Marcus has just exported a Maya scene file at 123 Happy Street

Aw man. Now everybody knows where I live..

Thanks Justin, I think I found what I was looking for; the memory fs is super cool.

​

[Matt Estela]

On 17/03/2015 5:24 AM, "Justin Israel" <justin...@gmail.com> wrote:
> Makes sense. We pretty much post everything from our lives to social networks now. Why not Maya scene file status? It's like the next Foursquare. "Marcus has just exported a Maya scene file at 123 Happy Street"

"Matt likes this"