alembic files being overwritten causes crash

323 views
Skip to first unread message

Helge Mathee

unread,
Apr 17, 2012, 6:22:05 AM4/17/12
to alembic-d...@googlegroups.com
Hey folks,

when alembic files are opened by a live reader with a handle to that
file and are overwritten by another exporter, the library crashes. is
there a way to determine if a file was changed / overwritten before
pulling on a sample, so that we could re-open the filehandle?

Thanks!

-H

Luke Emrose

unread,
Apr 17, 2012, 6:43:51 AM4/17/12
to alembic-d...@googlegroups.com
Hi Helge,

My original response to this question was a small novel of the reasons for it, but I think it's better that I ask why.

Alembic is designed for you to export data to have it basically cached for another process.

What workflow is encouraging you to open two file handles to the same file, which one in read and  one in write mode to the same file?

That's a recipe for disaster no matter which way you look at it.

Think about it, in windows, or Linux, or Mac, doing this would present you with a "file already in use" warning when you try and read/write to the file.

The reason it crashes in Alembic is that Alembic is not handling these cases for you, and it's potentially up for debate whether it should.

Personally I think that HDF5 instead should be modified to prevent you from even beginning such dangerous and flawed access patterns, but it does, it assumes that the code using HDF5 will perform this function.

Long story short, why do you feel you need to do this?  Because maybe there is another way to do what you are doing that prevents this at the source?

regards,

Luke

--
You received this message because you are subscribed to the Google
Groups "alembic-discussion" group.
To post to this group, send email to alembic-discussion@googlegroups.com
To unsubscribe from this group, send email to
alembic-discussion+unsub...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/alembic-discussion?hl=en

For RSS or Atom feeds related to Alembic, see:

http://groups.google.com/group/alembic-dev/feeds

http://groups.google.com/group/alembic-discussion/feeds



--
Luke Emrose
aka evolutionary theory
www.evolutionarytheory.com
www.soundcloud.com/evolutionarytheory
www.reverbnation.com/evolutionarytheory
http://www.facebook.com/pages/evolutionary-theory/110717958952413

Helge Mathee

unread,
Apr 17, 2012, 7:47:05 AM4/17/12
to alembic-d...@googlegroups.com, Luke Emrose
Hey Luke,

thanks a lot for your response.

In company pipelines, sometimes a file would be loaded by an artist doing lighting, for example. Then, the animator would export the same file another time and overwrite it. The next time the lighting artist changes the frame, the library crashes. The file will be on a network location accessible by multiple users at the same time.

I am not sure how to deal with this case really.

Thanks for your support!

-H


On 4/17/2012 11:43, Luke Emrose wrote:
Hi Helge,

My original response to this question was a small novel of the reasons for it, but I think it's better that I ask why.

Alembic is designed for you to export data to have it basically cached for another process.

What workflow is encouraging you to open two file handles to the same file, which one in read and  one in write mode to the same file?

That's a recipe for disaster no matter which way you look at it.

Think about it, in windows, or Linux, or Mac, doing this would present you with a "file already in use" warning when you try and read/write to the file.

The reason it crashes in Alembic is that Alembic is not handling these cases for you, and it's potentially up for debate whether it should.

Personally I think that HDF5 instead should be modified to prevent you from even beginning such dangerous and flawed access patterns, but it does, it assumes that the code using HDF5 will perform this function.

Long story short, why do you feel you need to do this?  Because maybe there is another way to do what you are doing that prevents this at the source?

regards,

Luke
On 17 April 2012 20:22, Helge Mathee <helge....@gmx.net> wrote:
Hey folks,

when alembic files are opened by a live reader with a handle to that file and are overwritten by another exporter, the library crashes. is there a way to determine if a file was changed / overwritten before pulling on a sample, so that we could re-open the filehandle?

Thanks!

-H

--
You received this message because you are subscribed to the Google
Groups "alembic-discussion" group.
To post to this group, send email to alembic-d...@googlegroups.com

To unsubscribe from this group, send email to
--
You received this message because you are subscribed to the Google
Groups "alembic-discussion" group.
To post to this group, send email to alembic-d...@googlegroups.com

To unsubscribe from this group, send email to

Luke Emrose

unread,
Apr 17, 2012, 7:03:27 AM4/17/12
to Helge Mathee, alembic-d...@googlegroups.com
OK, I'd like to comment on this line:

"Then, the animator would export the same file another time and overwrite it."

The reason I would HIGHLY discourage this is the following, surprisingly common example.

Let's say the director decides that the version the animator overwrote was the gold version, the perfect version, the one that he wants and won't settle for anything else in the world?

You just lost it.

In a pipeline that supports the changing minds of other people, sometimes it's better to disallow people overwriting files entirely.  It's safer, allows more traceability, and prevents exactly the situation you refer.

Why would you ever want to overwrite a file coming from another department?  Instead you should make a copy of that file and manipulate the copy.  Or, more correctly, open the original version in read mode, then write a modification of it out to another file in write mode.

Is there a reason you need to overwrite an existing file?  I don't think Alembic was designed with such a usage pattern in mind.  Indeed most cache formats aren't.

L

Helge Mathee

unread,
Apr 17, 2012, 8:06:02 AM4/17/12
to Luke Emrose, alembic-d...@googlegroups.com
Ok, this makes perfect sense. Thanks for the information.

-H

Steven Caron

unread,
Apr 18, 2012, 12:49:21 AM4/18/12
to alembic-d...@googlegroups.com
let me chime in for support of overwriting files. in your scenario we didn't loose the 'gold version' because we always version our source files which create caches or renders. we use tracking software like shotgun and other proprietary means for making sure this happens. so feel free to overwrite a file because we can always go back to the right version. we are a small team of people, if i am in charge of doing fx and animation for the shot i am actually lighting i would like it to update fluidly. we do use versioning extensively but we like to have a 'current' version which is just a pointer to the 'latest' version so users referencing the asset can always be up to date. at any time they can lock themselves to a version if they choose.

if alembic can't be designed to allow for this then i am sure we will re use some system thats already in place for 'versioning up' assets. but its still a valid request/workflow in my opinion.

s


On Tuesday, April 17, 2012 4:03:27 AM UTC-7, evolutionary theory wrote:
OK, I'd like to comment on this line:

"Then, the animator would export the same file another time and overwrite it."

The reason I would HIGHLY discourage this is the following, surprisingly common example.

Let's say the director decides that the version the animator overwrote was the gold version, the perfect version, the one that he wants and won't settle for anything else in the world?

You just lost it.

In a pipeline that supports the changing minds of other people, sometimes it's better to disallow people overwriting files entirely.  It's safer, allows more traceability, and prevents exactly the situation you refer.

Why would you ever want to overwrite a file coming from another department?  Instead you should make a copy of that file and manipulate the copy.  Or, more correctly, open the original version in read mode, then write a modification of it out to another file in write mode.

Is there a reason you need to overwrite an existing file?  I don't think Alembic was designed with such a usage pattern in mind.  Indeed most cache formats aren't.

L
On 17 April 2012 21:47, Helge Mathee  wrote:
Hey Luke,

thanks a lot for your response.

In company pipelines, sometimes a file would be loaded by an artist doing lighting, for example. Then, the animator would export the same file another time and overwrite it. The next time the lighting artist changes the frame, the library crashes. The file will be on a network location accessible by multiple users at the same time.

I am not sure how to deal with this case really.

Thanks for your support!

-H


On 4/17/2012 11:43, Luke Emrose wrote:
Hi Helge,

My original response to this question was a small novel of the reasons for it, but I think it's better that I ask why.

Alembic is designed for you to export data to have it basically cached for another process.

What workflow is encouraging you to open two file handles to the same file, which one in read and  one in write mode to the same file?

That's a recipe for disaster no matter which way you look at it.

Think about it, in windows, or Linux, or Mac, doing this would present you with a "file already in use" warning when you try and read/write to the file.

The reason it crashes in Alembic is that Alembic is not handling these cases for you, and it's potentially up for debate whether it should.

Personally I think that HDF5 instead should be modified to prevent you from even beginning such dangerous and flawed access patterns, but it does, it assumes that the code using HDF5 will perform this function.

Long story short, why do you feel you need to do this?  Because maybe there is another way to do what you are doing that prevents this at the source?

regards,

Luke
On 17 April 2012 20:22, Helge Mathee wrote:
Hey folks,

when alembic files are opened by a live reader with a handle to that file and are overwritten by another exporter, the library crashes. is there a way to determine if a file was changed / overwritten before pulling on a sample, so that we could re-open the filehandle?

Thanks!

-H

--
You received this message because you are subscribed to the Google
Groups "alembic-discussion" group.
To post to this group, send email to alembic-discussion@googlegroups.com

To unsubscribe from this group, send email to
--
You received this message because you are subscribed to the Google
Groups "alembic-discussion" group.
To post to this group, send email to alembic-discussion@googlegroups.com

To unsubscribe from this group, send email to

Luke Emrose

unread,
Apr 18, 2012, 1:06:35 AM4/18/12
to alembic-d...@googlegroups.com

Hi Steven,

Then I am confused what you are actually using alembic for.

Can't you just use files from your host application (houdini, xsi etc.) to do your changes and then export once to alembic for a render test?

For in progress work I'd probably recommend against any caching format. Something like Alembic is best used for "setting something in stone" - so that someone along the line can pick up something they can't modify and add their data to it and pass it on. I would really like to know more about how you have placed Alembic in your pipe to fully understand the issue.

As for using a pipeline that always points to latest versions, I can caution against that for other reasons too. If someone staying back late modifies the uvs of an object and publishes to current, then it will break existing renders on your farm.

Updating should always be a choice imo, not something that happens automatically. It makes real problems hard to pin down and introduces the possibility of bugs creeping into every shot in a pipeline simultaneously.

For a very small team it might be workable with tightly coupled people doing the same tasks intimately aware of the risk of making a change, but in the context of a bigger or growing company I've only ever seen that paradigm do damage.

Regards,

L

To post to this group, send email to alembic-d...@googlegroups.com

To unsubscribe from this group, send email to
alembic-discuss...@googlegroups.com

Andrew Lyons

unread,
Apr 18, 2012, 3:18:32 AM4/18/12
to alembic-d...@googlegroups.com
A couple of quick thoughts...

Copy-on-publish seems to be pretty standard across the larger studios
in the industry for many reasons - and not just for geometry caches.

There's various ways to put locks on files to prevent multiple handles
on a file. Opinions seem to differ on approaches to locking though.
Perhaps that's why it's not included in HDF5 or Alembic?

Cheers

--
=======================================
Andrew D Lyons | Digital Artist | http://www.tstex.com
=======================================

Steven Caron

unread,
Apr 19, 2012, 2:01:43 AM4/19/12
to alembic-d...@googlegroups.com
i appreciate your advice and i have experienced first hand the issues you outline. as i mentioned we are a smaller team and are aware of the risks.

we are still finding how alembic can fit in our pipeline, right now its with animation/fx caching to send to lighting. as i mentioned before we have tools in place for 'updating assets' and also 'locking an asset to a version' so in case you submit to the farm and dont want things to change mid render, or when finaling an asset.

in the end of the day at very least softimage shouldn't crash when the file does change. i hope helge can find a workaround to avoid the crash without compromising something to do so. if the alembic maintainers can provide some API level change to help him achieve these then great!

quick question, what does the maya reference implementation do when the stream changes underneath?

Luke Emrose

unread,
Apr 19, 2012, 2:10:17 AM4/19/12
to alembic-d...@googlegroups.com
If it's compiled against Alembic, it will do whatever alembic does, which, to my knowledge, would be exactly the same thing.  Crash.

The simplest initial solution would be to make Alembic stop the user from opening a second file handle of the opposite mode (i.e. read/write) to the same file.
i.e. attempting to open a read handle when that file is already open in write mode should generate a plug-in error that stops the user from completing the operation, rather than crashing.

A more complex solution could come after that initial fix maybe.

L

--
You received this message because you are subscribed to the Google
Groups "alembic-discussion" group.
To post to this group, send email to alembic-d...@googlegroups.com
To unsubscribe from this group, send email to
alembic-discuss...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/alembic-discussion?hl=en
 
For RSS or Atom feeds related to Alembic, see:
 
http://groups.google.com/group/alembic-dev/feeds
 
http://groups.google.com/group/alembic-discussion/feeds

daniel

unread,
Jan 11, 2017, 3:42:44 PM1/11/17
to alembic-discussion, evolution...@gmail.com

Hi all,

I just want to see if anything is planned on how this gets handled.

The problem I am facing is not only overwriting but the file handle changing. Say your network drive gets remapped or anything else that changes the file handle - if you want to access the loaded Alembic by changing the time line the application (in this case Maya) crashes.

Daniel

Lucas Miller

unread,
Jan 11, 2017, 8:19:10 PM1/11/17
to alembic-d...@googlegroups.com, Luke Emrose
No, there is no plan as the data is meant to be write once, read many times.
You would need to store the data in per frame files that would be
completely read on open, and closed once done.
This shuts the door (or makes less efficient) several other workflows.

Lucas
> --
> You received this message because you are subscribed to the Google Groups
> "alembic-discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to alembic-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Roeland Schoukens

unread,
Jan 17, 2017, 6:25:16 PM1/17/17
to alembic-discussion
I think this should be handled in a more graceful way.

About file locking:
 
Think about it, in windows, or Linux, or Mac, doing this would present you with a "file already in use" warning when you try and read/write to the file.

That is not the case, unless you explicitly implement file locking in your application.

On Linux this is very poorly supported. It can only work if all applications manipulating Alembic file explicitly use certain APIs to register file locks, and all applications reading Alembic files must explicitly check a file lock before opening a file. This adds boilerplate code to any code opening or saving Alembic files.

Secondly, the problem is that Alembic is crashing rather than just failing to read a file. Normally a file parser should never crash, regardless of the input it's receiving.

So since the assumption that files are not overwritten is so critical, those file locks are better implemented inside the Alembic library.

Also there's the point about network drives. These occasionally disconnect and reconnect, and file handles tend to go invalid after a while. I see this all the time when having a PDF file open in Foxit reader, and those PDF files are never overwritten. Is this going to cause an application to crash randomly when it has an Alembic file on a network drive open?

Finally, we can have our opinion that users shouldn't update files in place. Our users don't give a toss about our opinion. I can't possibly know all the ways they are using our programs. They may be going through a few iterations of export / import before something looks OK. They may keep assets in a version control system.

So, TL;DR: users may have legit reasons to overwrite files. Our application should not crash when that happens.

--
Roeland
Reply all
Reply to author
Forward
0 new messages