softimage and it's binary format

[Stefan Andersson]

Hello Everyone,

Something that is bothering me, and has been bothering me for a long while, is the constant use of binary files in Softimage.

1.) emdl
Would it be so wrong to have this as a ascii format? So that you can parse through the model and change data which might be needed?

2.) preset files
SERIOUSLY... if I save out a single shader I should be able to edit the contents in the file.

3.) scn files
Would be most helpful if this also could have THE OPTION of being ascii.

4.) dsprojectinfo
I would love if this could be a ascii file so that it can be edited or created without Softimage

Is there a way around this somehow? I've been lurking around to see if there was someone that had posted some ninja skills regarding this.

anyhow, insights and thoughts are welcomed

best regards
stefan andersson



--
Stefan Andersson | Digital Janitor

blog | showreel | twitter | LinkedIn | cell: +46-73-6268850 | skype:sanders3d

[Ciaran Moloney]

This has been a commonly requested feature over the years....

[Sajjad Amjad]

Some oil for the fire:
The terms 'non-trivial' and 'cost benefit to the customer' come to mind.

* ducks out *

[Ponthieux, Joey]

On 1/28/2013 9:29 AM, Stefan Andersson wrote:

Hello Everyone,

Something that is bothering me, and has been bothering me for a long while, is the constant use of binary files in Softimage.

3.) scn files
Would be most helpful if this also could have THE OPTION of being ascii.

--
Stefan Andersson | Digital Janitor

blog | showreel | twitter | LinkedIn | cell: +46-73-6268850 | skype:sanders3d

Second that!

Why?

I've been writing out Maya .ma files for almost 12 years. That would have been 14 years except in 98 and 99 I did not have the foresight to understand that Maya's binary .mb files might need to be edited. On rare occasions we ran into scene files that would get corrupted or otherwise take a mind of their own. ASCII .ma files could be edited and the scene resurrected, an activity that occurred more frequently as the files got really large. 

Other reasons? Old scene files(decade old), and which were not corrupted, but had old texture file links that were no longer valid could be edited to relink the textures to their new location. Old .ma file, new file server, Perl script, no problem.

The loss of hard disk space and extra time to load .ma files became trade-offs that were well worth the investment. As an added advantage interrogating the .ma scene structure yielded vast amounts of interesting information about Maya internals and structure. It was after all nothing but a huge Mel file.

An example of a significant benefit?  I once wrote a material file converter for Wavefront TAV .mtl files to Maya. Something Alias Wavefront never created...go figure. I used a .ma file to decipher what the materials should look like in Maya and converted the .mtl files straight to .ma. Without the ASCII scene information from the .ma file it would have been practically impossible to accomplish.

With this habit somewhat ingrained, I have often wondered what a Softimage scene file would look like in ASCII. I would welcome this capability in Softimage.

Going back even farther, I had my own database management tools written in Perl for Softimage 3D back in the day. So having ascii access to dsprojectinfo would also be useful.   As would the other files. Granted I can understand the performance issues related to having some of these files as ascii, but each should be optional.

Joey Ponthieux
LaRC Information Technology Enhanced Services (LITES)
Mymic Technical Services
NASA Langley Research Center

____________________________________________________________
Opinions stated here-in are strictly those of the author and
do not represent the opinions of NASA or any other party.

[Sajjad Amjad]

Just to clarify, those are some of the terms I heard from the AD end whenever someone brought this issue up.

Like Ciaran said, this request has been brought up frequently.

Anyone involved with trouble-shooting knows the benefits of having ASCII files, just need to convince the devs.

[ivan t]

Hi everyone,

Thanks for raising this up. I empathize the frustration that most of you have gone through (on the binary files). I will file in a request when I get in to office tomorrow.

Regards

Ivan

ivan...@nospam.autodesk.com (please remove nospam from email address)

[Stefan Kubicek]

I guess implementing an ASCII file format writer/reader is hard to do as an afterthought and difficult to justify for what is probably less than 1% of the user base that would actually require and benefit from it.

There has once been a push towards a 3dsMax ASCII file format by Borislav "Bobo" Petrov years ago, but it was just too time consuming to maintain and develop up to a level where it could actually be used reliably and professionally, and for simple things you could just use the FBX ascii file format.
http://www.scriptspot.com/bobo/darkmoon/bff/

For something like that I imagine you'd first need 100% script access to every minute detail of the scene. That is: Not only being able to read certain information, but also to create and connect objects in a non-linear fashion, however there are still places in Softimage where this is not possible (e.g. you cannot create an op and then connect it to some objects later, or change connections of an operator once they have been established, at least for the bigger part). That means that scene creation through scripting (assuming an ASCII file format for XSI would essentially be some sort of script, similar to a Maya ASCII file) would need to be linear, you'd have to be careful what you create at which point in time/the file, and that again makes it harder compared to just writing all the nodes out to the file and then start making connections to/from their parameters.

Just some thoughts, somebody correct me if I'm wrong.

Stefan

> Hello Everyone,
>
> Something that is bothering me, and has been bothering me for a long while,
> is the constant use of binary files in Softimage.
>
> 1.) emdl
> Would it be so wrong to have this as a ascii format? So that you can parse
> through the model and change data which might be needed?
>
> 2.) preset files
> SERIOUSLY... if I save out a single shader I should be able to edit the
> contents in the file.
>
> 3.) scn files
> Would be most helpful if this also could have THE OPTION of being ascii.
>
> 4.) dsprojectinfo
> I would love if this could be a ascii file so that it can be edited or
> created without Softimage
>
> Is there a way around this somehow? I've been lurking around to see if
> there was someone that had posted some ninja skills regarding this.
>
> anyhow, insights and thoughts are welcomed
>
> best regards
> stefan andersson
>
>
>


--

-------------------------------------------
Stefan Kubicek Co-founder
-------------------------------------------
keyvis digital imagery
Wehrgasse 9 - Grï¿œner Hof
1050 Vienna Austria
Phone: +43/699/12614231
--- www.keyvis.at ste...@keyvis.at ---
-- This email and its attachments are
--confidential and for the recipient only--

[Stefan Andersson]

You might want to check the backlog for the last 8-10 years of beta list discussions :)

If you remember to log it, please write back which REQ number.

best regards
stefan andersson

[ivan t]

Thanks Stefan , will do :)

The team is aware of this popular request (and many others). I will log it down.

Best Regards

Ivan

[David Barosin]

back in the day... hrcConvert ;)

[Matt Lind]

Whoa, the number would be significantly greater than 1%. More like 100%. While relatively few people would be writing code to modify the text format, everybody using it would benefit.

The single biggest issue I have right now in our pipeline is dealing with thousands of assets made over the past 8 years on this production that need to be used again in a current version of Softimage. This very production is projected to continue 10 more years. That means assets made in 2005 may need to be opened in the year 2023. There will have been too many architectural changes in Softimage 2023 to be able to open such files anymore. We've already hit this problem several times in the past few years getting stuff created in XSI 6.0 open in Softimage 2010 or later due to the deprecation of the particle system and changes in realtime shader architectures. If we have a text file format, we can at least do some work to make the old asset compatible with whatever version of Softimage is in use.

Without a text file format, I either have to roll my own - which I tried - or deprecate the asset and tell my art directory the team has to replace it with a new one. Say that 15,000 times and somebody high up won't be too happy.

I have written my own file format to do this work. While it's not too difficult to export data, it's damned near impossible to import the data and preserve the integrity mostly because Softimage doesn't expose the necessary hooks to rebuild the construction history. Operator port connections of often not exposed and the data needed to recreate the operator to behave exactly as during previous save operation is not possible either. Example: 'movecomponentOp'. I know it does a simple translation of affected points, but Softimage does not expose what that translation value(s) are. Although it's one of the simplest operators to roll out of the box, I cannot support it in a custom file format. It forces me to freeze everything on scene save and that makes some artists unhappy as much of what they work on is work in progress, or has operations in parts of the construction history other than the 'modeling' marker. Example: envelopes or 'shrinkwrap' operators. Rather import!
ant to know the intention to be able to rebuild it properly.

What Softimage desperately needs is a text file format that is fully accessible at the most granular level and supports edits by the end user. Not just trivial things like paths to texture bitmap paths, or filenames to write images produced by renders, but also modification of the construction history such as being able to deactivate an operator if it's known to induce crashes or is deprecated itself and needs to be replaced with something more modern.

Examples:

- replace the old deprecated Softimage particle system with ICE equivalent.
- replace shaders which have experienced architectural changes (e.g. realtime shaders created in XSI 5.x with an equivalent created in Softimage 2013x)
- deactivate operators known to induce crashes (due to corruption, perhaps).
- Add/remove objects/models from scene.
- Redirect operator targets (such as point constraints to affect different objects because original target no longer valid)
- Adjust renderpass partition memberships and overrides (because it's soooooo sloooooowww inside of the Softimage UI)
- delete crap that never belonged in the first place.


One very important aspect of having a text file format is the ability to use code to read/write/validate data in the file and have it work in the Software. Not trivial to implement, but highly important as there really needs to be an option to modify vast quantities of assets without having to do it inside of a call to xsibatch.exe. XSIbatch is much too slow to operate on hundreds of thousands of assets as we have to do. If a batch crashes for any reason, we just don't have the granularity to know where to restart the batch or get enough information about the error. One very important aspect of having a text file format is the ability to get valuable information from processing in a timely and effective manner.


There's more, but I think the point has been made. Text file format is critical to large scale production, but helps all productions.


Matt

Wehrgasse 9 - Gr�ner Hof

[Vladimir Jankijevic]

WORD! I think everything has been said so far. And just to make it clear enough for everybody out there: We need a ASCII file format. Yes we do! 

          Wehrgasse 9 - Grüner Hof

           1050 Vienna  Austria
         Phone:    +43/699/12614231
--- www.keyvis.at  ste...@keyvis.at ---
--  This email and its attachments are
--confidential and for the recipient only--

--
---------------------------------------
Vladimir Jankijevic
Technical Direction

Elefant Studios AG
Lessingstrasse 15
CH-8002 Zürich

+41 44 500 48 20

www.elefantstudios.ch
---------------------------------------

[Ed Manning]

+100!

And yes, that's "plus factorial 100."

One question -- wasn't .xsi supposed to be this?  And in what ways is it not? How did that happen?

Okay, yes, 3 questions.

[Gene Crucean]

| One question -- wasn't .xsi supposed to be this?

I don't think .xsi was ever supposed to be a full ascii scene description.

| How did that happen?

Autodesk happened. Then they killed everything that doesn't end in .fbx. YAY!!!!

--
Gene Crucean - Emmy winning - Oscar nominated VFX Supervisor / iOS-OSX Developer / Filmmaker / Photographer

** Freelance for hire **
www.genecrucean.com

~~ Please use my website's contact form on www.genecrucean.com for any personal emails. Thanks. I may not get them at this address. ~~

[jo benayoun]

I think, it has been made clear over the past why there is no ascii file formats ala Maya "ma".
Maya has the huge advantage to be build from the ground to the top using a command engine and so
makes easier to have such file formats.

I think it was one of the softimage dev who clearly explain that the emdl or scn files are dumps (serialized datas) from internal datastructures.  The downside of this is because Autodesk wont expose softimage internals, this makes the format closed and if your file gets corruptes, well thats all on you, but as a good point, we get faster load times.

We already have the tools to write custom exporter, though they are limited...  priority should go on exposing more from Softimage than actually asking for an ascii file format.

--jon

2013/1/28 Gene Crucean <emailgene...@gmail.com>

[ivan t]

Hi ,

This has been filed as SOFT-854 by Stefan Andersson.

Thanks

Ivan

[Stefan Kubicek]

Very good points Matt, and I didn't meant to say I wouldn't welcome it either, not just because of the merits of an ASCII file format itself but also in anticipation of it's positive side effects it's implementation would presumingly require like more operator transparency (i.e. non-linear editing of operator dependencies).

> Whoa, the number would be significantly greater than 1%. More like 100%. While relatively few people would be writing code to modify the text format, everybody using it would benefit.
>
> The single biggest issue I have right now in our pipeline is dealing with thousands of assets made over the past 8 years on this production that need to be used again in a current version of Softimage. This very production is projected to continue 10 more years. That means assets made in 2005 may need to be opened in the year 2023. There will have been too many architectural changes in Softimage 2023 to be able to open such files anymore. We've already hit this problem several times in the past few years getting stuff created in XSI 6.0 open in Softimage 2010 or later due to the deprecation of the particle system and changes in realtime shader architectures. If we have a text file format, we can at least do some work to make the old asset compatible with whatever version of Softimage is in use.
>
> Without a text file format, I either have to roll my own - which I tried - or deprecate the asset and tell my art directory the team has to replace it with a new one. Say that 15,000 times and somebody high up won't be too happy.
>

> I have written my own file format to do this work. While it's not too difficult to export data, it's damned near impossible to import the data and preserve the integrity mostly because Softimage doesn't expose the necessary hooks to rebuild the construction history. Operator port connections of often not exposed and the data needed to recreate the operator to behave exactly as during previous save operation is not possible either. Example: 'movecomponentOp'. I know it does a simple translation of affected points, but Softimage does not expose what that translation value(s) are. Although it's one of the simplest operators to roll out of the box, I cannot support it in a custom file format. It forces me to freeze everything on scene save and that makes some artists unhappy as much of what they work on is work in progress, or has operations in parts of the construction history other than the 'modeling' marker. Example: envelopes or 'shrinkwrap' operators. Rather impo!

> Wehrgasse 9 - Grï¿œner Hof

> 1050 Vienna Austria
> Phone: +43/699/12614231
> --- www.keyvis.at ste...@keyvis.at ---
> -- This email and its attachments are
> --confidential and for the recipient only--
>
>


--
-------------------------------------------
Stefan Kubicek Co-founder
-------------------------------------------
keyvis digital imagery

Wehrgasse 9 - Grï¿œner Hof

[Jason S]

I thaught it was a revenue thing to promote upgrading.

[Luc-Eric Rousseau]

There is really no technical reason why Softimage could not save the
equivalent of the current binary scene as ASCII, and it may not even
be slower. It's hard to explain why this project has never been
scheduled in the last 15 years without pointing fingers.

> priority should go on exposing more from Softimage than actually
> asking for an ascii file format.

.. and these projects often die often because argument like this,
which IMHO is a false dichotomy around core development vs the very
nebulous "more SDK access". It ignores the fact that core development
is done in a fraction of the time and benefit everyone plus the long
term viability of the product, and don't necessarily exclude SDK
support.

[Gene Crucean]

| It's hard to explain why this project has never been scheduled in the last 15 years without pointing fingers.

Excellent... let's point some fingers! :D

[Steven Caron]

at 'maya developers'?

[Stefan Andersson]

Finally I can say that it's Maya developers fault that its something wrong with softimage :)

/Stefan 

[Steven Caron]

technically you could have done that already, brent worked on maya while it was alias and alias/wavefront. and i think they got one or two others

[Szabolcs Matefy]

I had no time to run through the posts here, but there is one, and very important benefit, scene debugging…In the last scene I had lost several scene files, and wasn’t able to recover them, even the backups, and the versioned savings were unloadable. The built in scene debugging didn’t help too much…Not to mention, that in Maya we used to resolve (years ago) cross version compatibility with rewriting the header to the proper version

 

From: softimag...@listproc.autodesk.com [mailto:softimag...@listproc.autodesk.com] On Behalf Of Steven Caron
Sent: Tuesday, January 29, 2013 7:36 AM
To: soft...@listproc.autodesk.com
Subject: Re: softimage and it's binary format

 

technically you could have done that already, brent worked on maya while it was alias and alias/wavefront. and i think they got one or two others

[Stefan Andersson]

I also blame Pixar for the current status of the Animation Mixer.

/s

[Stefan Andersson]

Thanks Ivan!

I really hope that this option will be available to us soon. And we can see from the responses there are quite a few people who need it badly, and have requested it for a long time.

best regards
stefan andersson

[jo benayoun]

.. and these projects often die often because argument like this,
which IMHO is a false dichotomy around core development vs the very
nebulous "more SDK access".  It ignores the fact that core development
is done in a fraction of the time and benefit everyone plus the long
term viability of the product, and don't necessarily exclude SDK
support.

It sounds to me to be always the same arguments at the end (front-end tools vs SDK extensibility).

We are already capable of writing a custom exporter but suffer from inaccessible stuff.  Why would I like the team

to provide me an ascii file format while opening more the SDK would allow me to write my own + bring many other benefits in different areas other than IE?

Following this idea, why did you guys exposed the ToolSDK and not just provided user-friendly tools once a year (...)?

Considering the time it takes also to get updates or maintenance done on some parts of the software, I wouldn't like depending on the softimage

team to see what I am looking for implemented.

--jon

2013/1/28 Stefan Andersson <sand...@gmail.com>

[Eugen Sares]

Am 29.01.2013 08:57, schrieb jo benayoun:

.. and these projects often die often because argument like this,
which IMHO is a false dichotomy around core development vs the very
nebulous "more SDK access".  It ignores the fact that core development
is done in a fraction of the time and benefit everyone plus the long
term viability of the product, and don't necessarily exclude SDK
support.

It sounds to me to be always the same arguments at the end (front-end tools vs SDK extensibility).

We are already capable of writing a custom exporter but suffer from inaccessible stuff.  Why would I like the team

to provide me an ascii file format while opening more the SDK would allow me to write my own + bring many other benefits in different areas other than IE?

Following this idea, why did you guys exposed the ToolSDK and not just provided user-friendly tools once a year (...)?

Considering the time it takes also to get updates or maintenance done on some parts of the software, I wouldn't like depending on the softimage

team to see what I am looking for implemented.

--jon

Front-end-tools and SDK access shouldn't be mutually exclusive by all means.
The dev team is under time/budget restrictions, which is the main reason an SDK exists. Otherwise we would would just need to snap our fingers and the next needed tool would pop up with the next release.
What remains nebulous, for the usual stupid NDA reasons (investment fraud), is the internal priority list. If we knew what to expect, there wouldn't be double-tracking, and everybody would win. But sadly, this seems not to be realistic with a closed source application. The usual dilemma.

[Brent McPherson]

There is no central priority list from which we pull projects. As devs we obviously have our own ideas what should be done but there are many other business interests and opinions that go into deciding what gets done each release. You just hope that as a team you are striking the right overall balance for each release.

I know in the in the past we visited a number of you guys with our $100 test where we give you 100 virtual dollars to spend on features and you tell us what you would spend it on. It is fun because depending on who is in the room you can get wildly different opinions and the final result usually ends up looking quite different than what it was at the beginning of the test. I'm sure those of you who have participated can confirm that it is a harder exercise than you might have initially expected. ;-)

In the case of the tool SDK I think you would be surprised to know the history of that project and how it got developed... ;-)
--
Brent

From: softimag...@listproc.autodesk.com [mailto:softimag...@listproc.autodesk.com] On Behalf Of Eugen Sares
Sent: 29 January 2013 8:45 AM
To: soft...@listproc.autodesk.com
Subject: Re: softimage and it's binary format

[Xavier Lapointe]

"In the case of the tool SDK I think you would be surprised to know the history of that project and how it got developed... ;-)"

Oh please, tell us a story (;

.. I'm all ears.

[Eugen Sares]

Yeah, let's hear that story! =}

One basic question, for the spirit:
would you say that "anything is possible", or are there things (on the
known wishlist) that probably will never get fixed, simply because it
would mean too many changes of the "core architecture"? Like no surgery
on the open heart?
Or is there still enough room for any kind of improvement (counting out
starting "from scratch")?
The difference between being unwilling or unable...

Looking at 3ds max, you get the impression that this is the reason for
it's more or less stalled development. Hopeless kludge beyond repair...

[Brent McPherson]

Let's just say it was a lucky coincidence because it was also needed for another project/client.

The tool SDK is something that is mainly interesting to 3rd party devs and larger studios because it has a higher barrier to entry (C++ plugin) and although we had talked about it for a long time it never quite made the cut because of this.

As a software developer I will tell you that almost anything is possible if you are willing to commit the time & money. (and I'm sure it is the same in the production world) However, the architecture makes a big difference so with right architecture feature X might be 10 or 100 times easier to do in one product vs another. So architecture makes a big difference and does decide to a large extend how the software will evolve.

Multi-platform support is a good example. When XSI was developed Softimage was owned by Microsoft so the dev team made a decision to build directly on the Windows APIs. Therefore porting to a new platform would obviously incur a higher cost for Softimage. Maya on the other hand was designed to be multi-platform so the team invested in isolating the UI layer. This required more initial architecture investment but made it easier to port the product. I think both companies made the right call given their circumstances. At Alias we seriously considered taking a Mainwin-like approach (there were a few companies offering cross-platform windows API support at the time) since it was clear that Microsoft was going to be the dominant computing platform and supporting multiple platforms had an ongoing development cost. So having a higher cost means you need to have a stronger business case before you invest the extra effort in porting the product to a new platform.
--
Brent

[Luc-Eric Rousseau]

> It sounds to me to be always the same arguments at the end (front-end tools
> vs SDK extensibility).
> We are already capable of writing a custom exporter but suffer from
> inaccessible stuff. Why would I like the team
> to provide me an ascii file format while opening more the SDK would allow me
> to write my own + bring many other benefits in different areas other than
> IE?

Getting the team to add APIs so that you can write your own custom
format *instead* of doing an ASCII file format is just a wrong idea,
imho. I just hate these types of "please do work on this" discussion
where someone gets threatened by other development..

First, I think we actually already have those APIs anyway. I mean,
crosswalk and the various format supports are written with the SDK and
we even have apis to browse the connection stack. But more
importantly, you'll never write a full replacement for the XSI scene
format. It's never going to happen. You don't have the time, the
technical know-how or interest to do all of the work of saving every
bit in every corner of the software correctly and test that. And
people do not want to download your custom format plug-in, assuming it
would be free: they want the software to have it natively, tested and
supported, otherwise they'll never use it.

APIs are not related to problem in question here.

Native ascii file format is about dealing with scene corruption,
saving your ass on a deadline and the long term viability of Softimage
assets in the future. That shader corruption in softimage 2011
nonsense would have never happened if the scene file format was opened
and human readable. It doesn't make sense imho to put all this energy
creating assets and then putting them in black box you can never look
into.

[Vladimir Jankijevic]

WORD AGAIN! Thanks Luc-Eric! Putting even more weight to the subject and the request to finally just DO IT. Give us a real ASCII alternative. Not just a .scntoc

[Matt Lind]

> First, I think we actually already have those APIs anyway ....<snip>....

> But more importantly, you'll never write a full replacement for the XSI scene format. It's never
> going to happen. You don't have the time, the technical know-how or interest to do all of the
> work of saving every bit in every corner of the software correctly and test that. And people do
> not want to download your custom format plug-in, assuming it would be free: they want the
> software to have it natively, tested and supported, otherwise they'll never use it.

I don't think this assumption is 100% aligned with the request, or intent of the request.

You are correct we do not need to exactly replicate the file format, however we do need to replicate the functionality the file format provides - and in some cases change it because the original design has flaws. That is where the Softimage API falls short. One missing link is the ability to rebuild construction history because the necessary hooks are not exposed. As a result exported data must be compromised to a degree to make the workflow usable. This is the problem.

We need a file format which is open, accessible, modifiable, and can load/save data exactly the same as the native binary format, to maintain assets over the long haul, repair corruptions, and so on, but the APIs to do it to the level we need are not currently there.



Matt