Houdini 12 BGEO/GEO File Support

[Eric Falconer]

We've been trying to figure out a solution for transferring particle data from Houdini 12 to Maya 2012. We've explored the alembic path, but it currently doesn't support reading particle data back into Maya. Partio seems like a good fit so far, and Partio4Maya is a really great tool. The only problem is that partio doesn't seem to support the new json/bson GEO/BGEO formats in the newer version of Houdini. Although I'm fairly certain I could write the support for the new file formats into Partio, since we do not have access to the plugin's source we'd be unable to have a version of Partio4Maya that worked with the new version.

Do you have any plans for supporting the newer BGEO formats, or any plans of releasing the full source for the plugin itself so I can rebuild it with updated support?

[Michael Auerswald]

Is there a full description of the new geo/bgeo out yet? Last time I checked I was only able to find documentation for the old one.

I haven't tested them, but I think the partio4maya sources are in redpawfx's branch -  https://github.com/redpawfx/partio/tree/redpawfx/contrib/partio4Maya

Michael

[Colin Doncaster]

Hi Eric -

FWIW you can use gconvert to convert from the new BGEO format into the hclassic style that partio supports.

[John Cassella]

The Source for partio4Maya is available, just not in the  Official  disney  github repo for now, Its in my personal  github fork , (where you downloaded the binary package from)

I believe that  Andrew mentioned something about working on support for that new  H12  format  soon, however if you wanted to take a hack at it.. I don't think anyone would mind.  If you make your own github  fork  of the WDAS  partio repo, you can make changes to the format stuff, and  also  pull branches from my repo as well.
then when you have something  good to go, you   can submit a  pull request for the feature to the WDAS  origin  and Andrew  will merge it into the official repo.

Let me know if you want to help out  and we'll figure out what you need to get going!

thanks!
-johnc
(redpawfx)

[Eric Falconer]

Thank you so much for your responses. I'm a little new to working with houdini, so I wasn't aware of the gconvert, but that will definitely help. I was looking at your personal repository, but forgot to look under your specific branch.

I am interested in working on the H12 support, I will have to check with our studio and double-check that this is the direction we want to go in for getting particle data out of houdini, but assuming that's good then it would be great to contribute.

Eric

On Thursday, April 26, 2012 7:49:28 PM UTC-4, redpawFX wrote:

The Source for partio4Maya is available, just not in the  Official  disney  github repo for now, Its in my personal  github fork , (where you downloaded the binary package from)

I believe that  Andrew mentioned something about working on support for that new  H12  format  soon, however if you wanted to take a hack at it.. I don't think anyone would mind.  If you make your own github  fork  of the WDAS  partio repo, you can make changes to the format stuff, and  also  pull branches from my repo as well.
then when you have something  good to go, you   can submit a  pull request for the feature to the WDAS  origin  and Andrew  will merge it into the official repo.

Let me know if you want to help out  and we'll figure out what you need to get going!

thanks!
-johnc
(redpawfx)

[Andrew Selle]

Yes this is a good suggestion.  I am in the process of working on a plugin for the new BGEO (JSON based file format). It is a lot harder to parse than the old geometry format. Basically, I am implementing a generic parser for their Binary JSON spec, and I have that working based on their example in the public area.  I just need to interpret all the data and build a partio object and then I basically will have an importer that is independent of their libraries. Probably this will be available in the next few weeks, but I have been sidetracked lately. 

-Andy

[Colin Doncaster]

I'm sure you're already well aware of it Andrew, but a little reference at http://www.sidefx.com/docs/hdk12.0/hdk_ga_using.html#HDK_GA_FileFormat

Cheers

[Eric Falconer]

I have a quick question relating to performance. I have my initial H12 changes running, I'm able to load the new BGEO's directly and maya and at least all the position, velocity and custom attributes and stuff are coming through just fine.

The issue I'm having now is that when my FX guys got a hold of it, they put a modest 1-2 million particle export through it, and the performance is a lot slower than we were hoping. So I'm basically just trying to figure out if it's an inefficiency in how I"m dealing with the new file formats, or if it's an issue in general with going between partio's internal structure and maya's particle systems. Basically how many particles have you reasonably run through it without much delay between frames?

Still waiting for clearance to post my code, but will as soon as I get the go-ahead.

On Thursday, April 26, 2012 7:49:28 PM UTC-4, redpawFX wrote:

[Andrew Selle]

You could do a partconv to convert to the old format, and then see if
it is any faster with the old format. Also, make sure you are using
the partio particle iterators if you can which are a lot faster than
the data calls (which are virtual). By contrast the particle
iterators are far fewer virtual calls amortized over many trivial
inlined non-vritual things. However, the bottom line is that reading
particles incurs a copy and there is no way to really avoid this
without writing a custom back end that used the file in place using
mmap. This could be an interesting way to go, and in fact, that is why
ParticlesSimple is called that. You can write a custom back end that
plugs itself in when you read certain formats. Hope this makes some
sense.

We usually have not been using millions of particles with interactive
visualization, but we have rendered shots that have had up to about a
million per particle set.

-Andy

[Michael Auerswald]

Hey guys,

as a follow up - I'm not sure if everyone is aware of it, but it turns out to be very easy to still use the old file format within houdini. Just add a second File node and use the .bhclassic / .hclassic extension for the binary/ascii file format using the old H11 format.

It's just a matter of modifying the PaticleIO.cpp and adding two entries to the read / write maps:
data["bhclassic"]=readBGEO;
data["hclassic"]=readGEO;
...
data["bhclassic"]=writeBGEO;
data["hclassic"]=writeGEO;

Cheers
Michael

[Andrew Selle]

I believe there is also an environment variable that will make bgeo/geo write the old format by default.  However, that has compatibility implications. What we have done at disney is we have a rop that writes a .bhclassic and then a post convert step that renames it to a bgeo.

-Andy

[Michael Auerswald]

True, which - since the old format does not support all the features of the new one - is why I prefer to keep the name bhclassic and rather compile my partio lib anew. I want to be able to see at first glance whether I'm dealing with the old or (potentially) new format. But that's a matter of taste I guess.

I wasn't able to find a whole lot documentation, but there is an environment variable HOUDINI_GEOMETRY_FORMAT = hclassic that will switch the default behaviour to the old file format. However, that may cause issues when Houdini is trying to store a feature only supported in the new format. So for me, I just have a separate File SOP / ROP for the old format to exchange data.

Michael

--
Michael Auerswald | Voltairestr. 3 | 10179 Berlin | Germany
mic...@krautsourced.com | http://www.krautsourced.com
USt-ID: DE276454490

[Albert Omoss]

Is anyone still actively working on this? I've found the specs on the new file format and was going to try to implement it, but it seemed like a waste if someone else was already close.

New Houdini Geo/BGeo Specs:
http://www.sidefx.com/docs/hdk12.0/hdk_ga_using.html#HDK_GA_FileFormat

[Andrew Selle]

I had a reader pretty well underway. But it needs a lot of work. I can upload it into my branch tomorrow when I get to work.

[Andrew Selle]

I've checked into my current state of progress into my branch. It works with a simple point test that I am attaching. One thing I've heard is that the format might be changing again. For our production use we have prioritized using bhclassic for now because of this. Personally I prefer the old format, because even though it is limited, it is bone-head simple to parse. The new format is quite tricky (as you can see in my implementation). Nevertheless I am happy to help however I can... Unrelated I plan to do a major push on Partio right after the SIGGRAPH deadline (Jan 18) as well, so I'd love to hear everybody's requests now.

https://github.com/wdas/partio/tree/aselle

How to use
1. Run jsontest <filename>
This will currently try to read a binary json file and if successful it will write to a testdump.bgeo (old style bgeo)

Problems
1. Not really a reader. We need to change BGEO.cpp to read the magic code and go into the old format or new code (because extension is no longer sufficient for file format detection). This shouldn't be too harder.
2. With mesh geometry I believe the parser fails rather than the old bgeo reading just ignoring anything but points. This is the main reason I haven't deployed.
3. Needs a lot of testing
4. Needs ascii json support (should be easy)
5. Make a writer. I don't know how important this really is. Definitely lower priority than writing as long as houdini retains backward compatibility.

In doing this I've tried to stay true to my goal with partio to minimize dependencies to the bare minimum. I'm sure linking to the Houdini libraries would allow a much easier implementation of a reader. I've been hesitant to do this for now. I certainly would like to get to the point where file readers/writers are dynamically linked plugins instead of hard-coded into partio first.

-Andy

On Tue, Jan 8, 2013 at 3:43 PM, Albert Omoss <alber...@gmail.com> wrote:

[Albert Omoss]

Thanks for pushing that! 

I agree with your sentiments on not including the houdini libraries. I've forked the repo and will try to help in any way that I can.

[Ben Rogall]

Hi, I just wanted to check on the status of support of the Houdini 12+ bgeo format. Specifically are there any branches/forks which include a reader for  it? I have a Partio for Modo plugin that I am planning to update and I had a request for supporting the new bgeo format.

[Andrew Selle]

I picked it up again, and I found that I need to do some more work to support the format. I'm playing with it in my limited freetime. I hope to get something by the end of the year at the latest.

On Fri, Oct 25, 2013 at 8:06 AM, Ben Rogall <benr...@gmail.com> wrote:

Hi, I just wanted to check on the status of support of the Houdini 12+ bgeo format. Specifically are there any branches/forks which include a reader for  it? I have a Partio for Modo plugin that I am planning to update and I had a request for supporting the new bgeo format.

--
You received this message because you are subscribed to the Google Groups "partio-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to partio-discus...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.