Houdini Ready plugin work started

[Dan Wills]

Hi guys,
I finally bit the bullet and have started writing the Ready-Houdini
plugin that I suggested a long time ago, feels good to make some
progress :D

I've got something to compile and I /think/ it's properly linked.

Initially I'm just seeing if I can get the code from the commandline
example 'rdy' to work, to make sure that everything is functioning as
it should.

I'm getting a segfault at the call to:

OpenCL_utils::IsOpenCLAvailable()

It's a bit tricky to follow in a strace, but houdini writes a crash
log that suggests that the signal 11 is coming from clLibLoad, which
is a method in:
src/readybase/OpenCL_Dyn_Load.c

Now, houdini also uses openCL in a special mode of its fluid solver so
I'm wondering if that could be interfering with the way Ready loads
the lib.

Any thoughts or ideas on how best to debug this would be muchly appreciated :)

Thanks heaps,
Dan


Ps, this is the relevant bit of the crash report:

Crash report from dan; Houdini FX Version 13.0.198.21 [linux-x86_64-gcc4.6]
Traceback from Wed Nov 13 09:23:48 2013
Caught signal 11

AP_Interface::coredumpOutputLog(UTsignalHandlerArg) <libHoudiniUI.so>
AP_Interface::coredumpHandler(UTsignalHandlerArg) <libHoudiniUI.so>
UT_Signal::UT_ComboSignalHandler::operator()(int, siginfo*, void*)
const <libHoudiniUT.so>
UT_Signal::processSignal(int, siginfo*, void*) <libHoudiniUT.so>
_L_unlock_15 <libpthread.so.0>
clLibLoad <SOP_reactionDiffusion.so>
OpenCL_utils::LinkOpenCL() <SOP_reactionDiffusion.so>
OpenCL_utils::IsOpenCLAvailable() <SOP_reactionDiffusion.so>
SOP_reactionDiffusion::cookMySop(OP_Context&) <SOP_reactionDiffusion.so>
SOP_Node::cookMe(OP_Context&) <libHoudiniOP2.so>
OP_Node::cook(OP_Context&) <libHoudiniPRM.so>

[Andrew Trevorrow]

Dan:

> It's a bit tricky to follow in a strace, but houdini writes a crash
> log that suggests that the signal 11 is coming from clLibLoad, which
> is a method in:
> src/readybase/OpenCL_Dyn_Load.c
>
> Now, houdini also uses openCL in a special mode of its fluid solver so
> I'm wondering if that could be interfering with the way Ready loads
> the lib.

If your houdini code is already linking/loading the OpenCL lib somehow
then presumably the readybase code doesn't need to try and load it,
so you probably just need to disable the call to clLibLoad() in
OpenCL_utils::LinkOpenCL() which is in src/readybase/OpenCL_utils.cpp.
Just return CL_SUCCESS instead.

The same thing is done on a Mac -- this is because the OpenCL lib
is linked in (because it's available on every Mac running 10.6+)
rather than dynamically loaded as on Win/Linux.

Andrew

[Dan Wills]

Thanks heaps for the suggestion Andrew!

It's a bit hard to know whether my plugin would have access to the lib
that houdini loads.. I have -lOpenCL in my link flags, but I would
have thought that all the openCL linking that mattered would already
be sorted in libreadybase.a

I should write to sideFX to see if they have any advice about this too.

I did as you suggested, and now it gets past the clLibLoad call
(obviously) but it still crashes.

After that I isolated it to the call to clGetPlatformIDs (in
OpenCL_utils::IsOpenCLAvailable()). It seems like any CL call is
causing the problem.. I will do some more investigation.

On Wed, Nov 13, 2013 at 12:07 PM, Andrew Trevorrow <and...@trevorrow.com> wrote:
> Dan:

>
>> It's a bit tricky to follow in a strace, but houdini writes a crash
>> log that suggests that the signal 11 is coming from clLibLoad, which
>> is a method in:
>> src/readybase/OpenCL_Dyn_Load.c
>>
>> Now, houdini also uses openCL in a special mode of its fluid solver so
>> I'm wondering if that could be interfering with the way Ready loads
>> the lib.
>

> If your houdini code is already linking/loading the OpenCL lib somehow
> then presumably the readybase code doesn't need to try and load it,
> so you probably just need to disable the call to clLibLoad() in
> OpenCL_utils::LinkOpenCL() which is in src/readybase/OpenCL_utils.cpp.
> Just return CL_SUCCESS instead.
>
> The same thing is done on a Mac -- this is because the OpenCL lib
> is linked in (because it's available on every Mac running 10.6+)
> rather than dynamically loaded as on Win/Linux.
>
> Andrew
>

> --
> You received this message because you are subscribed to the Google Groups "reaction-diffusion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to reaction-diffus...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Trevorrow]

Dan:

> It's a bit hard to know whether my plugin would have access to the lib

> that houdini loads.. I have -lOpenCL in my link flags ...

Hmm, maybe try removing the -lOpenCL flag and put back the clLibLoad call.
If you want the readybase code to dynamically load the OpenCL lib then
I don't think it should be statically linked as well.

Andrew

[Dan Wills]

Hmm, still no dice, also I forgot to mention that I am using
-Wl,-Bstatic for the ready linkage and and -Wl,-Bdynamic for all the
rest (so -lOpenCL was already dynamically linked I reckon). My
thinking was that I want libreadybase.a to be statically linked but
the others all dynamic.

I also had trouble linking it until I added -fPIC (generate position
independent code) to the ready compile.

I yanked the link flags out of something that cmake made as part of
its build, which I realize might not be the best plan, but it got me
this far ;D

The two link lines in my makefile are:

LIBDIRS = -Wl,-Bstatic $(READYROOT)/libreadybase.a -Wl,-Bdynamic
LDFLAGS = -Wl,-Bstatic -L$(READYROOT) -lreadybase -Wl,-Bdynamic
-lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0
-lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo -lpango-1.0 -lfreetype
-lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0
-lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0
-lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo -lpango-1.0 -lfreetype
-lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0
-lvtkCommon -lvtkGraphics -lvtkIO -lvtkRendering -lvtkHybrid
-Wl,-rpath,/usr/X11R6/lib64

These get used by a little makefile-include system that ships with
Houdini, and come out looking like:

g++ -Wl,-Bstatic -L/home/dan/dev/ready/readyCommit/reaction-diffusion/
-lreadybase -Wl,-Bdynamic -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0
-lgio-2.0 -lpangoft2-1.0 -lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo
-lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0
-lgthread-2.0 -lrt -lglib-2.0 -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0
-lgio-2.0 -lpangoft2-1.0 -lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo
-lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0
-lgthread-2.0 -lrt -lglib-2.0 -lvtkCommon -lvtkGraphics -lvtkIO
-lvtkRendering -lvtkHybrid -Wl,-rpath,/usr/X11R6/lib64 -shared
SOP_reactionDiffusion.o -L/usr/X11R6/lib -Wl,-Bstatic
/home/dan/dev/ready/readyCommit/reaction-diffusion//libreadybase.a
-Wl,-Bdynamic -lGLU -lGL -lX11 -lXext -lXi -ldl -lpthread -o
SOP_reactionDiffusion.so

On Wed, Nov 13, 2013 at 1:30 PM, Andrew Trevorrow <and...@trevorrow.com> wrote:
> Dan:
>

>> It's a bit hard to know whether my plugin would have access to the lib

>> that houdini loads.. I have -lOpenCL in my link flags ...
>
> Hmm, maybe try removing the -lOpenCL flag and put back the clLibLoad call.
> If you want the readybase code to dynamically load the OpenCL lib then
> I don't think it should be statically linked as well.
>

[Dan Wills]

Had a think about this and it crossed my mind that having excess
linkage could be causing a problem, after all libreadybase.a is
already linked to all that stuff.

So I've now reduced it to:
LIBDIRS = -Wl,-Bstatic $(READYROOT)/libreadybase.a -Wl,-Bdynamic -lvtkIO

It needs vtkIO because it uses the Properties class (it also needs the
vtk include path for that reason)
That looks more reasonable but still has the exact same behavior :/

As a test I tried swapping the order so that it was
dynamic-then-static, like so:
LIBDIRS = -Wl,-Bdynamic -lvtkIO -Wl,-Bstatic $(READYROOT)/libreadybase.a

This gave an interesting linker warning:

/home/dan/dev/ready/readyCommit/reaction-diffusion//libreadybase.a(OpenCL_Dyn_Load.c.o):
In function `CL_LOAD_LIBRARY':
OpenCL_Dyn_Load.c:(.text+0x19): warning: Using 'dlopen' in statically
linked applications requires at runtime the shared libraries from the
glibc version used for linking

I would have thought that the glibc libs are always available on my
system so maybe that's safe to ignore..

Then it hits a linker error:

/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/../../../../x86_64-pc-linux-gnu/bin/ld:
/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/libstdc++.a(functexcept.o):
relocation R_X86_64_32 against `std::bad_exception::~bad_exception()'
can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/libstdc++.a: could not read
symbols: Bad value
collect2: ld returned 1 exit status

But then I realized that was because the linker wasn't being set back
to dynamic.. it goes back to linking when I fixed that, but still
crashes in houdini in the exact same way.

Cheers for the help Andrew, I really appreciate it! :D
Dan

[Tim Hutton]

I haven't looked at this in detail but just to note that it is often problematic to mix static and dynamic linked libraries. Try recompiling Ready (and the VTK and wxWidgets it uses) as dynamic and using that?

We compile as static to simplify the packaging - it's great to just hand out a single executable file.

--
Tim Hutton - http://www.sq3.org.uk - http://profiles.google.com/tim.hutton/

[Dan Wills]

That is a really good idea Tim, I will look into it :)

I've also written to sideFX to see if they might have any ideas..
fingers crossed.

Dan

[Dan Wills]

Awesome! SideFX have shown me how to get the openCL context that is
available in houdini, so I'm going to see if I can make a way to pass
that context into readybase.

Getting there ;D
d

[Dan Wills]

Cool, that worked! No more crashes! :D

I overloaded AbstractRD* CreateFromFile in SystemFactory with a
version that also accepts a cl_context object. Then I overloaded the
constructor for OpenCL_MixIn so that it can be passed in there as
well, and the passed-in one used instead of making a new one.

I guessed that this is probably something we'd want to keep separate
from the main ready build, so I introduced a #define
__EXTERNAL_OPENCL__ that I can use to switch to building
libreadybase.a that works this way.

Still lots of work to do (haven't seen any cell values inside houdini
yet), but it's progressing well. Thanks for all your help and
suggestions :)
Dan

[Dan Wills]

Further development: I got 'b reagent' cell values in the houdini
viewport, wohooo! :D (attached image is a screenshot).

[Tim Hutton]

Hi Dan,

I really appreciate your work in this direction - perhaps it will be useful in other software, not just Houdini.

A branch in svn is the best way to share experimental code. If we're happy with it all then we can merge the branch into mainline later. Go for it!
http://svnbook.red-bean.com/en/1.7/svn.branchmerge.using.html

Our branches live here:
https://code.google.com/p/reaction-diffusion/source/browse/#svn%2Fbranches

(I'll be making a new one for 0.6 soon.)

Tim

[Dan Wills]

Gday Tim,
Thanks so much for your guidance, I've started the branch, here:

http://code.google.com/p/reaction-diffusion/source/browse/#svn%2Fbranches%2FreadyHoudini

The plugin currently loads a vti, builds a rudimentary interface for
tweaking the parameters, and can read back the results into houdini as
either point-color, or voxel data (for 3d RD's). It's not super
optimized in terms of how it copies the data, so it's a little slower
than Ready at the moment, but still much faster than software solving!
:D

I'll post some results soon :)

I had some interest from a houdini guy: Kevin McNamara, who sounds
like he might be able to contribute (or at least help with testing)..
I'll see if he wants to join the list.

Thanks again for being so receptive :D and Ready 0.6 completely
rocks!! (those quantum demos are fascinating!)
Dan

[timb...@gmail.com]

Hi Dan

I stumbled across your work first on Vimeo then made my way eventually over here. Really amazing stuff.

I sent you a message on youtube but in case you don't get it there, I wondered if you might let me have a go with the Houdini plugin? I'm working on an art project at the moment that could really benefit.

Thanks for the inspiration anyway to you and all the other Ready development people

Tim

[Dan Wills]

G'day Tim, I did write a response to you on Youtube but I forgot to hit send, and when I came back later and pressed it, the session was invalid or something so it didn't go through, sorry about that!.

Unfortunately there isn't a windows build of readyHoudini yet.. as far as I know I'm the only person to have built it so far and I can only make it for linux. :/

*However*, there is another option - awhile back I implemented gray-scott in a houdini otl, and that was quite flexible and powerful (more so than the ready one at the moment) the only real drawback is that it's a bit slower, and you're limited to gray-scott (not too bad really ;)).

Since you seem interested I've attached the otls.. in order to see how they work, try this:

* Make a grid with at least 100x100 polys or so.

* Make a 'Reaction Diffusion Init' and wire it below the grid
* Assuming your grid is about 10 units on a side, set rad to 0.15 and Radius Difference to 0.07

* Make a 'Reaction Diffusion' and wire it to the init.

* When you cook the RD node it should show some dots starting to grow (300 iterations by default).
* If you want it to happen across time, put the rd node into a solver sop and turn the iterations down to ~10 or whatever you need.

* Check out the 'RD Map URL' parm on the RD node to get to Rob Munafo's awesome map - should help with getting the behavior you want.

I hope it works for you and I'd love to see anything you make with it!!
All the best, thanks for contacting me - always great to hear from like minded people :D

Dan

--

[cycblone]

Hi Dan

This work is really incredible. I am a houdini beginner. I am trying to follow your instructions, but don't know how to cook the setup? Which node do I need to add to do this?

Many thanks

To unsubscribe from this group and stop receiving emails from it, send an email to reaction-diffusion+unsub...@googlegroups.com.

[Dan Wills]

Hiya Cycblone (or should I say Paul? :)),

Thanks heaps for your enthusiastic interest! And good on ya for giving Houdini a go! I teach Houdini to groups of students and I totally appreciate how much there is to learn when you first start! Stick with it though, and it will pay you back many times over! :D

I'm assuming you are referring to my older Vops-based reaction-diffusion tools - The first thing is that you need to make sure you've done is to copy the otl files to the otls subdirectory of you user-houdini configuration.. on Linux (and assuming houdini17.0) this is a folder in your home-directory called something like ~/houdini17.0/otls, I'm not sure where it lives on other platforms I'm afraid. Copy the files there and they should be installed automatically when you start Houdini.
If all else fails you can install them manually by using "File->Import->Houdini Digital Asset" and browse to the otl/hda files to install them. That will only make them available in the current hip file though so it'd be worth figuring out the user-config/otls place for them to make them permanently available. Once that's done you should be able to follow the instructions in my earlier email (making a 'Reaction Diffusion Init' and a 'Reaction Diffusion' in Sops. Fair warning: These simulations can be a bit tricky to get going, and they sometimes need very specific initial conditions to kick off.. Gray-Scott isn't usually that hard though, for most settings of F and k, and that's what the 'Reaction Diffusion Init' otl is meant to help with).

I did make a compiled plugin 'readyHoudini' that is/was able to capture the result of a Ready sim to Houdini Volume Primitives in a Sop geometry.. however it's a bit of a trick to get that to compile!

I have also been working on a new method for fully importing Ready setups into Houdini that does not require a compiled plugin, but instead mainly relies on the "Gas OpenCL" Dop (a builtin part of Houdini). Being a Dops-based solution it suits RD-sims much better. In order to achieve this I have done quite a bit of work on the 'rdy' command-line utility (that comes with Ready) and added to it a bunch of arguments (via boost program_options) that can be used to print out the various bits of info about a Ready VTI file (reagent config, parameters, kernel source, reagent initial-states. I'll be adding the options part shortly) I also built supporting OTLs in Houdini that call the 'rdy' binary and bring in the various data. I need to check my changes into a branch and see if whoever is looking after the main Ready build will accept a pull request (a request to merge my changes with the main Ready code) ;D Fingers crossed on that I guess.. I think it will make a lot more things possible.

Best of luck on your RD-travels, and remember that it's very easy to emulate various RD things by simply feeding-back (think: Solver Sop) some perfectly normal pixel operations. For example repeatedly blurring and sharpening an image (in the right ratios) can already generate Turing patterns. Intermix this with various (blended) amounts of other filters such as edge-detection, emboss or dilate/erode, glass-distortion or warping and you'll find a vast amount of interesting things to explore ;D

All the best,

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to reaction-diffus...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "reaction-diffusion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reaction-diffus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.