LLVM ERROR: Error reading bitcode file: Invalid record

[Søren Ragsdale]

I've stumbled upon a fairly crazy issue. I've got this minimal shader:

surface dnDiffuse_srf () {
        // __texture ends up in CCout so that it can be composited and mixed
        Ci = debug("__N") * color(normalize(N));
}

When I run this within our production prman-19 OSL evaluation engine, it works fine as long as llvm_optimize=0. When I set llvm_optimize=1, I get:

X00004 [ dnOslMaterial ] Loaded "/u/sfr/tools/osl/default/osl-1.6/srf/dnDiffuse_srf.oso" (took 0.00s)
LLVM ERROR: Error reading bitcode file: Invalid record

Here's the crazy part. If I remove the 'normalize', it renders fine with llvm_optimize=1.

surface dnDiffuse_srf () {
        // __texture ends up in CCout so that it can be composited and mixed
        Ci = debug("__N") * color(N);
}

Not sure if it helps, but I've attached the LLVM code that gets emitted with llvm_debug=0/1. "opt0.ll" contains the version that works, "opt1.ll" contains the version that doesn't.

This is all running with:

Clang/LLVM 3.4.2

OSL 1.6.8

[Solomon Boulos]

(From an account that won't get bounced)

On Wednesday, August 12, 2015, Solomon Boulos <bou...@cs.stanford.edu> wrote:

I'd guess this is a bitcode version incompatibility (that error message comes from the bitcode reader); what version of LLVM is prman's OSL using?

Without the normalize you're likely not generating anything more than a call (which is extremely stable) and also doesn't rely on anything in the OSL standard library (the big chunk of bitcode).

You should be able to look at the bitcode using llvm-dis (the disassembler) but it might be a pain to get it out of the binary (presumably OSLs build step still shoves the bitcode into a constant array in the code).

Solomon

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.
To post to this group, send email to osl...@googlegroups.com.
Visit this group at http://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

[Søren Ragsdale]

We're not using (or rather we shouldn't be using) Pixar's OSL plugin. We should be using our own OSL plugin, which we've verified is linked to the same LLVM 3.4.2 that everything else is.

/user_data/test/oslllvm $ ldd /builds/dneg_prmanshadeops/0.18.6/4894c8c768/lib/prman/ris/bxdf/dnOslMaterial.so | grep -i llvmbit
libLLVMBitWriter.so => /builds/llvm/3.4.2/f53a4bc162/lib/libLLVMBitWriter.so (0x00007f66d4455000)
libLLVMBitReader.so => /builds/llvm/3.4.2/f53a4bc162/lib/libLLVMBitReader.so (0x00007f66d422b000)

The OSL ShadingSystem statistics reports the OSL version - I wish that it also reported the LLVM version it was running.

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@googlegroups.com.

[Solomon Boulos]

I think the build system doesn't throw away the bitcode, so can you run *your* llvm-dis on the OSL standard library bitcode? 

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.

To post to this group, send email to osl...@googlegroups.com.
Visit this group at http://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "OSL Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.

[Søren Ragsdale]

That's a good suggestion to investigate. (Current status: trying to figure out how to extract the LLVM bitcode from wherever liboslexec stashes it.)

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@googlegroups.com.

To post to this group, send email to osl...@googlegroups.com.
Visit this group at http://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@googlegroups.com.

[Søren Ragsdale]

For what it's worth I took the easy route. Upgraded our plugin to LLVM 3.5.2 and the problem disappeared.

[Solomon Boulos]

So wait, did you have both 3.4.2 and 3.5.2 installed on your system?  If so, I think that means that llvm-dis would have confirmed my fear: the bitcode was produced by a later llvm (3.5.2) than was linked into OSL (3.4.2).

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.

To post to this group, send email to osl...@googlegroups.com.
Visit this group at http://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.

To post to this group, send email to osl...@googlegroups.com.
Visit this group at http://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.

[Larry Gritz]

Agreed, I think Solomon has a good theory.

There is one module that is converted to bitcode when OSL is built (which is then folded as a big array into the C++ source that's compiled into the library).

If the version of clang used to compile that bitcode at build time is different than the version of LLVM that's linked against to interpret that bitcode at runtime, there could be a problem that sounds suspiciously like what you described. The cmake scripts should be grabbing them from the same place, but I can imagine weird installations where it could get confused.

--
Larry Gritz
l...@larrygritz.com

[Dan Kripac]

While the clang version clash theory may not be out of the question (i.e perhaps it has been loaded into process space by another external library?), Soren has been building OSL inside a contained environment that should only pick up the single version of LLVM that he is targeting.

Would the bytecode inside a compiled oso files affect this situation? We do not tend to recompile the oso files when experimenting with different LLVM versions.

[Solomon Boulos]

So what I've done on a different project is require people to specify LLVM_BUILD_DIR (OSL's make setup calls this LLVM_DIRECTORY which is fine as well), and then use the excellent llvm-config to get the related info:

 EXEC_PROGRAM(${LLVM_BUILD_DIR}/bin/llvm-config

                   ARGS --includedir OUTPUT_VARIABLE LLVM_INCLUDE_DIR)

   EXEC_PROGRAM(${LLVM_BUILD_DIR}/bin/llvm-config

                   ARGS --libs OUTPUT_VARIABLE LLVM_ARCHIVES)

      SET(LLVM_LIBRARY -L${LLVM_BUILD_DIR}/lib "${LLVM_ARCHIVES}" -lpthread -ldl)

   SET(LLVM_LINKER_APP ${LLVM_BUILD_DIR}/bin/llvm-link)

   SET(CLANG_APP       ${LLVM_BUILD_DIR}/bin/clang)

Honestly at this point, osl doesn't need to support llvm-g++ anymore and should probably just fail hard instead of letting it go anywhere on their system (https://github.com/imageworks/OpenShadingLanguage/blob/master/src/liboslexec/CMakeLists.txt#L82)

[Larry Gritz]

What's inside oso files has nothing to do with LLVM, and has been extremely stable across long time periods. The worst that can happen if the oso is out of sync, which can only happen if you compile (via "oslc") with a newer OSL release than you are using to render, is that you'll get a (non-LLVM) error message about an unknown built-in function referenced by the shader.