OSL 1.10 branched!

[Larry Gritz]

I have branched OSL RB-1.10 and tagged Release-1.10.1-beta1.

Yeah, I know I'd been talking about a "2.0" release for months, but the two features that were supposed to be the centerpiece of a 2.0 re-architecting don't seem quite ready to go (PTX is in master but incomplete, batched SIMD shading is fairly complete but on a very old branch and will be substantial work to rebase). Yet I looked at the release notes and saw that it's been almost a full year since 1.9 was released. That's just too long. So I'm branching what we have, calling it 1.10. And if a true 2.0 with the new features is ready in just a couple months, so be it.

I'm calling it a "Dec 1" release, so that's a very short beta period. I'm confident in the code, since it's what we've used at SPI all year, completely depending on it for Hotel Transylvania 3, The Meg, Smallfoot, and Spider-Man Into the Spider-Verse.

Now is the time for the rest of you to give it a spin (if you haven't been trying/using master all along) and let me know about any issues, especially those for which a fix might require breaking changes to the APIs, which we will not do once it becomes the "release" branch.

At the moment, master and RB-1.10 point to the same place, but they will diverge as soon as something needs to be committed to master that we don't think should be immediately backported to 1.10.

Dependency compatibility note: OSL 1.10 dropped support for LLVM 3.x (4.0 is the new minimum, though I recommend upgrading to the newest 7.0), and for OIIO 1.7 (1.8 is the new minimum, and I recommend 2.0 which is currently in beta).

Full release notes are below, detailing all the important changes compared to 1.9. Enjoy.



Release 1.10 -- Dec 1, 2018 (compared to 1.9)
--------------------------------------------------
Dependency and standards changes:
* **LLVM 4.0 / 5.0 / 6.0 / 7.0**: Support has been removed for LLVM 3.x,
added for 6.0 and 7.0.
* **OpenImageIO 1.8/2.0+**: This release of OSL should build properly against
OIIO 1.8 or 2.0. Support has been dropped for OIIO 1.7.

New back-end targets:
* **OptiX** Work in progress: Experimental back end for NVIDIA OptiX GPU ray
tracing toolkit. #861, #877, #902
- Build with `USE_OPTIX=1`
- Requires OptiX 5.1+, Cuda 8.0+, OpenImageIO 1.8.10+, LLVM >= 5.0 with
PTX target enabled.
- New utility **testoptix** is an example of a simple OptiX renderer
that uses OSL for shaders.
- Work is in progress to support the majority of OSL, but right now it
is restricted to a subset. All the basic math, most of the
standard library, noise functions, strings (aside from if you create
entirely new strings in the middle of a shader), and closures work.
The biggest thing not working yet is textures, but those are coming
soon.

New tools:
* **osltoy** : GUI tool for interactive shader editing and pattern
visualization (somewhat in the style of [Shadertoy](http://shadertoy.com).
#827, #914, #918, #926 (1.10.0)
* **docdeep** : This Python script (in src/build-scripts/docdeep.py) is an
experimental tool to scrape comments from code and turn them into
beautiful Markdeep documentation. (A little like a VERY poor man's
Doxygen, but markdeep!) Experimental, caveat emptor. #842 (1.10.0)

OSL Language and oslc compiler:
* In OSL source code, we now encourage the use of the generic "shader" type
for all shaders, and it is no longer necessary or encouraged to mark
the OSL shader with a specific type, "surface", "displacement", etc.
From a source code and/or `oslc` perspective, all shaders are the same
generic type. A renderer may, however, have different "uses" or "contexts"
and may additional runtime perform error checking to ensure that the
shader group you have supplied for a particular "use" does not do things
or access globals that are not allowed for that use. #899
* C++11 style Initializer lists. (#838) This lets you have constructs like

// pass a {} list as a triple, matrix, or struct
void func (point p);
func ({x, y, z});

// Assign {} to a struct if the types match up
struct vec2 { float x, y; };
vec2 v = {a,b};

// Compact 'return' notation, it knows to construct the return type
vec2 func (float a, float b)
{
return {a, b};
}

* osl now warns when it detects duplicate declarations of functions with
the exact same argument list, in the same scope. #746
* osl now correctly reports the error when you write to a user-function
parameter that was not declared as `output` (function params are by
default read-only, but a shortcoming in oslc has prevented that error
from being issued). #878 (1.10.0)
* Fix oslc crash with invalid field selection syntax. #835 (1.10.0/1.9.6)
* oslc fix to properly handle command line arguments if the shader file is
not the last argument on the command line. #841 (1.10.0/1.9.7)
* oslc: when using boost.wave for preprocessing, fix whitespace insertion
#840 and windows path separators #849. #841 (1.10.0/1.9.7)
* oslc: Fix bug/undefined behavior when trying to format/printf a struct.
#849 #841 (1.10.0/1.9.7)
* New rules for how polymorphic function variants are chosen: Matches are
now ranked in an objective way that no longer depends on declaration
order. Type coercions are preferred in the following order of descending
score: exact match, int -> float, float -> triple, spatial triple ->
spatial triple, any triple -> triple. If there is a tie for passed
arguments, return types will break the tie. If there is still a tie or
truly ambiguous case, a warning will be printed explaining the choices and
which was chosen. #844 (1.10.0)
* It is now a warning to define the same function twice in the same scope.
#853 (1.10.0)
* A shader input parameter marked with metadata `[[ int allowconnect = 0 ]]`
will disallow runtime connections via `ConnectShaders()`, resulting in an
error. #857 (1.10.0)
* oslc command-line argument `-Werror` will treat all warnings as hard
errors (failed compilation). #862 (1.10.0)
* `#pragma osl nowarn` will suppress any warnings arising from code on the
immediately following line of that source file. #864 (1.10.0)
* oslc error reporting is improved, many multi-line syntactic constructs
will report errors in a more intuitive, easy-to-understand line number.
#867 (1.10.0)
* Faster parsing of very large constant initializer lists for arrays in OSL.
We found an inadvertent O(n^2) behavior when parsing initializer lists.
It used to be that a constant table in the form an array of 64k floats
took over 10s to compile, now it is 50x faster. #901
* Resolution of ambiguous return type functions (such as noise) has been
improved when their results are used as arguments to type constructors.
#931 (1.10.1)

OSL Standard library:
* There has been a change in the appearance to Cell noise and Gabor noise.
This is to fix a bug that made an incorrect pattern for certain negative
exact-integer values for cellnoise, and in lots of places for Gabor noise.
The only way to fix it was to potentially change the appearance for some
shaders. Sorry. If this is a real problem, let us know, perhaps we can
make a build-time switch that will let you use the old buggy noise? But
since this is a "2.0" release, we figured it was as good a time as ever to
let it change to the correct results. #912 (1.10.0)

Shaders:
* Contributed shader library changes:
* mandelbrot.osl: computes Mandelbrot and Julia images. #827 (1.10.0)
* MaterialX support:
* Improved support for MaterialX 1.36: add sin, cos, tan, atan2, ceil,
sqrt, exp, determinent, ln, transpose, sign, rotate, transforms,
rgb/hsv convert, extract, separate, tiledimage. Rename exponent ->
power, pack -> combine, hsvadjust -> hueshift. Add some color2/4
mutual conversion operators. Fixes to ramp4, clean up texture mapping
nodes, fixes to triplanarprojection weighting. Extend add/sub/mul/div
to include matrices. #903, #904, #905, #907, #909 (1.9.10/1.10.0)

API changes, new options, new ShadingSystem features (for renderer writers):
* ShadingSystem API:
* It is now permitted to ConnectShaders a single component of a
color/point/vector/normal to a float and vice versa. #801 (1.10.0)
* An older version of ShadingSystem::execute, which had been marked
as deprecated since OSL 1.6, has been fully removed. #832 (1.10.0)
* `ShadingSystem::Shader()` now has all three parameters required (none
are optional), though the "use" parameter no longer has any meaning.
(It will be deprecated and removed in a future release.) #899
* ShadingSystem attributes:
* New `"allow_shader_replacement"` (int) attribute, when nonzero, allows
shaders to be specified more than once, replacing their former
definitions. The default, 0, considers that an error, as it always
has. #816 (1.10.0).
* New developer option `"llvm_output_bitcode"` dumps the bitcode for each
group, even if other debug options aren't turned on, and also any
dumped bitcode will save as text as well as binary. #831 (1.10.0)
* New attribute `"error_repeats"`, if set to non-zero, turns off the
suppression of multiple identical errors and warnings. Setting it
(even to its existing value) also clears the "already seen" lists.
#880, #883 (1.10.0/1.9.9/1.8.14)
* Shader group attributes:
* New attributes that can be queried with `getattribute()`:
`"globals_read"` and `"globals_write"` retrieve an integer bitfield
that can reveal which "globals" may be read or written by the shader
group. The meaning of the bits is given by the enum class `SGBits`
in `oslexec.h`. #899
* RendererServices API:
* Older versions of RendererServices texture functions, the old ones
with no errormessage parameter, which were documented as deprecated
since 1.8, are now marked OSL_DEPRECATED. #832 (1.10.0)
* Miscellaneous:
* liboslnoise: Properly hide/export symbols. #849 (1.10.0/1.9.7)
* The behavior of the "searchpath:shader" attribute, used for finding
`.oso` files when shaders is requested, has been modified. If no
search path is specified, the current working directory (".") will
be searched. But if there is a search path attribute specified, only
those directories will be searched, with "." only searched if it is
explicitly included in the search path value. #925 (1.10.0)

Bug fixes and other improvements (internals):
* The context's `texture_thread_info` is now properly passed to the
RenderServices callbacks instead of passing NULL. (1.10.0)
* Symbols are enbled in the JIT, allowing Intel Vtune profiler to correctly
report which JITed OSL code is being executed. #830 (1.10.0)
* ASTNode and OSLCompilerImpl -- add info() and message() methods to
complement the existing error and warning. #854 (1.10.0)
* Fix incorrect array length on unbounded arrays specified by relaxed
parameter type checking. #900 (1.9.10/1.10.0)
* oslc bug fix: the `regex_search()`/`regex_match()` functions did not properly
mark their `results` output parameter as write-only. This was never
noticed by anyone, but could have resulted in incorrect optimizations.
#922 (1.10.0)
* When reading `.oso` files, the parser is now more robust for certain ways
that the oso file might be corrupted (it's more forgiving, fewer possible
ways for it to abort or assert). #923 (1.10.0)
* Bug fixes related to incorrect reliance on OIIO's `ustring::operator int()`.
It's being removed from OIIO, so wean ourselves off it. #929 (1.10.0)
* Certain texture3d lookups with derivatives could crash. #932 (1.10.1)

Build & test system improvements:
* Appveyor CI testing for Windows. #849,852,855 (1.10.0/1.9.7)
* Our new policy is to disable `STOP_ON_WARNING` for release branches, to
minimize build breaks for users when a new compiler warning is hit. We
still enable it in development/master branches as well as any CI build
in any branch. #850 (1.10.0/1.9.7)
* Testsuite is now Python 2/3 agnostic. #873 (1.10.0)
* Build the version into the shared library .so names. #876
(1.8.13/1.9.8/1.10.0)
* Update to fix with OpenImageIO 1.9. #882,#889
* Flex/bison fixes on Windows. #891
* Fix Windows build break with iso646.h macros. #892
* Fix warnings on gcc 6. #896
* Fix errors building with MSVC. #898
* Fixes to build with clang 7, and to use LLVM 7. #910, #921 (1.10.0)
* Fix warnings on gcc 8. #921 (1.10.0)
* Build system: the variables containing hints for where to find IlmBase
and OpenEXR have been changed to `ILMBASE_ROOT_DIR` and `OPENEXR_ROOT_DIR`
(no longer `ILMBASE_HOME`/`OPENEXR_HOME`). Similarly, `OPENIMAGEIO_ROOT_DIR`
is the hint for custom OIIO location (no longer OPENIMAGEIOHOME). #928
* Eliminated some in-progress MaterialX tests, they weren't in good order,
we will do it differently if we want to add MX tests in the future. #928

Documentation:
* `osltoy` documentations in `doc/osltoy.md.html` (1.10.0).


--
Larry Gritz
l...@larrygritz.com

[Olivier Paquet]

Hi Larry,

Le mardi 20 novembre 2018 18:46:09 UTC-5, Larry Gritz a écrit :

Now is the time for the rest of you to give it a spin (if you haven't been trying/using master all along) and let me know about any issues, especially those for which a fix might require breaking changes to the APIs, which we will not do once it becomes the "release" branch.

 

Fixing it does not absolutely require breaking the API but I did come across a problem which might justify API changes recently. ShadingSystem::execute() receives a ShadingContext but ShadingSystem::optimize_group() does not. It creates its own internally which is stored in TLS (indirectly, through PerThreadInfo) and deleted only when the thread is terminated through some boost magic.

This is a problem because the ShadingContext destructor writes to the ShadingSystem, which means the ShadingSystem has to outlive the thread. This is not obvious at all when you look at the APIs and is a big problem if you want to use TBB threads to optimize shaders. The only way you can be certain they will not outlive the ShadingSystem is to not delete the ShadingSystem at all.

It would be enough for optimize_group() to receive a PerThreadInfo but as execute() gets a ShadingContext, it might make sense to give it that too. On the other hand, ShadingContext could also be made to not need to keep a reference to a valid ShadingSystem after it is pushed back into the pool. Or even better, the ShadingSystem could clean up the automatically allocated ShadingContext from all threads when it is destroyed. That's the behavior I would have expected from reading the headers and it would not require an API change after all.

Olivier

[Larry Gritz]

That sounds very reasonable. I'll take a look at it today.

BTW, adding new API calls that can be unambiguously distinguished from existing ones (such as having an additional parameter) is generally not a breaking change. I'm much more worried about removing calls, or changing the order or meaning of arguments in the existing calls, or changing the internal layout of any public data structures.

-- lg

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.
To post to this group, send email to osl...@googlegroups.com.
Visit this group at https://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--

Larry Gritz

l...@larrygritz.com

[Olivier Paquet]

Le mercredi 21 novembre 2018 11:28:44 UTC-5, Larry Gritz a écrit :

That sounds very reasonable. I'll take a look at it today.

No hurry, I've worked around it for now. I just thought I would not be the last one to hit this problem. 

 

BTW, adding new API calls that can be unambiguously distinguished from existing ones (such as having an additional parameter) is generally not a breaking change. I'm much more worried about removing calls, or changing the order or meaning of arguments in the existing calls, or changing the internal layout of any public data structures.

Indeed, it has some long term ugliness but it can be managed.

Olivier

[Larry Gritz]

Proposed change:

https://github.com/imageworks/OpenShadingLanguage/pull/936

--

Larry Gritz

l...@larrygritz.com