Extensible State/Pipeline Configuration with PBR, Lighting & Shadows

[Robert Osfield]

Hi All,

This month I begin work the last major change block of new functionality that I have planned for myself before VulkanSceneGraph-1.0 that I'm aiming for in early Spring. I have my own ideas what should go into this work, but welcome input from the community as well as help with developing and testing the functionality.

My aim with this block of work is to add support for support for multiple light sources, shadows and texture projection, with a set of configurable shader support with default set of shaders provided that provide support for PBR etc. 

The functionality will provide new scene graph VSG classes that will hold lighting, shadows, texgen  etc. settings, as well as new application classes for configuring the set up of state objects that are placed into the scene graph by vsgXchange loaders and classes like vsg::Builder. 

The state configuration will be done with a state cache so that can be shared via the vsg::Options when loading data, so that all the models you load will be able to reuse the same state, including graphics pipelines etc.  My intention is make this all customizable, either via configuration or by sub-classing, so if you have a novel requirement of how to map loaded data to final scene graph you'll be able to do this.

Over the festive break I was away from my dev system so took the opportunity to do some design work on paper.  Now code to show off yet.  I still have more research and design work before I dive in. 

I'm unlikely to come up with the ideal set of interface and implementation from the start so I'm expecting I'll be creating a series of branch of VulkanSceneGraph, vsgXchange and vsgExamples to explore different approaches.  I'll post on this thread to keep you all informed of the work, and call for testing/feedback.

I'd really appreciate test models/applications as well so if you'd like to coordinate your own work with these new additions then let me know what you models and applications are like, what their needs relating to the above functionality are etc.

Cheers,

Robert.

[Robert Osfield]

Hi All,

To help with assessing what we need to map for Lights and Cameras stored in 3d scenes loaded from Assimp I've created a Lights branch of vsgXchange to output the aiLight and aiCamera properties:

    https://github.com/vsg-dev/vsgXchange/tree/Lights

Assimp repository has a range of test models, on unix you can use find and vsgconv to list all the models:

     assimp/test/models$ find . -type f -exec vsgconv {} test.vsgb \;

An example of the console output:

filename = ./FBX/global_settings.fbx, mNumLights = 2, mNumCameras = 2
  light 0x559226ef6540
     light->mName = Lamp.001
     light->mType = 2 aiLightSource_POINT
     light->mPosition = 0, 0, 0
     light->mDirection = 0, -1, 0
     light->mUp = 0, 0, -1
     light->mAttenuationConstant = 0, mAttenuationLinear = 0, mAttenuationQuadratic = 8.8889e-05
     light->mColorDiffuse = 1, 1, 1
     light->mColorSpecular = 1, 1, 1
     light->mColorAmbient = 0, 0, 0
     light->mAngleInnerCone = 6.28319, mAngleOuterCone = 6.28319
     light->mSize = 0, 0
  light 0x559226f36ef0
     light->mName = Lamp
     light->mType = 2 aiLightSource_POINT
     light->mPosition = 0, 0, 0
     light->mDirection = 0, -1, 0
     light->mUp = 0, 0, -1
     light->mAttenuationConstant = 0, mAttenuationLinear = 0, mAttenuationQuadratic = 0.00222223
     light->mColorDiffuse = 1, 1, 1
     light->mColorSpecular = 1, 1, 1
     light->mColorAmbient = 0, 0, 0
     light->mAngleInnerCone = 6.28319, mAngleOuterCone = 6.28319
     light->mSize = 0, 0
  camera 0x559226ef4240
     camera->mName = Camera.001
     camera->mPosition = 0, 0, 0
     camera>mLookAt = 1, 0, 0
     camera->mUp = 0, 1, 0
     camera->mHorizontalFOV = 0.857556
     camera->mClipPlaneNear = 0.5, mClipPlaneFar = 500
     camera->mAspect = 1.77778
  camera 0x559226f3ba60
     camera->mName = Camera
     camera->mPosition = 0, 0, 0
     camera>mLookAt = 1, 0, 0
     camera->mUp = 0, 1, 0
     camera->mHorizontalFOV = 0.857556
     camera->mClipPlaneNear = 0.1, mClipPlaneFar = 100
     camera->mAspect = 1.77778

The aiCamera might be just mapped to a vsg::Camera, but I'll need to have a think about how best to handle Camera's embedded in the scene graph.

The aiLight might be best mapped to a single new vsg::Light class, or perhaps multiple light classes to handle the different types.  This will also need to map to uniforms passed to the GPU.

[Robert Osfield]

> The aiCamera might be just mapped to a vsg::Camera, but I'll need to have a think about how best to handle Camera's embedded in the scene graph.

Currently the VSG has a vsg::View node that "has a Camera" and is used to decorate a subgraph with all the settings required to set up the rendering of view of a scene.  The vsg::Camera itself just holds the projection and view matrix settings, it doesn't actually take any pictures, this is effectively the responsibility of the vsg::RecordTraversal/RenderGraph/View combination.

The aiCamera is used a container of camera settings as well, so has that similarity to the vsg::Camera. but differs in that it can be placed in the scene graph as other nodes.  We could have a vsg::CameraMount node that provides a way of hooking a Camera into the scene graph so that it can be tracked, but I'm now inclined to just making vsg::Camera subclass from vsg::Node rather than vsg::Object, and allow it to be placed directly into the scene graph.  Such a vsg::Camera wouldn't do anything, it'd just be a passive container of camera settings.

To wire things up so that a vsg::View tracks an in scene graph Camera I'm thinking that the vsg::View's camera could be set up to watch the in scene graph camera using a vsg::ViewMatrix subclass that has a NodePath to the in scene graph Camera, and accumulates this to compute the vsg::View's camera's view matrix on each new frame so automatically tracking any in scene graph updates - such as when you have an in scene graph attached to a car in the scene that is diving around a world, with the car decorated by a vsg::MatrixTransform that moves the whole car, and the vsg::Camera attached to car.

Having a ViewMatrix subclass that tracks a NodePath is something I've been considering already, so adding support for Camera in this node path should be a natural extension to this.  This would allow you to track any node in the scene graph, but also a named Camera if you so wished.

Thoughts?

[Robert Osfield]

Hi All,

On Mon, 3 Jan 2022 at 18:56, Robert Osfield <robert....@gmail.com> wrote:

The aiCamera is used a container of camera settings as well, so has that similarity to the vsg::Camera. but differs in that it can be placed in the scene graph as other nodes.  We could have a vsg::CameraMount node that provides a way of hooking a Camera into the scene graph so that it can be tracked, but I'm now inclined to just making vsg::Camera subclass from vsg::Node rather than vsg::Object, and allow it to be placed directly into the scene graph.  Such a vsg::Camera wouldn't do anything, it'd just be a passive container of camera settings.

To explore the possibility of having a vsg::Camera in the scene graph for the purposes of providing scene graph level container for camera controls I have change vsg::Camera so it subclasses from vsg::Node, added std::string name to it, added serialization support to vsg::Camera, vsg::LookAt and vsg::Perspective, and then added support for adding vsg::Camera into the scene graph created by the vsgXchange/assimp loader.

The work can be found in the CameraNode branch of VulkanSceneGraph and vsgXchange:

    https://github.com/vsg-dev/VulkanSceneGraph/tree/CameraNode

    https://github.com/vsg-dev/vsgXchange/tree/CameraNode

The next step that would make this functionality more useful is a vsg::FindCameras visitor that finds all the vsg::Camera in a scene graph and records the NodePath to each one, then a vsg::ViewMatrixTrackNodePath that can be assigned as the rendering Vriew's Camera's ViewMatrix so that what you see on screen follows the Camera in the scene graph.

I will need to come up with the right naming for these classes, consider the vsg::FindCameras and vsg::ViewMatrixTrackNodePath as a placeholder.  I welcome suggestions.

Cheers,

Robert.

 

 

[Robert Osfield]

Hi All,

In prep for restructuring shaders for multiple light sources I've refactored vsgXchange's Assimp ReaderWriter to use shaders converted from GLSL source to .cpp's using vsgconv.  I've also unified the shaders so that vsg::Builder and the Assimp loader now use the same base GLSL source.

I also added console help output for vsgconv:

$ vsgconv

Usage:
   vsgconv input_filename output_filefilename
   vsgconv input_filename_1 input_filefilename_2 output_filefilename
Options:
   --batch             # batch all vertex and texture data  
   --features          # list all ReaderWriters and the formats supported
   --features rw_name  # list formats sipportred  
   --no-compile --nc   # do not compile shaders to SPIRV
   --rgb               # leave RGB source data in it's original form rather than converting to RGBA  
   -v --version        # report version

The original GLSL shader source is found in vsgExamples:

     https://github.com/vsg-dev/vsgExamples/tree/master/data/shaders

To convert to .cpp you can use vsgconv thus:

    cd vsgXchange/src/assimp/shaders

    vsgconv --nc ~/Dev/vsgExamples/data/shaders/assimp.vert assimp_vert.cpp

    vsgconv --nc ~/Dev/vsgExamples/data/shaders/assimp_pbr.frag assimp_pbr_frag.cpp

    vsgconv --nc ~/Dev/vsgExamples/data/shaders/assimp_phong.frag assimp_phong_frag.cpp

Once converted to .cpp the shaders can be found in vsgXchange/src/assimp/shaders:

    https://github.com/vsg-dev/vsgXchange/tree/master/src/assimp/shaders

I used the --nc no shader compile option so the the VSG/vsgXchange at load time will determine the #define's required to enable/disable the features wanted and allow these to be used when compiling the GLSL source to SPRIV.

All the changes made to vsgXchange are:

    https://github.com/vsg-dev/vsgXchange/pull/89/files

--

The next step in this work will be to map the Assimp aiLight structure to VSG structures so we can load scene graphs with lights already configured/placed in the scene.  

Cheers,

Robert.

[Robert Osfield]

Hi All,

I have checked in new vsg::Light base class, and vsg::AmbientLight, DirectionalLight, PointLight and SpotLight subclass classes from this.and i[date vsgXchange so that Assimp's ai:ght to these. Serialisation support is provided.  The additions to the VulkanSceneGraph are:

    https://github.com/vsg-dev/VulkanSceneGraph/pull/395/files

Changes to vsgXchange:

    https://github.com/vsg-dev/vsgXchange/commit/fd35cc29eb02edd582b7b85a45e228f495994c8e

The new vsg::Light classes are nodes so can be added anywhere in the scene graph, but there are just passive containers - much like vsg::Camera, they store the settings for other classes to use when setting up the data to pass to Vulkan.

The next items of work will be to add support for mapping these vsg::Light nodes to corresponding uniform buffers/textures that are passed to the GPU so the shaders can do the appropriate lighting.  Lights are also related to shadows, so eventually we'll need to add infrastructure for this as well.  However, I need to get there step by step and resolve any challenges that arise on route rather than get ahead of myself. 

With the next blocks of work, extensibility will be an important consideration - while I want decent out of the box functionality to be available for lighting I know that different applications have different needs, one size doesn't fit all.  I would appreciate the community involvement in refining this functionality.  Even if what your needs will be isn't appropriate for building directly into the core VSG default functionality, knowing what you need helps make sure the core functionality is extensible in a way that enables rather than hinders specialising functionality for your particular needs.

Also please don't assume that I know everything about lighting and shadows.  You may well have more knowledge and experience that can help guide me in a better direction, be it from your own personal experience or knowing that other software or publication takes a better approach.

I would also appreciate test datasets or examples to use in testing as I develop this functionality.

Cheers,

Robert.

[Robert Osfield]

Hi All,

My focus on the new lighting functionality is now on how we collect the relevant lights in the scene that will affect what is rendered in the view frustum and then pack this light data into a form that can be passed to the GPU so that the shaders can use them to control lighting.  To enable this I have created a vsg::ViewDependentState class to store all required data that is refreshed on each frame.  This new class is  current found in the include/vsg/viewer/View.h header in the Lights branch of the VSG.

   https://github.com/vsg-dev/VulkanSceneGraph/blob/master/include/vsg/viewer/View.h

The vsg::View "has a" ViewDepentState  object that is made current by the RecordTraversal so that all the lights encountered when traversing the scene can be added to the appropriate ViewDependetState.  A single record traversal can be used to render multiple views so we need to make sure that each view gets all the correct state that is relevant to it.

Since I want the functionality to work seamlessly with large worlds it makes sense to transform the lights from local scene graph coordinates where the light is added into eye coordinates, doing the transform all in doubles to avoid precision issues associated with trying to manage lights in world coords.  This also means the shaders need to do all the lighting calculations in eye coords, so bear this in mind when trying to reuse example shaders from the web/books - they may be written with calculations done in world coords. 

Doing light calcs in world coords is fine if you can do a simple game demo where you have a single local coordinate frame for your world and all the vertex data stays close to this origin.  It doesn't work for large worlds, if you want to do big worlds you really need to do lighting calcs in eye coords.

The shaders that we'll provide out of the box will all do lighting calcs in eye coords so for your first encounters with the VSG you probably won't even need to concern yourself with this little detail about coordinate frames for doing light calcs in the shaders.

I'm now edging closer to passing data to the GPU and to update the shaders, this will be done using a vec4Array that is packed with the active lights that is passed as uniform or texture data to the GPU via a DescriptorImage/Buffer.  This state is cached in the ViewDepedentState and will be pushed/popped from the vsg::State stacks during the record traversal.  At least that's my plan, there are still lots of unknowns I need to resolve to make this work.

To help illustrate how lights are added to the scene graph, and for the purposes of testing while I develop this new functionality I have created a vsglights example that is now checked into the vsgExamples Lights branch.

The code that adds the lights is currently:

    // ambient light
    {
        auto ambientLight = vsg::AmbientLight::create();
        ambientLight->name = "ambient";
        ambientLight->color.set(1.0, 1.0, 1.0);
        ambientLight->intensity = 0.05;
        scene->addChild(ambientLight);
    }

    // directional light
    {
        auto directionalLight = vsg::DirectionalLight::create();
        directionalLight->name = "directional";
        directionalLight->color.set(1.0, 0.5, 0.5);
        directionalLight->intensity = 0.25;
        directionalLight->direction.set(0.0, 0.0, -1.0);
        scene->addChild(directionalLight);
    }

    // point light
    {
        auto pointLight = vsg::PointLight::create();
        pointLight->name = "point";
        pointLight->color.set(0.2, 1.0, 1.0);
        pointLight->intensity = 0.5;
        pointLight->position.set(bounds.min.x, bounds.min.y, bounds.max.z);

        // enable culling of the point light by decorating with a CullGroup
        auto cullGroup = vsg::CullGroup::create();
        cullGroup->bound.center = pointLight->position;
        cullGroup->bound.radius = vsg::length(bounds.max-bounds.min)*0.1;

        cullGroup->addChild(pointLight);

        scene->addChild(cullGroup);
    }


    // spot light
    {
        auto spotLight = vsg::SpotLight::create();
        spotLight->name = "spot";
        spotLight->color.set(0.0, 1.0, 0.0);
        spotLight->intensity = 0.5;
        spotLight->position = bounds.max;
        spotLight->direction = (bounds.min - bounds.max)*0.5 - spotLight->position;

        // enable culling of the spot light by decorating with a CullGroup
        auto cullGroup = vsg::CullGroup::create();
        cullGroup->bound.center = spotLight->position;
        cullGroup->bound.radius = vsg::length(bounds.max-bounds.min)*0.1;

        cullGroup->addChild(spotLight);

        scene->addChild(cullGroup);
    }

Note, none of the light nodes (just like most VSG nodes) are not culled automatically, if you want them to be view frustum/distance culled then you need to decorate them with a CullGroup or a LOD.  In the vsglights example I have the vsg::DirectionLight and vsg::AmbientLight nodes being added without any culling so no matter what the view is they will always be enabled, with the PointLight and SpotLight are placed below a CullGroup which has it's bounding sphere set the size that encompasses the field of effect of the light.

Unlike the OpenSceneGraph/OpenGL the lights in the VSG don't have light numbers that you have to match to GL_LIGHTi modes that you enable for those lights, instead you just have as many Light nodes as you want in the scene graph and on each frame the required lights for that frame are all packed into a light data array.  The shaders will check how many lights are active in each frame and then handle them. 

The only limits will be how large the light data array is allocated (something that you can configure when setting up your vsg::View/ViewDepdentState objects, and the performance limits of your shaders handling the lights.  In practice it'll all be down to the shader performance as the amount of light data even for thousands of lights per frame will be trivial to compute and pass to the GPU.

My expectation is that handling lights in the VulkanSceneGraph will actually be far simpler than with the OpenSceneGraph, gone will be issues of the fixed function pipeline being limited to 8 lights and having to enable/disable specific light numbers, or having to come up with a completely alternative way of collecting and packing light data and passing it to the GPU when using shaders.  Instead the VSG will have a set of light nodes that are mapped across to GPU structures and done so in a way that can be customised when needed (by subclassing from ViewDependentState.)

In this first pass of work on lights I don't intend to tackle shadows or any clever techniques for multi-pass or culling of lights in order to scale up to handle scenes with many lights active in the view frustum at one time whilst maintaining good performance.  I expect we'll be able to do a couple of dozen lights with reasonable performance, to scale things really up we'll need to discuss how we do this, whether this gets directly supported by the core VSG or how we provide the extensibility for users to add their own custom approaches. 

If you have advanced techniques you have in mind please feel free to suggest/explain/post links for further discussion in this thread.

Cheers,

Robert.

[Robert Osfield]

Hi All,

While washing the dishing last night I spotted a problem with my current approach of putting BindDescritptorSet/DescriptorSet into the vsg::ViewDepdendentState class and having this push/popped at the top level when handling the vsg::View node during the record traversal.  The problem is the approach would only scale to scenes that have a single PipelineLayout. 

For scenes that have multiple PipelienLayoyt's - such as what you'd have with multiple set of Graphics Pipeline active in a scene it wouldn't work.  So.... I'm going to have to come up with a scheme for connecting scene graph state with the state supplied by the ViewDependentState container.  There isn't yet a class in VSG that can manage this trick so I will have to come up with one.  The vsg::SwitchState is a bit related but isn't sufficient.

This is what R&D is all about... 

Robert.

[Tim Moore]

It will work if the PipelineLayouts are compatible (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#descriptorsets-compatibility ) Is it reasonable to demand that pipelines that use the lighting are compatible up to the descriptor set that contains the lighting stuff? I think it's reasonable for VSG to reserve at least one descriptor set, if not two, for its own use.

Tim 

This is what R&D is all about... 

Robert.

--
You received this message because you are subscribed to the Google Groups "vsg-users : VulkanSceneGraph Developer Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vsg-users+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vsg-users/CAFN7Y%2BVAgrB6ERu%2BbyVfpLHyc7uqza%3DwyyZh-B94KNTqGsKSTQ%40mail.gmail.com.

[Robert Osfield]

Hi Tim,

On Thu, 20 Jan 2022 at 12:14, Tim Moore <timo...@gmail.com> wrote:

On Thu, Jan 20, 2022 at 11:26 AM Robert Osfield <robert....@gmail.com> wrote:

It will work if the PipelineLayouts are compatible (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#descriptorsets-compatibility )

Thanks for the link. This should give us a bit of leeway, though not enough for all usage cases.

 

Is it reasonable to demand that pipelines that use the lighting are compatible up to the descriptor set that contains the lighting stuff? I think it's reasonable for VSG to reserve at least one descriptor set, if not two, for its own use.

The ideal for me is for the VSG to be able to provide a decent default state/shader set up so that basic scene graph usage can be done without customization, whilst still giving power users complete control to customise the default feature set, or replace it entirely. 

Part of this ideal would be that the system would enable users to steadily move from default configuration through to advanced usage without having to throw functionality, knowledge away. Instead the user would see a logical way that the basic system works and how to customise it.

So that's an arm waving way to say no it's not reasonable to demand pipelines work in any particular way to leverage the light stuff.  However, this might have to be a fallback if I can't figure out a clean and logical way of making all this functionality extensible.

During development of this functionality I'm happy to start with having elements hardwired and not extensible, and once the basic mechanics of the functionality is working then look at ways to make sure the interface and implementation all make sense and are extensible.

To get things to work as flexibly as I'd like I may need to refactor parts of the VSG beyond the classes that are obviously connected to lighting.  One example of this that I'm considering is you suggestion of moving the Descriptor::dstBinding and dstArrayElement out of Descriptor into DescriptorSet as this might help with decoupling the ViewDependentState container that needs to manage passing the light data to the GPU via  Descriptor, and that Descriptor being assigned at different places for different shaders.

At this stage it's all experimental with no commitment to any particular solution.

Cheers,

Robert.

[Robert Osfield]

HI All,

I've been quiet on the topic of the new lighting functionality as I've been busy implementing initial hardwired code between the vsg::Builder class and the new vsglights example, then bit by bit replacing the hardwiring with classes that enable looser coupling between the high level view dependent state that is updated each frame and the lower level state/pipeline set up used to render the geometries in the scene.

I've now got vsg::Builder and vsg::View/ViewDependentState coupled loosely via a shared vsg::DescriptorSetLayouts object used for pipeline setup, and a new vsg::BindViewDescriptorSets state command that uses the vsg::DescriptorSet that the high level ViewDependentState object manages.  The BindViewDescriptorSets determines which ViewDependentState to reference based on what is current during the RecordTraversal which the BindViewDescriptorSets is encountered, rather than being hardwired to a specific one. 

This approach should make it possible to have a single scene graph be used in multiple views with each view getting it's own set of light data.  This is important as the light nodes in the scene graph are view frustum culled (if decorated by CullNode/LOD etc.), and those remaining are transformed from model coords into eye coords - so each view needs it's own set of culled and transformed light data.  I haven't tested it yet, but that's the what I'm aiming for. I'll test this later in the week.

Tomorrow I'll look to adding support for the new lighting functionality to vsgXchange::Assump, and plan to achieve this by passing in the ViewDependentState's descriptorSetLayout via vsg::Options so the the loader can use this DescriptorSetLoyout when configuring it's graphics pipeline layout and assigning the required vsg::BindViewDescriptorSets to enable the runtime coupling of scene graph state to the view dependent state.

Once the Assimp loader is working I'll look at doing the same for the OSG loader.  The vsg::Builder and vsgXchange::Assimp shaders are related, while the current OSG ones are completely different so it may require a bit more work to unify the shaders used in the OSG so they are the same or at least compatible enough with the ones used by vsg::Builder and vsgXchange::Assimp.

I'll post some screenshots tomorrow.  All going well I hope to have this functionality wrapped up this week.

Cheers,

Robert.

[Robert Osfield]

HI All,

Today I struggled to adapt the Assimp PBR shaders to work with the new lighting functionality.  The shaders were hardwired to support just a single directional light, I've cobbled together modifications to make them more general purpose and now have it working with the new lighting functionality. 

This is the glTF example datasets running in vsglights with an ambient light, directional light, point light and spot light all working to illuminate the scene.

I'm not happy with how it's all coded though, both the Assimp C++ code and the associated shaders are not as optimised or flexible as they ideally should be.  It's rough and read enough for me not to check this work in.  What the code is calling out for is core VSG support for managing shaders and selection of different shader code paths via defines in a way that makes it easier for the likes of vsgXchange::OSG and Assimp loaders to be able to select which features they want and get a state combination that supports this.  This is a decent chunk of work in it's own right so I'll leave that for after I've completed more of the core VSG lighting support.

The work today has given me better feel for how the new lighting functionality is working, there are still a few areas that need work, but it's not far off from being ready for wider use.  Tomorrow I'll return to working on the core lighting/ViewDependentState functionality and finish off the work with the aim of checking the work into VSG and vsgExamples master.

Once that work is checked in I'll return to either the shader management side or look at the DescriptorSet/Descriptor binding refactor suggested by Tim Moore.

Cheers,

Robert.

[Robert Osfield]

One of the areas I haven't figured out how best to present to users is how to manage default lights.  With the OSG the osg::View "has a option" osg::Light and a mode, from include/osg/View header:

        /** Options for controlling the global lighting used for the view.*/
        enum LightingMode
        {
            NO_LIGHT,
            HEADLIGHT,
            SKY_LIGHT
        };

        /** Set the global lighting to use for this view.
          * Defaults to headlight. */
        void setLightingMode(LightingMode lightingMode);

        /** Get the global lighting used for this view.*/
        LightingMode getLightingMode() const { return _lightingMode; }

        /** Get the global light.*/
        void setLight(osg::Light* light) { _light = light; }

        /** Get the global lighting if assigned.*/
        osg::Light* getLight() { return _light.get(); }

        /** Get the const global lighting if assigned.*/
        const osg::Light* getLight() const { return _light.get(); }

We could do something similar with vsg::View, though we could do something more general purpose.  With the VSG I'm trying to make all the classes more general purpose and flexibility, so look for opportunities for approaches that facilitate this.

One approach I'm considering is to have a vsg::AbsoluteTransform node that reset the accumulated modelview to identity so that all it's children are now in eye space.  The children could then contain a combination of vsg::AmbientLight, vsg::DirectionalLight or vsg::PointLight nodes, or any HUD rendering as well.

Another approach would be to have a vsg::CameraTrackingTransform that computes the inverse of a specified Camera's view matrix.  If this node is placed below a vsg::View that has the same Camera that it tracks then it would in essence behave like a vsg::AbsoluteTransform.  However, what it could do better is for cases where you have multiple Views rendering a scene, such as when you have PowerWall or Stereo display in which you want the same lighting used for each view.  For this multiview displace each individual View has a Camera that is relative to a master Camera, and here we could assign this master Camera to the vsg::CameraTrackingTransform as well.

Another approach would be to have a set of View children that are all traversed before the View's Camera's ViewMatrix is pushed, so all those children would be effectively be in absolute transform.

While all these approach are more flexible than a single hardwired Light assigned to a View, they do make it less easy to get the light settings to modify them.  That's a general challenge for scene graphs - all the rendering is distributed in a graph that is hugely flexible but you have to go hunt for settings, so perhaps just making that hunt convenient is the way rather than adopting something less flexible.

Thoughts?

[Robert Osfield]

Hi All,

I ended up spending the whole day tracking down a perplexing bug, eventually tracking it down to a bug in vsg::BindViewDescriptorSets read/write methods.  Turns out just a couple of lines of code were wrong, alas this was a completely different place than I was expecting so I had to investigate a whole range of possibilities before homing in on the actual error. 

Aarrrrrrhh... just one of those days...

[Robert Osfield]

To wrap up the day I thought I have a quick experiment with using an vsg::AbsoluteTransform node to decorate an ambient and directional light in order to create a headlight effect.

The code for AutoTransform is simply:

namespace vsg
{
    class AbsoluteTransform : public Inherit<Transform, AbsoluteTransform>
    {
    public:

        dmat4 transform(const dmat4&) const override { return {}; }
    };
}

And the code to set up the headlight:

        if (add_headlight)

        {
            auto ambientLight = vsg::AmbientLight::create();
            ambientLight->name = "ambient";
            ambientLight->color.set(1.0, 1.0, 1.0);

            ambientLight->intensity = 0.1;

            auto directionalLight = vsg::DirectionalLight::create();

            directionalLight->name = "head light";
            directionalLight->color.set(1.0, 1.0, 1.0);
            directionalLight->intensity = 0.9;
            directionalLight->direction.set(0.0, 0.0, -1.0);

            auto absoluteTransform = vsg::AbsoluteTransform::create();
            absoluteTransform->addChild(ambientLight);
            absoluteTransform->addChild(directionalLight);

            group->addChild(absoluteTransform);
        }

Which is pleasingly simple.  So I think I'll add that to the core VSG tomorrow.

And this is what it looks like:

[Robert Osfield]

When setting an application with lighting the scene with either need vsg::Light nodes placed into the scene, or attached to the vsg::View. Something we can make it easier to set up is to provide convenient functions o.e.

  auto view = vsg::View::create();

  view->addChild( createHeadLight());

  view->addChild( scene );

Or have the a vsg::creatViewWithHeadlight() function.

Potentially one could have a collection of convenience functions that provide different common lights, Longer term I can envisage companion libraries that provide sun and moon light models that track time of day and place the sun/moon in the correct position and colour.

[Robert Osfield]

I have checked in the new vsg::AbsoluteTransform class to the Lights branch of the VSG.

I have generalized it a bit further so you can pass the matrix to use when resetting the view matrix.  This matrix will be identity by default so the headlight usage above will use this approach.

Being able to set the matrix in the AbsoluteTransform is helpful for handling the multiview case where we want a single headlight orientation where the view matrix are different for each view - such as when you have a powerwall or stereo rendering as you can initialize the AbsoluteTransform for each view to account for the view matrix offset.

[Robert Osfield]

I've just checked in a further improvement to how we can configure our pipelines to work with the new view dependent state - the addition of a vsg::ViewDescriptorSetLayout class that is a proxy to the View's ViewDependentState.descriptorSetLayout.  The active Vulkan descriptorSetLayout is determined during the compile traversal.  In combination with vsg::BindViewDescriptorSet you can now set up your pipelines at load/scene graph generation time without worrying about the specifics of light data that will be provided at runtime.

This means the setup is substantially cleaner and we're another step away from loaders and applications being hardwired for a fixed light source etc.  Now the scene can provide as many lights as it requires, and the application can add any extra it wishes - such as headlights.

The approach I've taken should handle support for clip plans, eye linear text and shadows, and user extensions via configuration or subclassing of ViewDependentState.  This will require further follow up work, but I feel the design side of view dependent state is now figured out so have a clearer path forward when tackling these features.

Changes to VSG and vsgExamples are Lights branches:

    https://github.com/vsg-dev/VulkanSceneGraph/compare/Lights

    https://github.com/vsg-dev/vsgExamples/compare/Lights

Changes to vsgXchange::Assimp are more experimental so under the Lights_Experimental branch:

  https://github.com/vsg-dev/vsgXchange/compare/Lights_Experimental

Items left to TODO

While the changes to the core VSG are inching closing to being ready to merge with master. I do have a few issues to resolve.  The first is multi-buffering of descriptors in ViewDependentState to handle the fact that you can have multiple frames being record, submitted and running on the GPU, so it's if you update the light data during the record it could change the light data that's being used on the GPU, but what is being rendered is actually a frame from 1 or 2 frames ago, not the current one being recorded.  

As the light data needs to be transformed from object space into eye space each frame inconsistencies each frame has to have it's own set of data, if you get the wrong frames light data then visually you see the influence of light sources moving around as you zoom in/out or pan.  It's a bit like having a drunk viewer.  One solution - and the one that the vsglights example deploys is to add waitForFences into the main loop:

   // rendering main loop
    while (viewer->advanceToNextFrame() && (numFrames < 0 || (numFrames--) > 0))
    {
        // pass any events into EventHandlers assigned to the Viewer
        viewer->handleEvents();

        viewer->update();

        if (frameToWait > 0 && waitTimeout > 0)
        {
            viewer->waitForFences(frameToWait, waitTimeout);
        }

        viewer->recordAndSubmit();

        viewer->present();

        numFramesCompleted += 1.0;
    }

While this works it does mean that we've got a wait which reduces the amount of parallelism that is possible and degrades performance, which is a big no no for performance oriented applications.

All is not lost, what I've done successfully before is to multi-buffer the DescriptorSet so each frame gets its own set of light data to work with.  This afternoon I'll tackle this.

--

Another that I need to address is that once your shaders are set up to use the light data and there are no lights in the scene you get no lighting, so everything appears black, which technically correct doesn't look that great as a default behaviour :-)

I mentioned yesterday in this thread that we can make it more convenient to add a headlight to a view by providing a creatHeadlight() convenience function, however, this does address the default behaviour issue.  While I can add to all the examples a little bit of vsg::View configuration code to add a headlight existing VSG applications will have black scenes with they are using the latest vsg::Builder or vsgXchange::Assimp that both now utilize the light data.

I could add a default headlight to the vsg::View in it's default constructor, but then if applications don't want this then they'd need to go in and remove it, which is really against how the VSG is generally intended to be used i.e. you aren't forced into a particular solution that you have to do extra work to undo if you don't want it.

What the VSG does already have is some convenience functions for setting CommandGraph and RenderGraph's with a View, creating the view automatically to set things up in a reasonable default way.  The functions are:

    /// convenience function that sets up RenderGraph inside primary CommandGraph to render the specified scene graph from the specified Camera view
    extern VSG_DECLSPEC ref_ptr<CommandGraph> createCommandGraphForView(ref_ptr<Window> window, ref_ptr<Camera> camera, ref_ptr<Node> scenegraph, VkSubpassContents contents = VK_SUBPASS_CONTENTS_INLINE);

    /// convenience function that sets up secondaryCommandGraph to render the specified scene graph from the specified Camera view
    extern VSG_DECLSPEC ref_ptr<CommandGraph> createSecondaryCommandGraphForView(ref_ptr<Window> window, ref_ptr<Camera> camera, ref_ptr<Node> scenegraph, uint32_t subpass);

And

    /// Convenience function that sets up RenderGraph and associated View to render the specified scene graph from the specified camera view.
    /// Assigns the WindowResizeHandler to provide basic window resize handling.
    extern VSG_DECLSPEC ref_ptr<RenderGraph> createRenderGraphForView(ref_ptr<Window> window, ref_ptr<Camera> camera, ref_ptr<Node> scenegraph, VkSubpassContents contents = VK_SUBPASS_CONTENTS_INLINE);

I'm thinking I can change these so they set up the vsg::View they create for you with a headlight subgraph.  If you don't want this default behaviour you just simply create the View, CommandGraph and RenderGraph yourself.  All the source code for these convenience functions is open source under the MIT licence so users are free to learn from this and adapt to their own requirements, or just set everything up explicitly as many of the vsgExamples already do.

Happy to take suggestions, if there are cleaner ways of managing this then I've very open to adopting them :-)

[Robert Osfield]

Hi All,

I have now checked in the new vsg::createHeadlight() function, and changes to the vsg::createCommandGraphForView(..) and createRenderGraphForView(..) that now take a bool assignHeadlight parameter that defaults to true.  This allows you to set it to false if you still want to use these convenience functions:

 /// convenience function that sets up RenderGraph inside primary CommandGraph to render the specified scene graph from the specified Camera view

    extern VSG_DECLSPEC ref_ptr<CommandGraph> createCommandGraphForView(ref_ptr<Window> window, ref_ptr<Camera> camera, ref_ptr<Node> scenegraph, VkSubpassContents contents = VK_SUBPASS_CONTENTS_INLINE, bool assignHeadlight = true);

And

    /// Convenience function that sets up RenderGraph and associated View to render the specified scene graph from the specified camera view.
    /// Assigns the WindowResizeHandler to provide basic window resize handling.

    extern VSG_DECLSPEC ref_ptr<RenderGraph> createRenderGraphForView(ref_ptr<Window> window, ref_ptr<Camera> camera, ref_ptr<Node> scenegraph, VkSubpassContents contents = VK_SUBPASS_CONTENTS_INLINE, bool assignHeadlight = true);

The implementation of which is simply:

ref_ptr<RenderGraph> vsg::createRenderGraphForView(ref_ptr<Window> window, ref_ptr<Camera> camera, ref_ptr<Node> scenegraph, VkSubpassContents contents, bool assignHeadlight)
{
    // set up the view
    auto view = View::create(camera);
    if (assignHeadlight) view->addChild(createHeadlight());
    if (scenegraph) view->addChild(scenegraph);

    // set up the render graph
    auto renderGraph = RenderGraph::create(window, view);
    renderGraph->contents = contents;

    return renderGraph;
}

This is all checked in the VulkanSceneGraph Lights branch.  For all the applications that use createCommandGraph/createRenderGraph rather than rolling their own vsg::View will now automatically have a headlight added to the View.  For apps/vsg examples that don't use these convenience functions loading vsgXchange::Assimp or vsg::Builder created scenes will come out black as they'll be unlit unless they have a light explicitly added to them (some 3d databases have lights.)

When updating the built in PBR shaders using vsgconv the automated build picked up a compile error under Windows due to a limit on handling string literals:

"d:\a\vulkanscenegraph\vulkanscenegraph\src\vsg\utils\shaders\assimp_pbr_frag.cpp(431): error C2026: string too big, trailing characters truncated (compiling source file D:\a\VulkanSceneGraph\VulkanSceneGraph\src\vsg\utils\Builder.cpp"

To fix this I had to refactor vsgXchange::cpp writer so that it breaks up the string written to the .cpp into a series of R"(...)" blocks.  This fix is checked into vsgXchange master and the Lights_Experimental branch.

Chasing down this Visual Studio "issue" sucked out enough of my afternoon that I no longer have time to implement multi-buffering of ViewDependentState's DescriptorSet.  I had hoped to wrap this part up this afternoon so I can merge the VSG Light branch to nicely bookend the week, alas coding is coding, doesn't often fit to the schedule aspirations you may harbour...

I'll wrap up the weekend with another screenshot, this time vsgmultiview with Sponza.gltf from the gltf example set, this is with a headlight added to the scene by vsglights then written to a .vsgb file to be loaded in vsgmultiview to illustrate how the new light system support multiviews without any need for coding from users.

vsglights Sponza/glTF/Sponza.gltf --headlight -o test.vsgb

vsgmultiviews test.vsgb

[Robert Osfield]

I have now implemented multibuffering of the descriptors/descriptors sets used within the new vsg::ViewDependentState, this avoids the need for using the sync in the main loop, this improves performance when running the vsglights test example - up to 47% faster in the small window test with vysnc off.  For normal apps you won't see such a big improvement but every little counts. 

The key improvement with the multibuffering is that the main loop is cleaned up and now will have stable lighting without any additions to user code. For me this is important as I don't want a complexity overload for users of the VulkanSceneGraph when adding basic features like lighting.

I am now pretty happy with the public interface and the implementation of the lighting so will do a code review and if it looks good merged with master today.

[Robert Osfield]

I have now completed my review and tidy up of the Lights branch of VulkanSceneGraph and vsgExamples and have merged them with respective master.

The Lights_Experimental branch of vsgXchange updates the vsgXchange::Assimp loader to make it capable that's capable of loading glTF PBR models. Below is an example of vsglights running on water bottle model from the glTF example set.  The vsglights example by default adds a vsg::AmbientLight, vsg::DirectionLight, vsg::PointLight and a vsg::SpotLight, you can see the three specular highlights for each of the light sources.

[Robert Osfield]

I have bumped the VSG master version to 0.2.7 to mark the addition of vsg::Light classes and vsg::ViewDependentState. 

vsgExamples master now requires this version of the VSGas well.