Imagery Layer adjustments/filtering during render

[Shahin Roboubi]

Imagery layer already provides some built in adjustments, like brightness/contrast/hue/saturation, which I believe are performed by shaders in the rendering pipeline. What is the best way to add my own custom adjustments to an imagery layer. For example, I would like to dynamically adjust the brightness of pixels per RGB channel (brighten blue, dim red, etc). Is there any way to add my own shader for a specific imagery layer (and pass parameters to it)? Or perhaps, is there another way to do this without shaders?

Thanks.

[Gabby Getz]

There is no hook for you to add your own shader per imagery layer. All the shading effects are done when rendering the globe surface, and are performed in the globe's shaders. 

If you would like to modify the Source code to achieve this effect, you would need to create new properties on the ImageryLayer itself, pass these values to the shaders, and modify the shaders.

The values are passed to the globe surface shaders in GlobeSurfaceShaderSet.js, and the shader files themselves are in GlobeVS.js and GlobeFS.js.

I've opened an issue to track adding the ability to do this in Cesium without modifying the source code (#6449)We would welcome a PR that is a generic solution, and would be more than happy to provide guidance and review code!

Thanks,

Gabby

[Shahin Roboubi]

So the shaders are at a global level, but the parameters are per layer... they must be otherwise I wouldn't be able to set the brightness/contrast/etc individually per layer. After some more thought I believe that modifying the shaders might not be appropriate for what I need. Ideally I would like to:

1. Adjust some parameters on an imagery layer

2. Have the image rendered according to these parameters

3. Have the imagery data be updated according to these parameters (for example if I create a histogram of the image, I want to see it updated)

From my understanding of shaders, #3 would not work as the shaders only apply modifications to pixels being rendered, and the source data (even in GPU memory) remains unchanged, right? So what I really need is to get my hands on the imagery layer texture data. That way I can modify it as needed and read it's value to produce analytics like an accurate histogram. Of course if the imagery is reloaded from the provider (as it would be during a zoom level change) I would have to reapply the adjustments to the new image, but that shouldn't be too hard. So... is there any way to get this imagery layer texture data?

Alternatively, I can implement my own imagery provider, wrapping the "real" one, and applying my adjustments to the image data. However I would need a way to tell cesium to re-request the image when adjustments are made, so that I don't have to wait for a zoom event to refresh the image.

Note, I found a similar post from a few years ago, any update on this: https://groups.google.com/forum/#!topic/cesium-dev/aysBQWn9c88

Thanks a lot for the quick replies!

[Gabby Getz]

Custom imagery adjustments or processing would definitely be a powerful feature! A few of the Cesium devs have been talking about it in regards to raster analysis. Doing the processing in the shader would most likely be the best option performance-wise. However we'd need to implement a better hook for the custom shader code. This would not modify the underlying imagery texture, and would just change how the imagery is rendered.

For the histogram however, it sounds like what you need to do is grabbing the texture itself like you suggested and then updating the UI.

ImageryProvider.loadImage is where the image itself is returned when requested.

It's possible trigger the image provider to update. We do that with time for WMTS Imagery. Take a look at the WebMapTileServiceImageryProiver.js source if you'd like to implement your own imagery provider.

[Shahin Roboubi]

I'm having some trouble finding the code that triggers cesium to re-fetch imagery... maybe I'm not looking in the right place? I basically need to tell cesium to fetch imagery data programatically, instead of as a result of a zoom level change. If this is done in WebMapTileServiceImageryProvider, could you help me find the lines of code that make the call (I assume periodically)?

Thanks.

[Gabby Getz]

WebMapTileServiceImageryProvider has an instance of TimeDynamicImagery, which watches for the specified interval and calls that requestImagery function in WebMapTileServiceProvider. Instead of using TimeDynamicImagery to trigger the request, you would trigger it when your UI changes.

Let me know if that doesn't sufficiently answer your question.

[Shahin Roboubi]

From what I can see, the TimeDynamicImagery only calls requestImage inside addToCache(). This is called when the WebMapTileServiceImageryProvider.requestImage tries to access an image that is not in cache, or when preloading the image for an upcoming time interval. 

After looking at the stack trace, the actual source of the imageryProvider.requestImage call (for any provider), and the consumer of the returned image, seems to always be the scene render function. Basically it looks like:

Scene.render() -> ... -> TileImagery.processStateMachine() -> ImageryLayer._requestImagery() -> anyProvider.requestImage()

The only way we can go from TileImagery.processStateMachine() to ImageryLayer._requestImagery() is if the tile has ImageryState.UNLOADED. So I think my problem is that I need to set the tile states to UNLOADED. I'm not sure how to do that if all I have is a handle to the Imagery Provider and the Imagery Layer. How does that happen for WebMapTileServiceImageryProvider? 

For the WMTS example, it seems to happen when the time is being advanced and also when a user manually adjusts the time interval. In the case of a WMS provided imagery layer, it only happens when a new zoom level of a given tile is needed.

Thanks.  

[Gabby Getz]

Maybe create something similar to TimeDynamicImagery which requests a new image periodically, and inside your custom provider requestImage function, set an isDirty flag that will only request after you make a change in your UI.