Refactoring DynamicScene

[Matthew Amato]

As Cesium has matured, interoperability with external data sources such as CZML, GeoJSON, KML, CityGML, and others has become increasingly important.  It's probably one of the most common topics on the mailing list. While we have made strides in this direction, larger changes need to take place in our internal architecture before we can really code full support for these formats.  In the interest of open development, I wanted to provide a full overview of where we are and where I see things going over the next few months.  Feedback, as always, is appreciated.

The current DynamicScene architecture in Cesium began with the sole purpose of visualizing our own still-in-progress CZML format. At the time we were only concerned with time-dynamic data. Time is a first-class citizen in Cesium and we wanted to be sure to showcase it. Many of us were also still fairly new to large scale JavaScript programming and we didn't have a clear picture on where it was all going.

Fast-forward 3 years and you get to where we are today, we have refactored parts of DynamicScene several times, added higher-level abstractions and plug-in support so that different data sources can be loaded regardless of format.  We have out of the box support for GeoJSON and in-progress support for KML, neither of which are time-dynamic.  Most importantly, we have a much clearer picture of what our ultimate goals are for both data sources and Cesium as a whole.

Along with the positive changes, we have accrued some technical debt that is preventing us from adding certain features. For example, the Geometry & Appearances system is not yet being utilized by the DynamicScene layer. Among other things, this prevents us from adding new geometry features to CZML, finishing up native KML support, or improving GeoJSON performance. Most of this is due to the fact we have been trying to make small, backwards compatible changes to the codebase slowly over time and these features are extremely hard to implement incrementally. Since the last thing we want to do is introduce lots of breaking changes every release, it is time to take a step back and perform a large-scale refactor of the code.

Before refactoring the code; it's important to take inventory of the issues we want to address.  Below is a list of the existing problems (in no particular order).  Keep in mind that this does not include additional features that we plan on adding, but only features that are hard to add in the existing architecture.  See the DynamicScene & DataSource roadmap for a full list of planned features.

1. DynamicScene and the objects contained in it are misnomers at this point, since we handle both static and dynamic data. This is easily changed, but is a breaking change that we will only want to make once.  Better names should make usability much easier for new users.
   
2. The current visualization system is very brute force, it re-evaluates almost everything every frame. We get away with this for sampled data, but it's incredibly inefficient for static or slow-changing data. The bookkeeping required to fix this is itself tedious and inefficient given the current architecture of DynamicObject. We need to fix this to efficiently support static geometry found, such as KML and CityGML.
   
3. Property objects do not have any sort of change notification, for example, there's no way to know that a sample was added to a SampledProperty. This makes batching visualization impossible because two equal properties can become unequal without us detecting it.
   
4. We have no way to know if a property is time-varying or not. We currently check for "instanceof ConstantProperty" in some places, but this is a hack. We also have no distinction between fast changing data (SampledProperty) and slow changing data (TimeIntervalCollectionProperty), which may be useful for caching purposes.
   
5. It's not currently possible to have two types of the same primitive registered on a DynamicObject, i.e. you can't have 5 separate polygons representing Hawaii as a single DynamicObject.
   
6. Since visualization relies on shared "position", "vertexPositions", and "orientation" properties at the root of DynamicObject, underlying graphics objects are limited in what they can do. For example, we can't currently support polygon hierarchies with holes with this method. This also makes it very hard to incrementally improve the current system without affected all objects that depend on these properties. It also means that objects are not self-contained, which makes tracking changes tedious and slow.
   
7. DynamicObject is a dumping ground that has every potential property registered on it. This makes it a  unwieldy. It would be better to support creation of streamlined mixed-in types just for the data being used.
   
8. We inconsistently use ES5 properties versus getter/setter functions. We should switch to ES5 exclusively for the entire DataSource layer.
   
9. There's no easy way to turn visualization of a DynamicObject "off" or turn individual graphics off. We added a `uiShow` property in the dataSourceBrower branch, but it's temporary until we do it for real.
   
10. The visualizers currently monkey patch DynamicObject instances with extra bookkeeping data. This most-likely affects performance and also creates potential conflicts if two visualizers use the same property names.
    
11. DataSources need to do a better job of throwing RuntimeErrors instead of DeveloperError, this is even more critical now that we are pre-processing out DeveloperError.
    
12. We currently use the `position` and `viewFrom` properties to track an object with the camera, but not every object has a position, and `viewFrom` is not adequately defined.
    

None of the above issues are particularly difficult to fix, but they will have a good number of implications throughout the code.  While there will certainly be some API breakage as a result, our goal is to make it trivial to migrate any existing Cesium applications over to the new code.  If you are familiar with the current system, the refactor system will be extremely similar.

Finally, I wanted to mention that since CZML is simply a DataSource plug-in into Cesium, none of these changes affect the CZML format or capabilities itself. However, once all of these changes are made, it will be easy to perform a similar refactoring on the CZML format in order to address issues that have become obvious in the last few years.  We may even potentially start a CZML version 2 in order to maintain backwards compatibility with the original unversioned format that's currently in use today.

Assuming no surprises, I think all of the above changes can be made in time for the April release of Cesium (maybe even March if things go really well).  However, given the nature of the changes, we will most likely ask those interested to check out the branch before then.  CZML enhancements would soon follow, either in the same release or the one right after (and KML after that).

Thanks for reading,

Matt

[umberto...@gmail.com]

Many thanks for the clarifications

Umberto

[d00d...@gmail.com]

Hello;

I'm just wondering what the status of this is? Specifically, I am interested in kml and czml with holes.

Thanks
Cody Smith

[Matthew Amato]

Wow, thanks for the trip down memory lane; this post is almost a year old and I think we got most of these done (and more).  That being said, KML is still heavily in development; you can try it out yourself by cloning and building the `kml` branch.  If you don't want to build Cesium yourself, we'll probably post a beta build for people to try early next year.

CZML holes are equally undone, but I'm also hoping to have those in within the next couple of weeks (this will also enable hole support in GeoJson and KML).

--
You received this message because you are subscribed to the Google Groups "cesium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cesium-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ha...@ucar.edu]

Greetings,

I'm enjoying using CesiumJS, it seems like a great application! Thanks for your work!

I am adding a data set that will be displayed as a polygon and I need to be able to add holes in the polygons. I would prefer to use CZML for this. It seems that CZML does not support holes in polygons yet, is that correct?

I found an 'eliminateHoles' function in the DotNet implementation of the czml-writer, which looks like it may be a work around until CZML supports holes. Is this the best way to do it for now?

Thanks,
-David

[Hannah Pinkos]

Thanks David!  Glad to hear Cesium is working well for you.

It looks like you're correct, CZML doesn't have support for polygons with holes yet.  I wrote up an issue here for us to add it: https://github.com/AnalyticalGraphicsInc/cesium/issues/3826

For now, I think using that function in the czml-writer is a good work around.

Best,

Hannah

[ha...@ucar.edu]

Hi Hannah,

Thanks for the quick reply! Just curious if you had an estimated completion date for this enhancement? It would be helpful for our project planning.

Thanks!
-David

[Hannah Pinkos]

Hi David,

I'm not really sure when we'd get a chance to work on it.  A lot of the team is really busy with 3D tiles and other projects right now.

I don't think the implementation would be too difficult, would you be interested in contributing?  I can point you to where the changes need to be made.

Just out of curiosity, what kind of project are you working on?

Best,

Hannah

[David Hahn]

Hi Hannah,

Yes, I think we could take a look at making the changes.  Just let me know how to get started.

We are building a display to visualize output from weather forecast models and possibly climate data as well.

-David

[Hannah Pinkos]

Hi David,

Upon looking into this, it looks a little more complicated than I initially thought.  I asked Matt if he has any suggestions for how to get started.

Thanks!

Hannah

[Tim Plummer]

Bump.

I work with David Hahn and am curious if there has been any thought given to this issue. I have noticed that holes have been implemented for geoJSON files in GeoJsonDataSource.js. They work well for geoJSON files, unfortunately other properties such as availability, height, and extrudeHeight are not supported for geoJSON files.

I have not looked at the issue in depth, but could the holes implementation in GeoJsonDataSource.js be used as a templet of how to add support for holes in CzmlDataSource.js?

Thanks,

Tim

[Hannah Pinkos]

Hi Tim,

Sorry I didn't give an update on this.  Because of the time dynamic nature of CZML, it turned out that processing polygons with holes was very complicated.  I took a stab at it at our recent code sprint, but was unable to come up with a good solution.

I think Scott (who has been making a bunch of CZML improvements lately) was planning to take a look at this as well.  I'm not sure how soon that would be though.

Meanwhile, you could try converting to GeoJSON and adding height and extrudedHeight to the entities after they are uploaded.  See the custom styling example in this demo: http://cesiumjs.org/Cesium/Apps/Sandcastle/index.html?src=GeoJSON%20and%20TopoJSON.html&label=DataSources

Is your data static or time dynamic? And do you have nested holes? (think concentric circles, ie a polygon with a hole with a polygon with a hole)

If you have static data with just one level of holes I could help you put together a temporary CZML solution that wouldn't be too complicated.  The time dynamic part is the biggest challenge in adding polygons with holes to CZML.

Best,

Hannah

[Tim Plummer]

Hi Hannah,

Thanks for the quick response.

I had previously looked through the styling example that you linked, but for some reason didn't consider emulating it to achieve what we want. Thanks for the reminder and I'll give it a try.

We do indeed have time varying data. The likelihood that we encounter a nested hole is pretty slim, although possible, and may be able to be ignored (I will have to check with the group). We do see polygons nested within holes quite often and will definitely need the ability to display those.

Thanks for the help,

Tim