Options Regarding Upcoming Terrain Data Changes

[Rob Milton]

We've been using cesiumjs for several years (since the beginning of the Cesium 1.x versions), and we're very excited about all that is happening with the product!  However, one of the big attractions to Cesium was that it was open source and free.  I work for a very small company that pushes hard for us to find very inexpensive solutions.

I recognize that there is definitely value in what Cesium offers, and I'm sure the new Cesium World Terrain will be extremely quality data.  But I've been tasked to investigate options for alternate terrain sources that are free or to define what it will take to "create" and host our own terrain data.  

One of my biggest challenges is that I don't really understand what the specs/requirements are for terrain data to be used by Cesium.  I'm also extremely interested in all of the "clamp to ground" capabilities that are being discussed and worked on, and I'm unclear if the only way these features will be supported is by using the new Cesium World Terrain data.

I have experimented with the "VRTheWorldTerrainProvider" and the various terrain data sets available from VT MAK / VR-TheWorld.  However, everything I've tried with this terrain data results in objects the should be "clamped to ground" moving significantly when the globe is tilted.  I've read several posts in the forums about this, but there does not seem to be a good solution (at least none I could find).

So I would very much appreciate if someone would be willing to define/discuss:

1. What options exist after September 1, 2018 for us to host our own terrain data and not lose the benefits of upcoming "clamp to ground" capabilities?
   
2. Where I can find specific "how to" information for creating terrain tile sets that can be used in/by Cesium?
• What are the specs/requirements for the terrain tiles?
• I've read a little about the STK Terrain Server.  Will it still be a valid option?  Will upcoming "clamp to ground" capabilities be supported if we go this route?
3. Any other suggestions related to these goals.

Thank you in advance for any help with this!

[Rob Milton]

By the way I'm sorry about my entire post being bold.  I don't know how that happened.  I would edit it if I could...

[Mark Erikson]

I am actually in the process of generating a complete from-scratch self-hosted Cesium terrain dataset right now.  I plan on writing a couple blog posts about it once I have some time free (probably not until May or June).

Long story short: use the https://github.com/geo-data/cesium-terrain-builder utility, and your choice of source terrain datasets.  EarthENV-DEM90 ( http://www.earthenv.org/DEM.html ) has 90m worldwide resolution, and ALOS30 ( http://www.eorc.jaxa.jp/ALOS/en/aw3d30/ ) has 30m worldwide resolution.  Both require a bunch of preprocessing before they're ready for the final generation steps. 

Also, I've made a bunch of improvements to the CTB tool over the last couple weeks, but haven't had a chance to file a PR yet.  Hope to do that in the near future as well.

For serving, the basic requirement is simply a static file server if you're writing the tiles straight to disk.  I've modified CTB to support writing to the MBTiles container format for ease of storage.  There's existing tools out there to serve data from MBTiles, or writing your own is relatively trivial (take XYZ from URL, query tile blob from the database, return it).

AGI's terrain servers have far greater resolution available, but that should do for a lot of use cases.

As for your other questions: I don't _think_ the "clamp to ground" stuff has any relation to what terrain provider you're using.  That's primarily Cesium being updated to correctly draw stuff relative to whatever terrain data it's loaded.

[Patrick Cozzi]

Hi Rob,

Thanks for the kind words on Cesium.

I just want to clarify that CesiumJS - the JavaScript library that everyone is using today for visualization - is still open source and free, and is getting a ton of investment by the core team.  There are no plans to change that.

I would ask that your employer at least consider evaluating the Cesium ion platform and Cesium World Terrain as a data provider for CesiumJS since it helps support open-source CesiumJS development that it sounds like they are benefiting from - and it will likely be, by far, the best data provider for CesiumJS, in terms of quality and performance, since the Cesium team is building it and knows the complete pipeline and formats.

As for the clamp to ground features, if they can be implemented efficiently client-side, they will be.  So far, textured polygons on terrain are all client side, keep an eye on #6393.

Thanks,

Patrick

[Rob Milton]

Thank you both for the replies...

Mark,

I appreciate the details and discussion on the approach you are taking.  We have access to a number of terrain sources, and I've been reading about the terrain builder and looking into options for converting and hosting our own terrain data.  If we were to pursue this path I'd like to be able to use a Cesium TerrainProvider and simply have it point to our data on our servers.

Patrick,

On one hand I would love to see our company use the new Cesium World Terrain.  I will do my best to present/explain pros and cons.  However, I also have to investigate answers to their questions about what it would take to host our own terrain data.

In the "Introducing Cesium World Terrain" article you stated:

This is the beginning of an upgraded terrain experience in CesiumJS that will culminate in terrain being enabled by default. First, more terrain data updates to improve resolution will roll out over the next several months. Second, new CesiumJS features such as polylines on terrain and textured polygons on terrain will roll out to make terrain support in CesiumJS first-class.

I wasn't sure if the mention of upcoming features such as polylines and textured polygons on terrain meant that these features would potentially not work with other terrain sources.  Can you clarify a little?

Also, in order for me to present a fair comparison of using the new Cesium World Terrain versus hosting our own terrain data I need to understand and present the requirements for converting/formatting/hosting our own terrain data.  Can you point me to resources that would explain the format and requirements for us to host our own terrain data.

Maybe a good look at the requirements to host our own data verses the cost of the new Cesium World Terrain would open possibilities.

Thanks for you help with this!

[Mark Erikson]

Yeah, you're conflating a couple different not-entirely-related topics there.

Cesium has various terrain provider adapter classes that let it request and parse different terrain data formats.  For example, the CesiumTerrainProvider class knows how to parse the "heightmap" and "quantized-mesh" file formats that the Cesium team has defined.  The GoogleEarthEnterpriseTerrainProvider and VRTheWorldTerrainProvider similarly parse terrain data from the formats provided by those servers.  Once that data has been parsed on the client side, Cesium can draw the 3D mesh appropriately.

For the CesiumTerrainProvider in particular, the question of which server it's requesting data from is a totally separate concern from how Cesium then renders the data, and how the rest of Cesium's rendering pipeline interacts with that rendered mesh (such as polylines/points/etc being drawn relative to the terrain mesh).

Cesium expects both quantized-mesh and heightmap terrain data to be available in the industry-standard TMS file structure, with individual tiles available at $SERVER/zoom/x/y.terrain .  Normally this is individual tile files on disk in that exact structure, so you would use a standard static file server (ie, literally any HTTP server).  However, since storing millions of individual tile files is inefficient, the MBTiles container format was created as a way to store all those individual files as blobs in an SQLite database ( https://github.com/mapbox/mbtiles-spec , https://wiki.openstreetmap.org/wiki/MBTiles ).  It was originally meant for use with tile map images, but works exactly the same if you stuff Cesium terrain tiles inside.  Serving tile files out of an MBTiles container requires a bit of logic for the server to query and retrieve the tile blobs from the database, but the logic itself is trivial.

What Patrick is saying is that AGI is improving several aspects of Cesium's terrain-related capabilities.  They have a new/updated AGI-hosted terrain dataset, and are continuing to improve the quality and contents of that terrain dataset.  They're also working on adding some much-requested rendering capabilities that relate to how Cesium handles terrain.  So, both terrain-related, but independent topics.

Per my comment earlier, there's not too much documentation out there on self-hosting terrain data, which is one reason why I plan to write some blog posts on the topic.  Here's the most relevant articles I've found in my own research:

https://cesiumjs.org/data-and-assets/terrain/formats/quantized-mesh-1.0/
http://blog.mastermaps.com/2014/10/3d-terrains-with-cesium.html
https://www.linkedin.com/pulse/fast-cesium-terrain-rendering-new-quantized-mesh-output-alvaro-huarte/
https://bertt.wordpress.com/2016/12/08/visualizing-terrains-with-cesium/

[Rob Milton]

Mark,

Thank you so much for taking the time to write this response and discuss these issues in this level of detail.  This goes a long way toward addressing the questions and confusion I've had related to what is required to host terrain data for Cesium.  I will need to spend some time digesting what you've written and the content of the links you've provided.

There is definitely more involved than I initially realized.  I will be very curious and interested to follow your blog posts on the topic when they become available.

I will use this information to continue discussions with my company on the topic (explaining the benefits of using the new Cesium World Terrain that become much more obvious when you see all that is required to host quality terrain data for Cesium that is served efficiently...  and also having the discussion related to the possibility of hosting our own data with a better understanding of what's involved).

Thanks again for the quality information!

[Patrick Cozzi]

Hi Rob & Mark,

On the two separate-but-related terrain topics: we are improving the terrain experience (clamping, resolution, performance, etc.) in Cesium to the point where terrain will be enabled by default with a great experience in terms of performance and capabilities.

Features like polyline clamping and textured polygons on terrain are generally client-side features (meaning open-source CesiumJS), but may benefit from a REST API that can return terrain metadata, e.g., the min/max height along a polyline.  Cesium ion (which hosts Cesium World Terrain) will implement these REST APIs to provide the best possible performance.  If other terrain data providers want to provide a first-class experience in Cesium, I expect they will implement them as well.  We certainty will make the REST APIs public to encourage an interoperable ecosystem.  As of now, the textured polygons are all client-side, but I can't commit to this for all possible features as it could end up being too limiting.

On more thing: in addition to hosting the premade Cesium World Terrain, Cesium ion will also allow users to drag & drop terrain sources to create new terrain tilesets for streaming to CesiumJS, which I imagine will be of interest to folks who want a very simple way to use their own terrain.  For example, here's a video showing a similar process with imagery.

Thanks,

Patrick

--
You received this message because you are subscribed to a topic in the Google Groups "cesium-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cesium-dev/frDruOjBuv0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cesium-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

twitter.com/pjcozzi

[Rob Milton]

Patrick, 

Thank you for the additional information.  I am compiling all this in preparation for discussions about how we move forward.  The information you and Mark provided is very helpful and will allow us to have much more meaningful conversations.

Rob