Handling large data sources

[Brad Hards]

As part of the WFS3 work, I've been thinking about whether OpenSphere could handle large data sets. For this purpose, large is 10^5 to 10^8 features.

Clearly having all that in the browser and trying to render that kind of data set is going to be a problem without some considered handling.

WFS3 supports filtering (e.g. by bounding box) and paging (e.g. 10 to 10000 items at a time), so that combined with a "hide above" setting (i.e. show a flag or outline until the user is zoomed in to either a specified level or a reasonable number of matched features) could help.

Analyses would need to work with paged data (e.g. generate the heatmap based on 10000 items at a time) or be disabled on very large data sets.

Has that been looked at previously? Is there anything that is already done?

Brad

[wallw-bits]

Features in OpenSphere are constrained by two main factors: render performance and memory for the metadata. On the OpenLayers end we are constrained mostly by the rendering performance. Cesium can render 300k billboards (point primitive is better but has other issues that prevent us from using it) without property fields. However, adding a hefty number of fields can quickly run the browser out of memory and so we limit that to 150k. The other problem that we have run into with responses of that size is that they can take a while to both generate on the server and to transfer over the network. Even if we were to make 2 orders of magnitude improvement to the rendering, that problem is not helped. We could potentially look into an option where we only request the most basic information back about a feature (id, geom, and time, perhaps) and only request other fields as the user views them.

If you start storing the full query on the server and only pulling a portion, then aggregate calculations such as the Timeline histogram must be calculated on the server as well since we do not have the data locally. Otherwise you limit those views to only the data that is currently loaded locally (and animating over the whole range may prove chunky if you don't buffer the next/previous pages temporally).

I have seen several extensions to OpenSphere limit what they load by zoom level or density. So it is certainly possible to dynamically page or load features as you move around. We also have some custom layers which can run aggregate things like histograms and total counts as the extent changes and those are subsequently shown to the user without loading any features (the map visualization is just raster tiles). This allows for some simpler analysis but on a much higher data volume. The downside is that there aren't any (at least that I know of) standards codified for that type of API.

[tjk...@gmail.com]

We are putting together a performance version of Cesium that handles ~500k full entities (camera tracking, Entity API, the works).  Rough beta in an app is here:

https://mapshot.app/

Check it out!

[Kevin Schmidt]

Cesium's PointPrimitive is much more efficient memory-wise than the Billboard primitive OpenSphere is using, so you'll be able to display far more points on the globe with that approach.

OpenSphere decided not to use it because shaving down the memory footprint came at a loss in capability. Point primitives do not have properties like eyeOffset, which we use to z-order primitives against one another. This allows us to control the perceived render order of coincident points based on the layer order in our Layers window. It also allows bringing a highlighted feature on top of all others when hovered in the Show Features window. The point primitive also does not support the various point shapes (square, triangle, icon, etc) in OpenSphere, though that wouldn't be a show stopper if we only used PointPrimitive when the default/point shape was selected.

If you're not concerned with any of these drawbacks, then PointPrimitive is certainly the most efficient way to display points in Cesium.

[TJ Koury]

Kevin,

These are not Point Primitives.  Click on one, they have eye offset et. al.  They are full Entities.

It is the same tech we are using for the Celestrak.com space catalog:

https://celestrak.com/cesium/orbit-viz.php?tle=/pub/TLE/catalog.txt&satcat=/pub/satcat.txt&pixelSize=3&samplesPerPeriod=90&referenceFrame=1

Open to partnering options if you guys want to use it.

--
You received this message because you are subscribed to a topic in the Google Groups "OpenSphere" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opensphere/aYZ_1mJnW6E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opensphere+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/opensphere/43dc6097-8f14-4b80-a10a-52ddb7840663o%40googlegroups.com.

[Kevin Schmidt]

Maybe I'm mistaken, but your code is using the entity.point property which is a PointGraphics object which does not support eyeOffset. The entity.billboard property is used to create a BillboardGraphics object, which would support eyeOffset is an analogous to what OpenSphere is using.

https://cesium.com/docs/cesiumjs-ref-doc/Entity.html

https://cesium.com/docs/cesiumjs-ref-doc/PointGraphics.html

https://cesium.com/docs/cesiumjs-ref-doc/BillboardGraphics.html

To unsubscribe from this group and all its topics, send an email to opens...@googlegroups.com.

[tjk...@gmail.com]

Meant to say “camera offset”, eye offset will be a function of whatever other primitives you add to the Entity. You can see the labels in the Celestrak.com page.

We are working on optimizing time dynamic volumes next, have some COVID - related visualizations we have been working on.

[tjk...@gmail.com]

Ha! I was writing my correction while you were asking for clarification, see below. The follows of posting on a phone...

[TJ Koury]

Looks like this is a popular topic so I posted an updated example here with 100k billboards:

https://mapshot.app/

[TJ Koury]

We just pushed an update to our Cesium distro c137.js that handles billboards better.

Drop it in and see if the performance meets your needs.