Support added to vsgXchange for reading data from the internet using libcurl

Robert Osfield

unread,

Mar 11, 2021, 6:47:49 AM3/11/21

to vsg-users : VulkanSceneGraph Developer Discussion Group

On Tuesday I began work on adding the ability to load data from http and https to the vsgXchange library using libcurl, and have just completed the first pass implementation of reading data along with support for storing data locally in a file cache. The new functionality is now merged with VulkanSceneGraph and vsgXchange master.

https://github.com/vsg-dev/vsgXchange/pull/29

This allows you to host your data on the web, or even github, and pull the data down on demand so you can do things like:

vsgviewer https://raw.githubusercontent.com/robertosfield/TestData/master/Earth_VSG/earth.vsgb

I've posted a video of this in action on youtube.

The vsgviewer example itself doesn't actually know anything about libcurl, it just reads a filename and the new vsgXchange::curl ReaderWriter automatically spots the https and downloads it pass it to the other ReaderrWriter to convert to a usable vsg data via the vsg::read(istream&, options) support.

I also added to the core VulkanSceneGraph a new vsg::Options::fileCache variable that you can set in your application, or use the --file-cache fileCacheDirectory command line options. The vsgXchange::curl class then uses this Options::fileCache path (when set) to write a copy of the download file to local directories.

vsgviewer https://raw.githubusercontent.com/robertosfield/TestData/master/Earth_VSG/earth.vsgb --file-cache ~/MyFileCache

When you run the viewer now the vsgXchange::curl will check the file cache directories for the requested files and if they exist locally just load them directly, if not found it'll download the data from the internet and store a copy in file cache, creating a directory structure with the file-cache directory that mirrors the path on the remote file paths.

At the bottom of this post I've pasted in the what the file cache contents are are a quick run.

The addition of http/https support to vsgXchange is another major feature that the OpenSceneGraph has that has been missing from the VulkanSceneGraph. Adding this feature is useful for end users, but also for myself in shaking down the core VulkanScneGraph and vsgXchange IO support so that it properly supports orthogonal requirements quite different than just standard loading of data from local file systems.

With the feature now working seamlessly (at least I hope as I've only tested it under Linux so far :-} we have can be confident that IO support in the VSG is now getting pretty close to all what we'll need for VSG-1.0.

Cheers,

Robert.

-- Results of running vsgviewer populating a file cache

Robert Osfield

unread,

Mar 11, 2021, 10:18:59 AM3/11/21

to vsg-users : VulkanSceneGraph Developer Discussion Group

Now that the VulkanSceneGraph has support for streaming data in over the internet I decided to create the classic "Hello World" to illustrate how easy it is to an application that has #multithreaded database paging, just 88 lines of code!

https://github.com/vsg-dev/vsgExamples/blob/master/examples/viewer/vsghelloworld/vsghelloworld.cpp

No need to download big databases, just let it all stream in from the internet automatically on demand. The new example is checked into vsgExamples master. You'll need VulkanSceneGraph and vsgXchange master as well.

Gavin McIntosh

unread,

Mar 11, 2021, 7:23:59 PM3/11/21

to vsg-users : VulkanSceneGraph Developer Discussion Group

Hi Robert,

Wow, that's a powerful new feature.

Just last night I was learning how to use QGIS to convert downloaded GeoTiffs to 3D to view them.

I can skip those steps now?

Add a menu option for pseudocolour singleband shading?

Been saving fridge temperature data for years, this is a way to visualize the data in 3D?

Predictive maintenance, use a threshold alarm value?

Easier to view the data remotely without downloading it all just to look at it.

One benefits of using Pi's for IoT data logging is the huge amount of data they can log.

Visualizing it has been the problem, seeing it as a 3D terrain map was my first idea years ago.

Never had the tool to do it till now.

Easy enough to take csv files?

Regards

Gavin

From: vsg-...@googlegroups.com <vsg-...@googlegroups.com> on behalf of Robert Osfield <robert....@gmail.com>
Sent: Thursday, March 11, 2021 9:47 PM
To: vsg-users : VulkanSceneGraph Developer Discussion Group <vsg-...@googlegroups.com>
Subject: [vsg-users] Support added to vsgXchange for reading data from the internet using libcurl

--
You received this message because you are subscribed to the Google Groups "vsg-users : VulkanSceneGraph Developer Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vsg-users+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vsg-users/05e5907e-f435-4c71-a364-432cfb8179dbn%40googlegroups.com.

Robert Osfield

unread,

Mar 12, 2021, 3:02:32 AM3/12/21

to vsg-...@googlegroups.com

Hi Gavin,

On Fri, 12 Mar 2021 at 00:24, Gavin McIntosh <g.mci...@griffith.edu.au> wrote:

Wow, that's a powerful new feature.

Indeed. Opens the door to storing and sharing data in lots of new ways.

Just last night I was learning how to use QGIS to convert downloaded GeoTiffs to 3D to view them.

I can skip those steps now?

I'm not familiar with QGIS, a quick search online it has quite a different role to the vsgXchange curl support.

The new functionality just enables loading of data from the http and https, it doesn't create anything from that data, so reading a DEM will just give you a vsg::Data with the elevation data stored in an Array2D. It'd be up to application or another plugin to create a mesh and texture it.

Right now though, only plugins that support reading for a istream are able to be read from http/https, so vsgGIS/GDAL integration won't be supported until we implement a GDAL::read(istream&) for vsgXchange.

Add a menu option for pseudocolour singleband shading?

Again this isn't what this feature is about, it's just low level loading of data.

Post processing of loaded data will be up to other tools/utilities. For the paged database that I've shown above I using VirtualPlanetBuilder to take the Nasa Blue Marble images and convert them into a OpenSceneGraph paged database, then used vsgconv to convert this paged database to a native VSG one, then I uploaded to github as an experiment.

Been saving fridge temperature data for years, this is a way to visualize the data in 3D?
Predictive maintenance, use a threshold alarm value?

Easier to view the data remotely without downloading it all just to look at it.

To do this you'd need to read the data and then create the mesh and any texturing yourself from the data. The new vsgXchange curl support isn't completely general purpose reading of http and https, as it immediately passes on the downloaded data to a stringstream and then passes this on to the other ReaderWriter to convert into usable data. For the instance of the paged earth data the files on the remote server are native .vsgb files, so the vsg::VSG::read(istream& ..) implementation converts the istream to the appropriate vsg::Objects.

Potentially you could write your own custom ReaderWriter that also implement the read(istream& ..) method and then figures out how to convert this data to an appropriate VSG scene graph.

One benefits of using Pi's for IoT data logging is the huge amount of data they can log.
Visualizing it has been the problem, seeing it as a 3D terrain map was my first idea years ago.

Never had the tool to do it till now.

Easy enough to take csv files?

How to convert CSV data into a 3d mesh is a whole different topic with a multitude of answers.

The bit that might be related to the vsgXchange curl support would be if you had CSV data accessible via http then you could write a custom ReaderWriter to take that data and convert it into a vsg::Array1D with float, vec2, vec3, vec4 types, or an vsg::Arra1D with float type, or you own subclass from osg::Object that is written specifically for your task.

Cheers,

Robert.

Robert Osfield

unread,

Mar 12, 2021, 6:07:23 AM3/12/21

to vsg-...@googlegroups.com

When testing the new http/https support I tried out the vsg::LoadPagedLOD functionality that vsgviewer runs when --load-levels I found that it didn't handle the protocol prefix or nesting directories so I've made improvements to LoadPagedLOD is address this. Fixes are checked into both VulkanSceneGraph master and vsgExamples master.

This enables preloading of PagedLOD required for the initial view so that all the you don't see the popping of LOD's as they are loaded when you first open up the viewer. You can combine this with file cache support as well i.e :

vsgviewer https://raw.githubusercontent.com/robertosfield/TestData/master/Earth_VSG/earth.vsgb --file-cache ~/MyFileCace --load-levels 5

For those familiar with the OpenSceneGraph paging support much of the VSG's paging support will be not too different from what you are used to - both have PagedLOD/DatabasPager classes and ability to store files loaded from http in a local file cache. The vsg::LoadPagedLOD utility doesn't have an equivalent though, technically one could be implemented, just never thought of wrapping up the functionality like that on the OSG side.

As well as avoiding the LOD popping on initial load up the vsg::LoadPagedLOD can also be used selectively during the run of an application. For instance if you application had a instant jump from location to location on a whole earth paged database you'd normally get lots of LOD popping, but instead you can set up the LoadPagedLOD with the new view frustum setting run it to load all the required tiles then render the frame loop as normal. There will be a pause while this loading happens, you can adjust how many levels to page in to manage this - this can mean once rendering starts again not all required tiles and loaded in so you still get some LOD popping, but you can avoid the biggest jumps.

Robert Osfield

unread,

Mar 12, 2021, 6:19:08 AM3/12/21

to vsg-...@googlegroups.com

While we now have the capability of loading data including paged databases from the internet the work is still in its infancy. It's that will need further work are:

Adding support into vsg::DatabasePager for managing local and http load requests via dedicated threads so that fast local file loads aren't held up by slower http loads. This was an important refinement to the OpenSceneGraph's DatabasePager as it made interacting with really large databases shared over the web more seamless with less awkward stalls when users moved quickly from place to place and back again.
Adding support for passing http authentication information to the https requests
Improved IO error handling in the vsgXchange::curl.
Multi-threaded vsg::LoadPagdLOD

While the original work on vsg::DatabasePager was funded in the early days of the VSG project, all this recent work on it and vsgXchange is self funded by myself or others contributing to vsgXchange. There's a limit to how much work I can self fund for the greater good so will have to move on for now. I'll start a separate topic on funding of VSG dev work as it's a wider topic beyond paging and http.

Robert Osfield

unread,

Mar 12, 2021, 2:44:43 PM3/12/21

to vsg-...@googlegroups.com

As a proof of concept/demonstrator of the VSG's PagedLOD and new curl support I have begun writing a vsgpagedlod example. It doesn't do anything interesting yet as I've just started work on it. My intention is for this example to be able download tiles from tile servers like openstreemap and readymap and render a globe as a dynamically generated database.

If you want to keep tabs on the work it's being checked into the PagedLOD branch of vsgExamples, the example can be found in vsgExamples/examples/nodes/vsgpagedlod.

Reply all

Reply to author

Forward