Sentinel 2 Imagery for Offline Cesium

[Evan]

Cesium Community,

I am trying to find high quality satellite imagery for an offline Cesium project. The computer cannot be connected to the internet, so depending on internet entities at runtime is out of the question. I am looking to download satellite imagery and transfer over to the computer to store on a local server for offline use with Cesium.

I have searched all around for free, open-source, high resolution (looking to get around zoom-level 10) imagery, and the most promising source that I came across was the Sentinel satellite imagery from the ESA. I noticed that in the newer versions of Cesium, they have included Sentinel-2 satellite imagery into the product. 

I was hoping someone in this forum has insight on how to get this data in a format that is compatible with Cesium (keep in mind it needs to be stored in a local drive on an off-the-network computer). I have searched the Amazon S3 web service, but it appears it is only raw data. I am looking for something that is true-color and in a format (i.e Tile Map Server, Web Map Server) that will be easily portable to Cesium.

Any information would be beneficial, thank you.

-Evan

[Mark Erikson]

I've been meaning to write a couple blog posts about the process of converting and serving self-hosted Cesium imagery and terrain.  Having said that, I also did a bit of research into commercial options, and the best dataset I found is from OpenMapTiles, at https://openmaptiles.com/downloads/dataset/satellite/ .  It's a worldwide dataset down to zoom level 13, 200GB, for $1200, and it sounds like it's already in a format suitable for serving.

Note that I haven't actually _used_ this dataset yet, as we're still going through internal paperwork shenanigans trying to get approval to purchase it, but it looks like a great option.

Since I don't seem to have time to write those blog posts atm, I'd be happy to answer any questions you have about self-hosting imagery and terrain.

[Evan]

Mark,

Thank you for the timely response, I have come across OpenMapTiles in my search as well, and it does look like it is the best option for serving the tiles to Cesium. The tiles from this service appear to come in the MBTiles format, I have tried serving MBTiles before to Cesium with OpenStreetMap (non-satellite) data. I struggled getting the tiles to appear using this format, but that could have also been user error, as I have read of other users serving MBTiles to Cesium. 

The only issue about this service would be the pricing, currently my project is working off of no budget for software, and are entirely operating off of open-source products. If we are to get funding, then purchasing this product would be ideal (assuming MBTiles works). But, since this looks like the most promising data set for Sentinel imagery, and the fact that they charge $1200, I have little hope in being able to find the same sort of data set at a lower price (AKA free). 

I have actually written an instructional paper about serving OpenStreetMap tiles to Cesium for offline-use. If you are interested in this topic, I would be willing to share it with you. 

Do you know the means in which the newer versions of Cesium are hosting the Sentinel-2 data? Does the company have a license from a provider to serve them via a web server, or does the company own the tile set themselves? 

-Evan

[Mark Erikson]

Lemme sketch out my understanding of the overall situation for clarity (apologies if some of this is info you're already familiar with):

- The Cesium client can be pointed at many different sources of imagery and terrain data, including arbitrary user-specified endpoints that follow standard TMS-format tile file structures.  So, if you have your own tilesets, you can self-host the data with an appropriate tile server tool, and point Cesium at that server.
- MBTiles is a container format to simplify the storage and transport of map tiles, as it allows storing thousands or millions of individual tile files in a single container.  It's a lot easier to move around one (or a few) multi-gig MBTiles files than it is millions of individual images.  There's many existing server applications that can serve up tiles from an MBTiles container, and also trivial to write your own.  (Our own homegrown MBTiles server is a couple hundred lines of Python, and most of that is logic for scanning a folder for available MBTiles  files that make up a dataset.)
- There's various free sources of worldwide imagery out there, but the quality is generally lower.  One older list of sources is at http://vterrain.org/Imagery/WholeEarth/ .  We've used NASA's BlueMarble, which gave us zoom level 8 tiles, as well as the TrueMarble set, which gave us zoom level 9.  Both of those had various visual quality issues that weren't really satisfactory, hence my recent investigation into other sources. 
- There _are_ datasets from satellites like Sentinel-2 and Landsat-8, but those involve lots of striped images that have overlap, varying cloud coverage, varying lighting, etc, and they're at a single resolution rather than a tilemap hierarchy.  My initial research did show sites where you can download some of those individual images, but it looked like trying to make a good worldwide composite image set to use as a source of tiles would be difficult. 
- AGI, the company behind Cesium, has stood up a new service called Cesium Ion that hosts pregenerated imagery and terrain data, as well as the ability to upload and convert user-provided data.  Looking at the Cesium Ion "Assets Depot" list, I do see a listing for a Sentinel-2 tileset down to zoom level 13.  However, I would assume this is something that AGI has spent time and money producing themselves based on available Sentinel-2 images, and not a dataset that is available for purchase or download.  Given that your requirements are for an offline dataset, I don't think this will be anything that will help you.

Mark

[Mark Erikson]

Hmm.  One additional new bit of info here.  Doing some searching for "sentinel-2 map tiles", I came across https://s2maps.eu/ and https://eox.at/2017/03/sentinel-2-cloudless-original-tiles-available/ , which looks as if someone has already done the work of taking mosaiced images and creating high-res images out of them.  The site appears to host pregenerated map tiles, but it also says that the source GeoTIFF images are available for download from an AWS S3 bucket using the "requestor pays" setting.

This actually appears to be the same dataset that was used to produce the OpenMapTiles product I linked.  So, it ought to be feasible to download the source GeoTIFFs from AWS, run them through a tile map conversion step, and self-host them, for free (at least, free dollars-wise, not counting time and effort :)  )

[Mark Erikson]

This evening I poked around a bit at the process of setting up an AWS account and trying to download some of the Sentinel-2 GeoTIFFs.  It's doable, but I'm pretty sure that the cost of purchasing the premade MBTiles container outright is way less than the time cost needed to download the images, massage them, and generate the tiles.

[Evan]

Mark,

It seems like you and I are fairly aligned with the structures that make up the tile map servers for Cesium. Much of what you said in the first of the three most recent responses is full of good information that any curious reader should definitely try to understand. One thing you mentioned that I was surprised I hadn't stumbled upon was the http://vterrain.org/Imagery/WholeEarth/ website. But like you said, a good majority of sources for satellite imagery that I have come across, tend to have visual quality issues that are not suitable for my application.

In response to your other replies, this seems very reminscent to when I was originally searching for satellite imagery tiles. I too have come across the AWS bucket of data, and it looks like there needs to be a lot of processing to the imagery to make it a suitable format for serving to Cesium. The tiles from AWS are from all different bands of electromagnetic spectrum, so combining them to make true color seems like a challenge in itself, not to mention that you would then need to stitch all the pictures together. I would have to agree with you that purchasing the MBTiles outright would far outweigh the cost of figuring out how to bring together all the raw data yourself. 

Unfortunately, it seems that the best way to get this high-resolution satellite imagery for the entire earth for offline use is to purchase the data yourself.

Now, from what I have experienced with MBTiles, they did not interface with Cesium correctly. Have you been able to serve a tileset in the MBTiles format to Cesium?

Thank you for your timely responses! 

-Evan 

[Mark Erikson]

Yes. Per my comment, there's plenty of existing MBTiles server tools out there - a quick search on Github will turn them up.

But, the basic principle is really simple.  Set up a Cesium imagery provider using `createTileMapServiceImageryProvider()`.  On the server side, handle a request by extracting the ZXY parameters from the URL, opening an SQLite connection to the MBTiles file, querying the database for that one row, extracting the blob, and returning it as an HTTP response. 

If your whole dataset is contained in a single MBTiles file, the logic is trivial.  If you've got multiple files (like zoom levels 0-8 in file A, level 9 in B, level 10 in C, etc), then you would need some logic for scanning a folder for available MBTiles files and figuring out which zoom levels are in which files at startup, and querying the right one when the request comes in.  That's where the bulk of the logic for our own homegrown MBTiles server lies.