Best way to generate Cloud Optimized GeoTIFF (COG)

[Julien Sabatier]

Hi all,

I'm trying to generate COG from our orthophotos.

My firsts attempt was to use directly the gdal_translate function and I faced some problems : 

- upper level are very slow to generate in GS/QGIS (not enough overviews?)

- WEBP not supported by GS

- gap between tiles on GS

So I ended trying this : 

gdalwarp -s_srs EPSG:2154 -t_srs EPSG:3857 -tap -tr 0.05 0.05 -of COG -co TILING_SCHEME=GoogleMapsCompatible -co BIGTIFF=YES -co COMPRESS=JPEG -co QUALITY=100 -co WARP_RESAMPLING=bilinear -co OVERVIEW_RESAMPLING=lanczos -co NUM_THREADS=ALL_CPUS -co OVERVIEWS=IGNORE_EXISTING source.vrt target/cog/rtge_cog.tif

But it's very long and produce a too huge amount of data (more than 6To at 70%  and still growing quickly) so I interrupted it.

After that, I read some different recipes I found on google to achive this.

Notably : 

- https://www.getbounds.com/blog/generating-cloud-optimized-geotiffs-and-raster-tiles-with-gdal/

- https://gist.github.com/palmerj/2e4a46fbcf0c97212e6e77fced22e885

So I made a script from thos methods : https://gitlab.agglo-lepuyenvelay.fr/-/snippets/1044

But when I run this on my datas (about 38k tiffs of about 16M for a total of 480G),

It is very slow when perform the gdaladdo (line 104) and take about a week for 10% of the level 2 overview.

So I wonder what are your recipes ?

Any advises to improve my script and make it faster?

Thanks in advance for yous answers =D

[FABRY Vincent]

Dear Julien,

Our recipe is avalaible here :  https://github.com/geo2france/cog-tips

Vincent

 

Vincent FABRY Agence Hauts-de-France 2020 2040 Service système d'information géographique et appui à la connaissance Administrateur des systèmes d'information +33374271583 - Standard. +33374270000 vincen...@hautsdefrance.fr 151 Avenue du président Hoover 59555 LILLE CEDEX 15 Mail Albert 1er 80000 AMIENS CEDEX

Pensez environnement : n'imprimez ce message que si nécessaire.

 

 

 

De : georc...@googlegroups.com <georc...@googlegroups.com> De la part de Julien Sabatier
Envoyé : mardi 26 septembre 2023 15:52
À : georchestra <georc...@googlegroups.com>
Objet : [georchestra] Best way to generate Cloud Optimized GeoTIFF (COG)

--
--
Vous avez reçu ce message, car vous êtes abonné au groupe
Groupe "georchestra" georc...@googlegroups.com
voir http://groups.google.fr/group/georchestra
 
Site web : http://www.georchestra.org

---
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes "georchestra".
Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse georchestra...@googlegroups.com.
Cette discussion peut être lue sur le Web à l'adresse https://groups.google.com/d/msgid/georchestra/00d81a71-63d9-4ab5-ad66-f2b943dee403n%40googlegroups.com.

[Landry Breuil]

my usecase is a bit different since i dont target GS, and i also dont
reproject (which is what takes an insane amount of time...) - also using
a recent version of gdal helps with native cog output support since 3.1
(and improved in 3.3 iirc - https://gdal.org/drivers/raster/cog.html)

https://github.com/IGNF/gdalCOG/blob/dev/script/gdal_COG.sh might also
give you inspirations..

my workflow is inspired from nicolas' scripts:

- if i have jp2 input (eg orthohr), first pass of converting all images
to tiff with gdal_translate using -of GTIFF -co TILED=YES -co
BIGTIFF=YES -co BLOCKXSIZE=$blocksize -co BLOCKYSIZE=$blocksize -co
COMPRESS=ZSTD -co PREDICTOR=2

- build a vrt with -addalpha -hidenodata -a_srs EPSG:2154

- use gdal_translate to make the COG directly:
-of COG -co COMPRESS=JPEG -co QUALITY=90 -co
OVERVIEW_RESAMPLING=BILINEAR -co BIGTIFF=YES -co BLOCKSIZE=$blocksize

-of COG takes care of producing the overviews so i dont need to worry
about them.

used gdal 3.5.2 in a freebsd jail on a monster truenas (with 300Gb ram
but spinning disks), converted a lot of various ecw bdortho/jp2
orthohr/jp2 scans in some weeks/months and i now have 3.9Tb of COG files
in the end, the largest one being 250gb. In comparison,
orthocraig10_le_puy_2013.cog.tif is 23Go.

> Any advises to improve my script and make it faster?

avoiding reprojection... using 860Go of uncompressed 5cm TIFs (~15k
images) on lyon metropole as input, i've made a 70Go COG in 28h. And i
keep all my COGs at their native projection.

on a sidenote, COG support in mapstore2 is coming, now testing it in
https://github.com/geosolutions-it/MapStore2/issues/9320
--
Landry Breuil
Responsable Informatique
04 44 05 12 42

----------------------------------------------------------------------------
Centre Régional Auvergne-Rhône-Alpes de l'Information Géographique
Hôtel de région
59 Boulevard Léon Jouhaux - CS 90706
63050 Clermont-Ferrand Cedex 2

https://www.craig.fr <https://www.craig.fr> - @GipCraig

----------------------------------------------------------------------------
> Support utilisateurs (tous les jours ouvrés de 8H30 à 12H30) : 09 72
62 25 31

[harry cr7]

The best code to run is

-co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES elevation.tif elevation_compreessed.tif

hope it helps

[Julien Sabatier]

Thanks for your answers.

I managed to create a COG (205Go) with : 

gdalbuildvrt mosaic.vrt -addalpha -hidenodata -a_srs EPSG:2154 target/last/*.tif

gdal_translate \
  --config GDAL_DISABLE_READDIR_ON_OPEN TRUE \
  -of COG \
  -co BIGTIFF=YES \
  -co OVERVIEW_RESAMPLING=BILINEAR \
  -co COMPRESS=JPEG -co QUALITY=85 \
  -co BLOCKSIZE=256 \
  -co NUM_THREADS=ALL_CPUS \
  mosaic.vrt rtge.tif

It work well when opening in QGIS.

The problem is when import it in GeoServer, it load only 1 band of the COG : https://ibb.co/JRmW1qH

It looks like it load a mask layer...

Then it render a black and white layer : https://ibb.co/MMnDk87

Here is the gdalinfo result, it looks good : https://pastebin.com/BGdMEKbP

Any idea why it doesn't work well with GeoServer ?

[Benjamin Chartier]

Exact same trouble with the GeoServer instance of the DataGrandEst portal when the COG is published on a remote server.

If the COG file is stored in the GeoServer file system every thing is ok.

It has been reported to Camptocamp team (just in case of a known workaround).

I don't have any clue about how to solve this.

Benjamin

[Julien Sabatier]

Thanks for the tip, I tried with a local filesystem COG and it "work".

But I'm despited that GeoServer handle COG so badly...

Using the same COG on QGIS (via URL), I've near-instant map's loading when navigate, but on GS it took 3~4 sec to load a full extent tiles.

Also as my source COG is in EPSG:2154, it appear that GS don't handle the reprojection well : https://ibb.co/QFP3RKC

I hope this is due to the youth of the plugin and will improve with time.

[François Van Der Biest]

At Camptocamp, we would be more than willing to contribute a fix to the geoserver codebase and/or COG plugin.

I'm pretty sure GeoSolutions would also welcome funding.

Feel free to get in touch.

Regards,

F.

Cette discussion peut être lue sur le Web à l'adresse https://groups.google.com/d/msgid/georchestra/9b783711-ae3e-4486-9dae-182c266c4102n%40googlegroups.com.

[Julien Sabatier]

Just found that on GS tracker : https://osgeo-org.atlassian.net/browse/GEOS-10574

So tried add "-Dorg.geotools.referencing.resampleTolerance=0" to my tomcat conf and it look like it solve the reprojection problem.