Rasterlite2 creating coverage / image import error

[VIns D]

Hi,

I am testing Spatialite/rasterlite2. I have got a couple of JPEG images which have the same properties, resolution... that I would like to store using rasterlite2. (example in attachment)

Problem: when trying to store my rasters I have got the following error (*** Error in `sqlite3': double free or corruption (fasttop)). Perhaps my coverage is not defined correctly?

Config: debian8, SQLite version: 3.16.2, SpatiaLite version: 4.3.0a , RasterLite2 version: 1.0.0-devel

My workflow:

1. In order to create my coverage I use gdalinfo to get the pixel size,srid and other parameters required. Here is the output:

Driver: JPEG/JPEG JFIF

Files: test_image_9.jpg

       test_image_9.jpg.aux.xml

Size is 5904, 5073

Coordinate System is:

GEOGCS["WGS 84",

    DATUM["WGS_1984",

        SPHEROID["WGS 84",6378137,298.257223563,

            AUTHORITY["EPSG","7030"]],

        AUTHORITY["EPSG","6326"]],

    PRIMEM["Greenwich",0,

        AUTHORITY["EPSG","8901"]],

    UNIT["degree",0.0174532925199433,

        AUTHORITY["EPSG","9122"]],

    AUTHORITY["EPSG","4326"]]

Origin = (115.610548499999993,-31.204588500000000)

Pixel Size = (0.000004500000000,-0.000004500000000)

Image Structure Metadata:

  COMPRESSION=JPEG

  INTERLEAVE=PIXEL

  SOURCE_COLOR_SPACE=YCbCr

Corner Coordinates:

Upper Left  ( 115.6105485, -31.2045885) (115d36'37.97"E, 31d12'16.52"S)

Lower Left  ( 115.6105485, -31.2274170) (115d36'37.97"E, 31d13'38.70"S)

Upper Right ( 115.6371165, -31.2045885) (115d38'13.62"E, 31d12'16.52"S)

Lower Right ( 115.6371165, -31.2274170) (115d38'13.62"E, 31d13'38.70"S)

Center      ( 115.6238325, -31.2160028) (115d37'25.80"E, 31d12'57.61"S)

Band 1 Block=5904x1 Type=Byte, ColorInterp=Red

  Overviews: 2952x2537, 1476x1269, 738x635

  Image Structure Metadata:

    COMPRESSION=JPEG

Band 2 Block=5904x1 Type=Byte, ColorInterp=Green

  Overviews: 2952x2537, 1476x1269, 738x635

  Image Structure Metadata:

    COMPRESSION=JPEG

Band 3 Block=5904x1 Type=Byte, ColorInterp=Blue

  Overviews: 2952x2537, 1476x1269, 738x635

  Image Structure Metadata:

    COMPRESSION=JPEG

 

2. Let's define the coverage based on the info above:

sqlite> SELECT RL2_CreateRasterCoverage('mycoverage','UINT8', 'RGB', 3, 'JPEG', 80, 512, 512, 4326,0.0000045); 

3. Let's load one raster 

sqlite> SELECT RL2_LoadRaster('mycoverage',  'test_image_9.jpg');

*** Error in `sqlite3': double free or corruption (fasttop): 0x00007fef931e1de0 ***

======= Backtrace: =========

/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7fef91e66bcb]

/lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7fef91e6cf96]

/lib/x86_64-linux-gnu/libc.so.6(+0x777de)[0x7fef91e6d7de]

/usr/lib/x86_64-linux-gnu/mod_rasterlite2.so(+0x6394e)[0x7fef9110e94e]

/usr/lib/x86_64-linux-gnu/mod_rasterlite2.so(+0x65cba)[0x7fef91110cba]

/usr/lib/x86_64-linux-gnu/mod_rasterlite2.so(rl2_load_raster_into_dbms+0x1f)[0x7fef9111113f]

/usr/lib/x86_64-linux-gnu/mod_rasterlite2.so(+0x79541)[0x7fef91124541]

sqlite3(+0x9997b)[0x7fef92dc697b]

sqlite3(+0xa1f8f)[0x7fef92dcef8f]

sqlite3(+0x1212e)[0x7fef92d3f12e]

sqlite3(+0x12423)[0x7fef92d3f423]

sqlite3(+0x12843)[0x7fef92d3f843]

sqlite3(+0x12f65)[0x7fef92d3ff65]

sqlite3(+0xcc22)[0x7fef92d39c22]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fef91e162b1]

sqlite3(+0xd76a)[0x7fef92d3a76a]

======= Memory map: ========

7fef7c000000-7fef7c021000 rw-p 00000000 00:00 0

 ...

I believed the coverage pixel size coverage was not setup correctly so I have tried rl2sniff which wouldn't help in this case:

image_format MD5_checksum image_path width height sample_type pixel_type bands no_data compression srid x_resolution y_resolution min_x min_y max_x max_y

JPEG test_image_9.jpg 5904 5073 UINT8 RGB 3 JPEG

I am wrong with the pixel size or the sample type "UINT8" ?

I have tried on a Windows machine and got an error as well.

 

Cheers

V.

[a.fu...@lqt.it]

On Wed, 13 Sep 2017 09:01:18 -0700 (PDT), VIns D wrote:
> Hi,
>
> I am testing Spatialite/rasterlite2. I have got a couple of JPEG
> images which have the same properties, resolution... that I would
> like
> to store using rasterlite2. (example in attachment)
>
> Problem: when trying to store my rasters I have got the following

> error_ (*** Error in `sqlite3': double free or corruption
> (fasttop))_.

> Perhaps my coverage is not defined correctly?
>

Hi,

receiving a "double free" message isn't a trivial error as many
others; it'ss a very serious symptom of some critical bug, and it
usually causes a sudden program crash due to heavily corrupted
memory (as it actually happens in your specific case).

note: you are testing a desperately obsolete version of RasterLite2.
the current development version is 1.1.0-devel, and it strictly
requires
libspatialite 4.5.0-devel.
if you really intend to perform any serious testing you necessarily
have to perform a custom build starting from the current sources
available from Fossil repositories.
the updated documentation about SQL functions is here:
https://www.gaia-gis.it/fossil/librasterlite2/wiki?name=sql_reference_list

coming to your sample: there is an obvious glitch in your workflow,
and I strongly suspect that it's the most reasonable cause
triggering the above bug.
all JPEG rasters always lack georeferencing informations (not
natively supported by the image format itself), so RasterLite2
always expects to get a reference to a companion WorldFile (*.jgw
suffix in the case of JPEG) when executing RL2_LoadRaster(); but
it completely lacks in your example, so you'll end up by
attempting to import a raster presenting undefined georeferencing,
and this can easily cause a severe malfunction.

> I believed the coverage pixel size coverage was not setup correctly
> so I have tried rl2sniff which wouldn't help in this case:

> ------------ <snip> -------------

As you can easily notice the "srid", "x_resolution", "y_resolution",
"min_x", "min_y", "max_x" and "max_y" columns are empty (=undefined).
the obvious cause is once again the absence of the expected
companion *.jgw WorldFile.

> I am wrong with the pixel size or the sample type "UINT8" ?
>

it's correct: JPEG images are usually base on 8-bit band
components.

> I have tried on a Windows machine and got an error as well.
>

not surprisingly, because the bug strictly depends on the
absence of a companion WorldFile, and this obviously is
a platform independent critical factor.

=========

I've quickly tested your sample on rasterlite2-1.1.0-devel,
and anything ran smoothly (once created an appropriate
WorldFile, a very trivial task just requiring some text
editor).
here is the exact log of my SQL queries:

---------------------------------------------------------
SELECT CreateRasterCoveragesTable();

SELECT RL2_CreateRasterCoverage('mycoverage','UINT8', 'RGB', 3,
'JPEG', 80, 512, 512, 4326, 0.0000045);

SELECT RL2_LoadRaster('mycoverage',
'c:/sandro_tests/test_image_9.jpg',
1, 4326, 1, 1);
SELECT SE_UpdateRasterCoverageExtent('mycoverage');
---------------------------------------------------------

you'll find attached to this mail:
1. the JPG WorldFile required by RasterLite2 for importing
JPEG files
2. a screenshot exemplifying a magnified detail of your
JPEG as shown by the new Map tool supported by
spatialite_gui-2.1.0-devel

bye Sandro

[VIns D]

Hi Sandro,

Thanks for your answer! 

It definitely sets me on the right path.

I will keep my progress posted here.

I have got 500gb of imagery that I would like to store. Looking at the benchmark section I was pretty impressed with the compression gain with rasterlite2.

Cheers

V.

[khanh dinh]

Hi Sandro,

Very interesting

you'll find attached to this mail:
1. the JPG WorldFile required by RasterLite2 for importing
    JPEG files

    

    -- Which tool do you use to get the file test_image_9.jgw  from the jpeg file test_image_9.jpg

        I'm running Linux Debian.

Regards

 

[a.fu...@lqt.it]

On Thu, 14 Sep 2017 19:29:05 -0700 (PDT), VIns D wrote:
> I have got 500gb of imagery that I would like to store. Looking at
> the
> benchmark section I was pretty impressed with the compression gain
> with rasterlite2.
>

Hi,

if you are mainly interested in strongly compressed RGB images
(digital aerial ortophotographs and alike) still preserving a
fairly good visual appearance you'll surely appreciate the
Jpeg2000 codec (based on OpenJPEG) by setting a quality index
as low as 5.

interesting new: the very recent openjpeg-2.2.0 (released
about one month ago) introduces several interesting
optimizations and is expected to be noticeably faster than
any previous version; quickly updating to OJ 2.2.0 is warmly
recommended.
the current development version of librasterlite2 available
from the Fossil repository already supports OJ 2.2.0

http://www.openjpeg.org/2017/04/27/Faster-OpenJPEG-is-on-track

hint: OpenJpeg still remains slower than jpeg-turbo, most
notably when compressing.
librasterlite2 supports a multi-threaded raster importer
that can effectively contribute to speed up the initial
setup of Jpeg2000 compressed Raster Coverages.

note: the default behavior of RL2 is always single-threaded;
in order to enable multi-threading you simply have to call
RL2_SetMaxThreads() and RL2_GetMaxThreads() before starting
to populate your Raster Coverage by importing individual input
image files from the file system.
if you are working on an Intel i5/i7 CPU having 8 logical
cores the best setting you can adopt for maximum speed
is "SELECT RL2_SetMaxThreads(8);" (or even a little bit
more aggressive "SELECT RL2_SetMaxThreads(12);".

hint #2: the latest development version of spatialite_gui
(2.1.0-devel) fully supports creating and populating
a Raster Coverage by using user friendly GUI wizards;
it could be surely worth to have a look ;-)

bye Sandro

[a.fu...@lqt.it]

On Fri, 15 Sep 2017 00:44:17 -0700 (PDT), khanh dinh wrote:
> Which tool do you use to get the file test_image_9.jgw  from
> the jpeg file test_image_9.jpg
> I'm running Linux Debian.
>

Hi Khanh,

in order to create the *.jgw WorldFile I've just used
the "geany" text editor (by manually copying&pasting
values from the gdalinfo output), but your are free
to use any other text editor such as "ed", "vi",
"emacs", "notepad" (on windows) or whatever else.

any WorldFile is just a simple text file and has
a fixed "format" based on six lines:
- line 1: the horizontal size of each pixel (measured
in Map Units).
- lines 2 and 3: rotation, usually set as zeroes.
- line 4: the vertical size of each pixel (measured
in Map Units, and usually expected to be a negative
value).
- line 5: the X coordinate (in Map Units)
corresponding to the centre of the upper left
pixel.
- line 6: the Y coordinate (in Map Units)
corresponding to the centre of the upper left
pixel.

the base name of the WorldFile must exactly match
the base name of the corresponding image file, and
the suffix must match the following correspondence
schema:
- TIFF (*.tif or *.tiff) = *.tfw worldfile
- JPEG (*.jpg or *.jpeg) = *.jgw worldfile
- Jpeg2000 (*jp2) = *.j2w worldfile


you could probably find useful reading this
post from GisStackExcahnge:
https://gis.stackexchange.com/questions/9421/creating-tfw-and-prj-files-for-folder-of-geotiff-files

bye Sandro

[VIns D]

Hi Sandro thanks for your advice and useful hint

The Jpeg2000 sounds like a great choice. 

COming back to your previous answer I have noticed something I am not too sure about:

SELECT CreateRasterCoveragesTable(); 

SELECT RL2_CreateRasterCoverage('mycoverage','UINT8', 'RGB', 3, 
        'JPEG', 80, 512, 512, 4326, 0.0000045); 

SELECT RL2_LoadRaster('mycoverage',   
'c:/sandro_tests/test_image_9.jpg', 
        1, 4326, 1, 1); 
SELECT SE_UpdateRasterCoverageExtent('mycoverage'); 

So here the coverage is defined with a resolution of 0.0000045 .

Then the following raise some questions 

SELECT RL2_LoadRaster('mycoverage',   
'c:/sandro_tests/test_image_9.jpg', 
        1, 4326, 1, 1); 

1. We can redefine the image resolution during import? as per your parameters 1,1 
   
2. The image loaded doesn't need to be of the same resolution as the coverage?

Cheers

V.

[mj10777]

On Friday, 15 September 2017 14:13:51 UTC+2, VIns D wrote:

Hi Sandro thanks for your advice and useful hint

The Jpeg2000 sounds like a great choice. 

COming back to your previous answer I have noticed something I am not too sure about:

SELECT CreateRasterCoveragesTable(); 

SELECT RL2_CreateRasterCoverage('mycoverage','UINT8', 'RGB', 3, 
        'JPEG', 80, 512, 512, 4326, 0.0000045); 

SELECT RL2_LoadRaster('mycoverage',   
'c:/sandro_tests/test_image_9.jpg', 
        1, 4326, 1, 1); 
SELECT SE_UpdateRasterCoverageExtent('mycoverage'); 

So here the coverage is defined with a resolution of 0.0000045 .

Then the following raise some questions 

SELECT RL2_LoadRaster('mycoverage',   
'c:/sandro_tests/test_image_9.jpg', 
        1, 4326, 1, 1); 

1. We can redefine the image resolution during import? as per your parameters 1,1 
   
2. The image loaded doesn't need to be of the same resolution as the coverage?

See:

https://www.gaia-gis.it/fossil/librasterlite2/wiki?name=sql_reference_list

SELECT RL2_LoadRaster

(

 -- chosen name of raster_coverage

 lower('1926.berlin_k4'),

 -- path to the file to be loaded

 '../berlin_images/1926.berlin.4000.3068.k4.tif',

 -- with world-file

 0,

 -- srid

 3068,

 -- pyramidize

 0,

 -- transaction

 1

);

 

[a.fu...@lqt.it]

On Fri, 15 Sep 2017 05:13:50 -0700 (PDT), VIns D wrote:
>> SELECT RL2_CreateRasterCoverage('mycoverage','UINT8', 'RGB', 3, 
>>         'JPEG', 80, 512, 512, 4326, 0.0000045); 
>

> So here the coverage is defined with a resolution of 0.0000045 .
>

all right, that's correct

> Then the following raise some questions 
>
>> SELECT RL2_LoadRaster('mycoverage',   
>> 'c:/sandro_tests/test_image_9.jpg', 
>>         1, 4326, 1, 1); 
>

> * We can redefine the image resolution during import? as per your
> parameters 1,1 
>

sorry, but you are wrongly interpreting the arguments
note: the signature of RL2_LoadRaster() abruptly changed in RL2-1.1.0,
and now is as follows:

RL2_LoadRaster ( coverageName String ,
sourcePath String ,
withWorldFile Integer ,
forceSRID Integer ,
pyramidize Integer,
transaction Integer )


so "..., 1, 4326, 1, 1" should be interpreted as:
* 1 = a companion WorldFile is required
* 4326 = the intended SRID (the Worldfile itself does
not contains any SRID value, so you necessarily have
to explicitly set one).
* 1 = immediately build a Section Pyramid while loading
* 1 = automatically handle BEGIN / COMMIT as appropriate

many SQL functions changed their signatures in RL2-1.1.0,
so a very good practice is the one to always carefully
checking the updated documentation you'll find here:

https://www.gaia-gis.it/fossil/librasterlite2/wiki?name=sql_reference_list


> * The image loaded doesn't need to be of the same resolution as the
> coverage?
>

it's no longer required in RL2-1.1.0
now you can create a Raster Coverages in three different
"resolution modes" (note: you can never change the "mode"
once the Coverage has been created):

1. "strict resolution mode": all Sections must have
exactly the same identical resolution
2. "approximative resolution mode": same as above,
but allowing for some moderate tolerance.
3. "mixed resolution mode": every individual Section
could legitimately have an arbitrary resolution
of its own.
Note: "mixed resolution" Coverages can just
support section-based Pyramids; monolithic
pyramids are strictly forbidden in this case.

bye Sandro

[VIns D]

Yes sorry I got confused :) 

Thanks.

Cheers

V.