Global Mapper Reproject

[Pompeo Mixon]

Theoriginal image is in MrSID format. I can convert that to either a ECW or a TIFF using mrsidextract that has been mentioned on the forum. In either case the image will eventually have to be uncompressed to be reprojected. The TIFF is about 34GB and the ECW is about 1.4GB. After reprojection I am going to recompress to ECW to use as a background layer.

Manifold will not import the 34GB TIFF. It won't even try. It immediately displays an error "Can't open file". I am guessing that it maxes out either my hardware or Manifold. I am running WinXP64 with 4GB of memory. Anyone know what the limits are on importing images? If it is a hardware issue then maybe I can remedy it.

Manifold imports the ECW just fine. Converting it to an RBG (i.e. uncompressing it) is another story. I have not run it long enough see how far it gets before it bombs out, but others on the forum have had troubles creating similarly sized uncompressed images.

I do have the huge image in a tiled format, but using those introduces a new set of problems. Originally I just reprojected each tile and created a mosaic. However, since the images were no longer orthogonal after reprojection and ECW doesn't support invisible pixels, I ended up with the remains of black wedges in the mosaic. The only way to get around this seems to be to create a mosaic after reprojecting, but before compression. However, this puts me right back in the same scenario of dealing with a large uncompressed image.

If you do follow the gdalwarp route then you could warp the SID straight to TIFF - but not sure what limits it can work with in hardware. I reckon your only hope with Manifold and the uncompressed TIFF is to get more RAM and do all you can to maximize the system's performance.

I almost forgot: also for the gdalwarp route, make sure you use tiling which will give you far better performance. Something like -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 - search on those argument names for discussion.

Any idea how much RAM one might need to handle a 34GB TIFF? I can upgrade by system from 4GB to 8Gb, but if the entire image needs to be in RAM at the same time this isn't going to help much. I would still have to increase the virtual memory to allow a 34 GB file to be loaded into "memory" (if it is really possible to allocate that much virtual memory in Windows). It would be incredibly slow, but it might work with enough time.

I would recommend that you use a subset for testing - I think -ts in gdalwarp lets you set the target size, so you can check that the conversion from MrSID (to TIF or ECW) is working as you expect. You probably want to look at compression options for either TIF or ECW too - see the -co creation options on the pages for those formats.

a TIFF is a compressed format, so that 34GB TIFF of yours might uncompress (as it does when being imported into Manifold) into, what? 100GB? Even bigger? Are you sure you have 400GB of free space on hard disk in all the right places (TEMP folder, pagefile, etc.)?

With current editions of Manifold it is doable, but just barely, and requires great patience. Your task is a very large one, the sort of thing that just a few years ago was not possible unless you spent hundreds of thousands of dollars.

You will need x64 hardware that has lots of free hard disk space. Think in terms of 1000GB disks in a striped array so you can have, say, 800GB of free space. I'd use a quad core processor, not in expectation of getting massive multi-threading going, but just so that no secondary tasks wasted any cycles I'd rather have devoted ot Manifold. I'd have at least 8 GB of RAM and, frankly, would probably try to find a "server" motherboard that would allow me to install 16 GB or 32 GB of RAM. RAM is cheap. I'd use XP x64 because it is the fastest and x64 Manifold. With all that it could still take a long time, hours or days, for each step.

Once I was done re-projecting the image I would save it within spatial DBMS so that in the future I could work with it very rapidly. I'd also save it to ECW just to have that as well, if ECW supports the projection I want to use.

Not at the present time. The right solution is just a bit in the future: we need an evolved Manifold that can not only store images in DBMS storage as it does now in tiles but also do things like reproject data that is stored within such tiles, doing the re-projection "in place."

There is also the problem of getting data from formats like TIFF. It would be nice to have a loader that could take a huge TIFF and, without loading the entire thing into Manifold, load bits of it in pieces for tiled upload into superfast DBMS storage. You could then point Manifold at a folder full of such large files and, without having to have huge free space to decompress the TIFFs, have Manifold spend the weekend uploading them into fast, working storage in the DBMS.

1. My original image compressed in MrSID. I have to use the free MrSIDextract.exe utility to extract it to an ecw or tif. The problem is that the resulting tifs are cannot be opened by any other software. I have tried ArcMap, ERMapper (I downloaded a trial to see if it would work), and Manifold. Manifold just displays "can't open file" while ArcGIS and ERMapper say that it is not a valid raster. When I first posted I thought it might be a size limitation of Manifold due to the 34GB tif, but since even ERMapper won't open it that doesn't seem to be the problem. Has anyone else run into similar problems using MrSIDextractor? I don't have problems with the .ecw extraction of the same image.

2. I did reproject some smaller test images using gdal. It is quite fast compared Manifold although I haven't timed it yet. The syntax is very simple and can produce a reprojected tif directly from a sid. The reprojected images worked fine.

I then reprojected the entire image without any problems on the hardware side. I am running WinXP 64 with 4GB RAM, and Intel Core 2 Duo 6450. It finished in 4 or 5 hours. However, again the resulting tif could not be opened by Manifold, ArcGIS, or ERMapper. GDAL doesn't seem to have a problem working with it. Is there something with large (34GB) tifs that might be causing these problems?

Ah! It would be a BigTIFF of course. GDAL only recently included support for this type. I don't think Manifold can read BigTIFF. You might test by creating a TIFF that is less than 4Gb, and see if Manifold and co. can read that. -co BIGTIFF=YES forces their creation for any size, but anything over the GeoTIFF limit will automatically use that option.

What are the software options for working with BigTIFF? I need to compress this 34GB tiff to ECW so I can use it in Manifold. Unfortunately, gdal has 500MB compression limit due to ERMapper codec licensing so it cannot compress the image.

LARGE_OK=YES: Under some circumstances this option can be set to allow compressing files larger than 500MB. It is the users responsibility to ensure that ERMapper licensing requirments for large file compression are being adhered to.

Global Mapper is a great companion tool for Manifold (or any GIS), and can convert sid's to ECW with no size limit, but it's not free (under $400). Or if jp2k is acceptable, you can convert to jp2k for free with FW Tools (or GDAL - FW Tools is a gui interface for GDAL). I posted a pdf with basic instructions for using FW Tools to convert sid's to jp2k here:

I have had to reproject large imagery files before. My requirement was to mosaic hundreds of ECW tiles (around 50GB total) and reproject at the same time. The output was also clipped to a vector polygon.

I used Global Mapper, which chugged away steadily, to mosaic/clip/reproject into BIL filetype (which didn't seem to have a file size limit). I then used an evaluation copy of ER Mapper to compress the BIL file to ECW.

The evaluation ER Mapper times out after 2 hours' use, however the compression seems to continue in the background if you leave it alone. I set it to batch compress, took about a week on P4 3.8GHz with 2GB RAM.

Global Mapper is what I use for any rester work including SID files. I have over 800gb of NY State sid files. Wish that Manifold would handle those files better. Even with all those clamis that Manifold can load large ECW files so much faster in v8, Global mapper still open rester files 3 times faster and any vector file more than 5 times faster. Amazing considering that it's written by couple guys. Unfortunetly it is useless for any GIS crunching and display.

Global Mapper script files allow the user to create custom batchprocesses that make use of the functionality built in to Global Mapper.From a script, one can import data in any of the numerous formats supported bythe software, reproject that data if desired, and export it to a new file.

Global Mapper script files consist of a series of command lines. Eachcommand line begins with a command. A series of parameter/value pairs shouldfollow the command. These pairs should be written as parameter=value. No spacesshould exist before or after the equal sign. Individual parameter/value pairsshould be separated by spaces. If a pair requires spaces internal to the value,quotes may be used around the entire value. For example, for a filename withspaces, the pair could look like FILENAME="c:\\my documents\\test.tif".Parameters that expect a value of YES or NO to enable or disable functionalitycan (starting with v13.1) be enabled with just the parameter name. So rather thansaying FLAG_PARAM_NAME=YES, you can just say FLAG_PARAM_NAME to get the samebehavior as specifying yes.

Command lines typically consist of one line each. To extend a command toanother line, use the backslash character (\) at the end of the line. There are a fewexceptions to this, including the DEFINE_PROJ andDEFINE_SHAPE commands andthe looping functionality provided by the DIR_LOOP_STARTand DIR_LOOP_END commands.

You can run a Global Mapper script file automatically be passing it on thecommand line to the Global Mapper .exe file. The script file will be run with nouser interface displayed and Global Mapper will immediately exit when the scriptfile completes processing. This allows you to easily run Global Mapper scriptsfrom another application or from a DOS batch file. Note that your script filesneed to have an extension of .gms for this to work.

3a8082e126