ESRI's new compressed LAS format (*.zlas)
942 views
Skip to first unread message
Martin Isenburg
Dec 30, 2013, 3:18:28 PM12/30/13
to LAStools - efficient command line tools for LIDAR processing, las...@googlegroups.com
Hello,
a few moments ago I got an email from a LAStools user at NOAA pointing
out an entry in the ESRI documentation that mentions the ".zlas"
format. The user writes "I spend a good portion of time on the ArcGIS
resources pages and today is the first time I have seen any mention of
this. Googling around doesn't return any other results either. Here is
a link to the documentation:
http://resources.arcgis.com/en/help/main/10.2/index.html#//00170000016v000000
I have heard rumors about this since Gene mentioned it in his blog
entry after ESRI's 3D Mapping and LiDAR Forum:
http://blog.lidarnews.com/esri-3d-mapping-and-lidar-forum
Back then I throught they were talking about LAZ and that my 1.5 years
of talking with them about supporting LAS was finally getting
somewhere. Turns out they have been cooking their own soup instead.
Here is what rumours I have heard about ESRI's new compressed *.zlas
format:
* it gets similar compression as LAZ
* it includes spatial indexing
* it maybe re-orders the points during compression
* its performance is like laszip.exe or better
* it will be available in ArcGIS 10.2.1
* it can be used without the LAS Dataset
* there will be a "free" Windows executable (see ArcGIS resources)
* there will be development libraries with an API available
* ESRI is contacting data providers to give them a heads up that their
clients will soon demand the new *.zlas format
My first thought was that this might be a reengineered version of the
LizardTech LiDAR Compressor but it is not. Seems to be ESRI's own
development. Does anyone else have more details on this?
Regards,
Martin @rapidlasso
--
http://rapidlasso.com - not lazzed enough for ESRI (-;
a few moments ago I got an email from a LAStools user at NOAA pointing
out an entry in the ESRI documentation that mentions the ".zlas"
format. The user writes "I spend a good portion of time on the ArcGIS
resources pages and today is the first time I have seen any mention of
this. Googling around doesn't return any other results either. Here is
a link to the documentation:
http://resources.arcgis.com/en/help/main/10.2/index.html#//00170000016v000000
I have heard rumors about this since Gene mentioned it in his blog
entry after ESRI's 3D Mapping and LiDAR Forum:
http://blog.lidarnews.com/esri-3d-mapping-and-lidar-forum
Back then I throught they were talking about LAZ and that my 1.5 years
of talking with them about supporting LAS was finally getting
somewhere. Turns out they have been cooking their own soup instead.
Here is what rumours I have heard about ESRI's new compressed *.zlas
format:
* it gets similar compression as LAZ
* it includes spatial indexing
* it maybe re-orders the points during compression
* its performance is like laszip.exe or better
* it will be available in ArcGIS 10.2.1
* it can be used without the LAS Dataset
* there will be a "free" Windows executable (see ArcGIS resources)
* there will be development libraries with an API available
* ESRI is contacting data providers to give them a heads up that their
clients will soon demand the new *.zlas format
My first thought was that this might be a reengineered version of the
LizardTech LiDAR Compressor but it is not. Seems to be ESRI's own
development. Does anyone else have more details on this?
Regards,
Martin @rapidlasso
--
http://rapidlasso.com - not lazzed enough for ESRI (-;
Jorge Delgado García
Jan 2, 2014, 3:55:09 AM1/2/14
to last...@googlegroups.com
This is a bad news. We need standard formats (open format) not another formats.
Happy New Year 2014.
Jorge Delgado
University of Jaén (Spain)
--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe
| Jorge Delgado García Vicerrector de Planificación, Calidad, Responsabilidad Social y Comunicación Universidad de Jaén Campus de las Lagunillas, Edif. Rectorado (B-1). 23071 Jaén (España) Teléfono/Phone: (+34) 953 212596 Fax: (+34) 953 212638 e-mail: vic...@ujaen.es http://www10.ujaen.es/conocenos/organos-gobierno/vicplan
|
Francesco Pirotti UNIPD
Jan 2, 2014, 7:14:26 AM1/2/14
to last...@googlegroups.com
Perfectly agree with Jorge! I guess ESRI wants complete control over the formats it uses.
Happy new year everyone!
Francesco
FRANCESCO PIROTTI
CIRGEO Interdepartmental Research Center in Geomatics
TESAF Department of Land, Environment, Agriculture and Forestry
Università di Padova
Via dell'Università, 16
35020 – Legnaro (PD),Italy
Tel: +39 049 827 2710
Cell.: +39 392 395 2067
Email: francesc...@unipd.it
Skype: francesco197576
Web: http://www.cirgeo.unipd.it/fp
CIRGEO Interdepartmental Research Center in Geomatics
TESAF Department of Land, Environment, Agriculture and Forestry
Università di Padova
Via dell'Università, 16
35020 – Legnaro (PD),Italy
Tel: +39 049 827 2710
Cell.: +39 392 395 2067
Email: francesc...@unipd.it
Skype: francesco197576
Web: http://www.cirgeo.unipd.it/fp
Kirk Waters - NOAA Federal
Jan 2, 2014, 7:41:42 AM1/2/14
to last...@googlegroups.com
I also agree. Open formats are particularly important for archive and I'll continue to send data for NOAA archive in open formats such as LAS and LAZ.
Regards,
Kirk
--
Kirk Waters, PhD | NOAA Coastal Services Center
Applied Sciences Program | 2234 South Hobson Ave
843-740-1227 | Charleston, SC 29405
843-740-1227 | Charleston, SC 29405
Stuart Mowbray
Jan 2, 2014, 9:31:16 AM1/2/14
to last...@googlegroups.com
I agree, I find it amazing how many companies still push proprietary formats for no good reason at all, but I guess it comes down to controlling/keeping their market hold.
Could be worse, they could have taken a proposed new variant of an open format and bastardised it for their own needs ... I’ll avoid naming names; you know who you are! ;-)
Regards,
Stuart
--------------------------------------------------------------------------------------------------
This email and any attachments are confidential and are for the use of the
addressee only. If you are not the addressee, you must not use or disclose the
contents to any other person. Please immediately notify the sender and
delete the email. Statements and opinions expressed here may not
represent those of the company. Email correspondence is monitored by
the company. This information may be subject to export control
regulation. You are obliged to comply with such regulations.
This email and any attachments are confidential and are for the use of the
addressee only. If you are not the addressee, you must not use or disclose the
contents to any other person. Please immediately notify the sender and
delete the email. Statements and opinions expressed here may not
represent those of the company. Email correspondence is monitored by
the company. This information may be subject to export control
regulation. You are obliged to comply with such regulations.
Renishaw plc (company number 1106260), Wotton Travel Limited (company
number 01973158) and Renishaw Advanced Materials Limited (company number 04632041),
are companies registered in England and Wales with a registered office
at New Mills, Wotton-under-Edge, Gloucestershire, GL12 8JR,
United Kingdom, Telephone +44 1453 524524.
--------------------------------------------------------------------------------------------------
Newcomb, Doug
Jan 2, 2014, 10:20:38 AM1/2/14
to last...@googlegroups.com
I agree. I personally do not think that having an api constitutes openness. I do not see such a proposed format as suitable for data exchange unless it is fully and publicly documented. Based on the description above, it would also seem redundant.
Doug
Doug Newcomb
USFWS
Raleigh, NC
---------------------------------------------------------------------------------------------------------
The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.
Martin Isenburg
Jan 2, 2014, 10:53:14 AM1/2/14
to LAStools - efficient command line tools for LIDAR processing
Wow, quite a "pre-release beating" that ESRI is getting. (-:
Remember, these are just rumors I have been collecting and we have yet to hear anything official from ESRI. That said, the statements in the documentation combined with the details that have leaked give us a reasonably good idea for what is ahead.
But I like to give ESRI the benefit of the doubt and assume their motivation was *not* to have control about the format. So what could a technical reason be? Is LAZ too slow for them? Probably not. Does LAZ not map well to their internal memory-mapping or whatever else paging system. That is my best guess. Maybe they are re-ordering the points during compression for more efficient (and spatially granular) area-of-interest queries directly from the compressed file.
However, it would have easily been possible to combine LASzip compression and spatial re-ordering to produce better LAZ files for memory-mapped spatially-indexed compressed LiDAR. And I would have happily adressed whatever LASzip was lacking as (compatible) extensions to the LAZ format. After all ... part of what created LAStools are small contracts like that. But instead they chose to invest serious money and man-power into creating an entirely new format.
Anyone want to speculate what other technical reasons there might be to not use (an adapted version) of LAZ …? Maybe they seperate out the compression of those parts of the per-point data that does not usually get visualized (e.g. GPS time) into a different bands so it does not have to get decompressed unless needed? That would seem reasonable. That is - in fact - an idea I have already been playing with for when LASzip gets extended to the new 1.4 point types.
Regards,
Martin @rapidlasso
Terje Mathisen
Jan 2, 2014, 12:14:08 PM1/2/14
to last...@googlegroups.com
Martin Isenburg wrote:
> Wow, quite a "pre-release beating" that ESRI is getting. (-:
>
> Remember, these are just rumors I have been collecting and we have yet
> to hear anything official from ESRI. That said, the statements in the
> documentation combined with the details that have leaked give us a
> reasonably good idea for what is ahead.
>
> But I like to give ESRI the benefit of the doubt and assume their
> motivation was *not* to have control about the format. So what could a
> technical reason be? Is LAZ too slow for them? Probably not. Does LAZ
> not map well to their internal memory-mapping or whatever else paging
> system. That is my best guess. Maybe they are re-ordering the points
> during compression for more efficient (and spatially granular)
> area-of-interest queries directly from the compressed file.
Any pipeline that starts by tiling the data will have pretty efficient
> Wow, quite a "pre-release beating" that ESRI is getting. (-:
>
> Remember, these are just rumors I have been collecting and we have yet
> to hear anything official from ESRI. That said, the statements in the
> documentation combined with the details that have leaked give us a
> reasonably good idea for what is ahead.
>
> But I like to give ESRI the benefit of the doubt and assume their
> motivation was *not* to have control about the format. So what could a
> technical reason be? Is LAZ too slow for them? Probably not. Does LAZ
> not map well to their internal memory-mapping or whatever else paging
> system. That is my best guess. Maybe they are re-ordering the points
> during compression for more efficient (and spatially granular)
> area-of-interest queries directly from the compressed file.
access to any given area, i.e. with something like 250x250 m tiles and 2
points/sq m you get less than 0.25M points in each file even if the
entire tile is in the overlap between two flight lines.
If you need to read data off disks, then you should do IOs of at least a
MB in order to get reasonable disk efficiency, and if you already have
the tile in memory then it doesn't matter.
>
> However, it would have easily been possible to combine LASzip
> compression and spatial re-ordering to produce better LAZ files for
> memory-mapped spatially-indexed compressed LiDAR. And I would have
> happily adressed whatever LASzip was lacking as (compatible) extensions
> to the LAZ format. After all ... part of what created LAStools are small
> contracts like that. But instead they chose to invest serious money and
> man-power into creating an entirely new format.
Right, this will turn out to have been quite silly. :-(
> However, it would have easily been possible to combine LASzip
> compression and spatial re-ordering to produce better LAZ files for
> memory-mapped spatially-indexed compressed LiDAR. And I would have
> happily adressed whatever LASzip was lacking as (compatible) extensions
> to the LAZ format. After all ... part of what created LAStools are small
> contracts like that. But instead they chose to invest serious money and
> man-power into creating an entirely new format.
>
> Anyone want to speculate what other technical reasons there might be to
> not use (an adapted version) of LAZ �? Maybe they seperate out the
> compression of those parts of the per-point data that does not usually
> get visualized (e.g. GPS time) into a different bands so it does not
> have to get decompressed unless needed? That would seem reasonable. That
> is - in fact - an idea I have already been playing with for when LASzip
> gets extended to the new 1.4 point types.
This is in fact a very good idea, it is a typical feature of in-memory
> Anyone want to speculate what other technical reasons there might be to
> not use (an adapted version) of LAZ �? Maybe they seperate out the
> compression of those parts of the per-point data that does not usually
> get visualized (e.g. GPS time) into a different bands so it does not
> have to get decompressed unless needed? That would seem reasonable. That
> is - in fact - an idea I have already been playing with for when LASzip
> gets extended to the new 1.4 point types.
databases that are stored in column order, instead of the normal
per-record order.
Terje
>
> Regards,
>
> Martin @rapidlasso
>
>
> On Thu, Jan 2, 2014 at 4:24 PM, Dwight Crouse <dwight...@tesera.com
> <mailto:dwight...@tesera.com>> wrote:
>
> I agree as well.� This is a redundant format, ESRI should just adopt
> the LAZ format that people are already successfully using.
>
> Dwight
>
> On Jan 2, 2014 8:21 AM, "Newcomb, Doug" <doug_n...@fws.gov
> Regards,
>
> Martin @rapidlasso
>
>
> On Thu, Jan 2, 2014 at 4:24 PM, Dwight Crouse <dwight...@tesera.com
> <mailto:dwight...@tesera.com>> wrote:
>
> I agree as well.� This is a redundant format, ESRI should just adopt
> the LAZ format that people are already successfully using.
>
> Dwight
>
> On Jan 2, 2014 8:21 AM, "Newcomb, Doug" <doug_n...@fws.gov
> <mailto:doug_n...@fws.gov>> wrote:
>
> I agree. �I personally do not think that having an api
> constitutes openness. �I do not see such a proposed format as
> suitable for data exchange unless it is fully and publicly
> documented. �Based on the description above, it would also seem
> redundant.
>
> Doug
>
>
>
> On Thu, Jan 2, 2014 at 7:41 AM, Kirk Waters - NOAA Federal
> <kirk....@noaa.gov <mailto:kirk....@noaa.gov>> wrote:
>
> I also agree. Open formats are particularly important for
> archive and I'll continue to send data for NOAA archive in
> open formats such as LAS and LAZ.
>
> Regards,
> Kirk
> --
>
> Kirk Waters, PhD � � � � � � � � � � � � � | NOAA Coastal
> Services Center
> Applied Sciences Program � � � � �| 2234 South Hobson Ave
> 843-740-1227 <tel:843-740-1227> � � � � � � � � � � � � � �
>
> I agree. �I personally do not think that having an api
> constitutes openness. �I do not see such a proposed format as
> suitable for data exchange unless it is fully and publicly
> documented. �Based on the description above, it would also seem
> redundant.
>
> Doug
>
>
>
> On Thu, Jan 2, 2014 at 7:41 AM, Kirk Waters - NOAA Federal
> <kirk....@noaa.gov <mailto:kirk....@noaa.gov>> wrote:
>
> I also agree. Open formats are particularly important for
> archive and I'll continue to send data for NOAA archive in
> open formats such as LAS and LAZ.
>
> Regards,
> Kirk
> --
>
> Kirk Waters, PhD � � � � � � � � � � � � � | NOAA Coastal
> Services Center
> Applied Sciences Program � � � � �| 2234 South Hobson Ave
> � �| Charleston, SC 29405 � �
> www.csc.noaa.gov/digitalcoast
> <http://www.csc.noaa.gov/digitalcoast>
> <mailto:francesc...@unipd.it>> wrote:
>
> Perfectly agree with Jorge! I guess ESRI wants complete
> control over the formats it uses.
> Happy new year everyone!
> Francesco
>
>
> On Thu, Jan 2, 2014 at 9:55 AM, Jorge Delgado Garc�a
> <jdel...@ujaen.es <mailto:jdel...@ujaen.es>> wrote:
>
> This is a bad news. We need standard formats (open
> format) not another formats.
>
> Happy New Year 2014.
>
> Jorge Delgado
> University of Ja�n (Spain)
>
> Perfectly agree with Jorge! I guess ESRI wants complete
> control over the formats it uses.
> Happy new year everyone!
> Francesco
>
>
> On Thu, Jan 2, 2014 at 9:55 AM, Jorge Delgado Garc�a
> <jdel...@ujaen.es <mailto:jdel...@ujaen.es>> wrote:
>
> This is a bad news. We need standard formats (open
> format) not another formats.
>
> Happy New Year 2014.
>
> Jorge Delgado
>
>
> 2013/12/30 Martin Isenburg
> <martin....@gmail.com
> <mailto:martin....@gmail.com>>
>
>
>
> *Jorge Delgado Garc�a*
>
> /Vicerrector de Planificaci�n, Calidad,
> Responsabilidad Social y Comunicaci�n/
>
> Universidad de Ja�n__
>
> Campus de las Lagunillas, Edif. Rectorado (B-1).
> 23071 Ja�n (Espa�a)____
> Campus de las Lagunillas, Edif. Rectorado (B-1).
>
> Tel�fono/Phone:�(+34) 953 212596�� Fax:�(+34) 953
> 212638__
>
> e-mail:�vic...@ujaen.es <mailto:vic...@ujaen.es>��____
>
> http://www10.ujaen.es/conocenos/organos-gobierno/vicplan��____
>
> __�
>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>
>
>
> --
> *FRANCESCO PIROTTI*
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>
>
>
> --
>
> *CIRGEO Interdepartmental Research Center in Geomatics*
> *TESAF Department of Land, Environment, Agriculture and
> Forestry*
>
> Universit� di Padova
> Via dell'Universit�, 16
> 35020 � Legnaro (PD),Italy�
>
> Tel:�+39 049 827 2710
> Cell.:�+39 392 395 2067
> Email:�francesc...@unipd.it
> <mailto:francesc...@unipd.it>
>
> Tel:�+39 049 827 2710
> Cell.:�+39 392 395 2067
> Email:�francesc...@unipd.it
> Skype: francesco197576
> Web:�http://www.cirgeo.unipd.it/fp�
> �
>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>
>
>
> --
> Doug Newcomb
> USFWS
> Raleigh, NC
> 919-856-4520 ext. 14 <tel:919-856-4520%20ext.%2014>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>
>
>
> --
> Doug Newcomb
> USFWS
> Raleigh, NC
> doug_n...@fws.gov <mailto:doug_n...@fws.gov>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
--
- <Terje.M...@tmsw.no>
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
--
"almost all programming can be viewed as an exercise in caching"
Loesch, Tim N (DNR)
Jan 2, 2014, 12:34:26 PM1/2/14
to last...@googlegroups.com
Here’s my perspective.
Initially I was dismayed by the news. The State of Minnesota has been using LAZ compression to store LiDAR data for several years. We won’t be changing how we store the data either as it makes no sense at all to go to a proprietary format however we have a lot of stakeholders that are heavy users of ESRI software.
After pondering this notion for a while I csme to the conclusion that it’s fine for ESRI to develop their own proprietary compression and indexing schemes. IF (and that’s a big if) the products prove to be better and faster in their software then I’ll welcome using it.
Competition is one of the things that drives innovation and improvement and I would expect Martin would likely not sit still if LASTOOLs compression and indexing were outdone.
I for one am not going to stop putting pressure on ESRI to read LAZ compressed data. I’m going to continue to send them e-mails and press for them to write into their software the ability to ingest LAZ compressed LiDAR data. They support all kinds of vector and raster formats and I see no reason for them not to support LAZ format if their users continue to demand it.
Keep up the good work Martin…
tim
Timothy N. Loesch | IT Business Services Manager
MN.IT Services @ Department of Natural Resources
Phone: (651) 259-5475 (w) |
tim.l...@state.mn.us |
Lat 44.95609, Long -93.08388
|
|
Information Technology for Minnesota Government |
Martin Isenburg
Jan 2, 2014, 1:16:33 PM1/2/14
to LAStools - efficient command line tools for LIDAR processing, las...@googlegroups.com
Hello,
as I watch my blog counter going crazy right now it may be sensible to move this discussion to
http://rapidlasso.com/2013/12/30/new-compressed-las-format-by-esri/
where I have copied & pasted some of the comments that were made in this mail thread here and elsewhere.
http://rapidlasso.com/2013/12/30/new-compressed-las-format-by-esri/
where I have copied & pasted some of the comments that were made in this mail thread here and elsewhere.
Regards,
Martin @rapidlasso
Martin Isenburg
Jan 7, 2014, 5:28:51 AM1/7/14
to last...@googlegroups.com, The LAS room - a friendly place to discuss specifications of the LAS format
Hello,
It is official now and I have updated the blog:
It is official now and I have updated the blog:
Apparently, Gene who has mentioned our rumors on our blog just received heads up from Clayton Crawford at ESRI that the LAS Optimizer is available for download here. The EzLAS.zip file contains a PDF with interesting details. Below a snapshot of the GUI and what the website says:
"This executable is used to optimize and compress LAS format lidar. It creates *.zlas files, an optimized version of LAS that's useful for archiving, sharing, and direct use. zLAS files are much smaller and more efficient to use, especially on the cloud and over networks, than regular LAS.
* The standalone executable does not require an ArcGIS install or license.
* The same executable is used for both compression and decompression.
* The download zip file contains more information and help in an included pdf document."
* The standalone executable does not require an ArcGIS install or license.
* The same executable is used for both compression and decompression.
* The download zip file contains more information and help in an included pdf document."
Regards.
Martin @rapidlasso
--
http://rapidlasso.com - fast tools to catch reality
Email: francesco.pirotti@unipd.it
Piotr Tompalski
Jan 8, 2014, 11:16:44 AM1/8/14
to last...@googlegroups.com, The LAS room - a friendly place to discuss specifications of the LAS format
Dear Martin and others,
I just run some short test on one file (ALS, XYZ, intensity, return number, source id; 24346429 points, 14 points/m2):
original las: 665 MB
converted to LAZ: 101 MB
converted to zlas: 105 MB
P.
Piotr Tompalski
Postdoctoral Research FellowHeidemann, Hans
Jan 8, 2014, 11:49:49 AM1/8/14
to last...@googlegroups.com
seems the real question is, will anybody other than esri be able to *effectively* read the .zlas format?
Karl
H. Karl Heidemann, GISP
Physical Scientist, Lidar Science
U.S. Geological Survey
Mundt Federal Building
47914 252nd Street
Sioux Falls, SD 57110
"Nothing matters very much, and very few things ... matter at all."
- Arthur James Balfour
Martin Isenburg
Jan 8, 2014, 12:11:07 PM1/8/14
to LAStools - efficient command line tools for LIDAR processing
seem the more real question is, why did Esri not have me to integrate LAX into LAZ to create a format with *.zlas functionality and *.laz compatiibility?
but it seems most of us think they know the answer ... (-;
Francesco Pirotti UNIPD
Jan 9, 2014, 3:02:40 AM1/9/14
to last...@googlegroups.com
One Format to rule them all, One Format to find them,
One Format to bring them all and in the ArcGIS bind them
:-)
Email: francesc...@unipd.it
Stuart Mowbray
Jan 9, 2014, 4:59:14 AM1/9/14
to last...@googlegroups.com
Results from a quick test with a fairly large LAS 1.2 file (30.8GB; 1,274,979,476 points; combined terrestrial scans; point data format 2: x, y ,z, intensity, RGB; Scale Factor 0.001 in x/y/z):
Original LAS file size: 30.8GB (33,149,466,689 bytes)
LAZ file size (default options): 7.78GB (8,356,090,204 bytes)
ZLAS file size (default options - “Scan file”/”Rearrange points”): 6.78GB (7,284,551,680 bytes)
Decompressing both files resulted in the original data from what I can see.
Thank goodness for solid-state drives! ;-)
So, a slight win in *this instance* from ZLAS, but for me, an open format will almost always be my preferred choice. The industry has too much to lose by letting one company dominate the market with a proprietary/closed format in my opinion. As mentioned, this was also a single run, so counts very little towards proving which compression technique is ‘best’, but I think that’s only half the issue here …
Dr Stuart Mowbray
Software Engineer
Group Engineering Software
☎+44 (0) 1904 736757 (internal ext: 36757)| ✉Stuart....@Renishaw.com
From: last...@googlegroups.com [mailto:last...@googlegroups.com]
On Behalf Of Francesco Pirotti UNIPD
Sent: 09 January 2014 08:03
To: last...@googlegroups.com
Subject: Re: [LAStools] ESRI's new compressed LAS format (*.zlas)
One Format to rule them all, One Format to find them,
:-)
--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe
Martin Isenburg
Jan 9, 2014, 6:48:31 AM1/9/14
to LAStools - efficient command line tools for LIDAR processing
Hello,
thanks Stuart. Please folks ... keep the test reports coming in ...
from the results so far it seems the ESRI compressor gives better
compression performance on terrestrial scans. which would make sense
since LASzip was optimized for airborne and mobile multi-return scans.
You can reassure yourself that the LAS files after decompression are
identical with
lasdiff -i original.las -i esri_decompressed.las
lasdiff -i original.las -i original.laz
but I am sure the ESRI developers made sure that they don't fail the
LAStools test. (-;
Also ... I was told that ESRI will be releasing an official statement
soon explaining why they have developed their own LiDAR compression
format. They really should do so. I have received and I continue to
receive a good number of personal (off-the-record) emails from people
all across the LiDAR industry who are expressing feelings of
"disappointment", "anger", "disgust", or even "hate" towards ESRI
about them apparently trying to sabotage our multi-year effort of
creating an open, free, and efficient compressed LiDAR exchange
format. It was explained to me that no one will so in public because
they do not want to be villanized by ESRI's public relations machinery
like ERDAS once did when they publically confronted ESRI way back
when. Must have been before my time ... (-:
thanks Stuart. Please folks ... keep the test reports coming in ...
from the results so far it seems the ESRI compressor gives better
compression performance on terrestrial scans. which would make sense
since LASzip was optimized for airborne and mobile multi-return scans.
You can reassure yourself that the LAS files after decompression are
identical with
lasdiff -i original.las -i esri_decompressed.las
lasdiff -i original.las -i original.laz
but I am sure the ESRI developers made sure that they don't fail the
LAStools test. (-;
Also ... I was told that ESRI will be releasing an official statement
soon explaining why they have developed their own LiDAR compression
format. They really should do so. I have received and I continue to
receive a good number of personal (off-the-record) emails from people
all across the LiDAR industry who are expressing feelings of
"disappointment", "anger", "disgust", or even "hate" towards ESRI
about them apparently trying to sabotage our multi-year effort of
creating an open, free, and efficient compressed LiDAR exchange
format. It was explained to me that no one will so in public because
they do not want to be villanized by ESRI's public relations machinery
like ERDAS once did when they publically confronted ESRI way back
when. Must have been before my time ... (-:
Stuart Mowbray
Jan 9, 2014, 9:05:14 AM1/9/14
to last...@googlegroups.com
Somebody asked privately regarding compression timing, so in the spirit of keeping everything public:
The running times on my desktop, with nothing else really taking up much processing power, memory, or disk IO at the time, were:
LASZip: 732.14 seconds to compress down to 7.78GB.
ESRI ZLAS (with “Rearrange Points”): 1410.09 seconds to compress down to 6.78GB
ESRI ZLAS (without “Rearrange Points”): 753.23 seconds to compress down to 7.70GB
Looking at the differences between LASZip and ZLAS w/out “Rearrange Points” (presumably this would be a fairer comparison to straightforward LAZ??), I’d say the gains in compression are minimal/bordering on negligible for this example. But again, this is one example.
I’m sure I didn’t need to specify that to millisecond resolution, but there you go :)
Regards,
Stuart
The running times on my desktop, with nothing else really taking up much processing power, memory, or disk IO at the time, were:
LASZip: 732.14 seconds to compress down to 7.78GB.
ESRI ZLAS (with “Rearrange Points”): 1410.09 seconds to compress down to 6.78GB
ESRI ZLAS (without “Rearrange Points”): 753.23 seconds to compress down to 7.70GB
Looking at the differences between LASZip and ZLAS w/out “Rearrange Points” (presumably this would be a fairer comparison to straightforward LAZ??), I’d say the gains in compression are minimal/bordering on negligible for this example. But again, this is one example.
I’m sure I didn’t need to specify that to millisecond resolution, but there you go :)
Regards,
Stuart
Martin Isenburg
Jan 10, 2014, 3:15:45 AM1/10/14
to LAStools - efficient command line tools for LIDAR processing, The LAS room - a friendly place to discuss specifications of the LAS format
Hello folks,
on the train to the airport I finally found some time to play with the
new LiDAR compressor of ESRI myself. First impression: solid! Nice
pice of work so my compliments to the many tireless hours the ESRI
developers must have worked on that. Compression is fast and
compression rates are state-of-the-art.
Note that the ESRI compressor uses mutli-threading (i.e. runs on
mutliple cores) whereas LASzip only uses one core. Up to know there
seemed little need to make LASzip any faster (given that it was
already 20 times faster than the LizardTech CompressorTM). And the
using of more than one core in LAStools has usually achieved by
processing more than one file with the '-cores 4' option. Adding
multi-threading to LASzip would be easily. So for the time being it is
more fair to compare decompression times that are single-core for both
LASzip and the ESRI compressor.
Also ... you should uncheck the "rearrange points" option for truly
"lossless" side-by-side comparison. Otherwise the ESRI compressor
reorders the points for better spatial access and/or improved
compression which is a great idea (and also implemented in
lassort.exe) but does not alow an apples to apples comparison. In the
following I refer to the reordered points as "lossy". This is a fair
term as the particular permutation of the n LiDAR points is "lost"
during reordering. As there are n! possible permutations that is a
huge loss in information from an information theoretical (i.e.
entropy) point of view.
I used the seven strips of airborne LAZ data that is used in the 3
LAStools tutorials for these experiments which is point type 0
(without GPS times). You can download them from here to run this
yourself:
http://rapidlasso.com/2013/04/20/tutorial-quality-checking/
http://rapidlasso.com/2013/10/13/tutorial-lidar-preparation/
http://rapidlasso.com/2013/10/20/tutorial-derivative-production/
Here are the resulting file sizes:
>> LAZ
9,968,439 LDR030828_211804_0.laz
12,292,371 LDR030828_212242_0.laz
8,834,097 LDR030828_212622_0.laz
11,770,079 LDR030828_213023_0.laz
10,967,043 LDR030828_213450_0.laz
3,518,051 LDR030828_213842_0.laz
9,500,706 LDR030828_214222_0.laz
==============
66,850,786 bytes
>> LOSSLESS ZLAS
10,309,052 LDR030828_211804_0.zlas
12,651,810 LDR030828_212242_0.zlas
9,176,510 LDR030828_212622_0.zlas
12,154,080 LDR030828_213023_0.zlas
11,231,231 LDR030828_213450_0.zlas
3,685,579 LDR030828_213842_0.zlas
9,870,689 LDR030828_214222_0.zlas
==============
69,081,161 bytes
>> LOSSY ZLAS
10,715,668 LDR030828_211804_0.zlas
13,153,141 LDR030828_212242_0.zlas
9,553,573 LDR030828_212622_0.zlas
12,588,417 LDR030828_213023_0.zlas
11,732,292 LDR030828_213450_0.zlas
3,808,598 LDR030828_213842_0.zlas
10,303,947 LDR030828_214222_0.zlas
==============
71,857,840 bytes
So in terms of compression rates it seems more or less a toss-up in
this experiment. Remeber that the *.zlas files also contain spatial
indexing information (like storing the *.lax file is inside the *.laz
file in LAStools speak) so the totals are fairly close.
I am happy to see that the lossless LAZ compression rates that I
cooked up back in 2007 as an individual are still the state-of-the-art
and have - seven years later - not been outdone by the development
team at ESRI ... (-:
Okay. Boarding to Singapore. Maybe more from Changi airport,
Martin @rapidlasso
PS: Below a typical "check" for lossless-ness with lasdiff.exe
--
>>>>>>>>> LOSSY (point order not preserved) <<<<<<<<<
lasdiff -i las\LDR030828_211804_0.las -i ezlasd_lossy\LDR030828_211804_0.las
checking 'las\LDR030828_211804_0.las' against
'ezlasd_lossy\LDR030828_211804_0.las'
headers are identical.
x: 272254812 273739350
y: 714699601 714419331
z: 140507 70964
intensity: 112 208
scan_angle_rank: 8 -4
point_source_ID: 120 118
point 1 of 2404613 is different
x: 272254845 273739353
y: 714699097 714419299
z: 141599 71006
intensity: 76 202
scan_angle_rank: 8 -4
point_source_ID: 120 117
point 2 of 2404613 is different
x: 272254850 273739355
y: 714698768 714419267
z: 141064 70999
intensity: 42 208
scan_angle_rank: 8 -4
point_source_ID: 120 118
point 3 of 2404613 is different
x: 272254830 273739357
y: 714698620 714419240
z: 139033 70990
intensity: 200 207
scan_angle_rank: 8 -4
point_source_ID: 119 118
point 4 of 2404613 is different
x: 272254890 273739360
y: 714697912 714419228
z: 141725 71023
intensity: 42 202
number_of_returns_of_given_pulse: 2 1
scan_angle_rank: 8 -4
point_source_ID: 120 118
point 5 of 2404613 is different
more than 5 points are different ... shutting up.
2404613 points are different.
both have 2404613 points. took 0.642 secs.
>>>>>>>>> LOSSLESS (point order preserving) <<<<<<<<<
lasdiff -i las\LDR030828_211804_0.las -i ezlasd_lossless\LDR030828_211804_0.las
checking 'las\LDR030828_211804_0.las' against
'ezlasd_lossless\LDR030828_211804_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2404613 points. took 0.633 secs.
lasdiff -i las\LDR030828_212242_0.las -i ezlasd_lossless\LDR030828_212242_0.las
checking 'las\LDR030828_212242_0.las' against
'ezlasd_lossless\LDR030828_212242_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2983331 points. took 0.786 secs.
lasdiff -i las\LDR030828_212622_0.las -i ezlasd_lossless\LDR030828_212622_0.las
checking 'las\LDR030828_212622_0.las' against
'ezlasd_lossless\LDR030828_212622_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2208166 points. took 0.375 secs.
lasdiff -i las\LDR030828_213023_0.las -i ezlasd_lossless\LDR030828_213023_0.las
checking 'las\LDR030828_213023_0.las' against
'ezlasd_lossless\LDR030828_213023_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2920460 points. took 0.78 secs.
lasdiff -i las\LDR030828_213450_0.las -i ezlasd_lossless\LDR030828_213450_0.las
checking 'las\LDR030828_213450_0.las' against
'ezlasd_lossless\LDR030828_213450_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2660474 points. took 0.623 secs.
lasdiff -i las\LDR030828_213842_0.las -i ezlasd_lossless\LDR030828_213842_0.las
checking 'las\LDR030828_213842_0.las' against
'ezlasd_lossless\LDR030828_213842_0.las'
headers are identical.
raw points are identical.
files are identical. both have 838865 points. took 0.178 secs.
lasdiff -i las\LDR030828_214222_0.las -i ezlasd_lossless\LDR030828_214222_0.las
checking 'las\LDR030828_214222_0.las' against
'ezlasd_lossless\LDR030828_214222_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2423315 points. took 0.598 secs.
on the train to the airport I finally found some time to play with the
new LiDAR compressor of ESRI myself. First impression: solid! Nice
pice of work so my compliments to the many tireless hours the ESRI
developers must have worked on that. Compression is fast and
compression rates are state-of-the-art.
Note that the ESRI compressor uses mutli-threading (i.e. runs on
mutliple cores) whereas LASzip only uses one core. Up to know there
seemed little need to make LASzip any faster (given that it was
already 20 times faster than the LizardTech CompressorTM). And the
using of more than one core in LAStools has usually achieved by
processing more than one file with the '-cores 4' option. Adding
multi-threading to LASzip would be easily. So for the time being it is
more fair to compare decompression times that are single-core for both
LASzip and the ESRI compressor.
Also ... you should uncheck the "rearrange points" option for truly
"lossless" side-by-side comparison. Otherwise the ESRI compressor
reorders the points for better spatial access and/or improved
compression which is a great idea (and also implemented in
lassort.exe) but does not alow an apples to apples comparison. In the
following I refer to the reordered points as "lossy". This is a fair
term as the particular permutation of the n LiDAR points is "lost"
during reordering. As there are n! possible permutations that is a
huge loss in information from an information theoretical (i.e.
entropy) point of view.
I used the seven strips of airborne LAZ data that is used in the 3
LAStools tutorials for these experiments which is point type 0
(without GPS times). You can download them from here to run this
yourself:
http://rapidlasso.com/2013/04/20/tutorial-quality-checking/
http://rapidlasso.com/2013/10/13/tutorial-lidar-preparation/
http://rapidlasso.com/2013/10/20/tutorial-derivative-production/
Here are the resulting file sizes:
>> LAZ
9,968,439 LDR030828_211804_0.laz
12,292,371 LDR030828_212242_0.laz
8,834,097 LDR030828_212622_0.laz
11,770,079 LDR030828_213023_0.laz
10,967,043 LDR030828_213450_0.laz
3,518,051 LDR030828_213842_0.laz
9,500,706 LDR030828_214222_0.laz
==============
66,850,786 bytes
>> LOSSLESS ZLAS
10,309,052 LDR030828_211804_0.zlas
12,651,810 LDR030828_212242_0.zlas
9,176,510 LDR030828_212622_0.zlas
12,154,080 LDR030828_213023_0.zlas
11,231,231 LDR030828_213450_0.zlas
3,685,579 LDR030828_213842_0.zlas
9,870,689 LDR030828_214222_0.zlas
==============
69,081,161 bytes
>> LOSSY ZLAS
10,715,668 LDR030828_211804_0.zlas
13,153,141 LDR030828_212242_0.zlas
9,553,573 LDR030828_212622_0.zlas
12,588,417 LDR030828_213023_0.zlas
11,732,292 LDR030828_213450_0.zlas
3,808,598 LDR030828_213842_0.zlas
10,303,947 LDR030828_214222_0.zlas
==============
71,857,840 bytes
So in terms of compression rates it seems more or less a toss-up in
this experiment. Remeber that the *.zlas files also contain spatial
indexing information (like storing the *.lax file is inside the *.laz
file in LAStools speak) so the totals are fairly close.
I am happy to see that the lossless LAZ compression rates that I
cooked up back in 2007 as an individual are still the state-of-the-art
and have - seven years later - not been outdone by the development
team at ESRI ... (-:
Okay. Boarding to Singapore. Maybe more from Changi airport,
Martin @rapidlasso
PS: Below a typical "check" for lossless-ness with lasdiff.exe
--
>>>>>>>>> LOSSY (point order not preserved) <<<<<<<<<
lasdiff -i las\LDR030828_211804_0.las -i ezlasd_lossy\LDR030828_211804_0.las
checking 'las\LDR030828_211804_0.las' against
'ezlasd_lossy\LDR030828_211804_0.las'
headers are identical.
x: 272254812 273739350
y: 714699601 714419331
z: 140507 70964
intensity: 112 208
scan_angle_rank: 8 -4
point_source_ID: 120 118
point 1 of 2404613 is different
x: 272254845 273739353
y: 714699097 714419299
z: 141599 71006
intensity: 76 202
scan_angle_rank: 8 -4
point_source_ID: 120 117
point 2 of 2404613 is different
x: 272254850 273739355
y: 714698768 714419267
z: 141064 70999
intensity: 42 208
scan_angle_rank: 8 -4
point_source_ID: 120 118
point 3 of 2404613 is different
x: 272254830 273739357
y: 714698620 714419240
z: 139033 70990
intensity: 200 207
scan_angle_rank: 8 -4
point_source_ID: 119 118
point 4 of 2404613 is different
x: 272254890 273739360
y: 714697912 714419228
z: 141725 71023
intensity: 42 202
number_of_returns_of_given_pulse: 2 1
scan_angle_rank: 8 -4
point_source_ID: 120 118
point 5 of 2404613 is different
more than 5 points are different ... shutting up.
2404613 points are different.
both have 2404613 points. took 0.642 secs.
>>>>>>>>> LOSSLESS (point order preserving) <<<<<<<<<
lasdiff -i las\LDR030828_211804_0.las -i ezlasd_lossless\LDR030828_211804_0.las
checking 'las\LDR030828_211804_0.las' against
'ezlasd_lossless\LDR030828_211804_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2404613 points. took 0.633 secs.
lasdiff -i las\LDR030828_212242_0.las -i ezlasd_lossless\LDR030828_212242_0.las
checking 'las\LDR030828_212242_0.las' against
'ezlasd_lossless\LDR030828_212242_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2983331 points. took 0.786 secs.
lasdiff -i las\LDR030828_212622_0.las -i ezlasd_lossless\LDR030828_212622_0.las
checking 'las\LDR030828_212622_0.las' against
'ezlasd_lossless\LDR030828_212622_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2208166 points. took 0.375 secs.
lasdiff -i las\LDR030828_213023_0.las -i ezlasd_lossless\LDR030828_213023_0.las
checking 'las\LDR030828_213023_0.las' against
'ezlasd_lossless\LDR030828_213023_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2920460 points. took 0.78 secs.
lasdiff -i las\LDR030828_213450_0.las -i ezlasd_lossless\LDR030828_213450_0.las
checking 'las\LDR030828_213450_0.las' against
'ezlasd_lossless\LDR030828_213450_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2660474 points. took 0.623 secs.
lasdiff -i las\LDR030828_213842_0.las -i ezlasd_lossless\LDR030828_213842_0.las
checking 'las\LDR030828_213842_0.las' against
'ezlasd_lossless\LDR030828_213842_0.las'
headers are identical.
raw points are identical.
files are identical. both have 838865 points. took 0.178 secs.
lasdiff -i las\LDR030828_214222_0.las -i ezlasd_lossless\LDR030828_214222_0.las
checking 'las\LDR030828_214222_0.las' against
'ezlasd_lossless\LDR030828_214222_0.las'
headers are identical.
raw points are identical.
files are identical. both have 2423315 points. took 0.598 secs.
Martin Isenburg
Jan 10, 2014, 5:53:29 PM1/10/14
to LAStools - efficient command line tools for LIDAR processing, The LAS room - a friendly place to discuss specifications of the LAS format
Hello from Singapore,
next round of testing was with more recent LiDAR data from a laser
penetration test flight with Asian Aerospace Services (AAS) and their
Airborne Sensors DA-42 plane with a RIEGL Q680i (and myself in the
plane) in the Khao Yai National Forest. Here some pictures from that
flight:
http://www.facebook.com/media/set/?set=a.576375162424932
The data was aligned, georeferenced, and exported to LAZ by RiPROCESS
and ground-classified by lasground.exe. We flew above dense tropical
rain forest so there are a large number of returns per laser pulse
(see lasinfo report at the end). Not sure if that had anything to do
with it but this data seems to (soft) crash the ESRI compressor with
an ERROR message when running with the rearrange points (lossy)
option, so I could not test this.
>> LAZ
101,259,295 Lost_Aircraft_UTM47_131104_055518_1g.laz
54,897,535 Lost_Aircraft_UTM47_131104_060155_1g.laz
66,896,049 Lost_Aircraft_UTM47_131104_060856_1g.laz
62,030,884 Lost_Aircraft_UTM47_131104_061334_1g.laz
63,069,630 Lost_Aircraft_UTM47_131104_062052_1g.laz
62,031,639 Lost_Aircraft_UTM47_131104_062547_1g.laz
==============
410,185,032 bytes
>> lossless ZLAS
105,848,202 Lost_Aircraft_UTM47_131104_055518_1g.zlas
56,497,395 Lost_Aircraft_UTM47_131104_060155_1g.zlas
68,838,869 Lost_Aircraft_UTM47_131104_060856_1g.zlas
63,620,718 Lost_Aircraft_UTM47_131104_061334_1g.zlas
64,760,326 Lost_Aircraft_UTM47_131104_062052_1g.zlas
63,526,193 Lost_Aircraft_UTM47_131104_062547_1g.zlas
==============
423,091,703 bytes
The picture is the same as before. The ESRI compressor (run in
lossless mode) has a very familiar feel in terms of performance as
compression rates and speeds are quite similar to LASzip. Would love
to peek at ESRI's code. Where is Julian LASsange when you need him? We
need some source code on http://ESRIeaks.laz ... (-;
Martin @rapidlasso
PS: next up ... speed measurements
--
D:\lastools\bin>lasinfo "ezlas\laz\Lost_Aircraft_UTM47 - Scanner 1 -
131104_055518_1 - originalpoints_g.laz"
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 1
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: 'LAStools (c) by Martin Isenburg'
generating software: 'lasground (131105) commercial'
file creation day/year: 310/2013
header size: 227
offset to point data: 407
number var. length records: 2
point data format: 1
point data record length: 28
number of point records: 18773818
number of points by return: 9689872 5512185 2420527 847869 237907
scale factor x y z: 0.001 0.001 0.001
offset x y z: 732989 1611682 -8292
min x y z: 729795.663 1602457.232 103.728
max x y z: 736599.043 1610202.656 625.539
variable length header record 1 of 2:
reserved 43707
user ID 'LASF_Projection'
record ID 34735
length after header 48
description 'GeoKeyDirectoryTag (mandatory)'
GeoKeyDirectoryTag version 1.1.0 number of keys 5
key 1024 tiff_tag_location 0 count 1 value_offset 1 -
GTModelTypeGeoKey: ModelTypeProjected
key 1026 tiff_tag_location 34737 count 24 value_offset 0 -
GTCitationGeoKey: WGS84, WGS84, UTM_North
key 2052 tiff_tag_location 0 count 1 value_offset 9001 -
GeogLinearUnitsGeoKey: Linear_Meter
key 2054 tiff_tag_location 0 count 1 value_offset 9102 -
GeogAngularUnitsGeoKey: Angular_Degree
key 3076 tiff_tag_location 0 count 1 value_offset 9001 -
ProjLinearUnitsGeoKey: Linear_Meter
variable length header record 2 of 2:
reserved 43707
user ID 'LASF_Projection'
record ID 34737
length after header 24
description 'GeoASCIIParamsTag (optional)'
GeoAsciiParamsTag (number of characters 24)
WGS84, WGS84, UTM_North
LASzip compression (version 2.2r0 c2 50000): POINT10 2 GPSTIME11 2
reporting minimum and maximum for all LAS point record entries ...
X -3193337 3610043
Y -9224768 124622
Z 155 8917539
intensity 49151 49151
edge_of_flight_line 0 1
scan_direction_flag 1 1
number_of_returns_of_given_pulse 1 7
return_number 1 7
classification 1 2
scan_angle_rank -37 36
user_data 0 0
point_source_ID 0 0
gps_time 67579721.241818 67579903.533960
WARNING: 61 points outside of header bounding box
WARNING: there are 53655 points with return number 6
WARNING: there are 11803 points with return number 7
overview over number of returns of given pulse: 4177687 6183316
4717974 2439848 921260 261906 71827
histogram of classification of points:
16904309 Unclassified (1)
1869509 Ground (2)
real max y larger than header max y by 1603.966000
real min z smaller than header min z by 8395.573000
next round of testing was with more recent LiDAR data from a laser
penetration test flight with Asian Aerospace Services (AAS) and their
Airborne Sensors DA-42 plane with a RIEGL Q680i (and myself in the
plane) in the Khao Yai National Forest. Here some pictures from that
flight:
http://www.facebook.com/media/set/?set=a.576375162424932
The data was aligned, georeferenced, and exported to LAZ by RiPROCESS
and ground-classified by lasground.exe. We flew above dense tropical
rain forest so there are a large number of returns per laser pulse
(see lasinfo report at the end). Not sure if that had anything to do
with it but this data seems to (soft) crash the ESRI compressor with
an ERROR message when running with the rearrange points (lossy)
option, so I could not test this.
>> LAZ
101,259,295 Lost_Aircraft_UTM47_131104_055518_1g.laz
54,897,535 Lost_Aircraft_UTM47_131104_060155_1g.laz
66,896,049 Lost_Aircraft_UTM47_131104_060856_1g.laz
62,030,884 Lost_Aircraft_UTM47_131104_061334_1g.laz
63,069,630 Lost_Aircraft_UTM47_131104_062052_1g.laz
62,031,639 Lost_Aircraft_UTM47_131104_062547_1g.laz
==============
410,185,032 bytes
>> lossless ZLAS
105,848,202 Lost_Aircraft_UTM47_131104_055518_1g.zlas
56,497,395 Lost_Aircraft_UTM47_131104_060155_1g.zlas
68,838,869 Lost_Aircraft_UTM47_131104_060856_1g.zlas
63,620,718 Lost_Aircraft_UTM47_131104_061334_1g.zlas
64,760,326 Lost_Aircraft_UTM47_131104_062052_1g.zlas
63,526,193 Lost_Aircraft_UTM47_131104_062547_1g.zlas
==============
423,091,703 bytes
The picture is the same as before. The ESRI compressor (run in
lossless mode) has a very familiar feel in terms of performance as
compression rates and speeds are quite similar to LASzip. Would love
to peek at ESRI's code. Where is Julian LASsange when you need him? We
need some source code on http://ESRIeaks.laz ... (-;
Martin @rapidlasso
PS: next up ... speed measurements
--
D:\lastools\bin>lasinfo "ezlas\laz\Lost_Aircraft_UTM47 - Scanner 1 -
131104_055518_1 - originalpoints_g.laz"
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 1
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: 'LAStools (c) by Martin Isenburg'
generating software: 'lasground (131105) commercial'
file creation day/year: 310/2013
header size: 227
offset to point data: 407
number var. length records: 2
point data format: 1
point data record length: 28
number of point records: 18773818
number of points by return: 9689872 5512185 2420527 847869 237907
scale factor x y z: 0.001 0.001 0.001
offset x y z: 732989 1611682 -8292
min x y z: 729795.663 1602457.232 103.728
max x y z: 736599.043 1610202.656 625.539
variable length header record 1 of 2:
reserved 43707
user ID 'LASF_Projection'
record ID 34735
length after header 48
description 'GeoKeyDirectoryTag (mandatory)'
GeoKeyDirectoryTag version 1.1.0 number of keys 5
key 1024 tiff_tag_location 0 count 1 value_offset 1 -
GTModelTypeGeoKey: ModelTypeProjected
key 1026 tiff_tag_location 34737 count 24 value_offset 0 -
GTCitationGeoKey: WGS84, WGS84, UTM_North
key 2052 tiff_tag_location 0 count 1 value_offset 9001 -
GeogLinearUnitsGeoKey: Linear_Meter
key 2054 tiff_tag_location 0 count 1 value_offset 9102 -
GeogAngularUnitsGeoKey: Angular_Degree
key 3076 tiff_tag_location 0 count 1 value_offset 9001 -
ProjLinearUnitsGeoKey: Linear_Meter
variable length header record 2 of 2:
reserved 43707
user ID 'LASF_Projection'
record ID 34737
length after header 24
description 'GeoASCIIParamsTag (optional)'
GeoAsciiParamsTag (number of characters 24)
WGS84, WGS84, UTM_North
LASzip compression (version 2.2r0 c2 50000): POINT10 2 GPSTIME11 2
reporting minimum and maximum for all LAS point record entries ...
X -3193337 3610043
Y -9224768 124622
Z 155 8917539
intensity 49151 49151
edge_of_flight_line 0 1
scan_direction_flag 1 1
number_of_returns_of_given_pulse 1 7
return_number 1 7
classification 1 2
scan_angle_rank -37 36
user_data 0 0
point_source_ID 0 0
gps_time 67579721.241818 67579903.533960
WARNING: 61 points outside of header bounding box
WARNING: there are 53655 points with return number 6
WARNING: there are 11803 points with return number 7
overview over number of returns of given pulse: 4177687 6183316
4717974 2439848 921260 261906 71827
histogram of classification of points:
16904309 Unclassified (1)
1869509 Ground (2)
real max y larger than header max y by 1603.966000
real min z smaller than header min z by 8395.573000
Martin Isenburg
Jan 13, 2014, 5:04:31 AM1/13/14
to last...@googlegroups.com, The LAS room - a friendly place to discuss specifications of the LAS format
Hello from the Philippines,
ESRI's decompressor (32 bit version) is 16 % to 20 % slower than LASzip. As mentioned before, the ESRI compressor uses multi-threading which has not yet been implemented for LASzip (would be easy enough to add so please go ahead an do it (-:) so a fair comparison is not possible at the moment. Anyone with access to a single-core machine? It would be nice to get some apples-to-apples timings for the compressor as well.
I compared the decompression speed on zLAS files *without* rearranged points (aka lossless) to that of LASzip so that both are decompressing the exact same file. Note that the ESRI decompressor also extracts *.lasx files but they are so tiny that they should hardly affect the timing. The exact logs are at the end of the file. I ran it twice across the same 13 files. The second set of timings (in brackets) is faster, likely because the compressed files are already in the file cache. LASzip beats ESRI by 25 (20) seconds needing only 149 (86) seconds instead of 174 (105).
While this seems a lot different it is not. Different implementations of LASzip also have 10 - 30 % speed variations. ESRI's solution delivers nearly the same compression rates and delivers at about the same decompression speeds as LASzip. Okay, dear ESRI, so what feature does your compressor have that cannot be created through a backward and forward compatible LASzip-based solution using a combination of LAZ and LAX (and some optional spatial sorting)? Would it not make more sense to integrate the community-chosen LAZ compressor into you community-used product?
Regards,
Martin @rapidlasso
D:\lastools\bin>ezlas_32 ezlas_lossless temp1 /D
Gathering folder information...
Processing files...
Files processed: 13 / 13
Process completed
Elapsed time: 174.36 seconds
D:\lastools\bin>laszip -v -i laz/*.laz -odir temp2/
2.89 secs to write 48097847 bytes for 'temp2/\LDR030828_211804_0.las' with 2404613 points of type 0
4.032 secs to write 59672207 bytes for 'temp2/\LDR030828_212242_0.las' with 2983331 points of type 0
3.156 secs to write 44168907 bytes for 'temp2/\LDR030828_212622_0.las' with 2208166 points of type 0
3.328 secs to write 58414787 bytes for 'temp2/\LDR030828_213023_0.las' with 2920460 points of type 0
3.094 secs to write 53215067 bytes for 'temp2/\LDR030828_213450_0.las' with 2660474 points of type 0
1.407 secs to write 16782887 bytes for 'temp2/\LDR030828_213842_0.las' with 838865 points of type 0
3.125 secs to write 48471887 bytes for 'temp2/\LDR030828_214222_0.las' with 2423315 points of type 0
31.408 secs to write 525667311 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_055518_1 - originalpoints_g.las' with 18773818 points of type 1
17.001 secs to write 287601735 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_060155_1 - originalpoints_g.las' with 10271476 points of type 1
20.673 secs to write 358081515 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_060856_1 - originalpoints_g.las' with 12788611 points of type 1
20.048 secs to write 324903419 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_061334_1 - originalpoints_g.las' with 11603679 points of type 1
18.907 secs to write 337341243 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_062052_1 - originalpoints_g.las' with 12047887 points of type 1
19.486 secs to write 322263103 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_062547_1 - originalpoints_g.las' with 11509382 points of type 1
needed 148.56 sec for 13 files
D:\lastools\bin>ezlas_32 ezlas_lossless temp1 /D
Gathering folder information...
Processing files...
Files processed: 13 / 13
Process completed
Elapsed time: 105.12 seconds
D:\lastools\bin>laszip -v -i laz/*.laz -odir temp2/
2.734 secs to write 48097847 bytes for 'temp2/\LDR030828_211804_0.las' with 2404613 points of type 0
3.094 secs to write 59672207 bytes for 'temp2/\LDR030828_212242_0.las' with 2983331 points of type 0
1.938 secs to write 44168907 bytes for 'temp2/\LDR030828_212622_0.las' with 2208166 points of type 0
3.14 secs to write 58414787 bytes for 'temp2/\LDR030828_213023_0.las' with 2920460 points of type 0
2.938 secs to write 53215067 bytes for 'temp2/\LDR030828_213450_0.las' with 2660474 points of type 0
1.078 secs to write 16782887 bytes for 'temp2/\LDR030828_213842_0.las' with 838865 points of type 0
2.797 secs to write 48471887 bytes for 'temp2/\LDR030828_214222_0.las' with 2423315 points of type 0
18.314 secs to write 525667311 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_055518_1 - originalpoints_g.las' with 18773818 points of type 1
9.281 secs to write 287601735 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_060155_1 - originalpoints_g.las' with 10271476 points of type 1
10.157 secs to write 358081515 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_060856_1 - originalpoints_g.las' with 12788611 points of type 1
8.204 secs to write 324903419 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_061334_1 - originalpoints_g.las' with 11603679 points of type 1
10.657 secs to write 337341243 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_062052_1 - originalpoints_g.las' with 12047887 points of type 1
11.438 secs to write 322263103 bytes for 'temp2/\Lost_Aircraft_UTM47 - Scanner 1 - 131104_062547_1 - originalpoints_g.las' with 11509382 points of type 1
needed 85.77 sec for 13 files
Martin Isenburg
Jan 14, 2014, 5:01:04 AM1/14/14
to LAStools - efficient command line tools for LIDAR processing, The LAS room - a friendly place to discuss specifications of the LAS format
Hello,
below a test report from Al Karlin (Southwest Florida Water Management
District) that was posted on LiDAR news. I suspect he may have left
the "rearrange points" button checked as I do not expect the ESRI
compressor to be that much slower that LASzip. I think we have
established that the ESRI compressor is essentially identical to
LASzip in terms of performance, which is why this "LAS clown" here
will keep on referring to the ESRI compressor as a "LAZ clone" ... (-;
http://blog.lidarnews.com/early-positive-reviews-for-esri-las-optimizer
"While this is NOT a rigorous test of the software, I tested one LiDAR
tile; 9,143,306 points; 250.016 KB in size; used EzLAS (x32 and x64),
LASzip, and MG4 (MrSID). My results suggest that the Esri software is
really quite competitive. As it is also very intuitive (though not
quite as simple as LASzip), I think that many will adopt it. My
preliminary results also suggest that it is A LOT faster than the MG4
compression/decompression. (My test computer was a 3.2 GHz Xeon
processor with 6 GB RAM and WIN7-64.)
Compression: (fastest to slowest)
LASzip – 3.2 sec: 42,985KB
EzLAS32 – 6.2 sec: 44,881 KB
EzLAS64 – 6.7 sec: 44,881 KB
MG4 – 1min,2 sec: 86908 KB
Decompression: (fastest to slowest)
LASzip – 6.5 sec: 250,015 KB
EzLAS32 – 7.6 sec: 250,016 KB
EzLAS64 – 7.6 sec: 250,016 KB
MG4 – 16.2 sec: 250,014 KB"
Regards,
LAS clown @rapidlasso
On Mon, Jan 13, 2014 at 11:04 AM, Martin Isenburg
below a test report from Al Karlin (Southwest Florida Water Management
District) that was posted on LiDAR news. I suspect he may have left
the "rearrange points" button checked as I do not expect the ESRI
compressor to be that much slower that LASzip. I think we have
established that the ESRI compressor is essentially identical to
LASzip in terms of performance, which is why this "LAS clown" here
will keep on referring to the ESRI compressor as a "LAZ clone" ... (-;
http://blog.lidarnews.com/early-positive-reviews-for-esri-las-optimizer
"While this is NOT a rigorous test of the software, I tested one LiDAR
tile; 9,143,306 points; 250.016 KB in size; used EzLAS (x32 and x64),
LASzip, and MG4 (MrSID). My results suggest that the Esri software is
really quite competitive. As it is also very intuitive (though not
quite as simple as LASzip), I think that many will adopt it. My
preliminary results also suggest that it is A LOT faster than the MG4
compression/decompression. (My test computer was a 3.2 GHz Xeon
processor with 6 GB RAM and WIN7-64.)
Compression: (fastest to slowest)
LASzip – 3.2 sec: 42,985KB
EzLAS32 – 6.2 sec: 44,881 KB
EzLAS64 – 6.7 sec: 44,881 KB
MG4 – 1min,2 sec: 86908 KB
Decompression: (fastest to slowest)
LASzip – 6.5 sec: 250,015 KB
EzLAS32 – 7.6 sec: 250,016 KB
EzLAS64 – 7.6 sec: 250,016 KB
MG4 – 16.2 sec: 250,014 KB"
Regards,
LAS clown @rapidlasso
On Mon, Jan 13, 2014 at 11:04 AM, Martin Isenburg
Martin Isenburg
Jan 14, 2014, 8:26:15 PM1/14/14
to last...@googlegroups.com, The LAS room - a friendly place to discuss specifications of the LAS format
Someone else wants to chime in on this discussion?
Seems "W2" does not approve of our efforts.
Jakub Krawczyk
Jan 14, 2014, 4:58:44 AM1/14/14
to last...@googlegroups.com
Hello you could try running them on virtualbox (or other) virtual machine and set the numbers of virtual cores to 1
--
kuba
--
Martin Isenburg
Feb 7, 2014, 3:16:55 AM2/7/14
to last...@googlegroups.com
Quick update: The front-lines harden as an unlikely coalition of open source knights, laser guardians, imperial agencies, and competing thugs forms a rebel movement against the approaching Desri Star who threatens the free world announcing that the dArcG is going to spawn parallelized LAZ clones. Encrypted instructions to Jedis spread like "Point Clouds on the Horizon" "Towards an Open Future" as the insurgency movement prepares the future of compression free of proprietary oppression. The clone wars might be starting soon. May the force be with LAZ ... (-;
For relevant cross-links check the updates at the end of http://rapidlasso.com/2013/12/30/new-compressed-las-format-by-esri/
Your LAS clown ... (o:
Martin @rapidlasso
Reply all
Reply to author
Forward
0 new messages