Yay! LiDAR for all of Netherlands now open (AHN1 and AHN2)

2,189 views
Skip to first unread message

Martin Isenburg

unread,
Mar 14, 2014, 8:11:33 AM3/14/14
to LAStools - efficient command line tools for LIDAR processing
Hello,

i heard tweets such as


and saw articles such as





but all as "zipped" compressed *.laz files of this form:

http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01dz1.laz.zip

These are ZIP files containing just a single LAZ file. So they are double-compressed. Having *.laz.zip files seems silly to me and prevents a lot of potential (aka streaming computations while downloading) and also requires everyone to create two copies of the downloaded data. Anyone know why it was done like this?

Also ... the first file I looked at (ahn2_uitgefilterd/u01cz1.laz.zip) seems to be an XYZ conversion from ASCII to LAZ (-> without time stamps, scan angles, return counts, point source IDs...). Is all the data like this? The lack of flightline information certainly hurts the usability quite a bit ...

Also ... they surely mean "raw" LiDAR when they say "raw" LiDAR. See attached pic for a large amount of LiDAR noise points just below the aircraft. You can tell roughly how low the aircraft was flying. Maybe we can reconstruct the flightlines from that ... (-:

Kudos to the Dutch for following the Finnish and Danish example of opening their national LiDAR!!! Oh ... and you Germany, France, Sweden, Italy, Norway, Austria, Switzerland, Spain, United Kingdom etc ... don't you see how increasingly uncool you are becoming ... (-; 

Regards,

Martin @rapidlasso

D:\lastools\bin>lasinfo u01cz1.laz -cd
reporting all LAS header entries:
  file signature:             'LASF'
  file source ID:             0
  global_encoding:            0
  project ID GUID data 1-4:   00000000-0000-0000-0000-000000000000
  version major.minor:        1.2
  system identifier:          'LAStools (c) by Martin Isenburg'
  generating software:        'lasmerge (version 130623)'
  file creation day/year:     236/2010
  header size:                227
  offset to point data:       227
  number var. length records: 0
  point data format:          0
  point data record length:   20
  number of point records:    12878037
  number of points by return: 12878037 0 0 0 0
  scale factor x y z:         0.01 0.01 0.01
  offset x y z:               100000 600000 0
  min x y z:                  141403.75 600000.00 -1.37
  max x y z:                  145000.00 601956.54 386.46
LASzip compression (version 2.1r0 c2 50000): POINT10 2
reporting minimum and maximum for all LAS point record entries ...
  X             4140375    4500000
  Y                   0     195654
  Z                -137      38646
  intensity           0          0
  return_number       1          1
  number_of_returns   1          1
  edge_of_flight_line 0          0
  scan_direction_flag 0          0
  classification      0          0
  scan_angle_rank     0          0
  user_data           0          0
  point_source_ID     0          0
number of last returns: 12878037
covered area in square units/kilounits: 3599944/3.60
point density: all returns 3.58 last only 3.58 (per square units)
      spacing: all returns 0.53 last only 0.53 (in units)
overview over number of returns of given pulse: 12878037 0 0 0 0 0 0
histogram of classification of points:
        12878037  never classified (0)

ahn2_raw_means_raw.png

Terje Mathisen

unread,
Mar 14, 2014, 8:26:40 AM3/14/14
to last...@googlegroups.com
Martin Isenburg wrote:
> and after some searching I found *all* of the LiDAR data directly linked
> here:
>
> http://geodata.nationaalgeoregister.nl/ahn1/atom/ahn1_uitgefilterd.xml
> http://geodata.nationaalgeoregister.nl/ahn1/atom/ahn1_gefilterd.xml

That's really good news. I have had some mail from .nl orienteers who
knew about this decision and wanted to use my code, together with
lastools, to generate orienteering base maps.
>
> http://geodata.nationaalgeoregister.nl/ahn2/atom/ahn2_uitgefilterd.xml
> http://geodata.nationaalgeoregister.nl/ahn2/atom/ahn2_gefilterd.xml
>
> but all as "zipped" compressed *.laz files of this form:
>
> http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01cz1.laz.zip
> http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01cz2.laz.zip
> http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01dz1.laz.zip
>
> These are ZIP files containing just a single LAZ file. So they are
> double-compressed. Having *.laz.zip files seems silly to me and prevents
> a lot of potential (aka streaming computations while downloading) and
> also requires everyone to create two copies of the downloaded data.
> Anyone know why it was done like this?

Using .zip includes a 32-bit crc check of every file, so there is some
error detection which I don't know if you get in .laz?

Terje

--
- <Terje.M...@tmsw.no>
"almost all programming can be viewed as an exercise in caching"

Howard Butler

unread,
Mar 14, 2014, 9:47:24 AM3/14/14
to last...@googlegroups.com
You can .zip a file without compressing it, essentially using .zip as a wrapper. I also produce .laz.zip data because it allows the data to make it through virus scans and the like, and I can also include laszip.exe in the package for people who do not have ready access to it. Of course because the LAZ is so much smaller than the equivalent LAS, adding in the executable doesn't even register as a significant cost.

A self-extracting LAZ would be a nice, if niche, feature.
signature.asc

Martin Isenburg

unread,
Mar 15, 2014, 2:46:03 AM3/15/14
to LAStools - efficient command line tools for LIDAR processing, The LAS room - a friendly place to discuss specifications of the LAS format
Hello,

> Martin Isenburg wrote:
>> and after some searching I found *all* of the LiDAR data directly linked
>> here:
>>
>> http://geodata.nationaalgeoregister.nl/ahn1/atom/ahn1_uitgefilterd.xml
>> http://geodata.nationaalgeoregister.nl/ahn1/atom/ahn1_gefilterd.xml
>
> That's really good news. I have had some mail from .nl orienteers who knew about this decision and wanted to use my code, together with lastools, to generate orienteering base maps.
>>
>> http://geodata.nationaalgeoregister.nl/ahn2/atom/ahn2_uitgefilterd.xml
>> http://geodata.nationaalgeoregister.nl/ahn2/atom/ahn2_gefilterd.xml
>>
>> but all as "zipped" compressed *.laz files of this form:
>>
>> http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01cz1.laz.zip
>> http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01cz2.laz.zip
>> http://geodata.nationaalgeoregister.nl/ahn2/extract/ahn2_uitgefilterd/u01dz1.laz.zip
>>
>> These are ZIP files containing just a single LAZ file. So they are
>> double-compressed. Having *.laz.zip files seems silly to me and prevents
>> a lot of potential (aka streaming computations while downloading) and
>> also requires everyone to create two copies of the downloaded data.
>> Anyone know why it was done like this?
>
> Using .zip includes a 32-bit crc check of every file, so there is some error detection which I don't know if you get in .laz?

Terje, that was one idea I was entertaining for LASzip's LAS 1.4 extension. This will be on the front burner soon.

About that: I had been trying to convince ESRI's management in a discussion that dragged on sine the MLK weekend to produce a joint LAZ 1.4 compressor that has whatever particular little extra features their LAZ clone has that is apparently so important to them (instead of the "lock-in" and "format control" that the LiDAR community has been suspecting) while at the same time being open source LAZ. I proposed to them what seemed a neat idea that is essentially "enabled" by the natural break in the LAS format due to the new point types in LAS 1.4: turn all future ESRI compressed LAS 1.0 to 1.3 content - under the hood - into upgraded LAS 1.4 files (using a jointly-developed LASzip extension to the LAS 1.4 point types incorporating many of their and our ideas) so that all of their content becomes automatically readable to everyone who switches their LASzip DLL's for the latest one that can also read compressed LAS 1.4. But it seems that discussion is not really going anywhere. Expect some final rabble-rousing open letter and/or columns articles from me on that topic before I lay it to rest for good. I certainly want to go on permanent record as having tried the hardest I could to prevent this unnecessary format fragementation from happening ...
 
You can .zip a file without compressing it, essentially using .zip as a wrapper. I also produce .laz.zip data because it allows the data to make it through virus scans and the like, and I can also include laszip.exe in the package for people who do not have ready access to it. Of course because the LAZ is so much smaller than the equivalent LAS, adding in the executable doesn't even register as a significant cost.

Howeard, that makes a lot of sense for single files - especially to include laszip or a small hillshaded thumbnail image. But for large archives with 500 billion points? If it was LAZ I could "peek" into the LiDAR file with 'curl' or 'wget' to see the header without downloading the file, check for VLR information, verify extends, or produce derivatives while downloading with '-stdin' piping. Is there a way to "unwrap" the zip in a piping fashion so I can inject it between  'curl' or 'wget' and the  '-stdin' piping of LAStools?

Regards,

Martin @rapidlasso

--
http://rapidlasso.com - fossfully guides your LiDAR sabers

Terje Mathisen

unread,
Mar 15, 2014, 3:34:51 AM3/15/14
to last...@googlegroups.com
Martin Isenburg wrote:
> Howeard, that makes a lot of sense for single files - especially to
> include laszip or a small hillshaded thumbnail image. But for large
> archives with 500 billion points? If it was LAZ I could "peek" into the
> LiDAR file with 'curl' or 'wget' to see the header without downloading
> the file, check for VLR information, verify extends, or produce
> derivatives while downloading with '-stdin' piping. Is there a way to
> "unwrap" the zip in a piping fashion so I can inject it between 'curl'
> or 'wget' and the '-stdin' piping of LAStools?

Yes, I'm pretty sure the unix-style zip tools allows piping data in/out,
but afair the zip format has all the catalog info at the very end, so
you would be limited to using this for actual zip files.

It is possible that another compression format, with interleaved file
header/data blocks would work better int his fashion, particularly if
we're only going to be using the CRC (or maybe a full MD5/SHA signature)
as an integrity check.

(Such a check is of course not valid until after you've read the entire
file! )

Re online access to huge files:

Isn't this area already covered by the company you linked to a few weeks
ago, i.e. subsecond 3D view into TB sized files?

This format must obviously contain quite a lot of preprocessing info,
not just .LAX style indexing!

Adam P

unread,
Aug 14, 2014, 11:27:45 PM8/14/14
to last...@googlegroups.com
After pondering this for some time, perhaps a fancy algorithm could recreate flight-lines by guessing which point pairs can confidently be treated as multiple returns for the same pulse, recreating the rays that resulted in these multiple returns, then using nearby rays to triangulate the corresponding aircraft position, with a healthy dose of noise reduction, auto-magic, and voodoo.

I wonder if anyone has explored this path yet?

Terje Mathisen

unread,
Aug 15, 2014, 3:35:00 AM8/15/14
to last...@googlegroups.com
Adam P wrote:
> After pondering this for some time, perhaps a fancy algorithm could
> recreate flight-lines by guessing which point pairs can confidently be
> treated as multiple returns for the same pulse, recreating the rays
> that resulted in these multiple returns, then using nearby rays to
> triangulate the corresponding aircraft position, with a healthy dose
> of noise reduction, auto-magic, and voodoo.
>
> I wonder if anyone has explored this path yet?

I've considered it, also in previous posts:

Given (x,y,z and angle triplets it is obviously possible to get a pretty
good estimate of the platform path. Even if each individual measurement
is very noisy (due to single-degree angle resolution), the fact that
we're getting more than a thousand of them every second means that we
can get at least better than meter-level estimates for the aircraft
positions.

Using the multiple (x,y,z) triplets from a single pulse would seem to
provide an even better angle estimate than the integer degree value
stored in the LAS/LAZ file, but only when you have some significant
delta Z offset, and at least for my files this seems to be rather rare:
Most pulses only give a single return. :-(
Reply all
Reply to author
Forward
0 new messages