Is there any command line tool to merge & flatten multiple shape files?
131 views
Skip to first unread message
Terje Mathisen
Feb 26, 2021, 2:50:39 PM2/26/21
to LAStools - efficient tools for LiDAR processing
My problem is as follows:
As I have described before I am using LAStools for all the initial
processing of public Norwegian lidar data (from https://hoydedata.no/ )
in order to generate our free topographic map of the entire country:
https://mapant.no/
(It should be obvious that LAStools is really crucial for making this
possible, with the LAZ format itself the key enabler: I have 25 TB
currently which would have required over 200 TB as LAS, i.e. out of the
question for a free/hobby/non-commercial project.)
The Norwegian mapping authority provides only very low-level access to
all the individual scanning projects and no combined/cleaned up version
using only the best data for each tile, so this is the step that I use
LAStools for.
Originally I just combined all the relevant projects (up to 7 or more
for a given spot), then thinned the resulting huge set down to 4
points/square meter using lasthin with step = 0.5
This worked fine as long as all the projects had high quality, but in
some parts of the country the only data we have is badly
height-calibrated to that it would end up 50-200 cm offset in height
from the good data, and this caused the ground surface (where good and
bad overlaps) to look like a huge boulder field with the same 50-200 cm
jumps up & down all over the place!
My next try was to use lasground_new on the combined data, this did give
us an OK ground surface, but now all the bad data turned up as very
dense vegetation, so I finally decided I had to use the project shape
files to clip all the data in priority order:
Starting with the oldest/worst project I would tile (200x200m tiles) the
input data, then clip it against the next (better) project.
In the next round the tiling process would take both the next input
project and the data that survived the previous clipping stage to make
the new input, then clip the combination, etc. until I got to the
final/best project which would be used exclusively wherever it was defined.
This works and is the current pipeline, but with up to 20+ projects in a
single 10x10km work area, the bottom layers will be re-tiled and
re-clipped over and over again, so I would much rather combine the shape
files!
I.e. project 0 (top/best) is just used as-is, project 1 is clipped
against project 0's shape file, then project 2 is clipped against the
combination of projects 0 & 1 shape files, and this is where it all
breaks down!
WARNING: polygon 2 has duplicate point at count 1625
ERROR: crossing constraint segments
ERROR: constraining 61 (590505.93/6648340.77) to 62 (590604.21/6648659.27)
of polygon 2 with 1069 points
I used GDAL with the ogr2ogr program to do shape file merging, this
results in a new single-layer shape file containing multiple overlapping
polygons, and this is rejected by lasclip, so my far faster workflow
doesn't currently work at all. :-(
I have not been able to find any command-line tool which can do the
merging and flatten the result down to a single set of non-overlapping
polygons. :-(
Any ideas?
Terje
--
- <Terje.M...@tmsw.no>
"almost all programming can be viewed as an exercise in caching"
As I have described before I am using LAStools for all the initial
processing of public Norwegian lidar data (from https://hoydedata.no/ )
in order to generate our free topographic map of the entire country:
https://mapant.no/
(It should be obvious that LAStools is really crucial for making this
possible, with the LAZ format itself the key enabler: I have 25 TB
currently which would have required over 200 TB as LAS, i.e. out of the
question for a free/hobby/non-commercial project.)
The Norwegian mapping authority provides only very low-level access to
all the individual scanning projects and no combined/cleaned up version
using only the best data for each tile, so this is the step that I use
LAStools for.
Originally I just combined all the relevant projects (up to 7 or more
for a given spot), then thinned the resulting huge set down to 4
points/square meter using lasthin with step = 0.5
This worked fine as long as all the projects had high quality, but in
some parts of the country the only data we have is badly
height-calibrated to that it would end up 50-200 cm offset in height
from the good data, and this caused the ground surface (where good and
bad overlaps) to look like a huge boulder field with the same 50-200 cm
jumps up & down all over the place!
My next try was to use lasground_new on the combined data, this did give
us an OK ground surface, but now all the bad data turned up as very
dense vegetation, so I finally decided I had to use the project shape
files to clip all the data in priority order:
Starting with the oldest/worst project I would tile (200x200m tiles) the
input data, then clip it against the next (better) project.
In the next round the tiling process would take both the next input
project and the data that survived the previous clipping stage to make
the new input, then clip the combination, etc. until I got to the
final/best project which would be used exclusively wherever it was defined.
This works and is the current pipeline, but with up to 20+ projects in a
single 10x10km work area, the bottom layers will be re-tiled and
re-clipped over and over again, so I would much rather combine the shape
files!
I.e. project 0 (top/best) is just used as-is, project 1 is clipped
against project 0's shape file, then project 2 is clipped against the
combination of projects 0 & 1 shape files, and this is where it all
breaks down!
WARNING: polygon 2 has duplicate point at count 1625
ERROR: crossing constraint segments
ERROR: constraining 61 (590505.93/6648340.77) to 62 (590604.21/6648659.27)
of polygon 2 with 1069 points
I used GDAL with the ogr2ogr program to do shape file merging, this
results in a new single-layer shape file containing multiple overlapping
polygons, and this is rejected by lasclip, so my far faster workflow
doesn't currently work at all. :-(
I have not been able to find any command-line tool which can do the
merging and flatten the result down to a single set of non-overlapping
polygons. :-(
Any ideas?
Terje
--
- <Terje.M...@tmsw.no>
"almost all programming can be viewed as an exercise in caching"
Michael Stimson
Feb 28, 2021, 8:22:48 PM2/28/21
to last...@googlegroups.com
https://gis.stackexchange.com/questions/266011/ogr2ogr-dissolve-shapefile might help you find what you're looking for. Another option is to erase existing polygons when appending using OGR2OGR https://gis.stackexchange.com/questions/151699/can-ogr2ogr-reverse-clip-or-clip-out-or-erase-or-difference-one-shapef.
Michael Stimson
Senior GIS Analyst
RPS | Australia Asia Pacific
D +61 7 3539 9694
E michael...@rpsgroup.com.au
-----Original Message-----
From: last...@googlegroups.com <last...@googlegroups.com> On Behalf Of Terje Mathisen
Sent: Saturday, 27 February 2021 4:09 AM
To: LAStools - efficient tools for LiDAR processing <last...@googlegroups.com>
Subject: [LAStools] Is there any command line tool to merge & flatten multiple shape files?
CAUTION: This email originated from outside of RPS.
My problem is as follows:
As I have described before I am using LAStools for all the initial processing of public Norwegian lidar data (from https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhoydedata.no%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tIyDTSLCpB0ZEdSTE7t7qN03js5b8YQ4fSmpXrlCX3Q%3D&reserved=0 ) in order to generate our free topographic map of the entire country:
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapant.no%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WNxLmgLYvvWZ5cGvu3RYWXVckv5yc%2BNqdKRQNn3pki8%3D&reserved=0
--
Download LAStools at
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flastools.org%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=t%2FN2mxjhiQXMxm60QrgUwUzv%2BZ1YoxNdOm1CNsaqZKo%3D&reserved=0
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frapidlasso.com%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zT4ssiUq9hNj6hPYWKwptn%2BKt7fBqZQiatacthEu4j4%3D&reserved=0
Be social with LAStools at
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffacebook.com%2FLAStools&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nXsW7LyTlX2Y7frFtriu%2FIu8iUIcBv9IagJuDL9JtEY%3D&reserved=0
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftwitter.com%2FLAStools&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ioyGzZuSsjm1egDkp%2BiVYKrGeVVjSHsFIvpmoMBQpFM%3D&reserved=0
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fgroups%2FLAStools-4408378&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ncVQZCZtWmx3o1RCfWgchFULcpUgwZoHxQiExmkUrgc%3D&reserved=0
Manage your settings at
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroups.google.com%2Fgroup%2Flastools%2Fsubscribe&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2osvvOO5SxApxt72LqQx2OOdai096JsqYQCbLFZ6B8M%3D&reserved=0
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Flastools%2Fdc063576-5644-782a-e1d0-fec79124d6bc%2540tmsw.no&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=a4PHxPPEHNeLJfpCb5e0kC07xCGkZhzx%2F9HgZLVrwSU%3D&reserved=0.
This email and its attachments may contain confidential and/or privileged information and is for the sole use of the intended recipient(s). The contents of this email must not be disclosed to or used by or copied in any way by anyone other than the intended recipient(s). If you are not the intended recipient, any use, distribution or copying of the information contained in this email and its attachments is strictly prohibited. Confidentiality and/or privilege in the content of this email is not waived. If you have received this email in error, please email the sender by replying to this message and immediately delete and destroy any copies of this email and any attachments. Please note that neither RPS Consultants Pty Ltd, any subsidiary, related entity ('RPS') nor the sender accepts any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The views or opinions expressed are the author's own and may not reflect the views or opinions of RPS
Michael Stimson
Senior GIS Analyst
RPS | Australia Asia Pacific
D +61 7 3539 9694
E michael...@rpsgroup.com.au
-----Original Message-----
From: last...@googlegroups.com <last...@googlegroups.com> On Behalf Of Terje Mathisen
Sent: Saturday, 27 February 2021 4:09 AM
To: LAStools - efficient tools for LiDAR processing <last...@googlegroups.com>
Subject: [LAStools] Is there any command line tool to merge & flatten multiple shape files?
CAUTION: This email originated from outside of RPS.
My problem is as follows:
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapant.no%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WNxLmgLYvvWZ5cGvu3RYWXVckv5yc%2BNqdKRQNn3pki8%3D&reserved=0
Download LAStools at
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flastools.org%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=t%2FN2mxjhiQXMxm60QrgUwUzv%2BZ1YoxNdOm1CNsaqZKo%3D&reserved=0
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frapidlasso.com%2F&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zT4ssiUq9hNj6hPYWKwptn%2BKt7fBqZQiatacthEu4j4%3D&reserved=0
Be social with LAStools at
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffacebook.com%2FLAStools&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nXsW7LyTlX2Y7frFtriu%2FIu8iUIcBv9IagJuDL9JtEY%3D&reserved=0
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftwitter.com%2FLAStools&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ioyGzZuSsjm1egDkp%2BiVYKrGeVVjSHsFIvpmoMBQpFM%3D&reserved=0
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fgroups%2FLAStools-4408378&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ncVQZCZtWmx3o1RCfWgchFULcpUgwZoHxQiExmkUrgc%3D&reserved=0
Manage your settings at
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroups.google.com%2Fgroup%2Flastools%2Fsubscribe&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2osvvOO5SxApxt72LqQx2OOdai096JsqYQCbLFZ6B8M%3D&reserved=0
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Flastools%2Fdc063576-5644-782a-e1d0-fec79124d6bc%2540tmsw.no&data=04%7C01%7Cmichael.stimson%40rpsgroup.com.au%7C132b435c23f74f4c057b08d8da8fcaa6%7C8091a96908434815991e2b531009928d%7C0%7C0%7C637499658417078596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=a4PHxPPEHNeLJfpCb5e0kC07xCGkZhzx%2F9HgZLVrwSU%3D&reserved=0.
This email and its attachments may contain confidential and/or privileged information and is for the sole use of the intended recipient(s). The contents of this email must not be disclosed to or used by or copied in any way by anyone other than the intended recipient(s). If you are not the intended recipient, any use, distribution or copying of the information contained in this email and its attachments is strictly prohibited. Confidentiality and/or privilege in the content of this email is not waived. If you have received this email in error, please email the sender by replying to this message and immediately delete and destroy any copies of this email and any attachments. Please note that neither RPS Consultants Pty Ltd, any subsidiary, related entity ('RPS') nor the sender accepts any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The views or opinions expressed are the author's own and may not reflect the views or opinions of RPS
Terje Mathisen
Mar 1, 2021, 4:17:55 AM3/1/21
to last...@googlegroups.com, Michael Stimson
Yes!
Thank you very much, the key was to use the sqlite command shown in the first link to generate a union of the polygons, so now I can run the much more efficient pipeline.
:-)
I ended up simplifying my logic so that I always do the union step even if there is just a single input shape file: It takes almost zero time and keeps the code more readable.
ogr2ogr "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/1174/1174_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/1023/1023_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/827/827_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/122/122_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/124/124_32N.shp"
ogr2ogr "work_33N_255000_6655000_1000/371/5.shp" "work_33N_255000_6655000_1000/371/5_temp.shp" -dialect sqlite -sql "select st_union(geometry) from '5_temp'"
Terje
Michael Stimson wrote:
Thank you very much, the key was to use the sqlite command shown in the first link to generate a union of the polygons, so now I can run the much more efficient pipeline.
:-)
I ended up simplifying my logic so that I always do the union step even if there is just a single input shape file: It takes almost zero time and keeps the code more readable.
ogr2ogr "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/1174/1174_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/1023/1023_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/827/827_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/122/122_32N.shp"
ogr2ogr -update -append "work_33N_255000_6655000_1000/371/5_temp.shp" "h:/hoydedata/124/124_32N.shp"
ogr2ogr "work_33N_255000_6655000_1000/371/5.shp" "work_33N_255000_6655000_1000/371/5_temp.shp" -dialect sqlite -sql "select st_union(geometry) from '5_temp'"
Terje
Michael Stimson wrote:
Reply all
Reply to author
Forward
0 new messages