Problem using lasinfo result as input in arithmetic expression
Jie Zang
I would love to use the output of lastool point spacing as a variable in an arithmetic expression. Somehow the result of such always returns a zero, which I guess may due to the type of lastool output is a string rather than a number.
My script:
::For example the output of point spacing is a parameter %y which is a result of FOR loop
set info=%y
set /A info=%info%*3
echo %info%
The result shown:
set info=0.29
set /A info=0*3
echo 0
Are there any suggestions for solution? Thanks ahead!
Best regards,
Jie
Perdue, Michael
Hello,
The problem you are running into is that DOS does not support floating point math. There are a number of ways to get around this, but the one I use involves GNU’s bc (calculator). Download GNU’s bc calculator for windows.
http://gnuwin32.sourceforge.net/packages/bc.htm
So, using bc inside a bat file (make sure bc is on the search path);
@echo off
set /P y=Enter a floating point value:
set info=%y%
set /A info=%info% * 3
echo DOS calculates %info%
set info=%y%
echo %info% * 3 | bc > info.txt
set /p info= < info.txt
del info.txt
echo using bc we calculate %info%
pause
Note that in DOS you cannot set the output of a command to a variable. So you need to write the variable to a file and the read that variable back from the file that you wrote it to.
Cheers,
Mike
--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe
Perdue, Michael
Hi Jie,
From what I can see below, there are a few of issues with your batch file;
1. Setting (unspecified) is exactly that. You have inserted “–step info” into your command instead of “–step %info%”, so you are literally passing the word info to the CLI switch.
2. The batch file looping over each file twice is expected behavior with the script as it’s written. You pass “las2dem -i %OUTPUTDIR%\try\*.laz” inside the for loop. As a result, each iteration of your for loop applies the selected values to all the laz files in the directory. This would get ugly as it grows quadratically. Four files results in 4x4/%cores% iterations. If you had 1000 files to work on… it’s gonna be a while. Also, the only iteration of the for loop that will matter is the last one. Successive iteration will overwrite the files from previous iterations and in the end you will end up with a set of files that all used the same set of values.
A couple of other observations:
1. The -wgs84 and –utm 33N switches are un-necessary. The output asc files have no georeferencing in them.
2. You are trying to set the –step and –spike_free switches to the same value. Is this what you want?
3. As far as I can tell, you are pulling the spacing info from the “all points” spacing rather than the “last only” point spacing. In the lasinfo files you sent, those two values are very close, but that isn’t always the case. I’m not very familiar with the spike_free canopy models and their construction, but I would verify that this is in fact what you want.
Finally:
1. If the intention is that you want to run a set of variables that have been custom fit to a specific tile/file, then you will not be able to use the “cores” switch to speed up the process. The for loop will need to determine values specific to a file and then pass las2dem a single file name. As there is only one input, you cannot use multiple cores to divide and conquer the workload. To the best of my knowledge, DOS has no way to run a for loop in parallel.
2. The work flow that you are designing is going to give you tiles/files that all use different step size/grid spacing. That seem to me that it is going to cause headaches further down the line.
3. Based on the lasinfo reports that you sent, some of the numbers don’t seem right. IE
number of first returns: 1006061
number of intermediate returns: 0
number of last returns: 3680231
number of single returns: 1006061
….
overview over number of returns of given pulse: 1006061 1153986 887027 443816 150101 32916 6324
I haven’t spent too much time thinking about it, but how do you have 887027 third returns and no intermediate returns???
These things said. This is how you would do what I think you are trying to achieve in a batch file. I would suggest regrouping and taking a different approach as the path you are currently on is going to lead to headaches with delayed variable expansion and a host of other issues that can make DOS batch programming a miserable experience. To make a couple of steps easier and generally improve abilities of DOS, I recommend you also install awk (Gawk), grep, find and sed from the GNUwin32 repository. However, the only two used below are awk and grep. Pure DOS methods are possible.
PART1: DOS rules
Move the workflow to a function, and then call the function. This allows you to avoid delayed variable expansion and helps keep you’re scripts more manageable/readable. I also recommend generating the files that you need as you need them, and then get rid of them once done. But that is my OCD need for filesystem tidiness shining through…
@echo off
set PATH=%PATH%;C:\LAStools\bin;
set PATH=%PATH%;C:\Program Files (x86)\GnuWin32\bin
set CHM_DIR=CHM
if not exist %CHM_DIR% ( mkdir %CHM_DIR% )
for %%j in (*.laz) do (
call :process_chm %%j
)
pause & goto:EOF
:: The function that is called in the for loop.
:process_chm (
:: strip the extension off the filename and use it as
:: a prefix
set prefix=%~n1
:: get information for file
lasinfo -compute_density -i %prefix%.laz 2> %prefix%.txt
:: get the average point spacing
grep spacing %prefix%.txt | awk "{print $4}" > spacing.txt
set /p spacing=< spacing.txt
del spacing.txt
:: multiply the average point spacing by 3
echo %spacing% * 3 | bc > spikes.txt
set /p spikes=< spikes.txt
del spikes.txt
:: no longer need the output from lasinfo so delete it.
del %prefix%.txt
las2dem -spike_free %spikes% ^
-step %spacing% ^
-i %prefix%.laz ^
-odir %CHM_DIR% ^
-oasc
) goto :eof
PART2: DOS sucks part1
If this is a one off exercise, the quickest way to do what you need is to extract the values that you need with the for loop that you already have and print the output to screen. Then use a text editor in column mode to fill in the rest of the command line and save it as a bat file.
If you are trying to automate a repetitive/tedious task and are bent on doing it in DOS, then go to part 1.
If you are trying to automate a repetitive/tedious task and are happy to leave DOS behind, then go to parts 3&4 or alternatively, pick up a good book on python/perl.
PART3: DOS sucks part2
If you are happy leaving DOS behind, you might want to give a POSIX shell a try. There are a number of options available that range in heft and setup requirements. I use Cygwin extensively, but msys is a lighter option that can be installed as part of osgeo4w. Apparently Windows 10 now has a native bash shell that can run all the unix tools natively, but I don’t have any experience with it.
Doing the same thing as the batch above in bash;
#!/bin/bash
chm_dir=CHM
if [ ! -d ${chm_dir} ]; then
mkdir ${chm_dir};
fi
for infile in *.laz; do
spacing=`lasinfo -compute_density -stdout -i ${infile} | grep spacing | awk '{print $4}'`;
spike=`echo ${spacing} | awk '{print $1*3}'`;
las2dem -spike_free ${spike} -step ${spacing} -i ${infile} -odir ${chm_dir} -oasc;
done
PART4: DOS sucks part3
If you choose one of the more robust shell clients, then it is possible to run GNU’s parallel to convert for loops to run in parallel. Parallel does run on Cygwin but you have to download it yourself and run “make install”. Parallel did not run properly on msys when I tested it there. It will auto detect how many cores are available and use them all, but the user can override this behavior.
#!/bin/bash
lastools_rocks_chm_function () {
infile=${1};
chm_dir=${2};
spacing=`lasinfo -compute_density -stdout -i ${infile} | grep spacing | awk '{print $4}'`;
spike=`echo ${spacing} | awk '{print $1*3}'`;
las2dem -spike_free ${spike} -step ${spacing} -i ${infile} -odir ${chm_dir} -oasc;
}
export -f lastools_rocks_function
chm_dir=CHM
if [ ! -d ${chm_dir} ]; then
mkdir ${chm_dir};
fi
parallel lastools_rocks_chm_function {} ${chm_dir} ::: *.laz
Cheers,
Mike
From: last...@googlegroups.com [mailto:last...@googlegroups.com] On Behalf Of Jie Zang
Sent: Tuesday, December 06, 2016 9:46 AM
To: LAStools - efficient tools for LiDAR processing <last...@googlegroups.com>
Subject: [LAStools] Re: Problem using lasinfo result as input in arithmetic expression
Hello Mike,
Thanks for your reply and suggestion. Using bc calculator as you suggested does work but then there I came across two new problems and would appreciate a lot if you could kindly point out the issues and solutions.
First, so glad and thanks that the calculator does work for one input floating point value (its final output is 0.89 which is the result of 0.29*3), but it does not work when I put this 0.89 back into las2dem as spike_free parameter. It shows:
setting (unspecified) spike-free freeze to 0 horizontal units (three times the step size)
Secondly, somehow I could not understand why and how, but with this external calculator, my loop repeats two times for each of the two txt file and does not follow the original logic flow.
Maybe I should show you my script clearer as follow and I attached those two txt files of lasinfo output that I used for getting %y:
set
PATH=%PATH%;C:\LAStools\bin;
set PATH=%PATH%;C:\Program
Files
(x86)\GnuWin32\bin
FOR /F
"tokens=4 USEBACKQ"
%y IN
(`find /i "spacing"
%OUTPUTDIR%\try\text\*.txt`) DO
(
@echo off
echo DOS calculates %info%
set info=%y
echo %info%
echo %info%*3
| bc > info.txt
set
/p info=< info.txt
del info.txt
echo using bc we calculate
%info%
las2dem -i %OUTPUTDIR%\try\*.laz
^ -spike_free
%info%
^ -step info
^ -wgs84
^ -utm
33N ^
-odir
%OUTPUTDIR%\try\CHM^
-oasc
^ -cores
%Cores%)
Thanks again and greetings,
Jie
Collins BK
Dear Mike,
It works excellently fine but only for the first tile (the loop isn't working).
I tried modifying it to;
@echo off
set PATH=%PATH%;C:\LAStools\bin;
set PATH=%PATH%;C:\Program Files (x86)\GnuWin32\bin
set CHM_DIR=CHM
if not exist %CHM_DIR% ( mkdir %CHM_DIR% )
call :process_chm %1
pause & goto:EOF
:: The function that is called in the for loop.
:process_chm (
:: strip the extension off the filename and use it as
:: a prefix
set prefix=%~n1
:: get information for file
lasinfo -compute_density -i %prefix%.laz 2> %prefix%.txt
:: get the average point spacing
grep spacing %prefix%.txt | awk "{print $4}" > spacing.txt
set /p spacing=< spacing.txt
del spacing.txt
:: multiply the average point spacing by 3
echo %spacing% * 3 | bc > spikes.txt
set /p spikes=< spikes.txt
del spikes.txt
:: no longer need the output from lasinfo so delete it.
del %prefix%.txt
las2dem -i %prefix%.laz ^ -spike_free %spikes% ^ -step 0.25 ^ -odir %CHM_DIR% ^ -oasc
) goto :eof
saving this as a batch file -> test.bat and then call the test .bat file in another .bat file with the following line of code;
Collins BK
Perdue, Michael
Hi Collins,
I see that you got your loop working for yourself via R, but for posterity, I’ll add my input to your original post.
What you were originally trying to accomplish by modifying the loop should have worked as you had it written.
for /F "eol=; tokens=1* delims=,. " %%i in (Tilenames.txt) do call test %%i
is a valid string.
Simply copying your code as you had written it below worked for me. However, the original bat file works for me as well and why it isn’t working for you is tough to answer without more details. As a result, I can only speak in generalities and give broad generic recommendations. The only thing I see that might trip someone up is that there is a pause statement in test.bat. This would cause the loop to pause and wait for user input at the end of each iteration and is probably not what you wanted.
Beyond that, the next thing to do is to start searching for the location where the failure is occurring.
Echo is your friend.
You can set @echo off to @echo on. This will cause each line in you batch file to echo back to the terminal. Personally, I’m not a big fan of this as it can generate a pile of output, but it will show you where things terminated. Personally, I prefer to manually echo commands back to the terminal at specific points so that I can see key points in the process. IE, something like;
for /F "eol=; tokens=1* delims=,. " %%i in (Tilenames.txt) do echo test %%i
To make sure your for loop is outputting what you intended. Then do the same inside test.bat to make sure your commands are being formed the way you intend. Something like.
set prefix=%~n1
echo prefix is set to %prefix%
:: get information for file
lasinfo -compute_density -i %prefix%.laz 2> %prefix%.txt
echo lasinfo -compute_density -i %prefix%.laz 2> %prefix%.txt
:: get the average point spacing
grep spacing %prefix%.txt | awk "{print $4}" > spacing.txt
set /p spacing=< spacing.txt
del spacing.txt
echo spacing is set to %spacing%
:: multiply the average point spacing by 3
echo %spacing% * 3 | bc > spikes.txt
set /p spikes=< spikes.txt
del spikes.txt
echo spikes is set to %spikes%
:: no longer need the output from lasinfo so delete it.
del %prefix%.txt
las2dem -spike_free %spikes% ^
-step %spacing% ^
-i %prefix%.laz ^
-odir %CHM_DIR% ^
-oasc
echo las2dem -spike_free %spikes% -step %spacing% -i %prefix%.laz -odir %CHM_DIR% -oasc
This way you can ensure that each command is being formed the way you intend it.
Test your files/executables existence before you call on them.
Other things that you should consider is adding tests to ensure that your batch file finds all the files/executables that you need are found. I didn’t add these tests to sample batch for clarity, but I always add them to my scripts in house. In DOS, something like;
for %%X in (laszip.exe) do (set FOUND=%%~$PATH:X)
if not defined FOUND (
@echo "Cannot find laszip.exe --Please repair the installation"
pause
GOTO Error
)
In the sample bat that I had written, I would be testing for bc, awk and las2dem.
These tests can save your bacon. Consider the simple batch file;
set LATOOLS_HOME=C:\LAS tools\bin
set PATH=%PATH%;%LATOOLS_HOME%
laszip -i *.las
del *.las
Ops, %LASTOOLS_HOME% has a typo in it. As a result, the laszip command will fail as it is not on the PATH. But DEL is built into DOS, so it is guaranteed to run like a champ. It might sound silly, but I saw it happen once. Fortunately our server supports snapshots, so it only took a couple of minutes to restore the 1000’s of files that bypassed the trashcan.
You can test if a specific file exists (if not exist (do something)…) and you can also test user input for validity;
:: Ask the user if they want to create google earth files (kmz's). If
:: they don’t enter Y or N, then “result” will not be set to true. The
:: invalid answer will be repeated to the user followed by a graceful exit.
set /p choice=Would you like to create kmz files for Google Earth? [Y/N]:
set result=false
if /i %choice% == N (set GoogleFiles=0 & set result=true)
if /i %choice% == Y (set GoogleFiles=1 & set result=true)
if /i %result% == false (
echo "%choice% is an invalid selection."
pause
goto Error
)
How much effort you put into this is your call. Most of the batch files I write are designed to be run by others, so I test everything for validity. This includes files that the batch expects as well as user input. In the long run, putting the effort into testing for valid input saves me time in support.
Know the difference between call and start.
The loop that you crafted to run test.bat will likely create issues when run over a large set of data. The reason lies in the difference between “call” and “start”. “Call” shares variables between the parent and child processes where “start” does not. To illustrate the differences consider the following examples;
Test.bat contains the following code
@echo off
set VAR=%VAR%+1
echo %VAR%
call.bat contains the following code;
@echo off
set VAR=1
for /L %%i in (1,1,5) do (
call test.bat
)
And when we execute call.bat , we get the following output;
1+1
1+1+1
1+1+1+1
1+1+1+1+1
1+1+1+1+1+1
While start.bat has the following code;
@echo off
set VAR=1
for /L %%i in (1,1,5) do (
start /b test.bat
)
And generates the following output;
Prompt>1+1
Prompt>1+1
Prompt>1+1
Prompt>1+1
Prompt>1+1
When you call your modified test.bat from a loop, PATH is being modified and returned to the parent process. As a result, PATH will get larger and larger as it accumulates crud from successive iterations. How big the variable is allowed to become and what happens when it overflows, I have no idea. But all variables that are created/used by a batch file will be returned to the parent process when using “call”. This can cause unexpected results and bugs that are hard to track down. In the case of “start”, variables in the parent process will pass to its children, but variables created/used in child processes will not pass back to its parent. Both cases have their uses, but you do need to put thought into when/where you use each case. In the situation that you were trying to solve, I would not have used one batch to call another: I simply would have substituted the for loop that you crafted with the one that was already inside the sample bat. However, I doubt that would have solved your issue.
Finally, if you are trying to craft more sophisticated bat files, I recommend you use an editor with syntax highlighting if you aren’t already. I use notepad++ with is open source and has the nice feature of being able to copy your highlighted syntax as html/RTF for easy to read inclusion in documents and e-mails.
Cheers,
Mike
From: last...@googlegroups.com [mailto:last...@googlegroups.com]
On Behalf Of Collins BK
Sent: Thursday, December 22, 2016 9:14 AM
To: LAStools - efficient tools for LiDAR processing <last...@googlegroups.com>
Subject: [LAStools] Re: Problem using lasinfo result as input in arithmetic expression
Dear Mike,
--