BayesWavEx software updates

[André Jalobeanu]

BayesWavEx - new versions and builds (change log)

[André Jalobeanu]

Version 1.1 beta 11/05/2018

This beta decodes Riegl SDF files, performs the extraction and writes LAS/LAZ in a single pass.

Automatic MTA resolution, outlier detection, advanced waveform processing and channel fusion included. 

More info on this blog post: http://bayesmap.com/we11b/

and on the WavEx page http://bayesmap.com/products/bayeswavex/ including link to PDF documentation.

[André Jalobeanu]

Version 1.1 beta build 11/09/2018 

This new build includes minor bug fixes and a few improvements:

• Included the Lambert Conformal Conic (LCC) projection - most US state planes are now supported
• Updated wkt parser to include new projection and better handle units
• Added option -em_flt to filter out emitted pulses affected by cross-talk from received waves (default 0)
• Fixed an issue happening in rare cases with complex waveforms and that could cause a crash

[André Jalobeanu]

Version 1.1 beta build 11/13/2018 

This new build includes minor bug fixes:

• Updated the method that prevents multiple instances from running, lock file no longer used, robust to crashes
• Better SDF format error handling
• Rejected swath list created instead of exiting after first failed extraction 

[André Jalobeanu]

Version 1.1 beta build 11/14/2018 

This new build includes minor updates and bug fixes:

• Added option -plot_po (along with -po_sub) to export subsampled and projected PO files in ASCII
• Improved robustness of SDF decoding for corrupt files
• Fixed issue - manual UTM zone not taken into account

[André Jalobeanu]

Version 1.4 build 09/19/2023 

• RiVLib updated to latest available version (2.7.0)
• Added support for HOM and OS projections
• New option -mta_amb_zone to manually set MTA zone in ambiguous cases (useful for low density power lines)
• New options -trunc_r1 and -trunc_r2 (argument in m) to filter out bad returns from truncated waveforms (SDF files)
• New option -set_withheld (enabled by default), use -set_withheld 0 to avoid setting the withheld flag in LAS files
• New option -drop_withheld (disabled by default), use it to filter out withheld points instead of setting the withheld bit in LAS files
• New option -withheld_ld to set low density points as withheld (RXP, high PRR MTA)
• New options -proj_test2 (arguments lon lat) and -proj_test3 (arguments lon lat h) to test projections and geoids 
• New expert option -cloud_r0 to set cloud filter threshold
• New option -inc_id to set LAS ID increment for file sequences
• New option -ch for channel override
• Other options and features synced with StripAlign (-plot_po, license management, build date in software name, etc)
• Improved stability of RXP processing by detecting and rejecting bad data sent by library (NAN range and timestamps)
• Improved reliability of channel cross-talk detection and filtering for waveform processing (SDF files)
• Improved compatibility with StripAlign internal calibration parameters I4 and I5
• Various minor bug fixes and improvements including display output, log files, LAS/LAZ export

This list only lists changes since Jan 2022. 

See updated documentation for full list of options and more details about usage.

[André Jalobeanu]

Version 1.5 build 03/11/2024

All the following updates apply to RXP (point cloud) input files.

• Robust time sync allowing LiDAR data to start before and stop after trajectory data (useful for UAS). This feature is not enabled when testing extraction without a PO file
• Cross-talk detection (between emitted and received pulses) to effectively filter low points (underground returns) and other artifacts. The new option -dr_xtalk (default 0.25) sets the distance threshold between emitted and received pulses. Detected cross-talks as marked as withheld
• Improved extraction robustness (reordering, bad data and duplicate point elimination). The updated documentation describes the counters for various kinds of removed points; the numbers are reported if option -llog is used (RXP log file)
• Time reordering of points  to avoid return number and number of returns inconsistencies in some (rare) cases
• Better handling of library exceptions to avoid losing entire output file when the input file is corrupted or when encountering bad or inconsistent data
• More consistent extra byte export, compatible with other software. Extra bytes are now "Reflectance" (dB), "Amplitude" (dB), "Deviation" and the type is signed short instead of unsigned short. "Amplitude" is replaced by "LinAmplitude" if option -alin is used (linear instead of logarithmic amplitude)
  
• New option -max_range (default 8000 m) sets the range limit corresponding to the scanner, and helps compute MTA bounds automatically rather than setting -mta_max manually (recommended to avoid MTA artifacts for short range UAV scanners)
• Fixed issue with intensity scaling introduced in one of the previous builds
• SDC conversion code deprecated, now only accepting SDF and RXP as inputs

Documentation updated. 

Better separation between SDF and RXP output format descriptions (LAS attributes and extra bytes), added new options, and improved install instructions (license info in persistent option file).

[André Jalobeanu]

Version 1.6 build 01/06/2025

RXP only

New features

• New cloud filtering algorithm, based on a method derived from MTA algorithm 2 (see option -cloud_mta_flt). Improvement in cloud/fog air point classification. At least 10 times lower false alarm rate. When enabling this filter, MTA algorithm 2 is automatically enabled. Only works with auto MTA (at least two MTA zones), otherwise previous algorithm has to be used (-cloud_flt)
• Now WavEx can sync files that can't be directly be synced with the Riegl library but have all the necessary information embedded in the RXP file (see option -pps). Use only if the first attempt fails to sync

New options

• New option -cloud_mta_flt (disabled by default) to use the new cloud filter
• New option -cloud_mta_min to cancel the effects of the filter if the rate of detected air points is below a certain threshold, argument is %, default 0 (apply filter to all scanlines when enabled)
• New option -cloud_mta_niter (default 3) to set the number of iterations of the new cloud filter, higher yields better filtering but also potentially a higher false alarm rate depending on the scene (tall trees etc.)
• New option -pps (disabled by default) to sync files that can't be directly synced with the library
• New option -next_day (enabled by default), try processing again with the next day date if the first time sync attempt failed, if the GPS date is unspecified and inferred from the file name, in case the date from the file name is wrong
• New option -ows (disabled by default) to force LAS timestamps to week seconds instead of adjusted GPS
• New option -skip_end followed by size in MB to skip the last MB of the file (if the processing ends with a library crash)
• New option -user to set user data for each point with value provided as argument

Updated option default values

• Updated default value for -cloud_max to 10
• Updated default value for -range_min to 0.25 (was 0.5) (minimum range filter, before MTA processing) which cuts the data loss rate in half

Improvements and updates

• RiVLib updated to latest available version (2.8.0)
• Outlier detection algorithm improved using order filter, can detect underground points closer to ground
• New option -out_ksigma for the new outlier filter method (default 4), in standard deviation units (reject points outside k standard deviations, for example 2 rejects 5% of points which can include good data, and a higher k value means lower false alarm rate but also less effective filtering)
• Improved MTA algorithm 1 success rate for partial scanlines, helps in cases when this algorithm is selected, or when it is used to initialize the more advanced algorithm 2
• Displaying an explicit warning message when all points are lost due to filtering (such as -range_near or -range_far) and not to time sync, also disabling the retry with next day feature in that case
• Updated minimum number of points per scanline to 3 instead of 32, avoid losing power lines over water
• Improved robustness to RiVLib library errors (SEGV) by adding option to skip end of file, with option -skip_end followed by the size (in MB) of the block of data to ignore at the end of the file, use with option below to determine optimal block size
• New option -llog2 to report read and left MB from RXP file to figure out size of block to skip in case of SEGV and try to minimize data loss
• Improved emitted/received cross-talk detection rate, now reaching 100% with all emitted pulses taken into account by improving library usage
• Now using class 29 for emitted/received cross-talk flagging
• Added two new initialization options for MTA algorithm 2 accessible via option -mta_algo with arguments 3 and 4. Select 3 when there are only 2 MTA zones to use both zones without preference (no effect if more than 2 zones). Select 4 to initialize with algorithm 0 instead of 1 (which is the default initialization)

Bug fixes

• Fixed issue preventing the outlier filter to work properly for rotated sensors (eg. mount angles 90 0 0)
• Fixed minor bug specific to MTA algorithm 2 and total loss of data due to filtering, was causing a crash
• Fixed minor bug, points marked as class 30 were not deleted by -w (remove withheld) but remained, marked as withheld

SDF only, new feature and improvements

• Implemented cloud filter in SDF waveform processing module (was only available for RXP point cloud files), enable with -cloud_flt (note: the new algorithm is not available for SDF files)
• Update to cloud filter for all file types to decrease the false alarm rate
• Fixed a counter (scanlines dropped) in the processing log; other minor improvements to SDF processing log
• New option -user to set user data for each point with value provided as argument, replaces info saved by the waveform processing module

Both RXP and SDF

• Updated exit codes to give more explicit info about type of error, now 128+code instead of 0 in case of failure. All platforms: ABRT FPE ILL INT SEGV TERM. Linux only: TRAP QUIT SYS BUS

Documentation

Documentation updated (WavEx16-doc.pdf) - last update before moving to Help Scout