> 1. When we set a segmentation angle, all boundary segments with an angle higher than that defined misorientation angle are considered grain boundaries?
If closed, they are grain boundaries, if not closed, they are stored in innerBoundary
> 2. Orientation gradient of EBSD data points at those grain boundaries become NaN, and GND density is calculated by MTEX considering the grain interior orientation data?
Yes. Have a look at the demo.
> 3. If 1 and 2 are correct then it would be better to choose a low segmentation misorientation angle (<5 degrees) than the standard 15 degrees? This will help to identify the LAGBs (in addition to HAGBs) and set their orientation gradient to NaN, consequently excluding LAGBs and HAGBs from the GND calculation?
Depends on the material and what sort of structures are actually represented by low angle boundaries. In the case you'd only want to see the effect of continuous curvature, that might be an option, however, your bulk result will obviously still depends on the segmentation angle.
Cheers,
Rüdiger