The NDVI values are low because there is not enough difference between the blue and red channels in the original photo. Healthy plants in a red-filtered photo should have a rich turquoise color like this:
The color is influenced strongly by the custom white balance, but also by the lighting and exposure of the photo. Custom white balance should be done while flooding the sensor with red light (red paper in direct sunlight). Over or under exposure changes the relationship between the red and blue channels (they become more similar to each other). Some practice is required to get meaningful NDVI results.
Chris