I am finetuning a FCN with my own dataset using LMDB as my backend and my model works though I need to improve the performance. I would like experiment with data augmentation even though on the Fully Convolutional Networks for Semantic Segmentation paper it's noted it yielded no marked improvement. In particular, I would like to try scaling, rotation, color, mirroring, cropping, vignetting, augmentations. My question is, do I have to perform the augmentations on both the data and the label layers?