I'm developing an instance segmentation algorithm for a particular class (Cows). I tested it on COCO and Pascal validation sets (71 image with cows in Pascal and 87 in COCO), and the results I got are rather confusing: State-of-the-art models - MaskRCNN and FCIS do very well on COCO and badly on Pascal. My method (finetuned from FCN8s weights + my ideas on MS COCO 2017 dataset, 1.9K+ images with cows) is the exact opposite:
MS COCO (AP@50% IoU threshold, mAP)
Pascal (AP@50% IoU threshold, mAP)
I'm not sure how to explain it. I've tinkered with the ground truth (removed and added back some small objects, fixed some contour bugs), but no big changes.