To answer your above questions directly, if you're in an organism with a large number of very sequence-similar genes (as opposed to isoforms of the same gene),
this can complicate the inference task. However, that is true of any model. With sailfish, I would expect that increasing the k-mer size could help
(they reported this, e.g. in the RNA-Skim paper). But this may not be ideal in this scenario since the mapping rate is already fairly low.
Given that you have a 100bp, paired-end strand-aware library, I
strongly suggest that you give
Salmon a try. It retains the speed benefits of Sailfish,
but has the potential of being significantly more accurate --- specifically with longer and/or paired-end reads. We've done quite a bit of testing on the software
so far, and already have a number of people using it "in the wild" reporting promising results. Certainly, it would be worth it to see if the results provided
by Salmon agree more strongly with your a priori expectation about which genes are highly expressed in different conditions. If you still see that the
genes appear to not be highly expressed, then this provides evidence that it may be due to a large number of other, relatively highly expressed genes.
However, if the estimates do agree better with your expectations, this is evidence that the improved model of salmon may provide a significant benefit
in your case. Please, keep us posted!
Best,
Rob