Hi Molly,
Transformations are typically used in ecological studies to remove the "horseshoe effect". This arises because population attributes at either end of an environmental gradient tend to be similar and so those ends are drawn together to form a horseshoe effect. It is hard to imagine how a similar process could apply to SNP data. So I agree with Stephen that transforming is unlikely to assist here.
A three dimensional plot is a good idea, and will give you some additional insight. Indeed, you should examine the scree plot to see how many dimensions are informative and which are just noise. Then, among the informative axes, you need to decide how far to delve down into deeper dimensions. A rule of thumb is to consider axes greater than 10% of variation explained (hence the horizontal line in dartR's scree plot).
That said, if there are three groups as your data suggest, they can be represented at a gross level in just two dimensions, so looking deeper will reveal departures from those groupings but probably not the gross story depicted in the two dimensions.
Without the context is is hard to advise further. However, one possibility is that there are two "clines" emanating from a source (or a "cline" disrupted by a barrier), contemporary or historical. Gaps in sampling arising from sampling limitations, or because intervening populations no longer exist, could be an issue obscuring the underlying process. If there were three discrete groups, I would not be suggesting this, but the arch you identify makes it a possibility. Have a look at the PCA I have attached. It represents a cline that is interrupted by a partial barrier (dispersal constriction at Brisbane) which leads to divergence patterns heading in different directions (an arch).
Without context, not saying this is an explanation for you, but it might be.
In any case, data talks and the pattern you see is crying out for a biological (popgen) explanation. Trying to make it go away with transformation is probably not the answer.
Good luck with it.
Arthur