The linker was taking up all the time (compilation of Catch's main is slow but doesn't take 10 to 20 minutes).
Disabling LTCG fixed the issue (although you need to also turn the GL compilation optimisation off as well, otherwise the linker automatically turns LTCG back on again.
Link time in release mode with GL & LTCG on: 966793 ms
Link time in release mode with GL & LTCG off: 1516 ms
Shaving a whole 16 minutes off the build - result!
Thank you