Hi Martin,
Your comment "starting a couple of months ago" caught my eye. I have four chromecast audios which started a similar issue around the same time, having worked perfectly for many years. The CCA on my main system uses the optical output, the others the audio output, but all seemed afflicted. There would be a single "click", rather like a digital version of a vinyl click, which might repeat after a few minutes or perhaps half an hour - not massive dropouts, but maybe related.
Working round one system, I replaced in turn the CCA, the USB power cable, the USB power supply and the headphone cable, all with no change. For no particular reason, I then used a cheap and cheerful powerbank instead of the mains supply, and bingo - no more clicks!
What is more, the slight "fuzziness" which has been on the main system promptly disappeared as well. Oddly, the fuzziness would only appear playing flac files from my NAS, whereas BBC sounds always sounded perfectly clear. My system is not complex - single drive Synology NAS, isp router, minim server, hifi cast. It is hard to quantify sound quality, but I have gone from thinking that I must upgrade soon, to 100% happy.
Hope this might help someone point us in the right direction!