You are absolutely right... I like to train just to improve my fitness more that for a specific training event... normally, I race 5-6 MTB Marathon races during the year, and in summer I could go for a sportive and do more long road rides..
So the season could be something like:
Since end March to end May: 2-4 XCM (MTB Marathon)
Summer: 1 o 2 Gran Fondo, and several long road rides with friends
September - Octobber: 2-3 XCM