Maybe instead you could use :caddfile, :caddbuffer, or :caddexpr to build a list using one format at a time?
I don't have a specific example, but yes, that's what I had in mind. Or better, a function called by a command.
When I made the suggestion, I thought you could get each compiler's output separately. Then it would be easy. But since all your compilers output to a single file, you'll probably actually need something like:
function ParseEverything()
lines=readfile(compiler_output_file)
call writefile(filter(copy(lines), 'v:val =~ ''pattern_to_match_compiler_x'''), compiler_x_output_file)
compiler X
caddfile compiler_x_output_file
" etc.
endfunction
So, maybe combining the errorformats is easier.
Alternatively, you could write a function or program to transform all the outputs into a common errorformat. In the Vim repository, runtime/tools has a ccfilter tool to do this for some compilers, you might look into that one.