FTR, the PR https://github.com/jenkinsci/plugin-compat-tester/pull/115 seems to have introduced a regression. We confirmed it while working on adding more Integration tests through https://github.com/jenkinsci/plugin-compat-tester/pull/140 (this PR is currently red, when it should be building fine).
Ramon Leon is going to file a report about that just now.
There was a bug in the implementation, fixed in https://github.com/jenkinsci/plugin-compat-tester/pull/140/commits/d6ce90e1c90a2db0f4b934ccc2784b66a26e8298
But JENKINS-57320 was introduced by another issue
https://github.com/jenkinsci/plugin-compat-tester/releases/tag/plugins-compat-tester-aggregator-0.1.0