The only supported way of building targets is with build.sh/Dockerfile
We considered allowing users to bring their own build as you describe, but decided against it (at least initially) due to maintenance burden.
We're taking a wait and see attitude to see how badly users want this feature.
However, if you really don't want this, you could try putting your own build in the build-out subdirectory of the workspace (pointed to by the env var GITHUB_WORKSPACE on github), and running the "run_fuzzers" action instead of running build_fuzzers first.
The main problem I for see with doing this, is ClusterFuzzLite will run all fuzzers instead of only affected fuzzers since that step is done in build_fuzzers.
I'm happy to help if this is a blocker for you, feel free to reach out again.