Let me answer these out-of-order (ahem):
2.
BOOM supports the full RV64G ISA, including the RISC-V privileged ISA. It can run programs bare metal, on top of the RISC-V pk (proxy kernel), and on top of the RISC-V Linux port.
3.
The *.hex files come from the riscv-tools/riscv-tests/{isa/benchmarks}. You don't have to pass the BOOM cpp emulator a hex file, it just helps get rid of the overhead costs of reading your host machine's memory to load in the binary. However, the latest updates to the new Privileged ISA v1.9 move BOOM to a self-booting environment, so the hex files will be gone soon (and sadly, benchmarks will run a tad slower due to the startup/boot cost, hehe).
1.
You can get a vcd dump from the C++ emulator ("make run-debug ..."). It will probably take a while to compile. I myself use the "vsim" path when I want to get waveforms, since it builds far faster and I find the Verilog easier to grok, if you have access to VCS. That won't give you the memory state though, which will be too big (as well as any memory "off chip" will be in the C++ test harness/dramsim2 model).
The nice thing about C++ though is you can write code in the test harness to perform all sorts of hacky stuff, to your heart's content (not recommended though).
In addition to using waveforms, I also debug by turning on "DEBUG_PRINTF" in boom/src/main/scala/consts.scala. It's incredibly cryptic, but unbelievably useful (grep for "printf" in the boom source to see what the printfs are doing).
0.
If your copy of BOOM is passing all of the tests and benchmarks, then it's likely you have created a malformed program or you are trying to feed it to BOOM incorrectly. I can't really diagnose this problem any further without knowing what the command is you typed to run your program. You should also test your program on Spike first before you try it on BOOM. If it doesn't run on Spike it won't run on BOOM!
"spike myBareMetalBinary"
"spike pk myNewLibBuiltBinary"
etc.
And "spike -d" gives you cycle-by-cycle output.
Hope that helps...