Thanks Farzad.
So, if I am reading the code correctly, at power-on reset, the "_start" symbol in CRT.s is what the ROM jumps to as part of the hardware reset sequence. That contains:
j _init
which jumps to this function in benchmarks/common/syscalls.c
_init(int cid, int nc)
Hence, every hart runs this _init() after reset. That _init(), in turn, jumps to
void thread_entry(int cid, int nc)
Which is overridden inside our code. So thread_entry acts as the "main" for our code.
Some implications -- first, the number of threads always equals the number of harts, and if we want to adapt existing code that uses posix threads, we remove all of the posix calls, and use thread_entry instead of thead_create. For the real "main" that coordinates, we put a clause inside the common thread_entry that checks the hart ID. So that "real main" only runs on, say, hart 0. And, we need to supply our own mutex implementation. Should work if the code isn't too complex and doesn't use fancy aspects of pthreads.