Backtracking in V8 is implemented using a stack. The relevant files
are in src/*regex*; it's part JS, part C++.
It's worth mentioning that irregexp, the regular expression parser,
normally emits native code but it also has a bytecode interpreter. I
don't think the interpreter is currently used (maybe by the
out-of-tree PPC port) unless you build with v8_interpreted_regexp=1.