It depends on how you are storing your vector<unsigned char> in hxcpp.
Generally, you do not need to tell haxe that it is a reference - just as long as haxe does not decide to make a copy of the variable you pass in at a bad time.
I'm guessing you would actually store your std::vector in a cpp.Reference, which is actually a pointer. In this case, you would type it as the same as your cpp.Reference.
It can be instructive to look at the generated code, and you should see that it is passing the cpp.Reference type directly, and in turn this reference should "autocast" to the actual std::vector&.
Good thing about cpp.Reference is that copying is ok, since it is really a pointer.
The key thing to remember is that the exerns are really to get the haxe compiler to type-check your code, not to literally represent the proto-types, although it is easier to be consistent, the closer you get to actual types.
Hugh