1) In your third example, where you cut the score short, it fits on the page because it does not have to fit in a custos. I guess (though not having your exact geometry I can't be sure) that with the custos it would break.
2) I consider explicit hyphenation a bad practice (except exceptions):
stan(f_//fO1e!fv_//fO1@e@f_@c_)(,)(ef!hv_//hvFE_0//hhh_)tem(ivHGhf_) (;)
rather than
stan-(f_//fO1e!fv_//fO1@e@f_@c_) (,) (ef!hv_//hvFE_0//hhh_)tem(ivHGhf_) (;) (your version)
because in the second case, gregorio will consider that the quarterbar is a quarterbar that sits between words, not inside a melisma, and it applies different rules in the two cases.
3) <nlba> tags are finnicky for sure; in any case they should span, not the whole line, but only the syllable or few syllables that you do not want breaking. I can't reproduce exactly your setup without the whole tex but the "logical" use of <nlba> in your case is
<nlba>stan</nlba>(f_//fO1e!fv_//fO1@e@f_@c_)(,)(ef!hv_//hvFE_0//hhh_)tem(ivHGhf_) (;)
On my computer it breaks cleanly:
175mm paperwidth + default margins for article class:
174mm paperwidth + default margins for article class:
You can also temporarily reduce the space around bars for just this score, so that the first segment of STANtem fits on your second line.
Matthias