Hi Jim,
The internals of the marking functions changed in Numbas v5 to accommodate alternative answers. The "mark student's answer" JavaScript now replaces Part.mark_answer, instead of Part.mark. Part.mark_answer takes two arguments, studentAnswer and scope, so you need to pass those when you call the built-in version. So the line should read:
this.__proto__.mark_answer.apply(this,[studentAnswer,scope]);
But you don't need JavaScript to do the marking you want. It could be achieved using a JME marking algorithm, and those are a lot more stable than the JavaScript API.
In fact, the question you've set up would be even easier to implement in explore mode: you could decide whether to show the "enter the factorisation" part based on the student's answer to "does this equation have a factorisation?"
I hope that helps.