It would be nice if .on('click',...) worked, as well as d3.mouse(...) when applied to x3dom elements.
As shown in the example code, the x3dom canvas's 'onclick' event handler fires instead, with an event that contains 3d and 2d coordinate information, so a workaround is to handle 'onclick' and to call .on('click').
In any case, the 'onclick' event is not of the type that d3.mouse(...) expects. I took at look at the d3 implementation, but I'm not yet sufficiently expert in javascript event handling to propose a change to d3.js. The commented-out event creation code the example .xhtml file might point in the direction of a solution.