Heather,
Some people recommend a criterion based on first axis length in DCA. But
a simple, graphical approach is to use a smoothing function (or
nonparametric regression) to plot species abundance or probability of
occurrence against the dominant predictors, then evaluate whether the
relationships are reasonably well described by straight lines or
not.
You can do this with kernel smoothers (as in the ordination overlay
function in PC-ORD, or via HyperNiche), or with GAMs, or other
smoothers.
-Bruce McCune