Hi Dusit
LIke Murray, I suspect that the shape of the detection function is not really predictable in advance. And depending on your detector density, you may not have much ability to distinguish between shapes on the basis of the capture data (higher detector density gives greater ability). You do however, seem to have good reason to suspect that animal ranges are different in the two habitats, and it these circumstances my starting point would be to use a detection function in which both g0/lambda0 and sigma depend on the mask covariate "habitat type" (or whatever you call it).
You can't use suggest.buffer() etc. with this kind of model, but you can quite easily look into the effect of buffer size by just looping through a series of fits in R, using a model like that above, and increasing buffer size on each fit. If you save the detection function parameter estimates, density and esa estimates inside the loop, you can then construct plots after the loop that show how these estimates change with buffer size and at what buffer size they (hopefully) stop changing much. (If you are not sure how to program this, I can help - or Murray may already have code to hand that does it.)
One thing to bear in mind: I suspect that you might end up needing quite a high-resolution mask (i.e. lots of mask points) because a suitable buffer for the animals with big ranges might give a very big mask for animals with small ranges and if you don't have enough grid points in the mask, it is concievable that you end up with so few grid points per range of the small-range animals that you get too crude an estimate of the small-range animals' esa. (The space that animals actually occupy is continuous - the mask is a discrete approximation to this space and if it is too discrete it can be a poor approximation.)
David