I tracked it down to an issue with setting the initial-scale to 0.4 (instead of 1)
When the initial scale is set to 1, everything looks right across all devices and browsers.
When the scale is set to less than 1, I seem to get unpredictable results between the stock android browser in Android 4.0 and the Chrome browser using rem units.
There is actually quite a bit of javascript used in the site so upon further inspection I suspect it could also be some javascript affecting the font sizing too.