This isn't the kind of oscillations you'd see if you had the gains wrong. Such a low OD is near the noise floor of the device though. If there's a positive offset to the OD then dilution would eventually exceed the washout rate and you'd just end up plotting noise. This is my best guess. My second best guess is that your gains are too low for such a small OD, but increasing gains makes the system more susceptible to noise.
Personally I think 0.1 is the lowest i'd attempt. Remember though that this is OD 650 at about 2cm path length so to adjust to "standard" OD600 1cm you multiply by about 0.7 for e. coli. Shorter wave lengths scatter more so if you tried a blue laser (make sure its dim enough!) that might also allow you to run lower cell densities for the same apparent OD.
Also on a side note, if you have slow growing cells you'll probably need to also make sure you humidfy your air (pass it through a bubbler before the chamber). Otherwise evaporation can be non-trivial at those low dilution rates.
Flocs are an unfortunate fact of life too :(
Taking effluent from the top of the culture selects for them eventually. If it's an issue the only thing you can do is take effluent from the middle of the culture, but then you'd need a pump for effluent and a liquid level sensor. Still, that can be rigged up without modifying any of the existing electronics if you're determined.
Chris