Robust lowess compared with matlab's

[Leon Sasson]

Hey there,

I'm sort of porting some functionality from our partner's matlab code and luckily statsmodels already has lowess.

That said, I noticed the output of statsmodel's lowess is less robust to outliers compared to matlab's.

I do know in the equivalent matlab code, it's using the "rlowess" method instead of just the "lowess" (`smooth(hrv_vals,0.1,'rlowess')`), which according to matlab's docs:

A robust version of 'lowess' that assigns lower weight to outliers in the regression. The method assigns zero weight to data outside six mean absolute deviations.

A few visual examples below. All of those plots have the matlab version above, and the python attempt below.

Any ideas how to proceed?

Best,

Leon Sasson

 

[josef...@gmail.com]

Did you compare the options for bandwidth and robustness iterations?

It could be that matlab and us (and R) use different default values.

I haven't looked at it in some time, but increasing frac will increase the bandwith (smoother) and it increases the number of iterations. 

I don't know without digging into the code if there is another hardcoded parameter internally for the variance estimate and cutoff for outliers.

one warning: there is a bug if there are a large number of tied x, i.e. the corner case where we have more tied x than the local sample (or something like this, I didn't manage to figure out the details.)

It won't affect continuous data with only unique or with just a few tied x values.

Josef