Filter "Detrend" in WEKA Investigator

[Anatoliy Zhadan]

Hi Peter!
Thanks for the new release.
Question about the Detrend filter. If I am not mistaken, the WEKA software does not have this filter. Can you advise where to find more information about how this filter works and what arguments? I found it very useful.

regards

Anatoliy

[Peter Reutemann]

> Thanks for the new release.

You're welcome. Since there are daily builds available, I don't tend
to make many releases.

> Question about the Detrend filter. If I am not mistaken, the WEKA software does not have this filter.

Correct. ADAMS contains a number of filters and classifiers that are
not available in Weka.

> Can you advise where to find more information about how this filter works and what arguments? I found it very useful.

Since we're dealing a lot with spectral data (eg NIR, MIR, XRF), we
add functionality that helps in that data domain.
Detrend currently has the following two correction methods available:
- Mean
computes the mean from all the amplitude/attribute values in the
specified range and subtracts it:
value := value - mean
https://github.com/waikato-datamining/adams-base/blob/master/adams-weka/src/main/java/weka/filters/unsupervised/attribute/detrend/Mean.java
- RangeBased
for each wave number/attribute range, the intercept/slope is
calculated and the new value is:
value := (value - intercept) / slope
https://github.com/waikato-datamining/adams-base/blob/master/adams-weka/src/main/java/weka/filters/unsupervised/attribute/detrend/RangeBased.java

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174 (office)
+64 (7) 577-5304 (home office)
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/

[Anatoliy Zhadan]

ok, i got it, thanks a lot

пн, 10 січ. 2022 р. о 21:56 'Peter Reutemann' via The ADAMS Flow User mailing list <theadams...@googlegroups.com> пише:

--
You received this message because you are subscribed to a topic in the Google Groups "The ADAMS Flow User mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/theadamsflow-user/kzJvR4witIE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to theadamsflow-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/theadamsflow-user/CAHoQ12%2BQ2Y7ELm9HpizjrHK_dy_jFB15V6Uwu9wUPcfAaCLFKA%40mail.gmail.com.

[Anatoliy Zhadan]

Hello Peter. I still come back to the issue of the DeTrend filter in ADAMS. I read your links. Maybe I'm missing something, but my impression is that it's like a math operation – “value := value – mean”, as you described. But when I repeated this mathematical operation as a filter mathematical expression - I got different results. Your Detrend filter provides radically better results for model correlation coefficients. You mentioned that you use this filter to process the spectral data, does your Detrend filter have something extra besides the “value := value – mean”? Maybe some operations with the data spectrum? Can I apply your Detrend to a regular number series but not spectral data?

regards

Anatoliy

[Peter Reutemann]

> Hello Peter. I still come back to the issue of the DeTrend filter in ADAMS. I read your links. Maybe I'm missing something, but my impression is that it's like a math operation – “value := value – mean”, as you described. But when I repeated this mathematical operation as a filter mathematical expression - I got different results. Your Detrend filter provides radically better results for model correlation coefficients. You mentioned that you use this filter to process the spectral data, does your Detrend filter have something extra besides the “value := value – mean”? Maybe some operations with the data spectrum? Can I apply your Detrend to a regular number series but not spectral data?

It is just a simple math's operation. Assuming you are using the
"Mean" detrend plugin, then per row in the dataset the amplitude
values are obtained, the mean from these calculated and then the mean
subtracted from each of the amplitude values. The "Mean" plugin
doesn't change the shape of the row, it merely adjusts the x values by
subtracting the (row) mean.

Today, I've added a new filted called "SimpleDetrend", which makes use
of the same plugins as Detrend. In contrast to Detrend, it does not
require you to have attributes named a certain way, you can simply
define the attribute range that you want to process/update. Just grab
the latest snapshot.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science

University of Waikato, Hamilton, NZ
Mobile +64 22 190 2375
https://www.cs.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/

[Anatoliy Zhadan]

Hi Peter. I see why I'm having trouble understanding your Detrend filter. "then per row in the dataset" - that is, the average is calculated not in columns but in rows. This is strange to me, because attributes are by default in columns. Perhaps this is a specific filter specifically for processing the spectral data that you mentioned, and there the database is organized differently.

Cheers, Anatoliy