How to replace outliers?

[Ivan Nepomnyashchikh]

Hello dear OpenPIV group.

I apologize at once for not providing a minimal reproducible piece of code (it was difficult to make). I will formulate my question rather conceptually instead.

Normally, we use the combination of the functions pyprocess.sig2noise_ratio , validation.sig2noise_val , filters.replace_outliers to replace the velocity vectors based on the signal to noise ratio.

But, even having filtered out the vectors based on their signal to noise ratio, we, still, can obtain velocity field where we can visually identify outlying velocity vectors. My question is about them. How do we deal with them?

For instance, consider the first OpenPIV tutorial. Scroll down and have a look at the very last velocity field. You can easily notice that all the velocity vectors are blue and looking downward, but there is one velocity vector which is looking sideways and is way longer than the rest of the vectors. That vector is the outlier. How do we replace it in OpenPIV?

Thank you in advance.

Ivan

[Alex Liberzon]

You need to define them properly - seeing is believing, but the code needs correct instructions. Try to manually select those vectors and understand what is wrong with them and how they escaped these filters. BTW, there already a variety of different filter propositions that you might want to add to OpenPIV.

Start with this tutorial and help us to understand what's happened there.

Thanks

Alex

--
You received this message because you are subscribed to the Google Groups "openpiv-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpiv-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openpiv-users/d1c8ec8b-045d-4763-8552-0ea9b6dc4b41n%40googlegroups.com.

[nepo...@oregonstate.edu]

I see, thank you, I’ll work on that.

Ivan

P.S. Taking an opportunity, let me update you briefly that I keep working on porting VortexFitting package to PIVPY. I’m almost there, I’m testing it. There are minor issues popping up here and there which I’m resolving. Also, in the meantime, I wrote my own code for Gamma1 and Gamma2 functions (vortex strength and scale), which I also want to push to PIVPY.

 

From: Alex Liberzon <alex.l...@gmail.com>
Sent: Tuesday, May 2, 2023 11:18 AM
To: Ivan Nepomnyashchikh <nepo...@oregonstate.edu>
Cc: openpiv-users <openpi...@googlegroups.com>
Subject: Re: [openpiv-users] How to replace outliers?

 

[This email originated from outside of OSU. Use caution with links and attachments.]

[Alex Liberzon]

This is wonderful, thanks. We need it asap in our work : ) https://photos.app.goo.gl/PRq3KQB1oMjHG5uZ8 

[Ivan Nepomnyashchikh]

For future readers, OpenPIV, in fact, is capable of doing what I'm asking about.

One needs to have a look at validate.py  file: https://github.com/OpenPIV/openpiv-python/blob/master/openpiv/validation.py

It can be broken onto two parts: 1) smoothing velocity field, 2) identifying outliers.

Smoothing is done using different Gaussian filters.

Outliers identifications has the following capabilities:

• having obtained the velocity field, use global threshold to mark the outlying vectors (functions global_val and global_std)
• having obtained just the correlations (without the velocity field yet), use signal to noise ratio to mark the correlations with peaks below the given threshold (function sig2noise_val); one has two options: either compare the peak to the next largest peak or to the mean of all the peaks
• having obtained the velocity field, use Westerweel's 1994 median test to mark outlying velocity vectors (function local_median_val)

IMPORTANT: any of the above mentioned functions only MARKS the vectors which, it thinks, are outliers. To remove the outliers once they have been marked, one has to use the function filters.replace_outliers.

I tried  local_median_val on my own data set and make it work in conjunction with filters.replace_outliers. But, boy, was it difficult to get the parameters right!

Pay attention to that when doing windef.py. That is because  windef.py makes use of  local_median_val   and if you don't have experience adjusting the parameters for  local_median_val , you could get unrealistic velocity field and might, falsely, decide that the entire windef.py doesn't work (like I did - that's why I stopped using windef.py).

That's why, I think, normalized Westerweel's 2005 median test is necessary: supposedly, it is easier to pick the parameters for it (I put it on my to-do list).

As per the OpenPIV example we discussed with Dr. Liberzon in this thread: I'm working on it.

The description of the median test can be found in the German PIV book on p.185 (the German PIV book is how I refer to Raffel, Willert, Werely, Kompenhans, "Particle image velocimetry", 2nd ed., Springer, 2007).

Ivan