Seeking clarification about np.unravel_index as used in pyprocess.find_first_peak(corr).

[Ivan Nepomnyashchikh]

Hello dear OpenPIV community.

I don't quite understand the function np.unravel_index and am seeking help with it.

The function pyprocess.find_first_peak(corr) given on lines 248-264 here, makes use of np.unravel_index.

According to the line 301 in the function pyprocess.find_second_peak (lines 267-317 here), it is assumed that np.unravel_index gives us a set of two.

My question is why it is assumed so. And what does it tell me about my correlation if it gives me a set of three?

Thank you in advance.

Ivan

P.S. Providing my code is problematic due to its overall complexity. That's why I tailored my question to the issue with np.unravel_index.

[Alex Liberzon]

numpy.unravel_index returns row and col indices https://numpy.org/doc/stable/reference/generated/numpy.unravel_index.html

it returns two arrays of the same shape as indices.

Isn't the docstring of these functions states clearly what the expected output?

[Ivan Nepomnyashchikh]

Thank you, Dr. Liberzon.

Upon a more thorough inspection, I conclude that the docstring states clearly what the expected output is.

Regarding the numpy docs, I can't understand what they wrote there. I have found several intelligible explanations in the answers to this StackOverflow post.

Now, to the point: I have managed to resolve my issue.

Let me tell you what it is and suggest something.

Like I said, my code is long and contains a lot of unrelated details. Therefore - lest to bother everyone with going through my code - let me explain the issue in words. Still, if anybody wants to see my code, I can, surely, publish it and try to explain what it does.

WORD EXPLANATION OF MY CODE (only the necessary part of it).

I take, say, 10 PIV pairs of images. Each pair consists of frame 0 and frame 1. I divide them into two groups of images: the first group of images contains only frames 0 from the original pairs (i.e. the first group contains 10 frames 0), the second group contains 10 frames 1. For each group, I pair the images. Thus - considering, for instance, the group with 10 frames 0 - by pairing them, I get 5 pairs. Then I apply pyprocess.moving_wind_array and pyprocess.fft_correlate_images to all the images. To speed up the process, I ended up working with Python lists and converting them to numpy arrays. But numpy would ask me to assign "object" dtype to the array when converting a Python list to a numpy array. Which I did.

WORD EXPLANATION OF MY ERROR.

I, personally, had (and have) on idea whatsoever what "object" dtype means. As long as it was working, I was happy. And it was working with everything, except for pyprocess.sig2noise_ratio(correlations). Whenever it met pyprocess.sig2noise_ratio(correlations) it would give me the following error:

File "C:\Users\nepomnyi\box\!Ongoing work\Larva turbulent dispersion\HUDLS with SoloPIV\24 August 2022 glass-silver particles\24 August 2022 glass-silver particles.py", line 552, in PIVanalysisEnsemble
    s2n = pyprocess.sig2noise_ratio(corrs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nepomnyi\Miniconda3\envs\jumpToOpenPIV0.25.0\Lib\site-packages\openpiv\pyprocess.py", line 535, in sig2noise_ratio
    (peak2_i, peak2_j), corr_max2 = find_second_peak(
                                    ^^^^^^^^^^^^^^^^^
  File "C:\Users\nepomnyi\Miniconda3\envs\jumpToOpenPIV0.25.0\Lib\site-packages\openpiv\pyprocess.py", line 315, in find_second_peak
    (i, j), corr_max2 = find_first_peak(tmp)
                        ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nepomnyi\Miniconda3\envs\jumpToOpenPIV0.25.0\Lib\site-packages\openpiv\pyprocess.py", line 264, in find_first_peak
    return np.unravel_index(np.argmax(corr), corr.shape), corr.max()
                                                          ^^^^^^^^^^
  File "C:\Users\nepomnyi\Miniconda3\envs\jumpToOpenPIV0.25.0\Lib\site-packages\numpy\ma\core.py", line 5885, in max
    result = self.filled(fill_value).max(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nepomnyi\Miniconda3\envs\jumpToOpenPIV0.25.0\Lib\site-packages\numpy\core\_methods.py", line 40, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>=' not supported between instances of 'float' and 'str'

Debugging this error was very difficult and took me 3 days. The problem with the debugging was that the complaining numpy function was written in C and it was very difficult to find the source code for it. I didn't manage to find the source code. But - by various probes - I managed to nail down the specific line of pyprocess.sig2noise_ratio(correlations) which was causing the problem and I managed to correctly guess the reason why it was complaining.

WORD EXPLANATION HOW TO RESOLVE THE ERROR.

Have a look here. The complaint is caused by the numpy.max() function which is called on line 315. Recall that I supply an "object" dtype numpy array to pyprocess.sig2noise_ratio(correlations). To make the long story short, I figured out that - on itself - numpy.max() works perfectly well with "object" dtype. But before I give my correlations numpy array to line 315, I do something to it on line 304. It turned out that whatever function corr.view(ma.MaskedArray) does to the "object" type numpy array, causes function numpy.max() to complain. I didn't manage to figure out what exactly happens to an "object" numpy array after corr.view(ma.MaskedArray) because it is written in C. But what I did find out was that if you supply "np.float64" type numpy array to it, everything works perfectly well and you don't get any complaints from line 315. Therefore, to resolve the error, I had to apply 

correlations.astype('float64') to my correlations array before supplying it to pyprocess.sig2noise_ratio(correlations).

MY SUGGESTION.

I think that - at your discretion, Dr. Liberzon - we can do one of the following.

   1) Just leave it as is. Whoever has the same problem, will find this post and resolve the problem.

   2) Add - what we call in Russia, defense from a fool - to the pyprocess.sig2noise_ratio(correlations) function by one of the two means:

         2a) just add a sentence to the function's docstring saying that the argument must be of "np.float64" dtype,

         2b) or add the following code between the lines 298 and 300:

                    if corr.dtype != 'float64':                      

                       corr = corr.astype('float64')

Thank you.

Ivan

[Alex Liberzon]

Dear Ivan,

Not sure I understand the problem. Please upload somewhere your 10 images or 20 images (if those 0 and 1) and add your code so we could look at the relevant error.

the error message "TypeError: '>=' not supported between instances of 'float' and 'str'" does not seem to be related to the unravel_index but to something with the dtype of your correlation. This is rather strange and requires some research. Please open an issue on Github and we move this discussion there to solve for a bug

What I'm puzzled is how do you get to something which is not float in correlation - this is a mystery.

Alex

--
You received this message because you are subscribed to the Google Groups "openpiv-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpiv-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openpiv-users/ac899a65-e13a-4dbd-956f-7cfa0e7c35bdn%40googlegroups.com.

[Ivan Nepomnyashchikh]

Thank you Dr. Liberzon.

1. I opened an issue (issue number 280 opened by nepomnyi)
2. I have created a WeShare link to my code and images.

Some explanations.

• Download Temp folder from the WeShare website, navigate to it, run the code (using the Powershell, you can do it by typing python analysis.py).
• I divide the original PIV pairs into frames 0 and frames 1 with the function framesRestructure.
• Then - using the function preconditionAllimages - I apply mask to all the images (the mask is defined in the main function) and I apply customDeen image preprocessing algorithm to all the images (customDeen makes use of stretchFilter).
• Then I apply ensembleAverage to all the images (which involves finding fft correlations, grouping the images, finding average correlation for each group).
• Then the function PIVanalysisEnsemble does PIV analysis using the correlations obtained on the previous step.
• Lines 412, 414, 420, 421 is where I create dtype='object'  numpy arrays.
  • I don't need it really in the current version of the code, but that part of the code used to be slightly different and I had to use  dtype='object'  there. In my current version of the code, I don't use dtype='object'  anymore.
•  Line 463 is where I convert dtype='object'  to np.float64  to avoid the error.
  •  If you comment out line 463, you'll get the error.

Ivan