Perfomance Metrics Homer3

[Joel Licea]

Hi everyone, 

We currently have a couple of pipelines, generated on Homer3, which we are trying to compare on our dataset. What features, if any, are available through Homer3 that can objectively show better performance of one pipeline over another? If not possible via Homer3, how are people comparing performance on various datasets? We do not have any simulated data or synthetic HRF data but any information on how to go about this issue would be greatly appreciated!! Thank you all in advance!

[Silvia Grazioli]

Hi Joel,

I don't know if this could be of any use to you, but here is a paper from our lab regarding a similar issue:  https://link.springer.com/chapter/10.1007/978-3-030-31635-8_28

Please let me know if I can help you any further

Silvia

On Thu, 17 Sep 2020, 17:46 Joel Licea, <joell...@gmail.com> wrote:

Hi everyone, 

We currently have a couple of pipelines, generated on Homer3, which we are trying to compare on our dataset. What features, if any, are available through Homer3 that can objectively show better performance of one pipeline over another? If not possible via Homer3, how are people comparing performance on various datasets? We do not have any simulated data or synthetic HRF data but any information on how to go about this issue would be greatly appreciated!! Thank you all in advance!

--
You received this message because you are subscribed to the Google Groups "Homer-fNIRS and AtlasViewer Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to homer-fnirs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/homer-fnirs/370ec3ed-665b-46ec-8ab8-7a48ae1e7722o%40googlegroups.com.

[Yucel, Meryem]

Hi Joel,

 

We have recently published (still provisional) such a dataset (resting + synthetic HRF) that you can use to test your different pipelines. You want to compare the estimated HRF with true HRF using different metrics (e.g. MSE / corr as in our paper).

 

https://www.frontiersin.org/articles/10.3389/fnins.2020.579353/full

 

Hope this helps.

 

Meryem

--

[Sergio Novi]

Hi Joel, 

one way to decided the most appropriate pipeline would be to check each preprocessing step separately (at least for a few subjects from your dataset).  

For example, suppose that you want to remove noisy channels, but you do not know which procedure or threshold (for a given method) is the most adequate for your data. 

You could try a few thresholds and verify if the deleted channels are indeed noisy channels.  In this way, you can define the best threshold or method for removing the bad channels. 

I think that checking each step is better than looking at the "final" results when you do not know the ground truth (as you would do on simulations). 

Best, 

Sergio Novi

To view this discussion on the web visit https://groups.google.com/d/msgid/homer-fnirs/CAHg47oz5vGPBL-k_akoeuBWpxv6UBY5ri-jRO2JRb4K-HkfjTw%40mail.gmail.com.

[Joel Licea]

Hi Yucel, 

I think this type of simulated data is what we need access to. Thank you so much for the response. I have downloaded the code to add simulated data to our channels but am having some trouble understanding how to incorporate it to our data set. Is this possible? 

Best, 

Joel

On Thursday, September 17, 2020 at 11:54:25 AM UTC-4, Yucel, Meryem wrote:

Hi Joel,

 

We have recently published (still provisional) such a dataset (resting + synthetic HRF) that you can use to test your different pipelines. You want to compare the estimated HRF with true HRF using different metrics (e.g. MSE / corr as in our paper).

 

https://www.frontiersin.org/articles/10.3389/fnins.2020.579353/full

 

Hope this helps.

 

Meryem

 

From: homer...@googlegroups.com <homer...@googlegroups.com> On Behalf Of Joel Licea
Sent: Thursday, September 17, 2020 11:47 AM
To: Homer-fNIRS and AtlasViewer Community <homer...@googlegroups.com>
Subject: Perfomance Metrics Homer3

 

Hi everyone, 

 

We currently have a couple of pipelines, generated on Homer3, which we are trying to compare on our dataset. What features, if any, are available through Homer3 that can objectively show better performance of one pipeline over another? If not possible via Homer3, how are people comparing performance on various datasets? We do not have any simulated data or synthetic HRF data but any information on how to go about this issue would be greatly appreciated!! Thank you all in advance!

--
You received this message because you are subscribed to the Google Groups "Homer-fNIRS and AtlasViewer Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to homer...@googlegroups.com.