Antonello's talk - performance comparisons

[Ross Gayler]

Hi Antonello,

Carrying on from our conversation: it might be worth trying to recast the performance analysis in terms of Signal Detection Theory - https://en.wikipedia.org/wiki/Detection_theory

Cheers

Ross

[Friedrich Sommer]

Sorry we missed Antonella's talk, somehow did not see an announcement mail.

So just commenting on Ross's comment in total ignorance of Antonella's talk:

The connection between VSA capacity analysis and signal detection theory has been made in our 2018 paper:

https://direct.mit.edu/neco/article-abstract/30/6/1449/8400/A-Theory-of-Sequence-Indexing-and-Working-Memory

Best,

/Fritz

--
You received this message because you are subscribed to the Google Groups "VSACommunity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vsacommunity...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vsacommunity/caf12078-0b9e-4c9a-9582-fbccfed2e1d6n%40googlegroups.com.

[Antonello Rosato]

Dear Ross,

thank you very much for this information, I will definitely look into it and see if I can find some appropriate datasets to redo the tests in these terms.

Antonello Rosato
________________________________________________

Postdoctoral Research Fellow

DIET Dept., University of Rome "La Sapienza"

Tel.: +39 06-44585745

Via Eudossiana 18, 00184 Rome, Italy
E-mail: antonell...@uniroma1.it

--

You received this message because you are subscribed to the Google Groups "VSACommunity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vsacommunity...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vsacommunity/caf12078-0b9e-4c9a-9582-fbccfed2e1d6n%40googlegroups.com.

________________________________________________________

Le informazioni contenute in questo messaggio di posta elettronica sono strettamente riservate e indirizzate esclusivamente al destinatario. Si prega di non leggere, fare copia, inoltrare a terzi o conservare tale messaggio se non si è il legittimo destinatario dello stesso. Qualora tale messaggio sia stato ricevuto per errore, si prega di restituirlo al mittente e di cancellarlo permanentemente dal proprio computer.
The information contained in this e mail message is strictly confidential and intended for the use of the addressee only.  If you are not the intended recipient, please do not read, copy, forward or store it on your computer. If you have received the message in error, please forward it back to the sender and delete it permanently from your computer system.

---

[Antonello Rosato]

Dear Friedrich,

thank you for sharing that paper.

I am not sure I see the specific connection with the performance analysis Ross was commenting on, but there are some insights that I think are useful for distributed learning problems.

I will look into it and see what I can find.

Antonello Rosato
________________________________________________

Postdoctoral Research Fellow

DIET Dept., University of Rome "La Sapienza"

Tel.: +39 06-44585745

Via Eudossiana 18, 00184 Rome, Italy
E-mail: antonell...@uniroma1.it

To view this discussion on the web visit https://groups.google.com/d/msgid/vsacommunity/CA%2B_aDzZtNRgKo9KFLmEzEttoBtxU_cNgqjyx7hZqa0fvNroE1g%40mail.gmail.com.

[Ross Gayler]

Hi Fritz,

Just supporting Antonello's last email:

My original comment about Signal Detection Theory was intended as a suggestion concerning a technical issue in assessing the performance comparison of multiple differing implementations of classification systems.

He had been comparing the performance in terms of accuracy of classification. The predicted classifications were generated from a per class "score" and application of winner-takes-all to the set of scores.

This reminded me of the conceptual model of SDT where there is a per-class distribution of scores on an underlying decision dimension. The discriminability of the classes is conceptualised as the separation of the respective distributions.

Decision making (classification) is treated separately in terms of setting decision thresholds on the decision dimension, or in this case, comparing the current samples from each distribution to find the maximum.

In psychology, the original motivation for using SDT, was that it separated the perceptual process (discriminability) from the decision process (selecting quantised outcomes).

Accuracy, as a measure, conflates the perceptual and decision processes, so might hide trends in the discriminability arising from the different implementations.

So my suggestion was aimed at a possible methodological improvement of the analysis - analysing the performance in terms of separation of class-specific distributions of "scores" *might* be more revealing than analysing accuracy.

Cheers

Ross

To view this discussion on the web visit https://groups.google.com/d/msgid/vsacommunity/CAPmqYN4DJQJDWJOCZyLsh0N3UiiZUZ5xFviEATTc0nMr0t3y0Q%40mail.gmail.com.

[Friedrich Sommer]

Hi guys,

The reference I sent makes specifically the connection between VSA decoding and signal detection theory.

Antonello apparently talked about comparing different classification methods, and obviously, SDT can be applied as well.

Sorry for sending my email with a lack of background...

Best, Fritz 

[Ross Gayler]

FS> The reference I sent makes specifically the connection between VSA decoding and signal detection theory.

That will be very useful (and will have proper maths, unlike my hand waving).

I also managed to muddy the water in my question in the talk, by bringing in a practical applications theme:

Antonello's work concerns distributed learning in IoT devices.

I proposed that taking an SDT view suggests separating discrimination (learning) from decision-making.

If the perceptual properties of the classes are constant across sites there is value in distributing learning across the IoT devices so that they benefit from each others' experience.

However, if the base rates (or the payoff matrix) of the classes vary across IoT sites, it would  be better to have the decision process vary by site (i.e. introduce site-specific class biases).

Cheers

Ross