Active Speaker Validation MAP

[Juan Leon]

Hi,

For the active speaker dataset: Is the MAP metric for the baseline strategy in the validation set available somewhere? the paper contains the MAP for the test set only.

 

Thanks

Juan

[Joseph Roth]

Juan,

I'm don't think we ever computed the mAP for the validation set.  We primarily used auROC when training and report that on the validation set in the paper.  I expect the mAP on the validation to be very similar to that on the test set based on the auROC being similar and the top result in the AVA workshop having identical mAP scores on validation and the test set.

--
You received this message because you are subscribed to the Google Groups "AVA Dataset Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ava-dataset-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ava-dataset-users/6c5b06ef-5303-496f-8989-b4c592daffe8%40googlegroups.com.

[Sourish Chaudhuri]

Juan,

         Thanks for bringing this up! This is a good point, and we should update the numbers to have consistent metrics on both datasets. As Joseph said, our experiments have shown a very similar result across the released eval split and the "secret" dataset for the challenge, and that seems to have been the case for other workshop participants too. We will update the paper on arxiv asap.

Thanks,

 Sourish

To view this discussion on the web visit https://groups.google.com/d/msgid/ava-dataset-users/CAJrbPAyfNAwUS4NMn1SD6zbmTv1gHYHagrq_bWz%2BtZAduqyaJw%40mail.gmail.com.

[Juan Leon]

Thanks!

Am Freitag, 1. November 2019 11:05:58 UTC-5 schrieb Sourish Chaudhuri:

Juan,

         Thanks for bringing this up! This is a good point, and we should update the numbers to have consistent metrics on both datasets. As Joseph said, our experiments have shown a very similar result across the released eval split and the "secret" dataset for the challenge, and that seems to have been the case for other workshop participants too. We will update the paper on arxiv asap.

Thanks,

 Sourish

On Thu, Oct 31, 2019 at 9:13 PM 'Joseph Roth' via AVA Dataset Users <ava-data...@googlegroups.com> wrote:

Juan,

I'm don't think we ever computed the mAP for the validation set.  We primarily used auROC when training and report that on the validation set in the paper.  I expect the mAP on the validation to be very similar to that on the test set based on the auROC being similar and the top result in the AVA workshop having identical mAP scores on validation and the test set.

On Sat, Oct 26, 2019 at 6:18 AM Juan Leon <fuank...@gmail.com> wrote:

Hi,

For the active speaker dataset: Is the MAP metric for the baseline strategy in the validation set available somewhere? the paper contains the MAP for the test set only.

 

Thanks

Juan

--
You received this message because you are subscribed to the Google Groups "AVA Dataset Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ava-data...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ava-dataset-users/6c5b06ef-5303-496f-8989-b4c592daffe8%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "AVA Dataset Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ava-data...@googlegroups.com.

[Juan Leon]

Sorry to bump the thread, but do you think the result in VAL will be available before the CVPR deadline?

[Sourish Chaudhuri]

Here are the comparable numbers on the validation set for the AVA-ActiveSpeaker data (where Table 7 of the arxiv paper contains the corresponding numbers for the test set in the ActivityNet challenge).

V-Static-f1: 0.441

V-Static-f10: 0.647

V-GRU-f2: 0.735

AV-Static-f1: 0.686

AV-Static-f10: 0.792

AV-GRU-f2: 0.863

Thanks,

 Sourish

To unsubscribe from this group and stop receiving emails from it, send an email to ava-dataset-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ava-dataset-users/0d9f49b7-b9e8-46b9-b8e2-dcdddcdde84d%40googlegroups.com.

[Sourish Chaudhuri]

Also, note per David's email a few days ago, the ActivityNet evaluation server is open, for anyone wanting to eval on the test set used in the guest task for the ActivityNet challenge.

[Baptiste Pouthier]

Hello,

If i understood correctly, the Tables 4, 5, 6 and the Figure 7 in the paper are the results obtained on the validation set. Am i right?

Thanks,

Baptiste

Le vendredi 1 novembre 2019 05:13:21 UTC+1, Joseph Roth a écrit :

Juan,

I'm don't think we ever computed the mAP for the validation set.  We primarily used auROC when training and report that on the validation set in the paper.  I expect the mAP on the validation to be very similar to that on the test set based on the auROC being similar and the top result in the AVA workshop having identical mAP scores on validation and the test set.

On Sat, Oct 26, 2019 at 6:18 AM Juan Leon <fuank...@gmail.com> wrote:

Hi,

For the active speaker dataset: Is the MAP metric for the baseline strategy in the validation set available somewhere? the paper contains the MAP for the test set only.

 

Thanks

Juan

--
You received this message because you are subscribed to the Google Groups "AVA Dataset Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ava-data...@googlegroups.com.

[Sourish Chaudhuri]

Hi Baptiste,

                   Yes, your understanding is correct. Tables 4-6 and Figure 7 are on the validation set whose labels have been publicly released.

Thanks,

 Sourish

To unsubscribe from this group and stop receiving emails from it, send an email to ava-dataset-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ava-dataset-users/47f29c63-e302-4e96-9a5b-1df855a92713o%40googlegroups.com.