DCASE 2018 - task 4 - Useful questions/answers

Nicolas Turpault

unread,

May 23, 2018, 11:57:34 AM5/23/18

to DCASE Discussions

Here is a short summary of some useful questions asked:

1) Problem downloading data ?

It is normal to have missing_files, and you need to send us a mail to get your "missing_files".

2) Baseline crashes on Sequencer:

Someone got this issue:

> Sequencer: Cannot create valid segment, adjust segment length and hop size, or use padding flag.
> srun: error: emblica: task 0: Exited with exit code 1

If you have it, it can probably comes from data, see in 3) how to check your data, and ask if you have some wrong files.

3) Is there a way to check data I downloaded:

Check the file attached with this post.

check_data.py can be launched and will create a check_data.log in which you can find :

if the metadata corresponds to your audio folder
if you have the same filenames and audio lengths with reference files

To use it, put all the files contained in the Archive.zip in dcase2018_baseline/task4/dataset folder and run check_data.py

4) Checking Audioset, I found that unlabel_out_of_domain files contains speech and children classes of the challenge classes (for example "Bark" which is associated with Dog):

You are right, nearly half of the annotations contain Speech. It was the only way to have an unlabel_out_of_domain set which was somehow representative of Audioset. Indeed, Speech is present in half of the segments of Audioset. Therefore, discarding speech would have also meant discarding many other classes and the variability of the set would have been penalized.

Regarding the child classes, this is something we did not consider while picking this subset. Therefore they can appear marginally.

Additionally, this dataset is unlabeled, so even if labels (of variable quality) exist in Audioset, you should not be allowed to use it during the challenge.
If you use them, we can evaluate your system but we would not be able to rank it against others.

5) What does fold means in the baseline system ?

Fold in this system is a trick to separate data used in the first pass system (weak annotations) with data used in the second pass system (strong annotations).

Fold 1 train set is the weak labels set

Fold 1 test set is the unlabel_in_domain set

Fold 2 train set is weak + unlabel_in_domain set

Fold 2 test set is strong labels set

Fold 2 evaluate set is filenames of strong labeled examples (used to simulate evaluation files)

6) Is there any trick involved during your training that is not documented because I created my own system in pytorch and cannot reproduce your results ?

No, since it is a baseline, we tried to not use tricks to have a clear system. However, if something is not clear to you, do not hesitate.

I had many other questions which I did not write in this post, but if you need a clarification of the baseline system or on the task, feel free to ask.

Archive.zip

Mario Alves

unread,

Aug 20, 2018, 8:44:53 AM8/20/18

to DCASE Discussions

Hi.
I'm not submitting to the challenge, but still trying to run the baseline of task 4 for selfstudy and prepare for next year and stuff.
I've got ~1000 missing files (csv attached), all because of unavailability, and exactly one with an "index error" that breaks the task4_crnn.py script thus:
77%|¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦? | 391/506 [04:19<01:16, 1.51it/s]Y

dgB3q7JhCTc_100.000_110.000.wav

list index out of range

100%|¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 506/506 [05:26<00:00, 1.55it/s]

Traceback (most recent call last):

File "task4_crnn.py", line 931, in <module>

sys.exit(main(parameters))

File "task4_crnn.py", line 96, in main

os.path.join("dataset", "audio", "test")

File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\datasets\datasets.py", line 661, in initialize

self.extract_packages()

File "D:\DCASE2018\Baseline\task4\Dataset_dcase2018.py", line 231, in extract_packages

train_meta_weak_fold2.save(filename=train_filename_fold2, csv_header=True)

File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\containers\metadata.py", line 1562, in save

value = item[field]

KeyError: 'filename'

Any tips on how to overcome these problems?
Thanks a lot.

item_access_error.log.csv

Mario Alves

unread,

Aug 21, 2018, 5:37:23 AM8/21/18

to DCASE Discussions

For the record, now it's a "type error" in another module.

(taskfourenv) D:\DCASE2018\Baseline\task4>python task4_crnn.py
Using TensorFlow backend.
2.2.2
2018-08-21 10:32:49.701982: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions tha
t this TensorFlow binary was not compiled to use: AVX2
[I] DCASE2018 / Task4

Traceback (most recent call last):
File "task4_crnn.py", line 931, in <module>
sys.exit(main(parameters))
File "task4_crnn.py", line 96, in main
os.path.join("dataset", "audio", "test")

File "D:\DCASE2018\Baseline\task4\Dataset_dcase2018.py", line 84, in __init__
    super(DCASE2018_Task4_DevelopmentSet, self).__init__(**kwargs)
File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\datasets\datasets.py", line 3088, in __init__
    super(AudioTaggingDataset, self).__init__(*args, **kwargs)
File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\datasets\datasets.py", line 494, in __init__
    self.load()
File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\datasets\datasets.py", line 507, in load
    self.load_crossvalidation_data()
File "D:\DCASE2018\Baseline\task4\Dataset_dcase2018.py", line 343, in load_crossvalidation_data
    self.process_meta_item(item=item)
File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\datasets\datasets.py", line 1138, in process_meta_item
    item.filename = self.relative_to_absolute_path(item.filename)
File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\dcase_util\datasets\datasets.py", line 1692, in relative_to_absolute_path
    return os.path.abspath(os.path.expanduser(os.path.join(self.local_path, path)))
File "D:\Anaconda3\envs\taskfourenv\lib\ntpath.py", line 114, in join
    genericpath._check_arg_types('join', path, *paths)
File "D:\Anaconda3\envs\taskfourenv\lib\genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

(taskfourenv) D:\DCASE2018\Baseline\task4>

Nicolas Turpault

unread,

Aug 27, 2018, 4:27:07 AM8/27/18

to DCASE Discussions

Hi,

I had the same kind of problems when I was not using the right version of dcase_util because some new functions and default parameters have been introduced.

Could you please tell me which version you are using ?

About the missing files, could you please launch "dataset/download_data.py" and send me the missing_files_[set].csv by mail.

Best,

Mario Alves

unread,

Aug 27, 2018, 5:27:30 AM8/27/18

to DCASE Discussions

Thanks a lot, Nicolas.
My version of dcase-util is 0.2.4
Missing files lists attached.
Long story in next post, maybe unecessary.

missing_files_weak.csv

missing_files_unlabel_out_of_domain.csv

missing_files_test.csv

missing_files_unlabel_in_domain.csv

Mario Alves

unread,

Aug 27, 2018, 5:56:30 AM8/27/18

to DCASE Discussions

DCASE_UTIL/PIP VERSION
Version of dcase_util = 0.2.4.
Pip is outdated, but surely that's not the issue.

    (taskfourenv) D:\>pip show dcase_util
    Name: dcase-util
    Version: 0.2.4
    Summary: A collection of utilities for Detection and Classification of Acoustic Scenes and Events
    Home-page: https://github.com/DCASE-REPO/dcase_util
    Author: Toni Heittola
    Author-email: toni.h...@gmail.com
    License: MIT
    Location: d:\anaconda3\envs\taskfourenv\lib\site-packages
    Requires: six, soundfile, pydot-ng, librosa, scipy, requests, future, numpy, validators, python-magic, matplotlib, tqdm, pyyaml
    You are using pip version 9.0.1, however version 18.0 is available.
    You should consider upgrading via the 'python -m pip install --upgrade pip' command.

DOWNLOAD_DATA
Running download_data.py gives the following error.

    (taskfourenv) D:\DCASE2018\Baseline\task4>python dataset\download_data.py
    [I] Download_data
    [I] Once database is downloaded, do not forget to check your missing_files
    [I] Test data

Traceback (most recent call last):

    File "dataset\download_data.py", line 206, in <module>
        download(test, result_dir, n_jobs=N_JOBS, chunk_size=CHUNK_SIZE)
    File "dataset\download_data.py", line 135, in download
        df = pd.read_csv(csv_file, header=0, sep='\t')
    File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f
        return _read(filepath_or_buffer, kwds)
    File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\pandas\io\parsers.py", line 405, in _read
        parser = TextFileReader(filepath_or_buffer, **kwds)
    File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\pandas\io\parsers.py", line 764, in __init__
        self._make_engine(self.engine)
    File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\pandas\io\parsers.py", line 985, in _make_engine
        self._engine = CParserWrapper(self.f, **self.options)
    File "D:\Anaconda3\envs\taskfourenv\lib\site-packages\pandas\io\parsers.py", line 1605, in __init__
        self._reader = parsers.TextReader(src, **kwds)
    File "pandas\_libs\parsers.pyx", line 394, in pandas._libs.parsers.TextReader.__cinit__ (pandas\_libs\parsers.c:4209)
    File "pandas\_libs\parsers.pyx", line 710, in pandas._libs.parsers.TextReader._setup_parser_source (pandas\_libs\parsers.c:8873)
    FileNotFoundError: File b'metadata\\test\\test.csv' does not exist

TASK4_CRNN
Ends with the following error (excerpt):

    77%|¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦?                      | 391/506 [04:44<01:23, 1.37it/s]Y

dgB3q7JhCTc_100.000_110.000.wav
list index out of range

    100%|¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 506/506 [09:59<00:00, 1.18s/it]
    'charmap' codec can't encode character '\u3010' in position 43: character maps to <undefined>

I have change the code to make it more robust to previous errors (TypeError, FileNotFound...) from subunits (Dataset_dcase2018.py), in hopes of getting the system running with the "incomplete" dataset, but no luck.
However, a "charmap codec" error seems strange as hell.
Probably the decase_utit version indeed.

DATASET_DCASE2018
I've instrumented this unit as follows (around line 341):

            # Process items
            for superfold in ['train', 'test', 'evaluate']:
                print("About to process ", len(self.crossvalidation_data[superfold][fold]), " items of ", superfold, " fold ", fold);
                print(".=item; *=TypeError, probably file missing; X=Item sans filename attribute");
                for item in self.crossvalidation_data[superfold][fold]:
                    print(".", end=''); #About to process item ", item);
                    if hasattr(item, 'filename'):
                        try:
                           self.process_meta_item(item=item);
                        except TypeError:
                           print("*", end=''); #"TypeError. Maybe file is missing! Process skipped.");
                    else:
                        print("X", end=''); #"Item has no filename; process skipped");

Rerunning task4_crnn.py with this intrument the output indicates a TypeError every other item (excerpt):

    About to process 551 items of test fold 2
    .=item; *=TypeError, probably file missing; X=Item sans filename attribute
    .*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*
    ..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..
    *..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*.
    .*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*
    ..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..
    *..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*..*

After this phase the output is as follows (complete):

[I] Download_data
[I] Once database is downloaded, do not forget to check your missing_files
[I] check files exist or download data
0%|                                                                                                               | 0/48 [00:00<?, ?it/s]U
sing TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
2.2.2
2018-08-27 09:48:39.252356: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions tha