Error while Preprocessing data for bob.paper.oneclass_mccnn_2019

[Shen Tak Wai]

Dear Bob developers,

I have a problem with the project bob.paper.oneclass_mccnn_2019.  According to the file bob.paper.oneclass_mccnn_2019/doc/running_mccnn-occl-gmm.rstm, I run the spoof.py

 ./bin/spoof.py protocol-grandtest mccnn-occl-gmm --execute-only preprocessing --sub-directory ~/Documents/data/WMCAtest/ --grid idiap

However, it showed

bob.learn.pytorch@2021-02-11 10:44:36,774 -- DEBUG: No pretrained file provided in the config, will try loading the commandline path!
Please advise how to config it. Thanks!

Best regards,

Wai

[anjit...@gmail.com]

Hi, 

You can Ignore this warning at the preprocessing stage. Is the pipeline running otherwise?

Anjith

[Shen Tak Wai]

Dear Anjith,

Thanks for your reply!

Only a empty folder "gridtk_logs/preprocess/" is obtained, so I guess I make something wrong in config.

Below shows the error messages.

Best Regards,

Wai

$ ./bin/spoof.py protocol-grandtest mccnn-occl-gmm --execute-only preprocessing --sub-directory ~/Documents/data/WMCAtest/ --grid idiap
bob.extension.config@2021-02-11 15:28:42,082 -- DEBUG: Loading configuration file `/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/bob/paper/oneclass_mccnn_2019/config/Method/Pipeline.py'...
bob.pad.face@2021-02-11 15:28:43,354 -- INFO: Light_cnn_model_file path: /nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/learn/pytorch/models/LightCNN_29Layers_checkpoint.pth.tar
extractor_file None
bob.learn.pytorch@2021-02-11 15:28:43,551 -- DEBUG: No pretrained file provided in the config, will try loading the commandline path!
bob.bio.base@2021-02-11 15:28:43,871 -- WARNING: The variable 'extractor_function' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_extractor_function') to suppress this warning
bob.bio.base@2021-02-11 15:28:43,871 -- WARNING: The variable 'network' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_network') to suppress this warning
bob.bio.base@2021-02-11 15:28:43,871 -- WARNING: The variable 'rc' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_rc') to suppress this warning
bob.pad.base@2021-02-11 15:28:44,107 -- ERROR: During the execution, an exception was raised: [Errno 2] No such file or directory: 'qsub': 'qsub'
Traceback (most recent call last):
  File "./bin/spoof.py", line 20, in <module>
    sys.exit(bob.pad.base.script.spoof.main())
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 293, in main
    detect_spoofing(args, command_line_parameters)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 240, in detect_spoofing
    retval = add_jobs(args, submitter)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 66, in add_jobs
    **args.grid.preprocessing_queue)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/bio/base/tools/grid.py", line 108, in submit
    **kwargs
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/sge.py", line 107, in submit
    job_id = self._submit_to_grid(job, name, array, dependencies, log_dir, verbosity, **kwargs)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/sge.py", line 71, in _submit_to_grid
    grid_id = qsub(command, context=self.context, name=name, deps=deps, array=q_array, stdout=log_dir, stderr=log_dir, **kwargs)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/tools.py", line 263, in qsub
    jobid = str_(sexec(context, scmd))
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/setshell.py", line 100, in sexec
    stderr=subprocess.STDOUT, env=E)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'qsub': 'qsub'

anjit...@gmail.com 在 2021年2月15日 星期一上午2:41:23 [UTC+8] 的信中寫道：

[anjit...@gmail.com]

Hi, 

The `--grid idiap` option works only at Idiap with SGE, you can omit that and run this as

./bin/spoof.py protocol-grandtest mccnn-occl-gmm --execute-only preprocessing --sub-directory ~/Documents/data/WMCAtest/ 

BTW, have you obtained the raw data already?, if you have downloaded the preprocessed data, then there is no need of this preprocessing step. 

Anjith

[Shen Tak Wai]

Yes, I have preprocessed data only. It includes "preprocessed-face-station_CDIT", "preprocessed-face-station_RGB" and a "documentation" folder.

"preprocessed-face-station_CDIT" and "preprocessed-face-station_RGB" folders have HDF5 data files.

~/.bob_bio_databases.txt is changed to

[YOUR_WMCA_DB_DIRECTORY] = /databases/WMCA/preprocessed-face-station_CDIT/

[YOUR_WMCA_ANNOTATIONS_DIRECTORY] = /databases/WMCA/documentation/

~/.bobrc is changed to

{

"bob.db.batl.directory": "~/Documents/data/WMCA/",
"bob.pad.face.database.batl.annotations_temp_dir": "~/Documents/data/WMCA/documentation/",
"bob.paper.oneclass_mccnn_2019.preprocessed_directory: "~/Documents/data/WMCA/face-station/,
"bob.paper.oneclass_mccnn_2019.cnn_output_dir": "~/cnn_outputdir/"

}

I try to run baselines in grandtest protocol of WMCA

$ ./bin/spoof.py      protocol-grandtest      mc-haralick-svm       --sub-directory ../grandtest/MC-Haralick-SVM/ -vvv

It showed below error. is it config problem?

Thanks and Regards,

Wai

$ ./bin/spoof.py      protocol-grandtest      mc-haralick-svm       --sub-directory ../grandtest/MC-Haralick-SVM/ -vvv

bob.extension.config@2021-02-19 11:26:53,618 -- DEBUG: Loading configuration file `/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/bob/paper/oneclass_mccnn_2019/config/Baselines/haralick_svm.py'...
bob.bio.base@2021-02-19 11:26:54,314 -- WARNING: The variable 'rc' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_rc') to suppress this warning
bob.pad.base@2021-02-19 11:26:54,632 -- INFO: - Preprocessing: processing 1679 data files from directory 'None' to directory 'temp/../grandtest/MC-Haralick-SVM/preprocessed'
bob.pad.base@2021-02-19 11:26:54,715 -- DEBUG: ... Processing original data file 'face-station/01.02.18/002_01_000_0_00.h5'
bob.pad.base@2021-02-19 11:26:54,717 -- ERROR: During the execution, an exception was raised: HDF5File - hdf5 constructor: C++ exception caught: 'cannot open file `face-station/01.02.18/002_01_000_0_00.h5''

Traceback (most recent call last):
  File "./bin/spoof.py", line 20, in <module>
    sys.exit(bob.pad.base.script.spoof.main())
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 293, in main
    detect_spoofing(args, command_line_parameters)

  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 267, in detect_spoofing
    if not execute(args):
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 161, in execute
    force=args.force)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/tools/preprocessor.py", line 86, in preprocess
    file_object, original_directory, original_extension)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/bio/video/preprocessor/Wrapper.py", line 66, in _read_video_data
    return biofile.load(directory, extension, frame_selector)
  File "/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/bob/paper/oneclass_mccnn_2019/database/batl.py", line 163, in load
    max_frames=self.max_frames)
  File "/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/src/bob.db.batl/bob/db/batl/models.py", line 187, in load
    with HDF5File(filepath, 'r') as hdf5_file:
RuntimeError: HDF5File - hdf5 constructor: C++ exception caught: 'cannot open file `face-station/01.02.18/002_01_000_0_00.h5''

anjit...@gmail.com 在 2021年2月16日 星期二下午5:02:41 [UTC+8] 的信中寫道：

[anjit...@gmail.com]

Hi, 

Please set all these variables to the <FOLDER_CONTAINING_PREPROCESSED_FILES> files:

bob.db.batl.directory, YOUR_WMCA_DB_DIRECTORY, bob.paper.oneclass_mccnn_2019.preprocessed_directory

For example: if the absolute path of one  file is:  ~/WMCA/preprocessed-face-station_CDIT/01.02.18/01.02.18$ 001_01_002_2_08.hdf5

Rename the folder "preprocessed-face-station_CDIT" to "face-station", and then the <FOLDER_CONTAINING_PREPROCESSED_FILES> is "~/WMCA/"

i.e., the folder containing the "face-station" folder. 

Now set this value for bob.db.batl.directory, YOUR_WMCA_DB_DIRECTORY, bob.paper.oneclass_mccnn_2019.preprocessed_directory

Regards, 

Anjith