Error while Preprocessing data for bob.paper.oneclass_mccnn_2019

57 views
Skip to first unread message

Shen Tak Wai

unread,
Feb 10, 2021, 11:49:37 PM2/10/21
to bob-devel
Dear Bob developers,

I have a problem with the project bob.paper.oneclass_mccnn_2019.  According to the file bob.paper.oneclass_mccnn_2019/doc/running_mccnn-occl-gmm.rstm, I run the spoof.py

 ./bin/spoof.py protocol-grandtest mccnn-occl-gmm --execute-only preprocessing --sub-directory ~/Documents/data/WMCAtest/ --grid idiap

However, it showed
bob.learn.pytorch@2021-02-11 10:44:36,774 -- DEBUG: No pretrained file provided in the config, will try loading the commandline path!
Please advise how to config it. Thanks!

Best regards,
Wai

anjit...@gmail.com

unread,
Feb 14, 2021, 1:41:23 PM2/14/21
to bob-devel
Hi, 
You can Ignore this warning at the preprocessing stage. Is the pipeline running otherwise?

Anjith

Shen Tak Wai

unread,
Feb 14, 2021, 10:10:37 PM2/14/21
to bob-devel
Dear Anjith,

Thanks for your reply!
Only a empty folder "gridtk_logs/preprocess/" is obtained, so I guess I make something wrong in config.
Below shows the error messages.

Best Regards,
Wai

$ ./bin/spoof.py protocol-grandtest mccnn-occl-gmm --execute-only preprocessing --sub-directory ~/Documents/data/WMCAtest/ --grid idiap
bob.extension.config@2021-02-11 15:28:42,082 -- DEBUG: Loading configuration file `/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/bob/paper/oneclass_mccnn_2019/config/Method/Pipeline.py'...
bob.pad.face@2021-02-11 15:28:43,354 -- INFO: Light_cnn_model_file path: /nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/learn/pytorch/models/LightCNN_29Layers_checkpoint.pth.tar
extractor_file None
bob.learn.pytorch@2021-02-11 15:28:43,551 -- DEBUG: No pretrained file provided in the config, will try loading the commandline path!
bob.bio.base@2021-02-11 15:28:43,871 -- WARNING: The variable 'extractor_function' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_extractor_function') to suppress this warning
bob.bio.base@2021-02-11 15:28:43,871 -- WARNING: The variable 'network' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_network') to suppress this warning
bob.bio.base@2021-02-11 15:28:43,871 -- WARNING: The variable 'rc' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_rc') to suppress this warning
bob.pad.base@2021-02-11 15:28:44,107 -- ERROR: During the execution, an exception was raised: [Errno 2] No such file or directory: 'qsub': 'qsub'
Traceback (most recent call last):
  File "./bin/spoof.py", line 20, in <module>
    sys.exit(bob.pad.base.script.spoof.main())
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 293, in main
    detect_spoofing(args, command_line_parameters)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 240, in detect_spoofing
    retval = add_jobs(args, submitter)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 66, in add_jobs
    **args.grid.preprocessing_queue)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/bio/base/tools/grid.py", line 108, in submit
    **kwargs
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/sge.py", line 107, in submit
    job_id = self._submit_to_grid(job, name, array, dependencies, log_dir, verbosity, **kwargs)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/sge.py", line 71, in _submit_to_grid
    grid_id = qsub(command, context=self.context, name=name, deps=deps, array=q_array, stdout=log_dir, stderr=log_dir, **kwargs)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/tools.py", line 263, in qsub
    jobid = str_(sexec(context, scmd))
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/gridtk/setshell.py", line 100, in sexec
    stderr=subprocess.STDOUT, env=E)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'qsub': 'qsub'


anjit...@gmail.com 在 2021年2月15日 星期一上午2:41:23 [UTC+8] 的信中寫道:

anjit...@gmail.com

unread,
Feb 16, 2021, 4:02:41 AM2/16/21
to bob-devel
Hi, 

The `--grid idiap` option works only at Idiap with SGE, you can omit that and run this as
./bin/spoof.py protocol-grandtest mccnn-occl-gmm --execute-only preprocessing --sub-directory ~/Documents/data/WMCAtest/ 
BTW, have you obtained the raw data already?, if you have downloaded the preprocessed data, then there is no need of this preprocessing step. 
Anjith

Shen Tak Wai

unread,
Feb 18, 2021, 10:52:34 PM2/18/21
to bob-devel
Yes, I have preprocessed data only. It includes "preprocessed-face-station_CDIT", "preprocessed-face-station_RGB" and a "documentation" folder.
"preprocessed-face-station_CDIT" and "preprocessed-face-station_RGB" folders have HDF5 data files.

~/.bob_bio_databases.txt is changed to
[YOUR_WMCA_DB_DIRECTORY] = /databases/WMCA/preprocessed-face-station_CDIT/
[YOUR_WMCA_ANNOTATIONS_DIRECTORY] = /databases/WMCA/documentation/

~/.bobrc is changed to
{
"bob.db.batl.directory": "~/Documents/data/WMCA/",
"bob.pad.face.database.batl.annotations_temp_dir": "~/Documents/data/WMCA/documentation/",
"bob.paper.oneclass_mccnn_2019.preprocessed_directory: "~/Documents/data/WMCA/face-station/,
"bob.paper.oneclass_mccnn_2019.cnn_output_dir": "~/cnn_outputdir/"
}

I try to run baselines in grandtest protocol of WMCA
$ ./bin/spoof.py      protocol-grandtest      mc-haralick-svm       --sub-directory ../grandtest/MC-Haralick-SVM/ -vvv

It showed below error. is it config problem?
Thanks and Regards,
Wai


$ ./bin/spoof.py      protocol-grandtest      mc-haralick-svm       --sub-directory ../grandtest/MC-Haralick-SVM/ -vvv
bob.extension.config@2021-02-19 11:26:53,618 -- DEBUG: Loading configuration file `/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/bob/paper/oneclass_mccnn_2019/config/Baselines/haralick_svm.py'...
bob.bio.base@2021-02-19 11:26:54,314 -- WARNING: The variable 'rc' in a configuration file is not known or not supported by this application; use a '_' prefix to the variable name (e.g., '_rc') to suppress this warning
bob.pad.base@2021-02-19 11:26:54,632 -- INFO: - Preprocessing: processing 1679 data files from directory 'None' to directory 'temp/../grandtest/MC-Haralick-SVM/preprocessed'
bob.pad.base@2021-02-19 11:26:54,715 -- DEBUG: ... Processing original data file 'face-station/01.02.18/002_01_000_0_00.h5'
bob.pad.base@2021-02-19 11:26:54,717 -- ERROR: During the execution, an exception was raised: HDF5File - hdf5 constructor: C++ exception caught: 'cannot open file `face-station/01.02.18/002_01_000_0_00.h5''

Traceback (most recent call last):
  File "./bin/spoof.py", line 20, in <module>
    sys.exit(bob.pad.base.script.spoof.main())
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 293, in main
    detect_spoofing(args, command_line_parameters)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 267, in detect_spoofing
    if not execute(args):
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/script/spoof.py", line 161, in execute
    force=args.force)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/pad/base/tools/preprocessor.py", line 86, in preprocess
    file_object, original_directory, original_extension)
  File "/nfs/home/twshen/anaconda3/envs/one_class_MCCNN/lib/python3.7/site-packages/bob/bio/video/preprocessor/Wrapper.py", line 66, in _read_video_data
    return biofile.load(directory, extension, frame_selector)
  File "/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/bob/paper/oneclass_mccnn_2019/database/batl.py", line 163, in load
    max_frames=self.max_frames)
  File "/nfs/home/twshen/Documents/bob.paper.oneclass_mccnn_2019/src/bob.db.batl/bob/db/batl/models.py", line 187, in load
    with HDF5File(filepath, 'r') as hdf5_file:
RuntimeError: HDF5File - hdf5 constructor: C++ exception caught: 'cannot open file `face-station/01.02.18/002_01_000_0_00.h5''





anjit...@gmail.com 在 2021年2月16日 星期二下午5:02:41 [UTC+8] 的信中寫道:

anjit...@gmail.com

unread,
Feb 22, 2021, 3:54:12 AM2/22/21
to bob-devel
Hi, 

Please set all these variables to the <FOLDER_CONTAINING_PREPROCESSED_FILES> files:

bob.db.batl.directory, YOUR_WMCA_DB_DIRECTORY, bob.paper.oneclass_mccnn_2019.preprocessed_directory

For example: if the absolute path of one  file is:  ~/WMCA/preprocessed-face-station_CDIT/01.02.18/01.02.18$ 001_01_002_2_08.hdf5
Rename the folder "preprocessed-face-station_CDIT" to "face-station", and then the <FOLDER_CONTAINING_PREPROCESSED_FILES> is "~/WMCA/"
i.e., the folder containing the "face-station" folder. 

Now set this value for bob.db.batl.directory, YOUR_WMCA_DB_DIRECTORY, bob.paper.oneclass_mccnn_2019.preprocessed_directory
Regards, 
Anjith
Reply all
Reply to author
Forward
0 new messages