Defining my own Database

[Guillermo Estrada]

Hi,

I'm trying to develop an interface for my face PAD database by following the How-To Guide provided in https://www.idiap.ch/software/bob/docs/bob/docs/stable//bob.pad.base/doc/implementation.html.

I got to implement the *File and *Database classes by following High Level Database Interface How-To Guide (https://www.idiap.ch/software/bob/docs/bob/docs/stable//bob.pad.base/doc/high_level_db_interface_guide.html).

However, I'm stuck in the stage of registering my new database as a new resource. Specially, I don't know what setup.py file of the bob.pad.face to modify in order to install the new database.

Thanks in advance,
Guillermo

[Amir Mohammadi]

Hi Guillermo,

You do not necessarily need to register your database as a new resource. Just pass the path to a file that contains the configuration file to spoof.py

I hope that helps.

Moreover, I would recommend that you create a file-list based database. It's very easy to create. You can see an example here: https://gitlab.idiap.ch/bob/bob.db.oulunpu/ . Once you create and install a package like bob.db.oulunpu, there is no need to modify bob.pad.face to register your database.

Thanks,

Amir

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guille]

Dear Amir,

Thanks a lot for your quick reply. I will modify my database by following the bob.db.oulunpu example package. I will notify you as soon as I have a result.

By the way, I tried to install bob.db.oulunpu package by running $ conda install bob.db.oulunpu in order to test it and to understand how it works. But I got the following message from conda:

PackagesNotFoundError: The following packages are not available from current channels:

  - bob.db.oulunpu

Current channels:

  - https://www.idiap.ch/software/bob/conda/linux-64
  - https://www.idiap.ch/software/bob/conda/noarch

Please, how could I install this package?

Best,
Guillermo

2018-05-14 12:12 GMT-03:00 Amir Mohammadi <183....@gmail.com>:

Hi Guillermo,

You do not necessarily need to register your database as a new resource. Just pass the path to a file that contains the configuration file to spoof.py

I hope that helps.

Moreover, I would recommend that you create a file-list based database. It's very easy to create. You can see an example here: https://gitlab.idiap.ch/bob/bob.db.oulunpu/ . Once you create and install a package like bob.db.oulunpu, there is no need to modify bob.pad.face to register your database.

Thanks,

Amir

On Sun, May 13, 2018 at 3:44 AM Guillermo Estrada <guil...@gmail.com> wrote:

Hi,

I'm trying to develop an interface for my face PAD database by following the How-To Guide provided in https://www.idiap.ch/software/bob/docs/bob/docs/stable//bob.pad.base/doc/implementation.html.

I got to implement the *File and *Database classes by following High Level Database Interface How-To Guide (https://www.idiap.ch/software/bob/docs/bob/docs/stable//bob.pad.base/doc/high_level_db_interface_guide.html).

However, I'm stuck in the stage of registering my new database as a new resource. Specially, I don't know what setup.py file of the bob.pad.face to modify in order to install the new database.

Thanks in advance,
Guillermo

--

-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---

You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to a topic in the Google Groups "bob-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bob-devel/Nvw6oQQNp98/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bob-devel+unsubscribe@googlegroups.com.

[Amir Mohammadi]

Hi,

This package has not been released yet.

You can install it using buildout or pip from source. 

Best,

Amir

-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---

You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---

You received this message because you are subscribed to a topic in the Google Groups "bob-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bob-devel/Nvw6oQQNp98/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bob-devel+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.

[Guillermo Estrada]

Hi Amir,

Thank you very much for your help. I created my database based on file list like in the bob.db.oulunpu example you suggested me. I installed my package locally using buildout.

In order to test my package I created two configurations files: the first, containing the object of my database (Database object, protocol and groups); the second, containing a preprocessor object to do FaceCropAlign on the input videos.

I run ./bin/spoof.py database_config.py preprocessor_config.py and I got the following error:

bob.pad.base@2018-05-15 08:13:12,292 -- ERROR: During the execution, an exception was raised: all_files() got an unexpected keyword argument 'add_zt_files'
Traceback (most recent call last):
  File "./bin/spoof.py", line 26, in <module>
    sys.exit(bob.pad.base.script.spoof.main())
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/pad/base/script/spoof.py", line 292, in main
    detect_spoofing(args, command_line_parameters)
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/pad/base/script/spoof.py", line 266, in detect_spoofing
    if not execute(args):
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/pad/base/script/spoof.py", line 158, in execute
    indices=biotools.indices(fs.original_data_list(groups=tools.groups(args)),
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/bio/base/tools/FileSelector.py", line 113, in original_data_list
    return self.database.all_files(groups=groups,add_zt_files=self.zt_norm)
TypeError: all_files() got an unexpected keyword argument 'add_zt_files'

Please, how could I address this problem?

Best,
Guillermo

[Amir Mohammadi]

Are you using bob 4?

Could you please send the output of "conda list bob"?

[Guillermo Estrada]

This is the output of "conda list bob":

# packages in environment at /home/guillermo/miniconda2/envs/bob_py3:
#
# Name                    Version                   Build  Channel
bob                       4.0.1            py36h2662e99_0    https://www.idiap.ch/software/bob/conda
bob.bio.base              3.3.0            py36hc028e29_0    https://www.idiap.ch/software/bob/conda
bob.bio.face              3.4.0            py36h25833c4_0    https://www.idiap.ch/software/bob/conda
bob.bio.video             3.4.1            py36hd9b8dee_0    https://www.idiap.ch/software/bob/conda
bob.blitz                 2.0.15           py36h42d9ab3_0    https://www.idiap.ch/software/bob/conda
bob.buildout              2.1.6            py36hdaa9b1e_0    https://www.idiap.ch/software/bob/conda
bob.core                  2.2.0            py36h42d9ab3_0    https://www.idiap.ch/software/bob/conda
bob.db.atnt               2.0.11           py36hd9b8dee_0    https://www.idiap.ch/software/bob/conda
bob.db.base               3.0.0            py36he3fedb2_0    https://www.idiap.ch/software/bob/conda
bob.db.msu_mfsd_mod       2.2.4            py36hd9b8dee_0    https://www.idiap.ch/software/bob/conda
bob.db.replay             3.0.4            py36hc1cdd12_0    https://www.idiap.ch/software/bob/conda
bob.db.replaymobile       1.1.4            py36hc1cdd12_0    https://www.idiap.ch/software/bob/conda
bob.extension             2.5.0            py36hf498ef1_0    https://www.idiap.ch/software/bob/conda
bob.io.base               3.0.3            py36hf5c1720_0    https://www.idiap.ch/software/bob/conda
bob.io.image              2.4.1            py36h9deff11_0    https://www.idiap.ch/software/bob/conda
bob.io.matlab             2.0.11           py36h3fe5594_0    https://www.idiap.ch/software/bob/conda
bob.io.video              2.1.1            py36he386153_0    https://www.idiap.ch/software/bob/conda
bob.ip.base               2.2.1            py36h191f71d_0    https://www.idiap.ch/software/bob/conda
bob.ip.color              2.0.11           py36h5093ba0_0    https://www.idiap.ch/software/bob/conda
bob.ip.dlib               1.0.4            py36h0862093_0    https://www.idiap.ch/software/bob/conda
bob.ip.draw               2.0.10           py36h63be7c4_0    https://www.idiap.ch/software/bob/conda
bob.ip.facedetect         2.1.5            py36hf256c7c_0    https://www.idiap.ch/software/bob/conda
bob.ip.facelandmarks      1.0.5            py36h2fbd5ee_0    https://www.idiap.ch/software/bob/conda
bob.ip.flandmark          2.1.7            py36hf256c7c_0    https://www.idiap.ch/software/bob/conda
bob.ip.gabor              2.0.12           py36h8b1e1b8_0    https://www.idiap.ch/software/bob/conda
bob.ip.qualitymeasure     1.1.0            py36h0b81b6f_0    https://www.idiap.ch/software/bob/conda
bob.learn.activation      2.0.11           py36h5093ba0_0    https://www.idiap.ch/software/bob/conda
bob.learn.boosting        2.0.14           py36h5c8395a_0    https://www.idiap.ch/software/bob/conda
bob.learn.em              2.1.1            py36he293cb3_0    https://www.idiap.ch/software/bob/conda
bob.learn.libsvm          2.1.0            py36h38daa3c_0    https://www.idiap.ch/software/bob/conda
bob.learn.linear          2.1.3            py36hba7be54_0    https://www.idiap.ch/software/bob/conda
bob.math                  3.1.1            py36h89a30c5_0    https://www.idiap.ch/software/bob/conda
bob.measure               3.0.0            py36h82bf406_0    https://www.idiap.ch/software/bob/conda
bob.pad.base              2.0.0            py36hd9b8dee_0    https://www.idiap.ch/software/bob/conda
bob.pad.face              2.0.1            py36hd9b8dee_0    https://www.idiap.ch/software/bob/conda
bob.sp                    2.0.11           py36hebb8551_0    https://www.idiap.ch/software/bob/conda

[Amir Mohammadi]

This is really weird. Could you please share your database_config.py preprocessor_config.py and the buildout configuration file too?

[Guillermo Estrada]

 ====== database_config.py contains ======

from bob.db.spoofdb import Database

ORIGINAL_DIRECTORY = "/home/guillermo/databases/SPOOFDB"

ORIGINAL_EXTENSION = ".avi"  # extension of the data files

database = Database(original_directory=ORIGINAL_DIRECTORY, original_extension=ORIGINAL_EXTENSION,

                    protocol='Protocol')

protocol = 'Protocol'

groups = ["train", "dev", "eval"]

 ====== preprocessor_config.py contains ======

sub_directory = 'results'

from bob.pad.face.preprocessor import FaceCropAlign
from bob.bio.video.preprocessor import Wrapper
from bob.bio.video.utils import FrameSelector

RGB_OUTPUT_FLAG = True # RGB output
USE_FACE_ALIGNMENT = False # use annotations
MAX_IMAGE_SIZE = None # no limiting here
FACE_DETECTION_METHOD = None # use annotations
MIN_FACE_SIZE = 50 # skip small faces

_image_preprocessor = FaceCropAlign(face_size = FACE_SIZE,
                                   rgb_output_flag = RGB_OUTPUT_FLAG,
                                   use_face_alignment = USE_FACE_ALIGNMENT,
                                   max_image_size = MAX_IMAGE_SIZE,
                                   face_detection_method = FACE_DETECTION_METHOD,
                                   min_face_size = MIN_FACE_SIZE)

_frame_selector = FrameSelector(selection_style = "all")

preprocessor = Wrapper(preprocessor = _image_preprocessor,
                       frame_selector = _frame_selector)

====== buildout configuration file ======

[buildout]
parts = scripts
develop = .
eggs = bob.db.spoofdb
extensions = bob.buildout
newest = false
verbose = true

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

===== Interesting Issue ======

Maybe this is not important, but during a testing of the example bob.db.oulunpu using an interactive session (./bin/python) I run the following code:

>>> from bob.db.spoofdb import Database

>>> ORIGINAL_DIRECTORY = "/home/guillermo/databases/SPOOFDB"

>>> db = Database(original_directory=ORIGINAL_DIRECTORY, protocol='Protocol')

>>> all_files = db.objects(protocol='Protocol')

>>> f = all_files[0]

>>> f.number_of_frames

>>> video = f.load()

at this line I got this error:

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/pad/face/database/database.py", line 36, in load
    return frame_selector(self.make_path(directory, extension))
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/bio/video/utils/FrameSelector.py", line 55, in __call__
    data = load_function(data)
  File "/home/guillermo/miniconda2/envs/bob_py3/lib/python3.6/site-packages/bob/io/base/__init__.py", line 149, in load
    return File(inputs, 'r').read()
RuntimeError: File - constructor: C++ exception caught: 'bob::io::video::avformat_open_input(filename=`train/1_1_01_1.avi') failed: ffmpeg reported -2 == `No such file or directory''

However, I resolved calling the load methods with the original directory as:

>>> video = f.load(ORIGINAL_DIRECTORY)

>>> type(video)
<class 'bob.bio.video.utils.FrameContainer.FrameContainer'>

Is this issue related with the TypeError: all_files() got an unexpected keyword argument 'add_zt_files' I described in the post above or the way to load the data files?

Best,

Guillermo

[Amir Mohammadi]

Guillermo,

That's not an issue. The load method needs the original directory.

It's implemented that way. It could have been implemented differently.

You can implement your database in a way that `video = f.load()` would

work.

What I don't understand is that your code is using the FileSelector class from bob.bio.base instead of bob.pad.base.

Looking at the code, the tools is imported locally https://gitlab.idiap.ch/bob/bob.pad.base/blob/v2.0.0/bob/pad/base/script/spoof.py#L18

and the fileselector (fs) is created from tools https://gitlab.idiap.ch/bob/bob.pad.base/blob/v2.0.0/bob/pad/base/script/spoof.py#L141

and the original_data_list of FS in bob.pad.base does not call db.all_files with zt_norm:
https://gitlab.idiap.ch/bob/bob.pad.base/blob/v2.0.0/bob/pad/base/tools/FileSelector.py#L102

However, I don't understand why your code (according to the traceback you provided) is going to bob.bio.base.tools instead.

[Guillermo Estrada]

Hi Amir,

I found why my code is using the FileSelector class from bob.bio.base instead of bob.pad.base. Sorry, I had modified the line 18 from .. import tools by from bob.bio.base import tools and I forgot to correct it.

After modified that line, I run my pipeline and I have not any error until now.

Thank you very much for all your help!

Best,

Guillermo