Invalid fields in the search

86 views
Skip to first unread message

Arye Kalter

unread,
Jun 23, 2020, 7:26:03 PM6/23/20
to phrasea-install-en

I installed version 4.1.1 using docker.
I imported lot of videos and photos into the system.
I cannot find anything using the searchbox, but when i want to see all without filter i can see it correctly.
I checked the search that the system send to elasticsearch, looks like the fields are invalid.
In the query, i see:
          "fields": [
            "caption.Object.de",
            "caption.Object.en^5",
            "caption.Object.fr",
            "caption.Object.nl",
            "caption.Object.light^10",
            "caption.Object.truncated",
            "caption.Category.de",
            "caption.Category.en^5",
            "caption.Category.fr",
            "caption.Category.nl",
            "caption.Category.light^10",
            "caption.Category.truncated",
            "caption.SupplCategory.de",
            "caption.SupplCategory.en^5",
            "caption.SupplCategory.fr",
            "caption.SupplCategory.nl",
            "caption.SupplCategory.light^10",
            "caption.SupplCategory.truncated",
            "caption.Keywords.de",
            "caption.Keywords.en^5",
            "caption.Keywords.fr",
            "caption.Keywords.nl",
            "caption.Keywords.light^10",
            "caption.Keywords.truncated",
            "caption.SpecialInstruct.de",
            "caption.SpecialInstruct.en^5",
            "caption.SpecialInstruct.fr",
            "caption.SpecialInstruct.nl",
            "caption.SpecialInstruct.light^10",
            "caption.SpecialInstruct.truncated",
            "caption.Date.light",
            "caption.Date",
            "caption.Date.truncated",
            "caption.Byline.de",
            "caption.Byline.en^5",
            "caption.Byline.fr",
            "caption.Byline.nl",
            "caption.Byline.light^10",
            "caption.Byline.truncated",
            "caption.BylineTitle.de",
            "caption.BylineTitle.en^5",
            "caption.BylineTitle.fr",
            "caption.BylineTitle.nl",
            "caption.BylineTitle.light^10",
            "caption.BylineTitle.truncated",
            "caption.City.de",
            "caption.City.en^5",
            "caption.City.fr",
            "caption.City.nl",
            "caption.City.light^10",
            "caption.City.truncated",
            "caption.Province.de",
            "caption.Province.en^5",
            "caption.Province.fr",
            "caption.Province.nl",
            "caption.Province.light^10",
            "caption.Province.truncated",
            "caption.Country.de",
            "caption.Country.en^5",
            "caption.Country.fr",
            "caption.Country.nl",
            "caption.Country.light^10",
            "caption.Country.truncated",
            "caption.OriginalRef.de",
            "caption.OriginalRef.en^5",
            "caption.OriginalRef.fr",
            "caption.OriginalRef.nl",
            "caption.OriginalRef.light^10",
            "caption.OriginalRef.truncated",
            "caption.Headline.de",
            "caption.Headline.en^5",
            "caption.Headline.fr",
            "caption.Headline.nl",
            "caption.Headline.light^10",
            "caption.Headline.truncated",
            "caption.Credit.de",
            "caption.Credit.en^5",
            "caption.Credit.fr",
            "caption.Credit.nl",
            "caption.Credit.light^10",
            "caption.Credit.truncated",
            "caption.Source.de",
            "caption.Source.en^5",
            "caption.Source.fr",
            "caption.Source.nl",
            "caption.Source.light^10",
            "caption.Source.truncated",
            "caption.Caption.de",
            "caption.Caption.en^5",
            "caption.Caption.fr",
            "caption.Caption.nl",
            "caption.Caption.light^10",
            "caption.Caption.truncated",
            "caption.CaptionWriter.de",
            "caption.CaptionWriter.en^5",
            "caption.CaptionWriter.fr",
            "caption.CaptionWriter.nl",
            "caption.CaptionWriter.light^10",
            "caption.CaptionWriter.truncated",
            "caption.Longitude.light",
            "caption.Longitude",
            "caption.Longitude.truncated",
            "caption.Latitude.light",
            "caption.Latitude",
            "caption.Latitude.truncated",
            "caption.CameraModel.de",
            "caption.CameraModel.en^5",
            "caption.CameraModel.fr",
            "caption.CameraModel.nl",
            "caption.CameraModel.light^10",
            "caption.CameraModel.truncated",
            "caption.FileName.de",
            "caption.FileName.en^5",
            "caption.FileName.fr",
            "caption.FileName.nl",
            "caption.FileName.light^10",
            "caption.FileName.truncated",
            "caption.Caption2.de",
            "caption.Caption2.en^5",
            "caption.Caption2.fr",
            "caption.Caption2.nl",
            "caption.Caption2.light^10",
            "caption.Caption2.truncated"
          ],

But in elastic search i dont see any of the variables in any of the records.
Example record from elastic search:

      "_source" : {
        "record_id" : 33,
        "collection_id" : 5,
        "uuid" : "a0cc6f00-b2dc-44cc-a01b-45aaccbf05e8",
        "flags_bitfield" : 0,
        "sha256" : "22f7fee1e123bd3f91fe114ffacc952ee20552f809d9896fb43c8130a2a95d84",
        "original_name" : "FILE NAME.jpg",
        "mime" : "image/jpeg",
        "type" : "image",
        "created_on" : "2020-06-21 00:35:21",
        "updated_on" : "2020-06-21 00:35:22",
        "coll_id" : "5",
        "collection_name" : "Photos",
        "width" : 1024,
        "height" : 683,
        "size" : 118815,
        "base_id" : "6",
        "databox_id" : 1,
        "databox_name" : "DATABOX NAME",
        "record_type" : "record",
        "title" : {
          "default" : "The title"
        },
        "caption" : {
          "Title" : [ "The title" ],
          "Creator" : [ "Name of the creator" ],
          "Subject" : [ "Historical", "war", "military" ],
          "Description" : [ "Paragraph" ],
          "Rights" : [ "copyright (c) 2017" ],
          "Filename" : [ "<THE FILE NAME>.jpg" ],
          "ArchiveDate" : [ "2020-06-21 00:35:22" ]
        },
        "caption_all" : [ ... some strings ... ],
        "metadata_tags" : {
          "Aperture" : 4,
          "CameraModel" : "Canon EOS 30D",
          "Channels" : 3,
          "ColorDepth" : 8,
          "ColorSpace" : 0,
          "FileSize" : 118815,
          "FlashFired" : false,
          "FocalLength" : 25,
          "Height" : 683,
          "HyperfocalDistance" : 8.3282760009793,
          "ISO" : 1600,
          "LightValue" : 4.6438561897747,
          "MimeType" : "image/jpeg",
          "Orientation" : 0,
          "ShutterSpeed" : 0.04,
          "Width" : 1024
        },
        "flags" : {
          "caption_filled" : false
        },
        "subdefs" : {
          "preview" : {
            "path" : "/var/alchemy/Phraseanet/datas/db_databox1/subdefs/2020/06/21/00000/33_preview.jpg",
            "width" : "1024",
            "height" : "683"
          },
          "thumbnail" : {
            "path" : "/var/alchemy/Phraseanet/datas/db_databox1/subdefs/2020/06/21/00000/33_thumbnail.jpg",
            "width" : "240",
            "height" : "160"
          }
        }
      },

I tried to add "caption.Title" into the fields array and send it directly to elasticsearch, then elasticsearch answered with some items.

The question, why the system is trying to search for invalid fields, and how can i solve it to make the search work as expected.

Gaspard Jacobson

unread,
Jun 24, 2020, 9:42:38 AM6/24/20
to phrasea-install-en
Hello,

As you have imported new documents to Phraseanet, you have to receate the elasticsearch index : On the Admin panel, click on Search engine settings and drop index. You can then create it.
You will also need to populate the newly created index as the documentary structure of the database as changed. To do so, on the Admin panel, go to Worker Manager and on the indexing tab choose the Databox to populate and click on indexing.

Also check in the production interface that you are looking for documents and not storie mode without any search filters on the type of documents (video, photo etc ...).
There is generally no operation to do directly in elasticsearch to make Phraseanet work.

I hope this will help.

Best regards,

Gaspard Jacobson

אריה קלטר

unread,
Jun 24, 2020, 6:48:10 PM6/24/20
to phrasea-i...@googlegroups.com
I tried to do both things:
drop the index and create it.
And i went to worker manager, indexing tab and clicked on the index button.

I did it before something like 5 hours, and i still dont see anything, just 5 items instead of more than 5243 records (in one collection i have 5243, in the second i have something like 25 records)
Before i deleted the index i saw all the 5243 records but i did not able to search for them, now i cannot see them at all.

Maybe it just still working to recreate the index?
How can i know?

‫בתאריך יום ד׳, 24 ביוני 2020 ב-16:42 מאת ‪Gaspard Jacobson‬‏ <‪gjac...@alchemy.fr‬‏>:‬
--
You received this message because you are subscribed to the Google Groups "phrasea-install-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/phrasea-install-en/eb2ebc63-8038-4057-8333-40830a96a518o%40googlegroups.com.

MD

unread,
Jun 25, 2020, 4:46:29 AM6/25/20
to phrasea-install-en
Hi, 

From the informations you give it seems like it is not a fresh install but an update !
Can you please confirme.
If yes from what version of Phraseanet ?
Have you ran an upgrade of the application using the bin/setup commande ?

Best regards

MD

Le jeudi 25 juin 2020 00:48:10 UTC+2, Arye Kalter a écrit :
I tried to do both things:
drop the index and create it.
And i went to worker manager, indexing tab and clicked on the index button.

I did it before something like 5 hours, and i still dont see anything, just 5 items instead of more than 5243 records (in one collection i have 5243, in the second i have something like 25 records)
Before i deleted the index i saw all the 5243 records but i did not able to search for them, now i cannot see them at all.

Maybe it just still working to recreate the index?
How can i know?

‫בתאריך יום ד׳, 24 ביוני 2020 ב-16:42 מאת ‪Gaspard Jacobs <
Hello,

As you have imported new documents to Phraseanet, you have to receate the elasticsearch index : On the Admin panel, click on Search engine settings and drop index. You can then create it.
You will also need to populate the newly created index as the documentary structure of the database as changed. To do so, on the Admin panel, go to Worker Manager and on the indexing tab choose the Databox to populate and click on indexing.

Also check in the production interface that you are looking for documents and not storie mode without any search filters on the type of documents (video, photo etc ...).
There is generally no operation to do directly in elasticsearch to make Phraseanet work.

I hope this will help.

Best regards,

Gaspard Jacobson

--
You received this message because you are subscribed to the Google Groups "phrasea-install-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install-en+unsub...@googlegroups.com.

MD

unread,
Jun 25, 2020, 6:03:58 AM6/25/20
to phrasea-install-en
Can you please see the logs of the Elasticsearch server.
The time it takes to populate the elasticsearch for ~ 5000 assets should be between 1 and 5 minutes.
Can you tel us about the installation environment ? (cpu, memory, Operating system, docker version ...) 
Dropping, recreating and populating the index can be made from the commande line interfaces.

with :

bin/console searchengine:index -dcp -vvv (-vvv for versose).


Last point : The Elasticsearch contener can be memory consumer. See below prerequisites on the host machine



Best regards

M.D

אריה קלטר

unread,
Jun 25, 2020, 3:01:58 PM6/25/20
to phrasea-i...@googlegroups.com
If it is clean install or not - i tried to make it as clean install, to remove all the things and create it from scrach.
But maybe i did not clean it correctly so it was partially upgrade.
If so, i cannot guess what left from the previous version and what was real clean.
I just know that i didn't have any documents after the "clean" install.
The old version was 4.0.0 or 4.0.1, the new version is 4.1.1.
Should i run the upgrade script in this case, to make sure that all is in the new version?

I executed the
bin/console searchengine:index -dcp -vvv (-vvv for versose).
command and it added all the data as expected.

Currently i am running it on c5n.xlarge EC2 instance, with EBS of SSD for the datas folder and all the other system files are on the root filesystem, in small SSD also.
What do you think?
It will work as expected with T2 instances?
I need to use SSD for the datas folder?
And what about other folders? maybe the mysql or the elasticsearch? temp folder?

Another question
In our previous system we inserted two texts, one into IPTC field 120 and one into IPTC field 121.
We want it to be in separate field, but currently it just append it to field 120 with seperator of "*** Local Caption ***"
How can i split it to another field?



‫בתאריך יום ה׳, 25 ביוני 2020 ב-13:04 מאת ‪MD‬‏ <‪pot...@gmail.com‬‏>:‬
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "phrasea-install-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/phrasea-install-en/2c26c9fe-65ac-483e-b771-8547f882638do%40googlegroups.com.

אריה קלטר

unread,
Jun 25, 2020, 3:09:10 PM6/25/20
to phrasea-i...@googlegroups.com
Maybe it will better to do clean install?
Or i can stay with the current installation?
If so, how can i clean it correctly?

‫בתאריך יום ה׳, 25 ביוני 2020 ב-22:01 מאת אריה קלטר <‪ary...@gmail.com‬‏>:‬

MD

unread,
Jun 26, 2020, 11:44:22 AM6/26/20
to phrasea-install-en
Hi, 

I think that you should keep your installation and make sure you've followed the "how to" upgrade process.

1 - make sure you have a backup of the MySQL database
2 - get the version of Phraseanet you want to install 
3 - switch the new source and the old ones
4 - copy the configuration directory content to the new sources "config/*"
5 - copy the lazaret and downloaded files fo the new sources "tmp/download and tmp/lazaret" 
6 - run the upgrade commande : bin/setup system:upgrade
7 - drop and recreate and populate the elasticsearch index : bin/console searchengine:index -dcp


Regards,

M.D
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install-en+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "phrasea-install-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install-en+unsub...@googlegroups.com.

אריה קלטר

unread,
Jun 27, 2020, 2:24:13 PM6/27/20
to phrasea-i...@googlegroups.com
I tried to run the upgrade script, it saying:
Upgrading... from version 4.1.1 to 4.1.1
                                                                                                                                     
  [UnexpectedValueException]                                                                                                          
  RecursiveDirectoryIterator::__construct(/var/alchemy/Phraseanet/config/custom_files): failed to open dir: No such file or directory



‫בתאריך יום ו׳, 26 ביוני 2020 ב-18:44 מאת ‪MD‬‏ <‪diouf...@gmail.com‬‏>:‬
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "phrasea-install-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "phrasea-install-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrasea-install...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/phrasea-install-en/53c7b35c-327a-4e26-bdbf-62f63763ca68o%40googlegroups.com.

אריה קלטר

unread,
Jun 27, 2020, 2:42:41 PM6/27/20
to phrasea-i...@googlegroups.com
And also, the backup script is not working.
It saying...


Phraseanet is going to be backup...
Generating config/db/ab_master_2020_06_27_18_39_53.sql ... Failed

                                                             
  [Symfony\Component\Debug\Exception\ContextErrorException]  
  Notice: Undefined variable: ok                            
                                                             

Exception trace:
 () at /var/alchemy/Phraseanet/lib/classes/module/console/systemBackupDB.php:45
 Symfony\Component\Debug\ErrorHandler->handleError() at /var/alchemy/Phraseanet/lib/classes/module/console/systemBackupDB.php:45
 module_console_systemBackupDB->doExecute() at /var/alchemy/Phraseanet/lib/Alchemy/Phrasea/Command/Command.php:63
 Alchemy\Phrasea\Command\Command->execute() at /var/alchemy/Phraseanet/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:264
 Symfony\Component\Console\Command\Command->run() at /var/alchemy/Phraseanet/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:868
 Symfony\Component\Console\Application->doRunCommand() at /var/alchemy/Phraseanet/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:191
 Symfony\Component\Console\Application->doRun() at /var/alchemy/Phraseanet/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:122
 Symfony\Component\Console\Application->run() at /var/alchemy/Phraseanet/lib/Alchemy/Phrasea/CLI.php:117
 Alchemy\Phrasea\CLI->runCLI() at /var/alchemy/Phraseanet/lib/Alchemy/Phrasea/CLI.php:133
 Alchemy\Phrasea\CLI->run() at /var/alchemy/Phraseanet/bin/console:174

system:backup-db [-t|--timeout TIMEOUT] [-g|--gzip] [-b|--bzip] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [--] <command> [<directory>]

The line is :
$res += $this->dump_base($this->getService('phraseanet.appbox'), $input, $output) && $ok;

$ok is really undefined, i tried to find it somewhere else in the code, did not find anything except of protected variable on one of the classes.

‫בתאריך שבת, 27 ביוני 2020 ב-21:23 מאת אריה קלטר <‪ary...@gmail.com‬‏>:‬

אריה קלטר

unread,
Jun 27, 2020, 2:56:31 PM6/27/20
to phrasea-i...@googlegroups.com
The problem was that the command mysqldump not found in the container phraseanet_phraseanet_1.
I installed mysql-client and it was ok, the backup is working as expected.
But still i don't understand what is the idea of the $ok there.

‫בתאריך שבת, 27 ביוני 2020 ב-21:42 מאת אריה קלטר <‪ary...@gmail.com‬‏>:‬

אריה קלטר

unread,
Jun 27, 2020, 3:18:24 PM6/27/20
to phrasea-i...@googlegroups.com
I checked the upgrade script, and it looks like it did not work because it is trying to search for folders to move, but it already in the place of the new version.
Should i do something if this is the case?

‫בתאריך שבת, 27 ביוני 2020 ב-21:56 מאת אריה קלטר <‪ary...@gmail.com‬‏>:‬

gjac...@alchemy.fr

unread,
Jun 30, 2020, 7:12:01 AM6/30/20
to phrasea-install-en
​Hello,

Can you try to launch the upgrade script from the worker using the following docker-compose command:
docker-compose -f docker-compose.yml run --rm worker bin/setup system:upgrade

Best regards,
Gaspard Jacobson
Reply all
Reply to author
Forward
0 new messages