Index and search after restoring backup

[Vincent]

Hi all

I have two instances of ICA AtoM 1.0.8 running. While installing them,
I got the warning: "Options not allowed in .htaccess files". I don't
know the implications of that, but... I have the experience that it is
possible to transfer data between the two applications, but that this
data is not searchable. It seems to be related with the command "php
symfony search:populate QubitSearch".

When I create a backup from the first application with a database ica1
using a command like "$MySQL/bin/mysqldump -Q -uroot -pmysqlpass -
hlocalhost ica1 > backup.sql", transfer the data to a second
application with a database ica2 using a command like "$MySQL/bin/
mysql -uroot -pmysqlpass -hlocalhost ica2 < backup.sql" and execute
"php symfony search:populate QubitSearch" in the directory of the
latter application... I don't find any results when searching with
terms that generate results in the first application.

The data can be found y bbrowsing the archival descriptions and/or use
the add/edit-menu when logged in, but searching does not seem
possible. I did not find any difference with regard to drafts or
published descriptions. (By the way, I found the implementation of EAD
in ICA AtoM in this way quite revolutionary.) However, when I execute
the "php symfony search:populate QubitSearch" in the directory of the
first application, the same happens there: the archival descriptions
are browsable and editable, but not searchable anymore.

Is there some way to fix this, or should I care about the htaccess-
files in the first place?

[Jesús García Crespo]

Hi Vincent!

On Thu, Nov 5, 2009 at 3:21 AM, Vincent <jan...@xs4all.nl> wrote:

I have two instances of ICA AtoM 1.0.8 running. While installing them,
I got the warning: "Options not allowed in .htaccess files". I don't
know the implications of that, but... I have the experience that it is
possible to transfer data between the two applications, but that this
data is not searchable. It seems to be related with the command "php
symfony search:populate QubitSearch".

I think that "Options not allowed in .htaccess files" message has nothing to do with this problem, although this is a known issue and Jack is working on it for 1.1 release.

 

When I create a backup from the first application with a database ica1
using a command like "$MySQL/bin/mysqldump -Q -uroot -pmysqlpass -
hlocalhost ica1 > backup.sql", transfer the data to a second
application with a database ica2 using a command like "$MySQL/bin/
mysql -uroot -pmysqlpass -hlocalhost ica2 < backup.sql" and execute
"php symfony search:populate QubitSearch" in the directory of the
latter application... I don't find any results when searching with
terms that generate results in the first application.

I followed all the steps you described but I couldn't reproduce this problem here.

However, when I run "php symfony search:populate QubitSearch" in the second installation, after sql import, I got his error message:

File "/www/icaatom_B/data/index/segments_6" is not readable.

I checked that this file didn't exist. I removed all the files in /www/icaatom_B/data/index and I run "php symfony search:populate QubitSearch" again and it worked. Now, icaatom_B installation is returning search results as expected.

Then, I rebuilt index in icaatom_A installation and it worked at the first time.

What output do you get when you run rebuild index task? You see when it is working because you see something like this:

QubitSearch >> Populating index...

QubitSearch >> Index erased.

QubitSearch >> admin inserted.

QubitSearch >> City of Vancouver. Office of the City Clerk inserted.

QubitSearch >> City of Vancouver Archives inserted.

...

Please, remember that files in data/index are created by web server so it is possible that when you run php cli to rebuild index you are not able to modify that files or create new ones inside data directory. I solved this running php CLI by sudo to get credentials of the user which is running httpd server in my machine.

What kind of data are you working with? When I did the fresh installations I imported data from http://www.ica-atom.org/demo.xml sample data.

Regards,

--
Jesús García Crespo

[Vincent]

Hi Jesús

On 6 nov, 22:08, Jesús García Crespo <cor...@sevein.com> wrote:
>
> I think that "Options not allowed in .htaccess files" message has nothing to
> do with this problem, although this is a known issue and Jack is working on
> it for 1.1 release.
>

I had already been searching for some information about htaccess (and
it looks interesting), but I will forget that for now. Thanks!

I don't get any errors when I run "php symfony search:populate
QubitSearch". The problem is that I can browse archival descriptions,
but I do not find them when I search them - using terms that are
inside them. Searching and finding them only seems to be possible for
a few documents and before running any mysql-command.

However, I figured out that I can specify terms as access points. When
I search documents and use these terms, I find the archival
descriptions that I have associated with them. It is also possible to
make a backup, transfer it and use these terms in another application,
finding the archival descriptions associated with them. In short, the
search functionality seems to work very well with the access points.

I had expected that the search functionality would unlock the archival
descriptions using all terms inside them (or at least most of them,
maybe excluding some terms in specific sections). At first sight, it
seems to be working this way, for a few documents and before running
any mysql-command. Now I wonder if it is meant to be this way.

My question now: what is searched by the search functionality? It is
meant to unlock access points and the content of the authority record?
Or does it use all content of the indexed archival descriptions? I
think it'll be the first; if it's the latter, it does not work quite
well - yet...

>
> What kind of data are you working with? When I did the fresh installations I
> imported data fromhttp://www.ica-atom.org/demo.xmlsample data.
>

I work with a collection from the National Archives of the
Netherlands. For a thesis, I wanted to compare the search
functionality of ICA AtoM and Nutch (from the Apache Foundation).

Nutch is a quite strait forward search engine, also meant to be a
search engine for everything that hits its way. When the emphasis of
the search functionality in ICA AtoM lies with working with access
points (which is a reasonable choiche), comparising them might not
make much sense...

Thanks in advance
Regards,

Vincent Jansen

[peterVG]

Hi Vincent,

This sounds like really interesting research. Please let us know more
about your findings and opinions as you get further.

I think this discussion is getting a little too technical for the ica-
atom-users list so I've replied in more detail to your questions on
the qubit-dev discussion list. Please see:

http://groups.google.com/group/qubit-dev/browse_thread/thread/1c021ca1cfa9d7d1

Cheers,

--peterVG

[David Juhasz]

Hi Vincent,

I just want to clarify that you do need to re-run the "php symfony
search:populate QubitSearch" task after running any mysql commands
directly on the database. This is because the search index is only
updated when updating archival descriptions through the application
interface (with a browser) or with the "search:populate" task above.

Also, I just wanted to confirm that you are searching for descriptions
that have been published - the search is not supposed to find archival
descriptions that have a publication status of "draft".

One last point is that the search page only finds results in your
*current culture*, so if your archival descriptions are in Dutch, then
you need to switch to the Dutch user interface to find your descriptions.

Adding access points to an archival description should certainly add to
the ability to find descriptions, but it shouldn't be necessary to add
access points to a description for it to be found in the search. All of
the text in the archival description should be added to the search index
on save, or when running the "search:populate" task.

You can also try out the search functionality with a large data set
(over 11,000 archival descriptions) at:
http://www.memorybc.ca/

This project has been adding access points to the description manually,
but the majority of the descriptions in the system don't have linked
access points.

Regards,

--
David Juhasz,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | P: 604.527.2056 | F: 604.521.2059

[Vincent]

On 11 nov, 00:39, peterVG <vangarderen.pe...@gmail.com> wrote:
> Hi Vincent,
>
> This sounds like really interesting research. Please let us know more
> about your findings and opinions as you get further.
>
> I think this discussion is getting a little too technical for the ica-
> atom-users list so I've replied in more detail to your questions on
> the qubit-dev discussion list. Please see:
>

> http://groups.google.com/group/qubit-dev/browse_thread/thread/1c021ca...
>
> Cheers,
>
> --peterVG
>

Hi Peter

I'll keep you informed about the results. With regard to the code...
that is something I want to understand one day as well, but at this
moment a little bit too technical for me too...

> Software Engineer, Artefactual Systems Inc.http://www.artefactual.com| P: 604.527.2056 | F: 604.521.2059

Hi David

I have a collection of 350 descriptions uploaded in ICA AtoM. I
switched the interface to my cultural language, published 3
descriptions and then tried the search function. I found much more
than only these 3 descriptions. In order to avoid problems in the
future, I'll publish them all.

It is a bit strange, I would not expect something like the necessity
of switching to the cultural language of the descriptions, but - it's
working!

Thanks!
Vincent

[David Juhasz]

On 12-Nov-09, at 10:34 PM, Vincent wrote:
>
> Hi David
>
> I have a collection of 350 descriptions uploaded in ICA AtoM. I
> switched the interface to my cultural language, published 3
> descriptions and then tried the search function. I found much more
> than only these 3 descriptions. In order to avoid problems in the
> future, I'll publish them all.

I should clarify that in ICA-AtoM 1.0.8 you can set user & group
permissions to allow users to see "draft" descriptions in the search.
By default "administrators", "editors" and "contributors" have "view
draft" permissions - so if you are logged in as the administrative
user, then you *will* be able to search on all "draft" descriptions as
well as the published ones.

> It is a bit strange, I would not expect something like the necessity
> of switching to the cultural language of the descriptions, but - it's
> working!

We have limited the search results to the current user culture to make
the results more relevant to the current user. We may in the future
expand the search to allow filtering by language, or searching all
languages - perhaps as part of a "faceted search" [1] implementation.

[1] => http://en.wikipedia.org/wiki/Faceted_search

Glad to hear the search is working for you!