Indexing errors in 2.2

[thearch...@gmail.com]

Greetings all,

So here’s the situation – before the new year, we were merrily working towards completing our final XML import into AtoM 2.2 until we reached some complex and lengthy XML files that maxed out our memory resources on the server. We corrected the memory problem, and started from the fail point, only to discover that some information objects were missing when we run

 

php symfony search:populate

 

what happens here is that we hit a “Couldn't find information object (id: ###)” error (similar to the one described in https://groups.google.com/forum/#!topic/ica-atom-users/Z8xPAMr0hiY, but under different circumstances). We confirmed that the object was “published” (each of them are), and ran

 

php symfony propel:generate-slugs

 

but rerunning the index still runs into the same error.  The only way we’ve been able to correct the error is to manually dig through the database and delete the offending row/entry (takes about 20-30 mins per error), then rerun the index, which inevitably hits another error.  The problem here is that we have no idea what we're really up against - is it 2 more or 2,000 more errors?

 

My question, then, is twofold:

a) is there any tool or process we can use to verify the database integrity? (we're concerned that the maxed out memory may have caused some cascading problems, or at least that's how it would appear)

b) is there a faster way to locate these errors without having to run the full index each time? (i.e. index everything and produce an error report for cases where information objects cannot be found - and perhaps the answer here would also address the first part of my question)

We are concerned that we would need to start the entire import process again from scratch, but this is a very daunting prospect (especially since we're so close to the end!).

Cheers,

Jeremy

[José Raddaoui]

Hello Jeremy,

there is another possible cause to that error, but it's really rare. Each 'information_object' row must have an 'object' row in the database. If the 'object' row is missing it will throw that error, but I don't know how this could happen in an import process.

Right now AtoM doesn't have any tool to check the database integrity, but you could find the information objects that will break the search index running the following query against the AtoM database:

SELECT io.id as io_id, obj.id as obj_id, st.status_id as pub_status_id, slug.slug

  FROM information_object io 

  LEFT JOIN object obj ON io.id=obj.id

  LEFT JOIN status st ON io.id=st.object_id AND st.type_id=158

  LEFT JOIN slug ON slug.object_id=io.id;

All resulting rows missing any of the column values will break the search index.

To add the missing slugs you could use a command line task from the AtoM folder:

php symfony propel:generate-slugs

To add the publication status you could use the following query:

INSERT INTO status (object_id, type_id, status_id, id, serial_number) VALUES ('{io_id}', '158', '159', NULL, '0');

In the third value, use 159 for draft or 160 for published.

To fix the missing 'object' row you could manually add that row:

INSERT INTO object (class_name, created_at, updated_at, id, serial_number) VALUES ('QubitInformationObject', '2016-01-31 03:57:44', '2016-01-31 03:57:44', '{io_id}', '0');

Or delete the row in the 'information_object' table (which should delete all related rows in other tables):

DELETE FROM information_object WHERE id={io_id};


After all you will need to fix any possible corruption to the nested set running the following task from the AtoM folder:

php symfony propel:generate-slugs

If there are no missing values in the first query or you can't fix the issues with this queries and tasks, you could send a dump from your database to my personal email so I can troubleshoot it locally.

Best regards,

Radda.

[Dan Gillean]

After all you will need to fix any possible corruption to the nested set running the following task from the AtoM folder:

php symfony propel:generate-slugs

I think for this task, Radda may have meant to say:

• php symfony propel:build-nested-set
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/17d22153-0fbb-462d-af70-2f9a86db2266%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[thearch...@gmail.com]

Thank you both for the assistance - we've been working through this, and have successfully located and fixed around 13 missing objects.  It's still a bit of a slog to wait for the index after each go, but it's still way faster than before.

Thanks again!

Jeremy