Problem upgrade 1.0.9 -> 1.1

[CIRA - Frédéric]

Hi,

First of all, congratulations for this first stable release!

I went successfully through the upgrade process on my local
installation yesterday. I fail doing the same on the remote one today.
First step:

$ php symfony propel:data-dump mig_datas.yml

results in a fatal error:

Fatal error: Unsupported operand types in /home/public/icaatom-1.0.9/
lib/vendor/symfony/lib/plugins/sfPropelPlugin/lib/addon/
sfPropelData.class.php on line 424

Such an error in the same context has been reported here [1], but no
solution was given.

Best regards,
Frederic

[1] https://groups.google.com/group/ica-atom-users/browse_thread/thread/826bc10b958ab05a?hl=en

[David Juhasz]

Hi Frederic,

Thanks for the congratulations on Release 1.1. :)

Thanks also for searching the discussion list for previous report of
the issue you are having, it really saves us time when users are
resourceful like that; Unfortunately your search didn't produce a
solution this time, but I appreciate the effort. :)

To be honest I can't remember the exact solution that I used with
Klaus' data, but I do remember that there was some corruption in his
data set. Please send me your data set at my personal email "david
at artefactual dot com" and I'll figure out what's happening and post
a general solution back to the discussion list.

Regards,

David Juhasz,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | P: 604.527.2056 | F: 604.521.2059

[Klaus Wendel]

Hi David,

apropos Klaus. Me again ;-)

My problems starts at step 1:


php /var/www/XXXdb/symfony propel:data-dump XXXdb_20101128.yml

results in:

>> propel dumping data to "/var/www/XXXdb/data/fixtures/XXXdb_20101128.yml"

Fatal error: Class QubitActor contains 4 abstract methods and must
therefore be declared abstract or implement the remaining methods
(ArrayAccess::offsetExists, ArrayAccess::offsetGet,
ArrayAccess::offsetSet, ...) in /var/www/XXXXdb/lib/model/
QubitActor.php on line 27

May I request your help again?

Regards,

Klaus

PS: However, the blank installation of 1.1. looks quite good!

> Software Engineer, Artefactual Systems Inc.http://www.artefactual.com| P: 604.527.2056 | F: 604.521.2059

[Jesús García Crespo]

Hi Klaus,

This is a known issue: http://code.google.com/p/qubit-toolkit/issues/detail?id=1188

You have to clear cache twice.

Regards,

On Sun, Nov 28, 2010 at 10:47 PM, Klaus Wendel <goo...@lednew.de> wrote:

Hi David,

apropos Klaus. Me again ;-)

My problems starts at step 1:


php /var/www/XXXdb/symfony propel:data-dump XXXdb_20101128.yml

results in:

>> propel    dumping data to "/var/www/XXXdb/data/fixtures/XXXdb_20101128.yml"

Fatal error: Class QubitActor contains 4 abstract methods and must
therefore be declared abstract or implement the remaining methods
(ArrayAccess::offsetExists, ArrayAccess::offsetGet,
ArrayAccess::offsetSet, ...) in /var/www/XXXXdb/lib/model/
QubitActor.php on line 27

 

--
Jesús García Crespo,

Software Engineer, Artefactual Systems Inc.

http://www.artefactual.com | +1.604.527.2056

[Klaus Wendel]

Hi Jesús,

thanks for that hint, that did it very well. Unfortunately, the next
issue don't let us wait. The "loading of migrated data"-step

php /var/www/htdocs/XXXdb/symfony propel:insert-sql

resulted into following error:

PHP 14. include_once() /var/www/htdocs/XXXdb/vendor/symfony/lib/
plugins/sfPropelPlugin/lib/vendor/phing/Phing.php:1006
Buildfile: /var/www/htdocs/XXXdb/vendor/symfony/lib/plugins/
sfPropelPlugin/lib/vendor/propel-generator/build.xml

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to
allocate 30720 bytes) in /var/www/htdocs/XXXdb/vendor/symfony/lib/
plugins/sfPropelPlugin/lib/vendor/phing/lib/Zip.php on line 2532

(In php.ini there is no memory limit resp. = -1.)

Thanks for your help,

Klaus
On Nov 28, 10:51 pm, Jesús García Crespo <je...@artefactual.com>
wrote:

[Jesús García Crespo]

Hi Klaus,

On Mon, Nov 29, 2010 at 12:17 PM, Klaus Wendel <goo...@lednew.de> wrote:

Hi Jesús,

 php /var/www/htdocs/XXXdb/symfony propel:insert-sql

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to
allocate 30720 bytes) in /var/www/htdocs/XXXdb/vendor/symfony/lib/
plugins/sfPropelPlugin/lib/vendor/phing/lib/Zip.php on line 2532

(In php.ini there is no memory limit resp. = -1.)

Symfony does not handle value -1 correctly, you have to set a positive integer big enough.

This is also a known issue :-), http://code.google.com/p/qubit-toolkit/issues/detail?id=1627

You can update your php.ini file or set this directive manually each time you run php-cli:

php -d memory_limit=512M symfony propel:insert-sql

Regards,

[Klaus Wendel]

> This is also a known issue :-),http://code.google.com/p/qubit-toolkit/issues/detail?id=1627

>
> You can update your php.ini file or set this directive manually each time
> you run php-cli:
>
> php -d memory_limit=512M symfony propel:insert-sql

Cool!!! It worked :-))

Thanks, Jesús

[Klaus Wendel]

> Cool!!! It worked :-))

D'oh! Oh no. Next issue:

After many hours of running ...

php -d memory_limit=2048M /var/www/htdocs/XXXdb/symfony propel:data-
load /var/www/htdocs/XXXdb/data/fixtures/
migrated_data_20101129112741.yml

>> propel load data from "/var/www/htdocs/XXXdb/data/fixtures/migrated_data_20101129112741.yml"

... the system halted with this statement:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry
'JnOb' for key 'slug_U_2'

What is to do now?

Regards,

Klaus

[David Juhasz]

Hi Klaus,

This is a bug that we caught late in the Release 1.1 testing phase,
but we though it was isolated to the import module and didn't consider
it critical enough to delay the release:
http://code.google.com/p/qubit-toolkit/issues/detail?id=1882

I've bumped the priority of the issue to "critical" for Release 1.2.

It occurs randomly, so starting the load again won't produce the same
key clash, but the chances of a clash increase with the amount of data
you are loading. When you say it takes several hours to load the
data, is that with the data indexing turned off? How many records are
in your system roughly (including actors, repositories, terms, and
functions)?

Cheers,

David Juhasz,

Software Engineer, Artefactual Systems Inc.

http://www.artefactual.com | P: 604.527.2056 | F: 604.521.2059

[David at Artefactual]

Frederic,

I've just finished analyzing and cleaning your data. The problem is
occurring because a number of q_object rows have been deleted from the
database, but the related rows that extend the base q_object
(q_information_object, q_actor, q_event) are still present in the
database. e.g.

mysql> select id from q_information_object where id = 686;
+-----+
| id |
+-----+
| 686 |
+-----+
1 row in set (0.00 sec)

mysql> select id from q_object where id = 686;
Empty set (0.00 sec)

This does not produce an error at the database level, but it does at
the application level.

As I mentioned before, Klaus Wendel experienced similar data
corruption in his database with previous versions of ICA-AtoM, so I
must assume that the data corruption is somehow being produced by the
ICA-AtoM application when deleting objects from the system. I'll have
to investigate the conditions that produce these corrupt objects
further.

In the meantime I will email the YAML dump file to your private email
so you can proceed with your migration.

[Klaus Wendel]

> This is a bug that we caught late in the Release 1.1 testing phase,
> but we though it was isolated to the import module and didn't consider
> it critical enough to delay the release:http://code.google.com/p/qubit-toolkit/issues/detail?id=1882
>
> I've bumped the priority of the issue to "critical" for Release 1.2.
>
> It occurs randomly, so starting the load again won't produce the same
> key clash, but the chances of a clash increase with the amount of data
> you are loading. When you say it takes several hours to load the
> data, is that with the data indexing turned off? How many records are
> in your system roughly (including actors, repositories, terms, and

Hi David,

to answer your qustion first: There are approximately 5000 records in
my database.

But the

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry

'OwNf' for key 'slug_U_2'

bug prevented me to go ahead. I tried it 5 or 6 times, but I never
succeeded in finishing this job. The randomly occurring event occurred
clocklike.

Is there a chance for me to get the database migrated before thursday?

Best regards,

Klaus

[David Juhasz]

Hi Klaus,

I'm just working on the duplicate key bug, so I should have a patch
available for you today.

[Klaus Wendel]

On Dec 7, 10:19 pm, David Juhasz <da...@artefactual.com> wrote:
> Hi Klaus,
>
> I'm just working on the duplicate key bug, so I should have a patch  
> available for you today.

Cool! Thank you very much David!!!

Very best regards,

Klaus

[David Juhasz]

Hi Klaus,

A patch for issue 1882 [1] is available here:

http://www.ica-atom.org/download/patches/release11/random_slug.patch

Unfortunately,  issue 1120 [2] is causing random keys to be generated for non-english records, instead of the human-readable permalinks that would normally be generated.  :(   I'm looking into this issue right now to see if I can resolve it.  This doesn't affect the ability to load data or access it, it's purely a cosmetic problem with the generated permalinks.

[1] => http://code.google.com/p/qubit-toolkit/issues/detail?id=1882

[2] => http://code.google.com/p/qubit-toolkit/issues/detail?id=1120

[Klaus Wendel]

Hi Dave,

> A patch for issue 1882 [1] is available here:http://www.ica-atom.org/download/patches/release11/random_slug.patch

Thank you very much for your emergency service! As root I tried to
apply with

patch -p0 < random_slug.patch

and got following results:

patching file lib/model/QubitObject.php
patching file lib/model/QubitSlug.php
Hunk #1 FAILED at 28.
Hunk #2 succeeded at 65 (offset 2 lines).
1 out of 2 hunks FAILED -- saving rejects to file lib/model/
QubitSlug.php.rej

What's wrong? lib/model/QubitSlug.php.rej is writable for user root.

Regards, Klaus

[Jesús García Crespo]

Hi Klaus,

The patch that you downloaded was incomplete. We fixed it today. Could you download it again, please?

http://www.ica-atom.org/download/patches/release11/random_slug.patch

This patch fixes two issues:

1) Duplicate slug generated, fatal error on insert

http://code.google.com/p/qubit-toolkit/issues/detail?id=1882

2) Permalinks for non-english records are randomized

http://code.google.com/p/qubit-toolkit/issues/detail?id=1905

Klaus, you should revert the old patch (make sure you do it before you download the new one):

$ patch -p0 -R < random_slug.patch

In case of doubts reverting the old patch, just copy the affected files (lib/model/QubitObject.php and lib/model/QubitSlug.php) from the tarball and then apply the new patch. For your information, this patch worked for us with really big data sets.

Regards,

On Wed, Dec 8, 2010 at 7:25 AM, Klaus Wendel <goo...@lednew.de> wrote:

Thank you very much for your emergency service! As root I tried to

apply with

patch -p0 < random_slug.patch

and got following results:

patching file lib/model/QubitObject.php
patching file lib/model/QubitSlug.php
Hunk #1 FAILED at 28.
Hunk #2 succeeded at 65 (offset 2 lines).
1 out of 2 hunks FAILED -- saving rejects to file lib/model/
QubitSlug.php.rej

What's wrong? lib/model/QubitSlug.php.rej is writable for user root.

Regards, Klaus

--
Jesús García Crespo

[Klaus Wendel]

Hallo Jesús,

many thanks for your second patch. It worked quite well for hours, but
a memory failure occurred at morning.

php -d memory_limit=1024M /srv/www/htdocs/XXXdb/symfony propel:data-
load /srv/www/htdocs/XXXdb/data/fixtures/
migrated_data_20101207210751.yml
>> propel load data from "/srv/www/htdocs/...igrated_data_20101207210751.yml"

Out of memory (Needed 8164 bytes)

[wrapped: SQLSTATE[HY000]: General error: 2013 Lost connection to
MySQL server during query]

1GB isn't enough? I cannot increase the memory limit very much on this
server.

Regards,

Klaus

[Jesús García Crespo]

Hi Klaus,

On Thu, Dec 9, 2010 at 10:18 AM, Klaus Wendel <goo...@lednew.de> wrote:

1GB isn't enough? I cannot increase the memory limit very much on this
server.

propel:data-load task needs lots of RAM but depends on the size of your data set.

Yesterday, I loaded a data set and its YAML file size was 54M. It needed more than 12G of RAM so I had to contract an Amazon E2C server for one hour. In my case, it took less than one hour because I disabled the search index, which I can build from my own machine.

Regards,

--
Jesús García Crespo,

Software Engineer, Artefactual Systems Inc.

http://www.artefactual.com | +1.604.527.2056

[David Juhasz]

Hi Klaus,

One thing you can also try is the new PHP 5.3 circular reference garbage collector:

http://ca3.php.net/gc_enable

More information on how it works can be found here:

http://php.net/manual/en/features.gc.php

N.B. We have not tried this ourselves, and make no guarantee of any kind that it will reduce the memory requirements of ICA-AtoM.

With that warning in mind, the Propel ORM which we use in ICA-AtoM makes heavy use of circular references (every foreign key relation is represented in the ORM with a circular reference), so hopefully being able to garbage collect these objects will reduce the memory footprint for your data load.

Regards,

David Juhasz,

Software Engineer, Artefactual Systems Inc.

http://www.artefactual.com | P: 604.527.2056 | F: 604.521.2059

[ErikaH]

Hello there,
I seem to be having similar issues with the upgrade from 1.0.9 to
1.1.


ltiarchives@ubuntu:~$ php /var/www/icaatom-1.1/symfony propel:migrate /
home/ltiarchives/Desktop/LTI\ Archives\ DB\ backup/Backup.yml

PHP Notice: Undefined index: type_id in /var/www/icaatom-1.1/lib/task/
migrate/QubitMigrate109.class.php on line 345
PHP Notice: Undefined index: type_id in /var/www/icaatom-1.1/lib/task/
migrate/QubitMigrate109.class.php on line 345
PHP Notice: Undefined index: value in /var/www/icaatom-1.1/lib/task/
migrate/QubitMigrate109.class.php on line 518
>> migrate Data migrated to version 1.1
>> migrate Migrated data written to "/home/...igrated_data_20101215161350.yml"

ltiarchives@ubuntu:~$ php /var/www/icaatom-1.1/symfony propel:insert-
sql
>> schema converting "/var/www/icaatom-1.1/config/schema.yml" to XML
>> schema putting /var/www/icaatom-1.1/config/generated-schema.xml
>> schema converting "/var/www/icaatom-1.1...Plugin/config/schema.yml" to XML
>> schema putting /var/www/icaatom-1.1/plu...generated-qbAclPlugin-schema.xml

WARNING: The data in the database related to the connection name
propel will be removed.

Are you sure you want to proceed? (y/N)

y


You must create a schema.yml or schema.xml file.


ltiarchives@ubuntu:~$ php /var/www/icaatom-1.1/symfony propel:data-
load /home/ltiarchives/Desktop/LTI\ Archives\ DB\ backup/
migrated_data_20101215161350.yml
>> propel load data from "/home/ltiarchive...igrated_data_20101215161350.yml"


Unable to execute INSERT statement. [wrapped: SQLSTATE[23000]:
Integrity constraint violation: 1062 Duplicate entry '30' for key
'PRIMARY']






I then tried clearing the cache. After clearing cache once:


PHP Fatal error: Class QubitTaxonomy contains 4 abstract methods and

must therefore be declared abstract or implement the remaining methods
(ArrayAccess::offsetExists, ArrayAccess::offsetGet,

ArrayAccess::offsetSet, ...) in /var/www/icaatom-1.1/lib/model/
QubitTaxonomy.php on line 21
QubitTaxonomy:
QubitTaxonomy_30:
class_name: QubitTaxonomy
created_at: '2010-06-29 15:11:47'
updated_at: '2010-06-29 15:11:47'
serial_number: '0'
id:
Fatal error: Class QubitTaxonomy contains 4 abstract methods and must

therefore be declared abstract or implement the remaining methods
(ArrayAccess::offsetExists, ArrayAccess::offsetGet,

ArrayAccess::offsetSet, ...) in /var/www/icaatom-1.1/lib/model/
QubitTaxonomy.php on line 21




I recognized this error from another user, and tried clearing the
cache twice. It then returned to above "unable to execute" error.
From looking at past problems with upgrading in the user group, it
seems like there are errors in my data file or it is corrupted in some
manner.
Any assistance would be greatly appreciated.
Sincerely,
Erika Heesen
Archivist, Leeds and the Thousand Islands Archives

[David Juhasz]

Hi Erika,

I successfully migrated and loaded your data without experiencing the
same error you did. I think the problem is in the insert-sql step:

> ltiarchives@ubuntu:~$ php /var/www/icaatom-1.1/symfony propel:insert-
> sql
>>> schema converting "/var/www/icaatom-1.1/config/schema.yml" to XML
>>> schema putting /var/www/icaatom-1.1/config/generated-schema.xml
>>> schema converting "/var/www/icaatom-1.1...Plugin/config/schema.yml" to XML
>>> schema putting /var/www/icaatom-1.1/plu...generated-qbAclPlugin-schema.xml
> WARNING: The data in the database related to the connection name
> propel will be removed.
>
> Are you sure you want to proceed? (y/N)
>
> y
>
>
> You must create a schema.yml or schema.xml file.
>

In your ICA-AtoM install directory, there should be a
"config/schema.yml" file. Can you please check your "config" directory
and see if the schema.yml file is missing? Are you installing from the
Release 1.1 tarball?

--

David Juhasz,
Software Engineer, Artefactual Systems Inc.

http://www.artefactual.com | P: +1.604.527.2056 | F: +1.604.521.2059

[David Juhasz]

Hi Erika,

Please send questions that don't include sensitive data to the ica-atom users discussion group (I've cc'd the group on this email) - this allows others to participate and benefit from the conversation.   The discussion list is also a good place to search for answers to your questions.

The error "abstract methods" error is a known problem, but it is easy to work around, please see http://code.google.com/p/qubit-toolkit/issues/detail?id=1188.

Also, you can manually delete any files in the "search/index" directory before running the search index task; the files are created by the indexing process.


Regards,

-- 
David Juhasz,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | P: +1.604.527.2056 | F: +1.604.521.2059 

On 12/20/2010 2:24 PM, Erika Heesen wrote:

Hi David,
Hmm.
I tried it with sudo and got the same "segments_3 is not readable" error. I checked, and the owner of those files is already www-data.

So I copied segments_2 in the index file, renamed it "segments_3" and ran it again. This gave me the following:

ltiarchives@ubuntu:~$ sudo mv /home/ltiarchives/Desktop/segments_3 /var/www/icaatom-1.1/data/index/
ltiarchives@ubuntu:~$ sudo php /var/www/icaatom-1.1/symfony search:populate QubitSearch
QubitSearch >> Populating index...
QubitSearch >> Index erased.
PHP Fatal error:  Class QubitActor contains 4 abstract methods and must therefore be declared abstract or implement the remaining methods (ArrayAccess::offsetExists, ArrayAccess::offsetGet, ArrayAccess::offsetSet, ...) in /var/www/icaatom-1.1/lib/model/QubitActor.php on line 28

Fatal error: Class QubitActor contains 4 abstract methods and must therefore be declared abstract or implement the remaining methods (ArrayAccess::offsetExists, ArrayAccess::offsetGet, ArrayAccess::offsetSet, ...) in /var/www/icaatom-1.1/lib/model/QubitActor.php on line 28
ltiarchives@ubuntu:~$


I don't have a 500 error anymore, but the search is still not returning any results. I tried clearing the cache and running it again, but got the same error message.
Cheers,
Erika

On Thu, Dec 16, 2010 at 6:08 PM, David Juhasz <da...@artefactual.com> wrote:

On 10-12-16 02:55 PM, Erika Heesen wrote:

Hi David,

The sql dump worked perfectly. Thank you so much.

I also checked - there is a schema.yml in the config directory (I installed from the tarball). Hope that helps ...

Now, I needed to do a rebuild of the search index as there are no search results for items that I know are in the database. However, I got this error message:
ltiarchives@ubuntu:~$ php /var/www/icaatom-1.1/symfony search:populate QubitSearch
 File '/var/www/icaatom-1.1/data/index/segments_3' is not readable.

And when I go to that location, segments_3 does not exist (but segments_2 does).
It also gives me a "500 Internal Server Error" message now when I try to search on my site. Any suggestions?
Cheers,
Erika

Hi Erika,

It sounds like a permissions problem. If you are running the search:populate task as a different user then the web user (usually "www-root") then you may need to run the populate task as the superuser (note the "sudo" command):

ltiarchives@ubuntu:~$ sudo php /var/www/icaatom-1.1/symfony search:populate

If the search:populate completes successfully and you still get a 500 error when you try a search, you may need to update the permissions on the search index directory:

ltiarchives@ubuntu:~$ sudo chown -R www-data:www-data /var/www/icaatom1.1/data/index

-- 
David Juhasz,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | P: +1.604.527.2056 | F: +1.604.521.2059 

[ErikaH]

Clearing the cache using the "php symfony cc && php symfony cc"
command worked.
I think it's important to note the '&&' for clearing the cache twice,
as I initially just entered "php symfony cc" twice in a row, which did
not solve the problem.
Cheers,
Erika

On Dec 20, 6:06 pm, David Juhasz <da...@artefactual.com> wrote:
> Hi Erika,
>
> Please send questions that don't include sensitive data to the ica-atom
> users discussion group (I've cc'd the group on this email) - this allows
> others to participate and benefit from the conversation.   The
> discussion list is also a good place to search for answers to your
> questions.
>
> The error "abstract methods" error is a known problem, but it is easy to

> work around, please seehttp://code.google.com/p/qubit-toolkit/issues/detail?id=1188.

>
> Also, you can manually delete any files in the "search/index" directory
> before running the search index task; the files are created by the
> indexing process.
>
> Regards,
>
> --
> David Juhasz,

> Software Engineer, Artefactual Systems Inc.http://www.artefactual.com| P: +1.604.527.2056 | F: +1.604.521.2059

> Software Engineer, Artefactual Systems Inc.http://www.artefactual.com| P: +1.604.527.2056 | F: +1.604.521.2059