Problem migrate ica-atom.

[Marcello Henrique]

Hello,

I have the same problem his [0], you do help me?
Attachment my ".yml" file and message error is:

Unable to execute INSERT statement. [wrapped: SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`icaatom2/q_information_object`, CONSTRAINT `q_information_object_FK_2` FOREIGN KEY (`level_of_description_id`) REFERENCES `q_term` (`id`) ON DELETE SET NULL)]

[0] http://groups.google.com/group/qubit-dev/browse_thread/thread/ae8163ac123e800c#

Hugs!

--
Marcello Henrique
Associação Software Livre de Goiás (www.aslgo.org.br)
Cercomp - UFG (www.cercomp.ufg.br)

[Evelyn McLellan]

Hi Marcello,

Thanks for your question. I've forwarded it to one of our developers, who will get back to you soon.

Evelyn McLellan
ICA-AtoM Community Manager

[Jack Bates]

Hi Marcello - thanks for posting your YAML file - I tried loading it
into ICA-AtoM 1.0.7 and reproduced the same error - but I'm not sure
what went wrong : (

If you don't mind posting an SQL dump of your old database, maybe we
could try recreating a YAML file?

On Aug 11, 2:40 pm, Evelyn McLellan <epmclel...@gmail.com> wrote:
> Hi Marcello,
>
> Thanks for your question. I've forwarded it to one of our developers, who
> will get back to you soon.
>
> Evelyn McLellan
> ICA-AtoM Community Manager
>

> On Tue, Aug 11, 2009 at 2:11 PM, Marcello Henrique <fara...@gmail.com>wrote:
>
> > Hello,
>
> > I have the same problem his [0], you do help me?
> > Attachment my ".yml" file and message error is:
>
> > Unable to execute INSERT statement. [wrapped: SQLSTATE[23000]: Integrity
> > constraint violation: 1452 Cannot add or update a child row: a foreign key
> > constraint fails (`icaatom2/q_information_object`, CONSTRAINT
> > `q_information_object_FK_2` FOREIGN KEY (`level_of_description_id`)
> > REFERENCES `q_term` (`id`) ON DELETE SET NULL)]
>
> > [0]

> >http://groups.google.com/group/qubit-dev/browse_thread/thread/ae8163a...

[Marcello Henrique]

Hi Jack,

In Attachment .sql file.

Thanks for reply.

[Marcello Henrique]

Any news?

Thanks!

2009/8/17 Marcello Henrique <far...@gmail.com>:

>> --~--~---------~--~----~------------~-------~--~----~
>> You received this message because you are subscribed to the Google Groups
>> "ICA-AtoM Users" group.
>> To post to this group, send email to ica-ato...@googlegroups.com
>> To unsubscribe from this group, send email to
>> ica-atom-user...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.ca/group/ica-atom-users?hl=en
>> -~----------~----~----~----~------~----~------~--~---

[David at Artefactual]

Hi Marcello,

Sorry it's been so long with no response - thank you for you patience.

I'm sorry, but I'm confused as to your goal - your .sql file seems to
be from ICA-AtoM version 1.0.7, so presumably that's your current
version of the database. Are you trying to migrate to ICA-AtoM
version 1.0.8, or are you trying to use the .yml dump file for data
backup?

If you are trying to upgrade to ICA-AtoM 1.0.8, you can find full
instructions here:
http://www.qubit-toolkit.org/wiki/index.php?title=Upgrading

Using the .yml file for database backup is not a good idea, because of
problems with how data-dump orders the data. Your .yml file is a
great example of the ordering problem - because the data is in the
wrong order, the data-load task fails due to foreign key constraints.
The migration task (in the Upgrading link above) re-orders the data in
the .yml data to avoid this problem.

For ICA-AtoM you should use mysqldump to create a .sql file (as you
have done above) for data backup.
http://www.qubit-toolkit.org/wiki/index.php?title=Data_backup



Regards,

David Juhasz,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | P: 604.527.2056 | F: 604.521.2059

[Marcello Henrique]

Hi David,

I'm tried upgrade version 1.0.7 to 1.0.8 by follow steps by [0], now
got this error:

# php symfony propel:migrate /data/backup_2009-11-20.yml
>> migrate Initial data version 1.0.7
>> migrate Data migrated to version 1.0.8
>> migrate Migrated data written to "data/migrated_data_v108.yml"
# mv data/migrated_data_v108.yml data/fixtures/
# php symfony propel:insert-sql
...
# php symfony propel:data-load
>> propel load data from "/data/www/default/data/fixtures"

Fatal error: Zend_Search_Lucene_Storage_Directory_Filesystem::require_once():
Failed opening required
'Zend/Search/Lucene/Storage/File/Filesystem.php'
(include_path='/data/www/default/plugins/sfLucenePlugin/lib/vendor:/data/www/default/lib/vendor/symfony/lib/plugins/sfPropelPlugin/lib/vendor:/data/www/default:/data/www/default/lib/vendor/symfony/lib:/data/www/default/lib/vendor/symfony/lib/plugins/sfPropelPlugin/lib/task:/usr/share/php:/usr/share/pear')
in /data/www/default/plugins/sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Storage/Directory/Filesystem.php
on line 348

You have more advices? Thanks!

[0] http://www.qubit-toolkit.org/wiki/index.php?title=Upgrading

> --

>
> You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.

> To post to this group, send email to ica-ato...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ica-atom-users?hl=.

[Jesús García Crespo]

Hi Marcello,

On Fri, Nov 20, 2009 at 9:41 AM, Marcello Henrique <far...@gmail.com> wrote:

# php symfony propel:data-load
>> propel    load data from "/data/www/default/data/fixtures"

Fatal error: Zend_Search_Lucene_Storage_Directory_Filesystem::require_once():
Failed opening required
'Zend/Search/Lucene/Storage/File/Filesystem.php'
(include_path='/data/www/default/plugins/sfLucenePlugin/lib/vendor:/data/www/default/lib/vendor/symfony/lib/plugins/sfPropelPlugin/lib/vendor:/data/www/default:/data/www/default/lib/vendor/symfony/lib:/data/www/default/lib/vendor/symfony/lib/plugins/sfPropelPlugin/lib/task:/usr/share/php:/usr/share/pear')
in /data/www/default/plugins/sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Storage/Directory/Filesystem.php
on line 348

You have more advices? Thanks!

I think now you have two issues who are affecting your migration at data-load step.

Firstly, what you see, a fatal error message when try to open a required file [1]. This is not very important, but it hides the real problem.

Secondly, the real problem, it could be whatever, but at this step, the most common problem is that the open file limit was exceed. In my system, this limit was 1024 files but data-load requires a bit more than that, depending on how big is the data set you are importing. This is a known problem [2].

To increase this limit, you can try to run this command: "ulimit -n 50000", to set 50,000 as a new limit. To check if the limit was set correctly, run "ulimit -n" to get the current value.

Hopefully, ulimit may help you to jump to next step, :-).

[1] - http://code.google.com/p/qubit-toolkit/issues/detail?id=1137
[2] - http://code.google.com/p/qubit-toolkit/issues/detail?id=1054

Voçe viu? Obrigado! :-)

--
Jesús García Crespo

[Marcello Henrique]

Hello my friends,

As you predicted I went to the next step, now my problem is memory
ica-atom consumed 512Mb quickly, I am surprise with this greedy, he is
have a limit? because just have 512Mb memory my machine?

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried
to allocate 86 bytes) in
/data/www/default/plugins/sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Index/SegmentInfo.php
on line 311

Usted está en Europa o América? Estoy en Brasil, la tierra de carnaval! :-)

2009/11/20 Jesús García Crespo <cor...@sevein.com>:

> --
>
> You received this message because you are subscribed to the Google Groups
> "ICA-AtoM Users" group.
> To post to this group, send email to ica-ato...@googlegroups.com.

> To unsubscribe from this group, send email to
> ica-atom-user...@googlegroups.com.

[Jesús García Crespo]

Hi again, Marcello!

On Fri, Nov 20, 2009 at 11:57 AM, Marcello Henrique <far...@gmail.com> wrote:

As you predicted I went to the next step, now my problem is memory
ica-atom consumed 512Mb quickly, I am surprise with this greedy, he is
have a limit? because just have 512Mb memory my machine?

It is a limit set by PHP called "memory_limit" [1] to prevent bad written scripts for eating up all available memory on a server. Imagine in a shared server if one user runs a script like this one! This is why I think migration shouldn't be run in production servers. Of course, you can increase this limit. I guess your machine has more than 512Mb, isn't it? But don't set PHP to use all your system memory!

You may know a PHP configuration file called "php.ini" where you set all the PHP environment parameters, extension and limits. You can get easily the location of this file running:

$ php -i | grep "Loaded Configuration File" | awk '{print $5}'

In my case, php.ini location is: "/opt/local/etc/php5/php.ini". Edit that file and search for "memory_limit" directive. Replace the limit with a new higher value and check that new value was set correctly running this command:

$ php -i | grep "memory_limit"

Then, run data-load task again.

Take into account that you can find other PHP script execution limits [2].

[1] - http://ca2.php.net/manual/en/ini.core.php#ini.memory-limit

[2] - http://qubit-toolkit.org/wiki/index.php?title=PHP_script_execution_limits

Usted está en Europa o América? Estoy en Brasil, la tierra de carnaval! :-)

I'm Spanish but I am living in Savador da Bahia now, closed to you! E eu estou aprendendo um pouco de Português. Infelizmente, eu vou a voltar para a Espanha antes do Carnaval. I really would like to visit Brasilia, it is a nice architecture model, don't you think so?

Tchau!

 

--
Jesús García Crespo

[David Juhasz]

Hi Marcello,

To limit the amount of memory consumed when you are loading data, you
can also turn off the search indexing functionality:
http://www.qubit-toolkit.org/wiki/index.php?title=Search#Loading_large_data_sets

This should allow you to get the data into your database, then you can
rebuild the search index as a separate process.

[Marcello Henrique]

David,

I followed the following steps:

1. patch -p0 < disable_search_index.patch
2. php symfony propel:insert-sql
* Remove all datas
3. php symfony propel:data-load
* Working fine.
4. patch -p0 -R < disable_search_index.patch
5. php symfony search:populate QubitSearch
* It took a long and soon after "Allowed memory size exhausted"

Am I doing something wrong?

[Marcello Henrique]

Hi Jesús,

I understand fine php, and my phisical limit is 512Mb and php use 90%
this value.

Back subject, I'm 250Km from Brasília, I'm Goiânia principal city of
the state of Goiás and i like Brasília is a beatiful city.

2009/11/20 Jesús García Crespo <cor...@sevein.com>:

[Jesús García Crespo]

Hi Marcello,

On Mon, Nov 23, 2009 at 8:10 AM, Marcello Henrique <far...@gmail.com> wrote:

David,

I followed the following steps:

1. patch -p0 < disable_search_index.patch
2. php symfony propel:insert-sql
 *  Remove all datas
3. php symfony propel:data-load
 * Working fine.
4. patch -p0 -R < disable_search_index.patch
5. php symfony search:populate QubitSearch
 * It took a long and soon after "Allowed memory size exhausted"

As you can see, rebuilding index task is very memory intensive. I ignore if this can be optimized. For sure this will be resolved for 1.1 release. In the meanwhile, you could try to run this task under a machine with more memory.

If you don't have a machine like this in hand, you could try Amazon EC2, where you can run a machine with 32GB o RAM for $1.20 per hour. This may help you!

Regards,

--
Jesús García Crespo

[Marcello Henrique]

I'm already tried with 2 Gb but yet not got it! Wait that's this
version get quickly.

Thanks!

2009/11/23 Jesús García Crespo <cor...@sevein.com>: