migrate stalled out

133 views
Skip to first unread message

jde

unread,
Dec 14, 2011, 3:40:01 PM12/14/11
to ICA-AtoM Users
During my upgrade to icaatom-1.2.0, I ran propel:migrate, and it seems
to have stalled out. I've been waiting about 35 minutes with no
change. Below is the command and output:

# php -d memory_limit=2G symfony propel:migrate /var/www/icaatom-1.1/
data/fixtures/datadump_20111214.yml
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib64/
php/modules/imagick.so' - /usr/lib64/php/modules/imagick.so: undefined
symbol: php_set_error_handling in Unknown on line 0
PHP Warning: date(): It is not safe to rely on the system's timezone
settings. You are *required* to use the date.timezone setting or the
date_default_timezone_set() function. In case you used any of those
methods and you are still getting this warning, you most likely
misspelled the timezone identifier. We selected 'America/Los_Angeles'
for 'PST/-8.0/no DST' instead in /var/www/icaatom-1.2.0/vendor/symfony/
lib/config/sfRootConfigHandler.class.php on line 91
>> migrate Initial data version 62

Does it just take a long time, or should I be seeing something else by
now?

Also, if you have suggestions to resolve the warnings in the output
above, I'd be happy to hear those too.

Thank you,
Jill Earles

jde

unread,
Dec 14, 2011, 4:02:22 PM12/14/11
to ICA-AtoM Users
Apparently it's still running. Just got additional output:
>> migrate Data migrated to Release 1.1 v75
and then the path it was written to.

Does this mean it did not migrate to be compatible with
icaatom-1.2.0? I ran the command from my 1.2.0 directory, but with a
path to the data dump that was in the 1.1 directory.

David Juhasz

unread,
Dec 14, 2011, 4:23:10 PM12/14/11
to ica-ato...@googlegroups.com
On 11-12-14 01:02 PM, jde wrote:
> Apparently it's still running. Just got additional output:
>>> migrate Data migrated to Release 1.1 v75
> and then the path it was written to.
>
> Does this mean it did not migrate to be compatible with
> icaatom-1.2.0? I ran the command from my 1.2.0 directory, but with a
> path to the data dump that was in the 1.1 directory.

No, that's perfect Jill, it's just a glitch in the output - Release 1.1
v75 = Release 1.2. :)

I'm surprised it took that long though, it usually only takes a few
minutes. Maybe you have a very large database?

As for the error messages (if you are still interested. I typed them up
in response to your first post):

What OS are you running? To get rid of the "imagick.so" error in Ubuntu
or Debian Linux you can install the PHP imagemagick library using:

sudo apt-get install php5-imagick


To get rid of the date warning you need to set a timezone in the php
configuration file, see
<http://php.net/manual/en/datetime.configuration.php>


Regards,
David

--
David Juhasz,
Software Engineer

Artefactual Systems Inc.
www.artefactual.com

jde

unread,
Dec 14, 2011, 7:04:08 PM12/14/11
to ICA-AtoM Users
Yes, it seems to be a very large database - the data load is still
running after almost 3 hours.

This is running on Ubuntu 10.04. I'll try the things you mentioned
once the data load is complete.

Thank you!

Jill

jde

unread,
Dec 15, 2011, 3:30:38 PM12/15/11
to ICA-AtoM Users
Last night mysql crashed during the data load and I had to reinstall
it and start again. The propel load data task has now been running
for about 2.5 hours, and when I check the database via phpmyadmin, I
can see that there are no records in any of the tables. I'm not
seeing any errors in the mysql or apache logs, and the output from the
data load command isn't showing anything.

The last command I issed was:

nohup php -d memory_limit=50000M symfony propel:data-load data/
fixtures/migrated_data_20111214125840.yml
nohup: appending output to `nohup.out'

And the output in nohup.out is:


PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib64/
php/modules/imagick.so' - /usr/lib64/php/modules/imagick.so: undefined
symbol: php_set_error_handling in Unknown on line 0

>> propel load data from "data/fixtures/migrated_data_20111214125840.yml"

Do you have any suggestions for how to determine what's wrong, or how
to get the migrated data into the database?

Thank you!

Jill

jde

unread,
Dec 15, 2011, 3:33:07 PM12/15/11
to ICA-AtoM Users
P.S. I was wrong about the OS (confused this with another server).
It's RHEL 5.6.

jde

unread,
Dec 15, 2011, 3:54:27 PM12/15/11
to ICA-AtoM Users
Also, in case it helps, the size of the
migrated_data_20111214125840.yml file is just under 63 MB.

David Juhasz

unread,
Dec 16, 2011, 2:55:29 PM12/16/11
to ica-ato...@googlegroups.com
Hi Jill,

I recommend disabling the search index when doing the data load:
http://www.qubit-toolkit.org/wiki/index.php?title=Search#Disabling_the_search_index_for_Release_1.2

I've added a note to this effect to the upgrading documentation:
http://www.qubit-toolkit.org/wiki/index.php?title=Upgrading#Load_migrated_data

Regards,
David Juhasz

jde

unread,
Jan 5, 2012, 1:07:30 PM1/5/12
to ICA-AtoM Users
Hi David,
I disabled the search index and tried loading the migrated data again.
 MySQL crashed and could not restart, so I had to uninstall and
reinstall it again.  This has happened a few times as I've attempted
this.  Any idea what would cause this?
Regards,Jill
On Dec 16 2011, 11:55 am, David Juhasz <da...@artefactual.com> wrote:
> Hi Jill,
>
> I recommend disabling the search index when doing thedataload:http://www.qubit-toolkit.org/wiki/index.php?title=Search#Disabling_th...
>
> I've added a note to this effect to the upgrading documentation:http://www.qubit-toolkit.org/wiki/index.php?title=Upgrading#Load_migr...
>
> Regards,
> David Juhasz
>
> On 11-12-15 12:54 PM, jde wrote:
>
>
>
>
>
>
>
>
>
> > Also, in case it helps, the size of the
> > migrated_data_20111214125840.yml file is just under 63 MB.
>
> > On Dec 15, 12:33 pm, jde<jilldenaeear...@gmail.com>  wrote:
> >> P.S.  I was wrong about the OS (confused this with anotherserver).
> >> It's RHEL 5.6.
>
> >> On Dec 15, 12:30 pm, jde<jilldenaeear...@gmail.com>  wrote:
>
> >>> Last nightmysqlcrashedduringthedataloadand I had to reinstall
> >>> it and start again.  The propelloaddatatask has now been running
> >>> for about 2.5 hours, and when I check the database via phpmyadmin, I
> >>> can see that there are no records in any of the tables.  I'm not
> >>> seeing any errors in themysqlor apache logs, and the output from the
> >>>dataloadcommand isn't showing anything.
> >>> The last command I issed was:
> >>> nohup php -d memory_limit=50000M symfony propel:data-loaddata/
> >>> fixtures/migrated_data_20111214125840.yml
> >>> nohup: appending output to `nohup.out'
> >>> And the output in nohup.out is:
> >>> PHP Warning:  PHP Startup: Unable toloaddynamic library '/usr/lib64/
> >>> php/modules/imagick.so' - /usr/lib64/php/modules/imagick.so: undefined
> >>> symbol: php_set_error_handling in Unknown on line 0
> >>>>> propel    loaddatafrom "data/fixtures/migrated_data_20111214125840.yml"
> >>> Do you have any suggestions for how to determine what's wrong, or how
> >>> to get the migrateddatainto the database?
> >>> Thank you!
> >>> Jill
> >>> On Dec 14, 4:04 pm, jde<jilldenaeear...@gmail.com>  wrote:
> >>>> Yes, it seems to be a very large database - thedataloadis still
> >>>> running after almost 3 hours.
> >>>> This is running on Ubuntu 10.04.  I'll try the things you mentioned
> >>>> once thedataloadis complete.
> >>>> Thank you!
> >>>> Jill
> >>>> On Dec 14, 1:23 pm, David Juhasz<da...@artefactual.com>  wrote:
> >>>>> On 11-12-14 01:02 PM, jde wrote:
> >>>>>> Apparently it's still running.  Just got additional output:
> >>>>>>>> migrate  Datamigrated to Release 1.1 v75
> >>>>>> and then the path it was written to.
> >>>>>> Does this mean it did not migrate to be compatible with
> >>>>>> icaatom-1.2.0?  I ran the command from my 1.2.0 directory, but with a
> >>>>>> path to thedatadump that was in the 1.1 directory.

jde

unread,
Jan 5, 2012, 4:44:16 PM1/5/12
to ICA-AtoM Users
Could this be a RAM issue? I saw something in this message:
https://groups.google.com/group/ica-atom-users/msg/2a25fa367d0bc0d9?hl=en,
that indicates that it might be. My YAML file for the migrated data
is close to 63M. That message says they needed 12G of RAM for a 53M
YAML file. Will I need that much RAM or more to get this to work?

On Jan 5, 10:07 am, jde <jilldenaeear...@gmail.com> wrote:
> Hi David,
> I disabled the search index and tried loading the migrateddataagain.
>  MySQLcrashed and could not restart, so I had to uninstall and

jde

unread,
Jan 5, 2012, 5:31:04 PM1/5/12
to ICA-AtoM Users
I did not get an out of memory error during the data load.  Just:
Unable to execute INSERT statement. [wrapped: SQLSTATE[HY000]: General
error: 2006 MySQL server has gone away]
On Jan 5, 1:44 pm, jde <jilldenaeear...@gmail.com> wrote:
> Could this be a RAM issue?  I saw something in this message:https://groups.google.com/group/ica-atom-users/msg/2a25fa367d0bc0d9?h...,

Jesús García Crespo

unread,
Jan 5, 2012, 5:45:55 PM1/5/12
to ica-ato...@googlegroups.com
I have seen that error before but in a very different context.

You can find more information here: http://dev.mysql.com/doc/refman/5.0/en/gone-away.html

But I don't have a solution off the top of my head. I'd use top or htop to monitor the memory usage in your machine, if you are running out of memory you may should need to get more server. I personally did some big migrations using Amazon E2C, where you can get a 64G machine just for some hours, then run the search index separately in your own machine as this task is not so memory consuming.

Regards,

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To post to this group, send email to ica-ato...@googlegroups.com.
To unsubscribe from this group, send email to ica-atom-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ica-atom-users?hl=en.




--
Jesús García Crespo,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | +1.604.527.2056
Reply all
Reply to author
Forward
0 new messages