EC2/EBS issue for db persistence

[Chag Will]

Hi,

I have my instance running on EC2 and I have EBS volume for persistence. But as per the following link, only /opt/appscale gets backed up: 

https://github.com/AppScale/appscale/wiki/Automated-Data-Persistence

But, when I install mysql (apt-get install mysql-server), it is installed somewhere else. So, when I restart the instance, the installation as well as the data gets vanished.

Please tell me how to bring mysql to /opt/appscale so that I can persist data. Thank you.

[Meni Vaitsi]

Hello,

You could create a directory in /opt/appscale/ and try to set the DATADIR option in a MySQL option file (method described here: https://dev.mysql.com/doc/refman/5.5/en/option-files.html) so that it points to that directory. Then restart the server and see if the data is saved to the new location.

Let us know if that works. :)

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems Inc.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Chag Will]

Hi Meni,

I tried that to set the DATADIR in etc/mysql/my.cnf. But the issue is, the installation of mysql is under '/usr' directory and I don't think I can change BASEDIR (installation directory) to point to some other directory. So, when I did 'appscale down' followed by 'appscale up', the data directory I created under /opt/appscale is present but mysql server itself is gone as the previous instance is destroyed and a new instance is created.

So, it seems I should not use 'apt-get' to install mysql if I want it to be installed in a directory that I want. So, I am going to download binary installable (tar file) from mysql downloads online, then install it manually in the required directory. 

Hopefully, the 'untar'ing will just install the server in /opt/appscale.. :)

I will let you know..Thanks.

[Chag Will]

Hi,

Sorry for the delayed response. As mentioned in my previous update, I tried to install mysql manually under /opt/appscale, but that process turned out to be tedious and it seemed like it will not be easy to do any updates in future if I take that route to install mysql.

Then, while reading online I realized that we can take an AMI snapshot of the appscale VM image at any state (i.e., after installing some custom software). As I am a new EC2 user, I did not know that before otherwise I could have tried that first. 

So, I came back to your original solution of pointing DATADIR to somewhere in /opt/appscale, then took AMI image snapshots so that I won't lose mysql installation next time I do 'appscale up'. 

I got a new issue now. Once I start my custom appscale AMI instance with 'appscale up', it is giving me following error: [Errno 111] Connection refused

If I switch back to the original appscale ami( ami-0ef4bf66), I don't have issues.

Issue details:

---------------

shell> ssh -i /home/vagrant/.appscale/appscale2bce9326ccab4f11a83b1330249d7551.key -o LogLevel=quiet -o NumberOfPasswordPrompts=0 -o StrictHostkeyChecking=no -o UserKnownHostsFile=/dev/null root@ec2-[IP address].compute-1.amazonaws.com

       stdin str: service monit start

       stdout buffer: /tmp/tmpHec0ZC

shell> ssh -i /home/vagrant/.appscale/appscale2bce9326ccab4f11a83b1330249d7551.key -o LogLevel=quiet -o NumberOfPasswordPrompts=0 -o StrictHostkeyChecking=no -o UserKnownHostsFile=/dev/null root@ec2-[IP address].compute-1.amazonaws.com

       stdin str: monit start -g controller

       stdout buffer: /tmp/tmp42cK3X

Please wait for the AppController to finish pre-processing tasks.

Please wait for AppScale to prepare your machines for use.

[Errno 111] Connection refused

A log with more information is available at

/home/vagrant/.appscale/log-4338e1b8-cfbf-41d5-9a2b-9dc63edebb85.

--------------

That log file shows this: 

--------------------

stacktrace : Traceback (most recent call last):

  File "/usr/local/appscale-tools/bin/appscale", line 61, in <module>

    appscale.up()

  File "/usr/local/appscale-tools/bin/../lib/appscale.py", line 252, in up

    AppScaleTools.run_instances(options)

  File "/usr/local/appscale-tools/bin/../lib/appscale_tools.py", line 412, in run_instances

    instance_id)

  File "/usr/local/appscale-tools/bin/../lib/local_state.py", line 361, in update_local_metadata

    role_info = acc.get_role_info()

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 183, in get_role_info

    self.server.get_role_info, self.secret)

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 116, in run_with_timeout

    function, *args)

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 116, in run_with_timeout

    function, *args)

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 116, in run_with_timeout

    function, *args)

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 116, in run_with_timeout

    function, *args)

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 116, in run_with_timeout

    function, *args)

  File "/usr/local/appscale-tools/bin/../lib/appcontroller_client.py", line 118, in run_with_timeout

    raise exception

error: [Errno 111] Connection refused

exception : error

locale : en_US

tools_version : 2.2.0

platform : Linux-3.2.0-23-generic-x86_64-with-Ubuntu-12.04-precise

message : [Errno 111] Connection refused

runtime : CPython

-------------------

Please note I just added mysql installation to the original appscale ami: ami-0ef4bf66

and created a new ami.

Please let me know..Thanks much.

[Chag Will]

When I detached EBS volume and restarted the instance again, I see a different version of same error:

Received status from head node:     Currently using 0.0 Percent CPU and 12.40 Percent Memory

    Hard disk is 42 Percent full

    Is currently: load_balancer, taskqueue_master, zookeeper, db_master, taskqueue, memcache, database, shadow, login, appengine

    Database is at ec2-52-4-244-2.compute-1.amazonaws.com

    Is in cloud: cloud1

    Current State: Starting up AppManager

    Hosting the following apps: none

[Errno 111] Connection refused

Waiting 1 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 2 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 4 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 8 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 16 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

[Errno 111] Connection refused

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

So, I did CTRL+C, then it gave this:

Waiting 20 second(s) for ec2-52-4-244-2.compute-1.amazonaws.com:4343 to open

^CTraceback (most recent call last):

  File "/usr/local/appscale-tools/bin/appscale", line 61, in <module>

    appscale.up()

  File "/usr/local/appscale-tools/bin/../lib/appscale.py", line 252, in up

    AppScaleTools.run_instances(options)

  File "/usr/local/appscale-tools/bin/../lib/appscale_tools.py", line 432, in run_instances

    options.verbose)

  File "/usr/local/appscale-tools/bin/../lib/remote_helper.py", line 268, in sleep_until_port_is_open

    time.sleep(sleep_time)

KeyboardInterrupt

[Meni Vaitsi]

Are you on a cloud deployment on EC2 by running appscale up from your local machine?

Do you use different AppScalefiles for the two AMIs?

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems Inc.

[Chag Will]

Are you on a cloud deployment on EC2 by running appscale up from your local machine?

--Yes, I am using cloud deployment on EC2, and I run appscale commands from local.

Do you use different AppScalefiles for the two AMIs?

--I use same Appscalefile. I am just changing AMIs to test:

#machine : 'ami-0ef4bf66'

#machine : 'ami-9cdae0f4'

#machine : 'ami-94c6fcfc'

machine : 'ami-7e202516'

#machine : 'ami-ba2623d2'

The original 'ami-0ef4bf66' has no problems anytime.

Thank you.

[chris....@appscale.com]

Hi Chag,

The easiest thing to do would be to use AWS's RDS service. That way, you can point any mysql client to a RDS instance. However, that does incur extra costs, and what you're trying is possible without using RDS.

The issues you've been running into so far have likely been because doing an 'appscale up' alters the instance state. Here's the steps I took to create a new AMI without having to do an 'appscale up'.

1. Launch Instance from AWS Console
1. Select ami-2c80bc44 (This is the AppScale 2.3.0 image. I think you need to be in US East to select it.)
2. Select an instance type with >3G ram (I used m3.medium).
3. Add an EBS volume.
4. Select or create a security group that allows all TCP and UDP traffic (AppScale manages it's own firewall).
2. After launching, login to the instance as the ubuntu user.
1. Install mysql-server.
2. Stop mysql.
3. Move the mysql data directory to the EBS volume and update my.cnf.
4. Update /etc/apparmor.d/usr.sbin.mysqld.
3. From the AWS Console, create an image from the instance.
4. Start AppScale with the new AMI and EBS volume (which you'll need to detach from the original instance).
   

[Chag Will]

Hi Chirs & Meni,

Wow..this is amazing. I just followed steps from Chris and my custom AMIs are working without issues. Yes, it seems taking AMI snapshot by running instance with 'appscale up' does not seem to be the right way, as 'appscale up' seems to change the state of the instance which is in someway causing a faulty AMI being created. 

I appreciate your expertise and great help.

Thanks,

-CW

[Meni Vaitsi]

We're glad that this solves your issue. Please let us know if you run into any more problems.

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems Inc.

--

[Chag Will]

Hi Meni/Chris,

I have a follow up question. I need to start mysql at system startup. So, I ran the combination of these commands and created new ami's:

chkconfig mysql on

chkconfig --add mysql

echo "/etc/init.d/mysql start" >> /etc/rc.local

update-rc.d mysql defaults

None of them gave desired result, so mysql doesn't start automatically on the new EC2 instance. 

Any ideas? Thank you.

[Chag Will]

Also, in a normal situation those commands may have worked. But, my datadir is pointing to EBS volume, and it may be possible that the system is starting mysql before EBS volume is attached to the instance, in which case it would fail.

[Meni Vaitsi]

Hey there,

So first of al,l you need to look at what the error is for starting mysql. Either check mysql server's log location or redirect the output from the rc.local command to a file.

As a second option, I would suggest to put the command for starting the mysql server in your AppScalefile, in the section user_commands, so that it starts it on appscale up after the volume is attached.

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems Inc.

On Wed, Apr 22, 2015 at 11:37 AM, Chag Will <chag...@gmail.com> wrote:

Also, in a normal situation those commands may have worked. But, my datadir is pointing to EBS volume, and it may be possible that the system is starting mysql before EBS volume is attached to the instance, in which case it would fail.

--

[Chag Will]

Hi Meni,

I have been trying several commands under user_commands in Appscale file, like these:

service mysql start

mysqld_safe & (also without &)

update-rc.d mysql defaults

echo "/etc/init.d/mysql start" >> /etc/rc.local

and so on.

On console, I do see those commands are picked up. For ex:

Running user-specified commands at ec2-[IP address].compute-1.amazonaws.com

shell> ssh -i /home/vagrant/.appscale/appscaleffb80708e9664fd583666b2db5373851.key -o LogLevel=quiet -o NumberOfPasswordPrompts=0 -o StrictHostkeyChecking=no -o UserKnownHostsFile=/dev/null root@ec2-[IP address].compute-1.amazonaws.com

       stdin str: echo "mysqld_safe &" >> /etc/rc.local

       stdout buffer: /tmp/tmp1IeUe6

But once the instance is started, I get the following error in /var/log/mysql/error.log:

===========================

150423  6:15:09 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.

150423  6:15:09 [Note] Plugin 'FEDERATED' is disabled.

/usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist

150423  6:15:09 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.

150423  6:15:09 InnoDB: The InnoDB memory heap is disabled

150423  6:15:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins

150423  6:15:09 InnoDB: Compressed tables use zlib 1.2.3.4

150423  6:15:09 InnoDB: Initializing buffer pool, size = 128.0M

150423  6:15:09 InnoDB: Completed initialization of buffer pool

150423  6:15:09  InnoDB: Operating system error number 13 in a file operation.

InnoDB: The error means mysqld does not have the access rights to

InnoDB: the directory.

InnoDB: File name ./ibdata1

InnoDB: File operation call: 'create'.

InnoDB: Cannot continue operation.

I did try mysql_upgrade as suggested in error, but the issue is still there.

Then, When I manually start mysql server with any of above commands, the log is clear with no error:

===========================

150423  6:36:08 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.

150423  6:36:08 [Note] Plugin 'FEDERATED' is disabled.

150423  6:36:08 InnoDB: The InnoDB memory heap is disabled

150423  6:36:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins

150423  6:36:08 InnoDB: Compressed tables use zlib 1.2.3.4

150423  6:36:08 InnoDB: Initializing buffer pool, size = 128.0M

150423  6:36:08 InnoDB: Completed initialization of buffer pool

150423  6:36:08 InnoDB: highest supported file format is Barracuda.

150423  6:36:08  InnoDB: Waiting for the background threads to start

150423  6:36:09 InnoDB: 5.5.41 started; log sequence number 1595685

150423  6:36:09 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306

150423  6:36:09 [Note]   - '127.0.0.1' resolves to '127.0.0.1';

150423  6:36:09 [Note] Server socket created on IP: '127.0.0.1'.

150423  6:36:09 [Note] Event Scheduler: Loaded 0 events

150423  6:36:09 [Note] /usr/sbin/mysqld: ready for connections.

Version: '5.5.41-0ubuntu0.12.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

[Meni Vaitsi]

Hi,

It looks like there is a permissions issue based on the error output. 

AppScale runs as root so you need to make sure that mysql components are owned by root.

Do you ssh as ubuntu into your instance when you run the command by hand?

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems Inc.

[Chag Will]

Hi Meni, I use 'appscale ssh' to login as root and run the mysql command manually.I have changed the permissions of the datadir folder as mentioned in this link: https://www.centos.org/forums/viewtopic.php?f=14&t=44526&start=10

Didn't seem to work.

Since I am running mysql as a root user manually, I am not sure If I still need to change the folder ownership to root, but I will definitely try that as well. I will let you know. Thanks very much. 

[Chag Will]

Hi Meni, 

Permissions could be one issue but that is resulting from other real issue:- After trying few more things, the issue turned out to be DATADIR. So, I have created a fresh AMI with default datadir (/var/lib/mysql) and 'sysv-rc-conf mysql on' (this sets mysql to start at bootup). I used this image to launch the instance and 

mysql server is started automatically without issues. Then, I have other AMI with datadir pointing to EBS volume and 'sysv-rc-conf mysql on' was also run. When I used this image, mysql doesn't start. Further, I tried 'user-commands' to start mysql server. That didn't work either. Then, one of the 'user-commands' was:

'cp -p /opt/appscale/rc.local /etc' and it gave an error saying it can't find /opt/appscale: 

cp: cannot stat `/opt/appscale/rc.local': No such file or directory

So, it seems that '/opt/appscale' from EBS disk was not found during the booting process hence failing to start mysql. May be that directory is attached at the end or after reboot.

So, in a way it makes sense why this has been failing. I will probably use a shell script to start mysql after the reboot or I will just run it manually for now. Thanks so much for providing timely answers and guidance.

Cheers.

[Meni Vaitsi]

Hi Chag,

We're always here to help if you need to take another shot at it. We can set up some time and have a closer look.

Thanks for using AppScale.

-Meni

--

Meni Vaitsi

Software Engineer
AppScale Systems Inc.

[Chag Will]

Hi, I think I am ok for now but I am sure I will need some more help on the way later. Thanks very much.