MySQL LVM 64 Issues

[mikeytag]

As of 5 AM PST this morning, my mysqllvm64 master instance would hang
on inserts. Selects were fine, but nothing would get inserted. I
noticed that this all started when it received a bundle trap. We do a
bundle every 5 hours. Fearing this as the culprit I unchecked the
bundling box for now. However, on that screen I noticed that it says
my db instance (custom mysqllvm64 role) is Based on role: mysql.
Maybe this is the culprit? Did Scalr somehow switch my db instances
into operating as mysql roles instead of mysqllvm64 roles?
My farm id is 517 if that helps at all.
Thanks,
Mike

[Alex Kovalyov]

Guys will be able to check this in few hours only..
Enough space on instance? Using InnoDB? What's in mysql logs?

[mikeytag]

Thanks Alex,

InnoDB = yes
/var/log/mysql.log and /var/log/mysql.err both blank.
/var/log/messages only had info about the bundle process
/var/log/daemon.log only had info about snmpd udp connectins

Unfortunately, in a desperate attempt to get back online, I have
terminated all instances and am initializing a new master right now.
Unfortunately, that process takes about 5 hours. Our database size is
around 60GB.

Mike

On Oct 23, 1:30 pm, Alex Kovalyov <alex.koval...@gmail.com> wrote:
> Guys will be able to check this in few hours only..
> Enough space on instance? Using InnoDB? What's in mysql logs?
>

[mikeytag]

Alex,
It seems that Scalr does the bundles and backups on the master
instance. Is there any way to switch this or give an option to doing
it on a slave? I am thinking along the lines of:

1. Pull slave out of DNS and /etc/aws/hosts/mysql
2. Bundle data
3. Bring slave back

That way the master never encounters extra load to bundle and upload
the instances. Our application is very write heavy, so our master
instance is under high load, even with an m1.xlarge instance. (Crazy I
know ;)) Moving bundling to a slave instance would help preserve any
precious resources on the master and I would gladly pay the extra cost
to have an extra slave running if it helps with the stability of the
master.

The truth is we are bumping up against the limits of a master/slave
replication based architecture as our writes are continuing to build.
I am looking into alternative designs including sharding, ScaleDB
(beta), etc. In the mean time, it would be nice to squeeze every last
ounce out of the master that I can.

Mike

On Oct 23, 1:30 pm, Alex Kovalyov <alex.koval...@gmail.com> wrote:

> Guys will be able to check this in few hours only..
> Enough space on instance? Using InnoDB? What's in mysql logs?
>

[Steve Ru]

Are you using EBS?

[Alex Kovalyov]

We're going to move both operations to slave, if more than one mysql
instances running.