Looking at ZRM with XtraBackup plugin

[Henrik Ingo]

Hi Lachlan, any others on the list

So I had a look at using ZRM together with the new xtrabackup plugin
from vmfarms:
http://blog.vmfarms.com/2011/03/innodb-hot-backups-with-mysql-zrm-for.html

For the question whether we should just abandon XBM and focus on this
as an existing and older tool, I don't think there is an easy answer.
But perhaps working on XBM is the way to go...

Pro ZRM:

- It's an existing tool and has a user base.
- It does work. Also the xtrabackup plugin from vmfarms seems to work.

Against ZRM:

- The open source version is not maintained anymore, it was released
6 years ago. With MySQL 5.5 it apparently tries to back up Performance
Schema and dies.
- Documentation is a bit wieldy (but at least they have it :-). It
took me a long time before understanding anything of what I'm supposed
to do.
- I didn't come out feeling particularly fond of it. Examples:
- Due to the history of supporting all the hacky backup methods we
historically used, it is now this frankenstein tool which will
intelligently choose to back up some databases with xtrabackup and
some with mysqldump. Just on MyISAM table is enough to fall back to
mysqldump for the whole database.
- I end up with half of the database copied by xtrabackup, the other
in a mysqldump sql file. Restoration would be unnecessarily complex
(note that innobackupex will instead copy the myisam data files, not
dumping the sql).
- Setting MySQL general log to on, I don't see that tables are ever
locked. So I'm not so sure the recorded binlog position has any value
in a real world scenario. (again, innobackupex does this correctly)

I'm sure I could configure away some of the above issues (like getting
a copy of myisam tables instead of a sql dump). But the first
impression wasn't perfect.

So it seems building XBM as a layer on top of innobackupex is a good
ground to build upon - especially as you've already come quite far.

henrik
--
henri...@avoinelama.fi
+358-40-8211286 skype: henrik.ingo irc: hingo
www.openlife.cc

My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

[Lachlan Mulcahy]

Hi Henrik,

I'm so sorry that I _completely_ missed this email.

I really need to get my filters on this gmail account setup better -- as it stands I am receiving everything from the bugs list on xtrabackup all in amongst the rest.

I had been a little side-tracked getting my new US Visa. There was a bit of stress around that, but it is all sorted out now and I'm slowly coming back to reality.

I am glad to hear that you think XBM still warrants development :)

Also, I am glad to see that we at least took some time to evaluate ZRM -- Hopefully we can take learnings of good things away from it, but leave the bad behind.

At the moment I am sidelined on some other tasks at work, but hoping to get them finished over the next 2 weeks or so -- then I will be coming back around to XBM.

The next step that I am working on will be a configurator so that setting up and managing things isn't so unwieldy. With the increased complexity of backup strategies and parameters for those strategies, I am noticing that using the DB to set things up is just more and more painful. Even for me as a guy who wrote the code.

Also, as an aside, I have started reading a book "PHP Objects, Patterns and Practice" (3rd ed). I'm hoping that it will help round out some of my coding knowledge/ideas since I have learned most of my PHP from a PHP4 book many years ago and just osmosis since then :)

How are things on your side?

Lachlan

[Henrik Ingo]

On Wed, Jul 6, 2011 at 7:19 PM, Lachlan Mulcahy
<lachlan...@gmail.com> wrote:
> How are things on your side?

Well, July is vacation.

Like I indicated earlier it seems I will not be able to spend more
working time together with XBM in the future. I hope my reviews and
small contributions are still helpful.

> I am glad to hear that you think XBM still warrants development :)

Well, I think once there is a better interface for configuration, it
can quite easily take the place of the best open source alternative
out there.

Even if there now is an xtrabackup plugin for ZRM, the fact that ZRM
itself is not maintained anymore (the open source code base) means
that there is pretty much no competition for this. You're clearly
filling a need.

[Mikael Fridh]

Completely without weighing the options I just thought I'd mention the Holland backup framework which I find quite attractive, with it's plugin architecture:

http://hollandbackup.org/
http://docs.hollandbackup.org/amsterdam/backup-sets/xtrabackup.html#xtrabackup

Plus, it has a very nice tag line.

And on the topic of configuration in the database, you tend to be forced into building very complex UIs or tools just to allow people to edit the configuration when it's so much easier to do it file-based and allow for traditional xbm.d/*.conf style config which invites the user to build his configuration using his favourite tools, for loops or even using configuration management such as puppet. I'm not saying it can't be done in the database as well, but I say priority-wise I would personally like my configuration on file rather than using a web UI. The Web UI can come later on...

And why waste the effort when VIM has the best UI in the world :)

[Lachlan Mulcahy]

Hi Mikael,

Holland looks like another viable option for a backups, but I'm not sure it offers the rich level of integration with XtraBackup that I'd like to see.

eg. It seems to have a concept of retention (how many backups to keep) but I'm not sure the XtraBackup plugin will know when to take a full backup, or an incremental (or if it even supports incrementals).

The sweet spot I'm hoping to hit with XBM is more of a focus on XtraBackup so that you can really take advantage of things like full backups an incrementals and controlling what your strategy should be for keeping and collapsing them.

So far (to my knowledge) both ZRM and Holland don't do this …

In other words.. I'm still feeling OK that my plugging away at this thing is not just reinventing someone else's wheel :D

Thought I admit that I do worry every time someone mentions another project that I'll realise something already does this pretty well and the /facepalm will follow :)

Lachlan

[Mikael Fridh]

On Thu, Jul 28, 2011 at 8:48 PM, Lachlan Mulcahy
<lachlan...@gmail.com> wrote:
> Hi Mikael,
>
> Holland looks like another viable option for a backups, but I'm not sure it offers the rich level of integration with XtraBackup that I'd like to see.

I do agree, I think it's hard to cover database backups in another
tool which does "everything". It's probably not hard to get the data
_in_ to a generic backup tool, but getting it back out again -
processing, incrementally applying things etc while keeping it
consistent in _that_ backup software's catalog - it feels very
difficult.

> eg. It seems to have a concept of retention (how many backups to keep) but I'm not sure the XtraBackup plugin will know when to take a full backup, or an incremental (or if it even supports incrementals).
>
> The sweet spot I'm hoping to hit with XBM is more of a focus on XtraBackup so that you can really take advantage of things like full backups an incrementals and controlling what your strategy should be for keeping and collapsing them.
>
> So far (to my knowledge) both ZRM and Holland don't do this …
>
> In other words.. I'm still feeling OK that my plugging away at this thing is not just reinventing someone else's wheel :D
>
> Thought I admit that I do worry every time someone mentions another project that I'll realise something already does this pretty well and the /facepalm will follow :)

I haven't seen anything specialized like this in the wild, and if
there was, just do everything it does a little better!

--
Mikael Fridh