Roadmap kind of planning

33 views
Skip to first unread message

Henrik Ingo

unread,
Mar 30, 2011, 9:49:08 AM3/30/11
to xtrabackup-mgr
Hi Lachlan

Having successfully worked on non-trivial new code yesterday, I
thought now is a good time to share what I want/need to do next so
that I can use this. (Ok, some are unimportant cleanup/refactoring,
but throwing them in for completeness.)

1) Move stuff from config.php to a config mysql table - it is a more
GUI friendly place to keep such things, also prefer to have 1 place
for configs, not two. This can be done in a way that doesn't change
how current code works, I'll add a utility class that just populates
the $config array from the database so it is identical to what it
looks like now. The knowledge of how to connect to mysql must of
course remain in config.php :-)

2) make --datadir optional
a) innobackupex doesn't need --datadir=".$sbInfo['datadir_path'], it
can pick it up from my.cnf
b) xtrabackup doesn't need to know the datadir at all for --prepare phase.
I'd like to make it possible to leave this parameter empty. (In
backupSnapshotTaker and scheduledBackup classes.)

3) Ability to have different backup strategy and method
- backup strategy (just full dump, continuous increment, full dump +
increments cycle...)
- backup interval (nightly, weekly, monthly... both for new fresh
full dump and increments)
- backup method (Maybe I want to do a mysqldump once in a while.
Having a logical backup as fallback option makes me sleep better at
night.

The cron_expression kind of is the interval already.

What you have now I call "continuous increment" strategy. I'm planning
to do a variation where I keep a staging copy to do incremental
backups on, but then also make a copy where I do the crash recovery
part of xtrabackup --prepare to the end. Knowing that it succeeded
with that gives me confidence that once I need to do recovery, mysqld
will actually start on the data and not hit a recovery bug.

To enable this, what is currently backupSnapshotTaker must be
abstracted away, so I can create many such classes.
- create "backupTaker" interface or abstract class.
- create a backupTakerFactory class that returns a backupTaker of the
type requested. Type must be defined in scheduled_backup table or
somewhere.
- more complex than just a suite of classes implementing/inheriting
one interface. It seems to me strategy/interval/method are separate
dimensions and can be combined arbitrarily.

4) Binary log (PiT)
Related to the previous point, I also need to copy the binary log from
the server, to enable point in time recovery.
Bonus for creating a CLI/GUI for such recovery - that is likely way
into the future :-) (Currently our recovery command is just cp/scp
anyway!)

5) Verification
I want to verify the backup after it completes.
- copy the database and start a mysql server on top of it.
- I'm planning to use something like maatkit checksum script to make
a "before and after" kind of verification. Details to be investigated.
- could also do sanity checks for instance that size of data is
approximately right compared to origin and/or last weeks backup.

6) Retention policy
- once backup is verified, copy it to some final destination (like in
another datacenter, to tape, whatever).
- delete old backups

7) choosing xtrabackup binary should be automatic.
It disturbs me that you have to define the mysql_type_id in
scheduled_backups. innobackupex figures the mysql version out
automatically, and stores it in a file xtrabackup_binary that is
stored in the backup directory (along with other meta data such as
checkpoints, binlog_info...). From there you can find out the correct
xtrabackup binary to use in following steps (like for --prepare).

So I realise from your earlier email you are using this to sometimes
select the "wrong" xtrabackup version to work around recovery bugs
when doing --prepare. The suggestion to bundle innobackupex and
xtrabackup into xbm could be a solution allowing us to fix such
problems. In any case, I'd like the following behavior:
- identifying mysql version and using correct xtrabackup binary
should be transparent to end user.
- when applying logs and doing incremental backups one should use the
same xtrabackup binary using the information recorded in
xtrabackup_binary.
- if MySQL server version changes, the step should fail. user must
start again with full backup.
- the previous point requires that in all steps we re-check the mysql
version (except for just doing --prepare where mysqld is not involved)
and abort if major version is different.
- it would actually be a useful innobackupex feature to also store
MySQL version string in the backup metadata. I'm thinking here of the
recovery scenario, in disaster recovery the specific version could be
assumed to be a known good mysqld version that should be used. (I'm
thinking here of purely hypothetical environments that run a mess of
different MySQL versions in production :-) I'll see if I can submit
such a thing upstream.

The mysql_types table might still be needed in MySQL so XBM can match
detected mysqld version string and version of xtrabackup binary. But
it shouldn't be a foreign key of scheduled_backups table - the
decision should be invisible to user. If we need workarounds, let's
hide them in the code somewhere instead :-)


#) cleanup: In some tables it seems more natural to use a short string
as id and primary key rather than auto_increment. In particular the
parameter to ./xbm-backup -s ... I should be able to use something
like hostname instead of magic number.

#) When ssh:ing into the remote host, XBM should create an empty
working directory and cd into it, then run innobackupex and xtrabackup
from there. It is not nice to create lot's of files in the home
directory where you end up after login.


Of the above list, a simple version of (3) and (5) are the ones I
intend to mostly work on next, but I'm likely to do some simple ones
like (1) and annoying ones like (7) too.

henrik
--
henri...@avoinelama.fi
+358-40-5697354 skype: henrik.ingo irc: hingo
www.openlife.cc

My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

Lachlan Mulcahy

unread,
Mar 30, 2011, 1:09:10 PM3/30/11
to xtrabac...@googlegroups.com
Hi Henrik,

My comments inline...

On 30/03/2011, at 6:49 AM, Henrik Ingo wrote:

> Having successfully worked on non-trivial new code yesterday, I
> thought now is a good time to share what I want/need to do next so
> that I can use this. (Ok, some are unimportant cleanup/refactoring,
> but throwing them in for completeness.)

No problem :)

> 1) Move stuff from config.php to a config mysql table - it is a more
> GUI friendly place to keep such things, also prefer to have 1 place
> for configs, not two. This can be done in a way that doesn't change
> how current code works, I'll add a utility class that just populates
> the $config array from the database so it is identical to what it
> looks like now. The knowledge of how to connect to mysql must of
> course remain in config.php :-)

I agree with this - originally I think I was just throwing this together to get the quick proof of concept stuff going. It should be really easy to plug this in.

By the sounds of it this is something you're happy to work on?

> 2) make --datadir optional
> a) innobackupex doesn't need --datadir=".$sbInfo['datadir_path'], it
> can pick it up from my.cnf
> b) xtrabackup doesn't need to know the datadir at all for --prepare phase.
> I'd like to make it possible to leave this parameter empty. (In
> backupSnapshotTaker and scheduledBackup classes.)

This seems like a perfectly reasonable request to me. Less configuration is better... let's try to make it as easy as possible for people with simple installations and allow overrides where necessary.

> 3) Ability to have different backup strategy and method
> - backup strategy (just full dump, continuous increment, full dump +
> increments cycle...)

I like this idea as just continually incrementing is not great -- eventually you worry that you've gotten far enough away from the seed that something could have gone wrong.

It's nice to take a fresh full backup snapshot on a regular interval and then increment from that.

> - backup interval (nightly, weekly, monthly... both for new fresh
> full dump and increments)

As you said below.. the cron_expression is kind of the interval already. The reason that I picked cron_expression rather than something a little more "friendly" was to avoid reinventing the wheel. Cron already does scheduling well and it exists almost everywhere and trying to convert between "english" and a cron_expression is actually pretty difficult.

I went to see if there was a nice way to do it -- like if someone had written some classes or something, but it seems a pretty tricky problem.

For now I'd say lets stick with using cron expression methodology and cron and focus on the other features that should make a stronger difference.

> - backup method (Maybe I want to do a mysqldump once in a while.
> Having a logical backup as fallback option makes me sleep better at
> night.
>
> The cron_expression kind of is the interval already.
>
> What you have now I call "continuous increment" strategy. I'm planning
> to do a variation where I keep a staging copy to do incremental
> backups on, but then also make a copy where I do the crash recovery
> part of xtrabackup --prepare to the end. Knowing that it succeeded
> with that gives me confidence that once I need to do recovery, mysqld
> will actually start on the data and not hit a recovery bug.

It seems like this could be a feature or option to that particular method.

Maybe we have a concept of methods and then methodOptions which are similar key/value type pairings to the way the config class would work. Maybe we can use some object inheritance there.

This way we can have a basic continuous increment and then an option to perform some verify step at the end.

> To enable this, what is currently backupSnapshotTaker must be
> abstracted away, so I can create many such classes.
> - create "backupTaker" interface or abstract class.
> - create a backupTakerFactory class that returns a backupTaker of the
> type requested. Type must be defined in scheduled_backup table or
> somewhere.
> - more complex than just a suite of classes implementing/inheriting
> one interface. It seems to me strategy/interval/method are separate
> dimensions and can be combined arbitrarily.

I would guess that strategy and method would be more closely paired.

eg. Continuous increment strategy cannot be employed if your method is full dump.

> 4) Binary log (PiT)
> Related to the previous point, I also need to copy the binary log from
> the server, to enable point in time recovery.
> Bonus for creating a CLI/GUI for such recovery - that is likely way
> into the future :-) (Currently our recovery command is just cp/scp
> anyway!)

Yes this is something that we will actually need ourselves.

> 5) Verification
> I want to verify the backup after it completes.
> - copy the database and start a mysql server on top of it.
> - I'm planning to use something like maatkit checksum script to make
> a "before and after" kind of verification. Details to be investigated.
> - could also do sanity checks for instance that size of data is
> approximately right compared to origin and/or last weeks backup.

This seems like it could be handled by adding hooks for pre and post scripts?

> 6) Retention policy
> - once backup is verified, copy it to some final destination (like in
> another datacenter, to tape, whatever).
> - delete old backups
>
> 7) choosing xtrabackup binary should be automatic.
> It disturbs me that you have to define the mysql_type_id in
> scheduled_backups. innobackupex figures the mysql version out
> automatically, and stores it in a file xtrabackup_binary that is
> stored in the backup directory (along with other meta data such as
> checkpoints, binlog_info...). From there you can find out the correct
> xtrabackup binary to use in following steps (like for --prepare).
>
> So I realise from your earlier email you are using this to sometimes
> select the "wrong" xtrabackup version to work around recovery bugs
> when doing --prepare. The suggestion to bundle innobackupex and
> xtrabackup into xbm could be a solution allowing us to fix such
> problems. In any case, I'd like the following behavior:
> - identifying mysql version and using correct xtrabackup binary
> should be transparent to end user.
> - when applying logs and doing incremental backups one should use the
> same xtrabackup binary using the information recorded in
> xtrabackup_binary.

Interestingly enough I have been told by Vadim that evidence suggests that "xtrabackup" binary itself should work fine for all cases. So it may be that we just use that for everything.

Also, there is a bug currently where xtrabackup_binary file is not correctly copied when you do a streaming backup -- instead it gets created in the working dir.

So for now at least we can't rely on this file being there -- hopefully we can just simplify the whole thing and run "xtrabackup" binary only and do away with the other complications! :)

> - if MySQL server version changes, the step should fail. user must
> start again with full backup.

This makes some sense, but perhaps in this case we should do something like:

* If minor version change only - send an alert and continue taking incrementals -- allow user action to determine whether to start fresh
* If major version change - send an alert - maybe behaviour should be configurable? eg. stop completely and wait for user action OR proceed with taking a new seed.

> - the previous point requires that in all steps we re-check the mysql
> version (except for just doing --prepare where mysqld is not involved)
> and abort if major version is different.

mysqld is not exactly involved, but xtrabackup is kind of mysqld-ish inside anyway :)

> - it would actually be a useful innobackupex feature to also store
> MySQL version string in the backup metadata. I'm thinking here of the
> recovery scenario, in disaster recovery the specific version could be
> assumed to be a known good mysqld version that should be used. (I'm
> thinking here of purely hypothetical environments that run a mess of
> different MySQL versions in production :-) I'll see if I can submit
> such a thing upstream.
>
> The mysql_types table might still be needed in MySQL so XBM can match
> detected mysqld version string and version of xtrabackup binary. But
> it shouldn't be a foreign key of scheduled_backups table - the
> decision should be invisible to user. If we need workarounds, let's
> hide them in the code somewhere instead :-)

We'll see about whether types will be needed. I'm still running into a bunch of roadblocks on my end -- so far my simple test has shown that using "xtrabackup" binary only, even for Percona Server 5.0.77, is working just fine.

I need to do some more tests -- another problem I'm having now is that on some hosts sshd is not returning the session after the INCREMENTAL xtrabackup is taken so we just get stuck waiting for ssh to come back.

> #) cleanup: In some tables it seems more natural to use a short string
> as id and primary key rather than auto_increment. In particular the
> parameter to ./xbm-backup -s ... I should be able to use something
> like hostname instead of magic number.

I'd suggest we keep the use of integer IDs for internal DB level linking purposes, but add a feature like shortname or somesuch so that users can identify things by those names themselves.

I would guess only a few things like the scheduled backup name or storage volume would need that.

> #) When ssh:ing into the remote host, XBM should create an empty
> working directory and cd into it, then run innobackupex and xtrabackup
> from there. It is not nice to create lot's of files in the home
> directory where you end up after login.

Agree here. Especially if we're copying our own xtrabackup binary / innobackupex scripts.

> Of the above list, a simple version of (3) and (5) are the ones I
> intend to mostly work on next, but I'm likely to do some simple ones
> like (1) and annoying ones like (7) too.

It all sounds like viable and worthwhile work to me and very much along the lines of the stuff I need and want to add.

I would suggest maybe we go about creating some individual "Issues" against the Google Code project for these action items/coding tasks. I have done this previously by listing them as "Enhancement" type.

Then we can keep track of them more individually.

What do you think?

Lachlan

Henrik Ingo

unread,
Mar 30, 2011, 4:04:39 PM3/30/11
to xtrabac...@googlegroups.com
On Wed, Mar 30, 2011 at 8:09 PM, Lachlan Mulcahy
<lachlan...@gmail.com> wrote:
>> 1) Move stuff from config.php to a config mysql table - it is a more
> I agree with this - originally I think I was just throwing this together to get the quick proof of concept stuff going. It should be really easy to plug this in.
>
> By the sounds of it this is something you're happy to work on?
>

I would have done it today but wrote this instead :-)

>> 3) Ability to have different backup strategy and method

> I like this idea as just continually incrementing is not great -- eventually you worry that you've gotten far enough away from the seed that something could have gone wrong.
>
> It's nice to take a fresh full backup snapshot on a regular interval and then increment from that.

Just feels so much safer :-)

>> What you have now I call "continuous increment" strategy. I'm planning
>> to do a variation where I keep a staging copy to do incremental
>> backups on, but then also make a copy where I do the crash recovery
>> part of xtrabackup --prepare to the end. Knowing that it succeeded
>> with that gives me confidence that once I need to do recovery, mysqld
>> will actually start on the data and not hit a recovery bug.
>
> It seems like this could be a feature or option to that particular method.

It could. But I don't think you can cover all imaginable strategies
with just adding more options onto one mega class. I'll work on this
soonish, then you can comment on actual code. Both inheritance or just
completely different objects implementing the same interface will
eventually allow much more versatile behavior.

But as a short term solution I don't plan to overdo it. I will
implement one mega-class if it is satisfactory for me :-)

>> To enable this, what is currently backupSnapshotTaker must be
>> abstracted away, so I can create many such classes.
>> - create "backupTaker" interface or abstract class.
>> - create a backupTakerFactory class that returns a backupTaker of the
>> type requested. Type must be defined in scheduled_backup table or
>> somewhere.
>> - more complex than just a suite of classes implementing/inheriting
>> one interface. It seems to me strategy/interval/method are separate
>> dimensions and can be combined arbitrarily.
>
> I would guess that strategy and method would be more closely paired.
>
> eg. Continuous increment strategy cannot be employed if your method is full dump.

"Just full dumps" is also a strategy.

Mysqldump is a method. Mysqldump is a limited tool, so it can only do
"just full dumps".

Xtrabackup can do continuous increment, just full dump and weekly
cycle of full dump + daily increments, etc...

> Also, there is a bug currently where xtrabackup_binary file is not correctly copied when you do a streaming backup -- instead it gets created in the working dir.
>

Bug number? This works for me - I'm pretty sure. I did use streaming.
But I'm using the 1.5 release candidate version.

> So for now at least we can't rely on this file being there -- hopefully we can just simplify the whole thing and run "xtrabackup" binary only and do away with the other complications! :)
>

This system of having different binaries always felt fishy - now that
you suggest using "wrong" binary... even worse!

>> - if MySQL server version changes, the step should fail. user must
>> start again with full backup.
>
> This makes some sense, but perhaps in this case we should do something like:
>
> * If minor version change only - send an alert and continue taking incrementals -- allow user action to determine whether to start fresh
> * If major version change - send an alert - maybe behaviour should be configurable? eg. stop completely and wait for user action OR proceed with taking a new seed.
>

I'd say the minor version change could be user configurable. Doing
increments over major version change seems like something we could
authoritatively forbid from XBM completely - just require a fresh full
backup in such cases. You only do major version upgrades like once a 3
years.

>> - the previous point requires that in all steps we re-check the mysql
>> version (except for just doing --prepare where mysqld is not involved)
>> and abort if major version is different.
>
> mysqld is not exactly involved, but xtrabackup is kind of mysqld-ish inside anyway :)

Each incremental backup is taken from a running mysqld. I want to
check the version of that mysqld. I know xtrabackup doesn't connect to
the mysqld, I want to do it for additional safety.

> I need to do some more tests -- another problem I'm having now is that on some hosts sshd is not returning the session after the INCREMENTAL xtrabackup is taken so we just get stuck waiting for ssh to come back.
>

I think I experienced that, but it was when I had other problems (like
home directory not writable) so didn't pay attention.

>> #) cleanup: In some tables it seems more natural to use a short string
>> as id and primary key rather than auto_increment. In particular the
>> parameter to ./xbm-backup -s ... I should be able to use something
>> like hostname instead of magic number.
>
> I'd suggest we keep the use of integer IDs for internal DB level linking purposes, but add a feature like shortname or somesuch so that users can identify things by those names themselves.
>
> I would guess only a few things like the scheduled backup name or storage volume would need that.

I'm ok with this. Insisting on integer primary keys is a valid style
for schema design. I only care about the end user experience.

The reason I suggested making the short name the primary key directly
is that currently the mysql tables are the user interface :-)
Especially if you look at my xbm-conftool.

> It all sounds like viable and worthwhile work to me and very much along the lines of the stuff I need and want to add.
>
> I would suggest maybe we go about creating some individual "Issues" against the Google Code project for these action items/coding tasks. I have done this previously by listing them as "Enhancement" type.
>
> Then we can keep track of them more individually.

I will do so tomorrow. Wanted to hear comments first, will integrate
what has been agreed to the issues.

Lachlan Mulcahy

unread,
Mar 30, 2011, 4:33:38 PM3/30/11
to xtrabac...@googlegroups.com, Henrik Ingo
Hi Henrik,


On 30/03/2011, at 1:04 PM, Henrik Ingo wrote:

> On Wed, Mar 30, 2011 at 8:09 PM, Lachlan Mulcahy
> <lachlan...@gmail.com> wrote:
>>> 1) Move stuff from config.php to a config mysql table - it is a more
>> I agree with this - originally I think I was just throwing this together to get the quick proof of concept stuff going. It should be really easy to plug this in.
>>
>> By the sounds of it this is something you're happy to work on?
>>
>
> I would have done it today but wrote this instead :-)

hah! Fair enough!

>>> What you have now I call "continuous increment" strategy. I'm planning
>>> to do a variation where I keep a staging copy to do incremental
>>> backups on, but then also make a copy where I do the crash recovery
>>> part of xtrabackup --prepare to the end. Knowing that it succeeded
>>> with that gives me confidence that once I need to do recovery, mysqld
>>> will actually start on the data and not hit a recovery bug.
>>
>> It seems like this could be a feature or option to that particular method.
>
> It could. But I don't think you can cover all imaginable strategies
> with just adding more options onto one mega class. I'll work on this
> soonish, then you can comment on actual code. Both inheritance or just
> completely different objects implementing the same interface will
> eventually allow much more versatile behavior.
>
> But as a short term solution I don't plan to overdo it. I will
> implement one mega-class if it is satisfactory for me :-)

Sorry - It seems like I was not clear.

I agree completely with your idea of having a number of subclass modules so that you can choose method, etc.

What I meant is that perhaps these modules/methods could also have applicable "methodOptions" - which would be like a set of options and possible values paired to a particular method.

Then a scheduled_backup would have a method and then a bunch of methodOption settings which are like an option_name and value.

>>> To enable this, what is currently backupSnapshotTaker must be
>>> abstracted away, so I can create many such classes.
>>> - create "backupTaker" interface or abstract class.
>>> - create a backupTakerFactory class that returns a backupTaker of the
>>> type requested. Type must be defined in scheduled_backup table or
>>> somewhere.
>>> - more complex than just a suite of classes implementing/inheriting
>>> one interface. It seems to me strategy/interval/method are separate
>>> dimensions and can be combined arbitrarily.
>>
>> I would guess that strategy and method would be more closely paired.
>>
>> eg. Continuous increment strategy cannot be employed if your method is full dump.
>
> "Just full dumps" is also a strategy.
>
> Mysqldump is a method. Mysqldump is a limited tool, so it can only do
> "just full dumps".
>
> Xtrabackup can do continuous increment, just full dump and weekly
> cycle of full dump + daily increments, etc...

Agreed. So there is some correlation there to figure out.

>> Also, there is a bug currently where xtrabackup_binary file is not correctly copied when you do a streaming backup -- instead it gets created in the working dir.
>>
>
> Bug number? This works for me - I'm pretty sure. I did use streaming.
> But I'm using the 1.5 release candidate version.

Bug is here: https://bugs.launchpad.net/percona-xtrabackup/+bug/723318

Doesnt work for me using 1.4 -- have not tried 1.5.

>> So for now at least we can't rely on this file being there -- hopefully we can just simplify the whole thing and run "xtrabackup" binary only and do away with the other complications! :)
>>
>
> This system of having different binaries always felt fishy - now that
> you suggest using "wrong" binary... even worse!

Yeah -- I will take a hack that works over a broken system any day ;-)

Some sacrifices are unfortunate, but need to be made. With enough testing we will find what to use.


>>> - if MySQL server version changes, the step should fail. user must
>>> start again with full backup.
>>
>> This makes some sense, but perhaps in this case we should do something like:
>>
>> * If minor version change only - send an alert and continue taking incrementals -- allow user action to determine whether to start fresh
>> * If major version change - send an alert - maybe behaviour should be configurable? eg. stop completely and wait for user action OR proceed with taking a new seed.
>>
>
> I'd say the minor version change could be user configurable. Doing
> increments over major version change seems like something we could
> authoritatively forbid from XBM completely - just require a fresh full
> backup in such cases. You only do major version upgrades like once a 3
> years.

Agree - major version changes should probably be forbidden. options could be to just stop completely or automatically start taking new full backup.

>>> - the previous point requires that in all steps we re-check the mysql
>>> version (except for just doing --prepare where mysqld is not involved)
>>> and abort if major version is different.
>>
>> mysqld is not exactly involved, but xtrabackup is kind of mysqld-ish inside anyway :)
>
> Each incremental backup is taken from a running mysqld. I want to
> check the version of that mysqld. I know xtrabackup doesn't connect to
> the mysqld, I want to do it for additional safety.

I think I'm getting confused here - I guess what you're saying is that you want to make sure that you don't end up taking a full backup from one version and then applying an incremental of another version?

>> I need to do some more tests -- another problem I'm having now is that on some hosts sshd is not returning the session after the INCREMENTAL xtrabackup is taken so we just get stuck waiting for ssh to come back.
>>
>
> I think I experienced that, but it was when I had other problems (like
> home directory not writable) so didn't pay attention.

I figured out the issue. The stream buffer that the process called from PHP was writing to got full, which caused writes from that process to block and prevent the sub process from returning. I committed a change to fix that now -- basically we make sure we periodically read from the stream while doing checks on whether the sub process is still running.

>>> #) cleanup: In some tables it seems more natural to use a short string
>>> as id and primary key rather than auto_increment. In particular the
>>> parameter to ./xbm-backup -s ... I should be able to use something
>>> like hostname instead of magic number.
>>
>> I'd suggest we keep the use of integer IDs for internal DB level linking purposes, but add a feature like shortname or somesuch so that users can identify things by those names themselves.
>>
>> I would guess only a few things like the scheduled backup name or storage volume would need that.
>
> I'm ok with this. Insisting on integer primary keys is a valid style
> for schema design. I only care about the end user experience.
>
> The reason I suggested making the short name the primary key directly
> is that currently the mysql tables are the user interface :-)
> Especially if you look at my xbm-conftool.

Yep - sounds good - today and yesterday I burned a lot of time on that stream/blocking issue but hopefully will get to check out more today or tomorrow.


>
>> It all sounds like viable and worthwhile work to me and very much along the lines of the stuff I need and want to add.
>>
>> I would suggest maybe we go about creating some individual "Issues" against the Google Code project for these action items/coding tasks. I have done this previously by listing them as "Enhancement" type.
>>
>> Then we can keep track of them more individually.
>
> I will do so tomorrow. Wanted to hear comments first, will integrate
> what has been agreed to the issues.

Great!

Thanks again for all your contributions ! It is greatly appreciated -- hopefully we will both walk away with something much better than if we'd just done our own thing in the dark :)

Lachlan

Reply all
Reply to author
Forward
0 new messages