On Sun, Jun 2, 2013 at 4:51 PM, Jesse House <
jesse...@gmail.com> wrote:
> WAL-e chef cookbook
>
> I have been working on a WAL-e chef cookbook off and on the last few weeks,
> it is finally ready for some review from the WAL-e experts
Very cool.
I have one warning, that I think you might be taking care of here but
isn't entirely clear. It hasn't been written down in full many times,
so bear with me:
When converting a standby into a (writable) primary, it is somewhat
dangerous to archive back into the same prefix that one originates
from. Consider the following topology:
primary1 (writing to prefix1 on timeline1)
standby1 (reading from prefix1)
standby2 (reading from prefix1)
At some point, one decides to promote standby1, now called primary2:
primary1 (writing to prefix1 on timeline1)
standby2 (reading from prefix1)
primary2 (writing to prefix1, on timeline2)
Note how primary2 is on another timeline: the way that works is it is
an integer incremented by postgres when it is told to diverge from its
primary.
This is sort of okay, but precarious, because the next step introduces
corruption, which is promoting standby2:
primary1 (writing to prefix1 on timeline1)
primary2 (writing to prefix1 on timeline2)
primary3 (writing to prefix1 on timeline2)
Now we can see that the two latter primaries are on the same timeline
and in the same prefix, which can result in WAL-overwriting, which
results in corruption. An ugly solution to this that is in use by me
is to give every standby a separate read and write prefix. The same
situation would evolve as follows:
primary1 (writing to prefix1 on timeline1)
standby1 (reading from prefix1)
standby2 (reading from prefix1)
primary1 (writing to prefix1 on timeline1)
standby2 (reading from prefix1)
primary2 (writing to prefix2, on timeline2)
primary1 (writing to prefix1 on timeline1)
primary2 (writing to prefix2 on timeline2)
primary3 (writing to prefix3 on timeline2)
The ease of making this mistake is among one of the most bad things
about wal-e's design to date (and doesn't occur if it's only used in
simple cases, which is what wal-e was originally used for), but nobody
has made a proposal that seems easy to use, is general (particularly
with regard to treatment of credentials), and sands off this rough
edge, and my interest is merely a moral one since I've worked around
it in production. So, until then, I think as input to the 'come out
of hot standby' step one needs a wal-e prefix that is intended to be
the new spot to write things to, and if possible it'd be nice to
recheck that it isn't the same as the read-url.