Possible to exclude pg_logs from base push?

[psyc...@gmail.com]

Hi All,

I'm running into an issue where mt wal-e base-push will not complete due to large log files in the pg_log directory, is there anyway to to exclude these?  See error below:

wal_e.operator.backup WARNING  MSG: blocking on sending WAL segments
        DETAIL: The backup was not completed successfully, but we have to wait anyway.  See README: TODO about pg_cancel_backup
        STRUCTURED: time=2017-02-16T12:55:50.170735-00 pid=38947
NOTICE:  pg_stop_backup cleanup done, waiting for required WAL segments to be archived
NOTICE:  pg_stop_backup complete, all required WAL segments have been archived
wal_e.main   ERROR    MSG: Attempted to archive a file that is too large.
        HINT: There is a file in the postgres database directory that is larger than 1610612736 bytes. If no such file exists, please report this as a bug. In particular, check pg_log/postgresql-2017-02-01.log, which appears to be 18506123065 bytes.

Thanks in advance.

[Daniel Farina]

On Thu, Feb 16, 2017 at 9:00 AM <psyc...@gmail.com> wrote:

 

Hi All,

I'm running into an issue where mt wal-e base-push will not complete due to large log files in the pg_log directory, is there anyway to to exclude these?  See error below:

 

I've committed a patch to avoid this by default in future versions of WAL-E some time ago. It'll be in v1.1

Maybe I should release it soon.

Until then, you could use "master" at its current commit: it should be stable.

[psyc...@gmail.com]

Hi Daniel, thanks for the info. Are you referring to https://github.com/wal-e/wal-e/pull/278, where you can set PARTITION_MAX_SZ as a variable?  I could try this but I'd rather exclude the directory all together, as the logs are multiple gigabytes in size.

[Daniel Farina]

On Thu, Feb 16, 2017 at 9:39 AM <psyc...@gmail.com> wrote:

Hi Daniel, thanks for the info. Are you referring to https://github.com/wal-e/wal-e/pull/278, where you can set PARTITION_MAX_SZ as a variable?  I could try this but I'd rather exclude the directory all together, as the logs are multiple gigabytes in size.

No, I mean https://github.com/wal-e/wal-e/commit/8362eb815cd427b9b6bb09a2b421e9a276dbdce8, which has already been committed. 

[psyc...@gmail.com]

Oh, awesome! That's in v1.0.3 already, which I just installed. Didn't realize it was available as under your Github releases it still says 1.0.2.  I'll see how it goes tonight!

Thanks.

[psyc...@gmail.com]

Guess, I was wrong it's not in v1.0.3.

[Daniel Farina]

On Thu, Feb 16, 2017 at 9:56 AM <psyc...@gmail.com> wrote:

Guess, I was wrong it's not in v1.0.3.

No, it's enough a behavior change that it'll be a in a to-be-released v1.1.

I'm suggesting that you could pip install the current commit in master as a workaround. I will vouch that it's thought to be stable: the only reason I've held off is to integrate some pull requests.

[psyc...@gmail.com]

I was able pip install the commit as suggested.  Look forward to seeing how the backup performs tonight.

Thanks.

[psyc...@gmail.com]

Wal-e worked great off of the master version last night! Thanks for your help.