backup-push failing wal_e.main ERROR MSG: Attempted to archive a file that is too large.

[dhav...@gmail.com]

This is my first post, kindly redirect me to the correct place, if it is not the right one. 

I am getting following error while uploading to s3 using backup-push, due to the size of the object is more. Yes, there is a object more than 2 gb exists.

Please help me how to solve it.  

wal_e.main   INFO     MSG: starting WAL-E

        DETAIL: The subcommand is "backup-push".

        STRUCTURED: time=2015-11-09T08:59:37.441763-00 pid=3644

wal_e.operator.backup INFO     MSG: start upload postgres version metadata

        DETAIL: Uploading to s3://bucket/basebackups_005/base_00000004000000040000002E_00000040/extended_version.txt.

        STRUCTURED: time=2015-11-09T08:59:37.784639-00 pid=3644

wal_e.operator.backup INFO     MSG: postgres version metadata upload complete

        STRUCTURED: time=2015-11-09T08:59:37.845454-00 pid=3644

wal_e.operator.backup WARNING  MSG: blocking on sending WAL segments

        DETAIL: The backup was not completed successfully, but we have to wait anyway.  See README: TODO about pg_cancel_backup

        STRUCTURED: time=2015-11-09T08:59:37.862014-00 pid=3644

NOTICE:  pg_stop_backup complete, all required WAL segments have been archived

wal_e.main   ERROR    MSG: Attempted to archive a file that is too large.

        HINT: There is a file in the postgres database directory that is larger than 1610612736 bytes. If no such file exists, please report this as a bug. In particular, check base/13056/49161, which appears to be 2147483648 bytes.

        STRUCTURED: time=2015-11-09T08:59:39.034341-00 pid=3644 

Details: 

wal-e version

0.8.1

postgresql version: 

9.4.0

In case  you need further details please let me know. 

[dhav...@gmail.com]

Thanks.

I will be able to solve it. 

[Daniel Farina]

On Mon, Nov 9, 2015 at 4:34 AM <dhav...@gmail.com> wrote:

Thanks.

I will be able to solve it. 

How? Did you have extra files that were not Postgres' in the directory? Because that size limit takes advantage of the fact that Postgres is supposed to have a maximum file size limit of about 1G... 

[DHAVAL JAISWAL]

actually the file size of some object is more than 2 GB. Increasing the size of PARTITION_MAX_SZ has solved this.   

--

Thanks & Regards
Dhaval Jaiswal

[Daniel Farina]

On Tue, Nov 10, 2015 at 3:29 AM DHAVAL JAISWAL <dhav...@gmail.com> wrote:

actually the file size of some object is more than 2 GB. Increasing the size of PARTITION_MAX_SZ has solved this.   

Which ones?  The directory containing the file would answer this question.

I'm curious because for the longest time Postgres had a restriction that files would not be too large for a FAT32 file system to handle, and I rely on this to segment the backup. 

[DHAVAL JAISWAL]

Which ones?  The directory containing the file would answer this question.

>> files inside the base directory.   e.g. data/base/13056/14056 (in my case having size of 2 gb)

I'm curious because for the longest time Postgres had a restriction that files would not be too large for a FAT32 file system to handle, and I rely on this to segment the backup.

>> i did set with-segsize=2 at the time of configuring PostgreSQL. 

[Daniel Farina]

On Thu, Nov 12, 2015 at 10:01 AM DHAVAL JAISWAL <dhav...@gmail.com> wrote:

Which ones?  The directory containing the file would answer this question.

>> files inside the base directory.   e.g. data/base/13056/14056 (in my case having size of 2 gb)

I'm curious because for the longest time Postgres had a restriction that files would not be too large for a FAT32 file system to handle, and I rely on this to segment the backup.

>> i did set with-segsize=2 at the time of configuring PostgreSQL. 

Ah. I see. Well, I think you are the first try to use that (and reported it), but yes, you'll need to mess with the max partition size, as you did, and beyond that you probably should be good to go.