a 6024-bytes file in an incremental backup?

[Satoshi Nagayasu]

Hi,

I'm testing pg_rman.

According to the user manual, incremental backup would take a backup
for "modified files or blocks". I understood that it would mean
incremental backup must have files consisting of 8kb blocks.

However, my incremental backup contains a 6024-bytes file in it.
What does it mean?

I took two backup sets, a full backup and an incremental backup
after that.
------------------------------------------------------------------
bash-3.2$ /usr/pgsql-9.0/bin/pg_rman show -B /var/lib/pgsql/9.0/backups
============================================================================
Start Time Total Data WAL Log Backup Status
============================================================================
2012-08-25 17:26:16 0m ---- 3416kB 67MB ---- 67MB OK
2012-08-25 17:08:00 12m 5435MB ---- 989MB ---- 5535MB OK
------------------------------------------------------------------

The full backup contains a 58236-bytes (7-blocks) file.
It seemed to be ok.
------------------------------------------------------------------
bash-3.2$ ls -l
/var/lib/pgsql/9.0/backups/20120825/170800/database/base/111236/112660
-rw------- 1 postgres postgres 58236 Aug 25 17:08
/var/lib/pgsql/9.0/backups/20120825/170800/database/base/111236/112660
------------------------------------------------------------------

The incremental backup contains a 6024-byte (smaller than 1-block) file.
What's happend?
------------------------------------------------------------------
bash-3.2$ ls -l
/var/lib/pgsql/9.0/backups/20120825/172616/database/base/111236/112660
-rw------- 1 postgres postgres 6024 Aug 25 17:26
/var/lib/pgsql/9.0/backups/20120825/172616/database/base/111236/112660
------------------------------------------------------------------

This file is a primary key index file.
So, it must consist of one or more blocks.
------------------------------------------------------------------
bash-3.2$ /usr/pgsql-9.0/bin/oid2name -d testdb -o 112660
From database "testdb":
Filenode Table Name
----------------------------
112660 testdb_logs_pkey
bash-3.2$
------------------------------------------------------------------

And the production file also consists of 14-blocks - it's ok, too.
------------------------------------------------------------------
bash-3.2$ ls -l /var/lib/pgsql/9.0/data/base/111236/112660
-rw------- 1 postgres postgres 114688 Aug 25 17:41
/var/lib/pgsql/9.0/data/base/111236/112660
bash-3.2$
------------------------------------------------------------------

I need to know more about "incremental backup" in this context.
What does the incremental backup mean? And how does it work?

Regards,
--
Satoshi Nagayasu <sn...@uptime.jp>
Uptime Technologies, LLC. http://www.uptime.jp

[Tomonari Katsumata]

Hi,

thank you for testing pg_rman.
the incremental backup works like bellow.

--
1. getting mtime and lsn from previous backup(fullbackup).
2. comparing mtime of the file with previous backup's one.
3. if it is not same, the file is taken for the incremental backup.
   (if it is same, skipping to backup the file.)
4. next, reading the whole pages of the file.
   and comparing lsn getting 1 and lsn from the page header.
5. if lsn from the page header is less than lsn getting 1, the page is
   not modified. so the page is skipped to be taken.
6. here if not skipped, the page is written as backup.
   but, data page has unused space and it is not necessary to restore.
   so the unused space is not written(*).
   (*) this behavior has nothing to do with fullbackup or incrementalbackup.
--


I'm not sure the sequence of your action,
I think the process 6 made smaller file than 8192kB.

does this behavior match your test scenario?

regards,

2012/8/25 Satoshi Nagayasu <sn...@uptime.jp>

[Satoshi Nagayasu]

Hi,

Thanks for the detailed explanation.

(2012/08/27 11:05), Tomonari Katsumata wrote:> Hi,

>
> thank you for testing pg_rman.
> the incremental backup works like bellow.
>
> --
> 1. getting mtime and lsn from previous backup(fullbackup).
> 2. comparing mtime of the file with previous backup's one.
> 3. if it is not same, the file is taken for the incremental backup.
> (if it is same, skipping to backup the file.)
> 4. next, reading the whole pages of the file.
> and comparing lsn getting 1 and lsn from the page header.
> 5. if lsn from the page header is less than lsn getting 1, the page is
> not modified. so the page is skipped to be taken.
> 6. here if not skipped, the page is written as backup.
> but, data page has unused space and it is not necessary to restore.
> so the unused space is not written(*).
> (*) this behavior has nothing to do with fullbackup or
> incrementalbackup.
> --
>
>
> I'm not sure the sequence of your action,
> I think the process 6 made smaller file than 8192kB.

Oh, an unused space in the single page would not be written
into the backup file?
I didn't imagine that, and I think I have to understand
how it works precisely, including managing backup catalog...

BTW, I'm considering another reason which might make
my backup file smaller than a 8k block.

I guess it has come from file compression. I didn't have
any way to confirm it, but my file might be compressed.

When I ran pg_rman with "-v" option, I found pg_rman
was telling me that files were going to be compressed
as below.
------------------------------------------------
-bash-3.2$ pg_rman backup -v -b full -B /var/lib/pgsql/9.0/backups -D
/var/lib/pgsql/9.0/data
========================================
backup start
----------------------------------------
# configuration
BACKUP_MODE=FULL
WITH_SERVERLOG=false
COMPRESS_DATA=false
----------------------------------------
backup destination is initialized.
INFO: database backup start
(...snip...)
(6/1215) base/1/11581 compressed 102528 (96.27% of 106496)
(7/1215) base/1/11581_fsm copied 24576
(8/1215) base/1/11581_vm compressed 32 (0.39% of 8192)
(9/1215) base/1/11583 compressed 11200 (68.36% of 16384)
(10/1215) base/1/11583_fsm copied 24576
(11/1215) base/1/11583_vm compressed 32 (0.39% of 8192)
(12/1215) base/1/11585 compressed 300 (1.83% of 16384)
(13/1215) base/1/11586 copied 16384
------------------------------------------------

Surprisingly, despite the configuration told me
"COMPRESS_DATA=false" and I didn't specify "-Z" option,
pg_rman told me those files were being compressed.

According to the user manual, file compression would
be enabled only when specifying "-Z". But as I experienced,
pg_rman seems to enable file compression by default.

Is this expected?

Regards,

(2012/08/27 11:05), Tomonari Katsumata wrote:
> Hi,
>
> thank you for testing pg_rman.
> the incremental backup works like bellow.
>
> --
> 1. getting mtime and lsn from previous backup(fullbackup).
> 2. comparing mtime of the file with previous backup's one.
> 3. if it is not same, the file is taken for the incremental backup.
> (if it is same, skipping to backup the file.)
> 4. next, reading the whole pages of the file.
> and comparing lsn getting 1 and lsn from the page header.
> 5. if lsn from the page header is less than lsn getting 1, the page is
> not modified. so the page is skipped to be taken.
> 6. here if not skipped, the page is written as backup.
> but, data page has unused space and it is not necessary to restore.
> so the unused space is not written(*).
> (*) this behavior has nothing to do with fullbackup or
> incrementalbackup.
> --
>
>
> I'm not sure the sequence of your action,
> I think the process 6 made smaller file than 8192kB.
>
> does this behavior match your test scenario?
>
> regards,
>
>

> 2012/8/25 Satoshi Nagayasu <sn...@uptime.jp <mailto:sn...@uptime.jp>>

> Satoshi Nagayasu <sn...@uptime.jp <mailto:sn...@uptime.jp>>