Barman 2.0 Docker test - WAL shipping settings
1,157 views
Skip to first unread message
col...@plevium.ca
Oct 1, 2016, 11:39:36 PM10/1/16
to Barman, Backup and Recovery Manager for PostgreSQL
I am testing out Barman 2.0 for a Docker setup. My 'barman check' command is returning:
root@7d609cc628d9:/# barman check testdb
Server testdb:
WAL archive: FAILED (please make sure WAL shipping is setup)
PostgreSQL: OK
superuser: OK
PostgreSQL streaming: OK
wal_level: OK
replication slot: OK
directories: OK
retention policy settings: OK
backup maximum age: OK (no last_backup_maximum_age provided)
compression settings: OK
failed backups: OK (there are 0 failed backups)
minimum redundancy requirements: OK (have 0 backups, expected at least 0)
pg_basebackup: OK
pg_basebackup compatible: OK
pg_basebackup supports tablespaces mapping: OK
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: OK
archiver errors: OK
I can't figure out why the FAILED message on WAL archive. As far as I am understanding that with a streaming only setup the 'archive_command' is not required to be set.
Here is a full Github repo with all the necessary code/Dockerfiles/etc... https://github.com/collinpeters/barman-2-docker. With Docker installed you should be able to run the two shell files from within the two directories (the last few commands I am just running from within the Barman Docker container.
Can you tell me what I am missing or doing wrong?
/Collin
Gabriele Bartolini
Oct 2, 2016, 4:47:31 AM10/2/16
to pgba...@googlegroups.com
Hi Colin,
I just spot a bug in the documentation. Look at this section:
That section should be at a higher level (not a subsection of WAL archiving). That check fails until a WAL file has been shipped. Run a switch-xlog command and then wait for cron to kick in.
Let us know if this fixes it. Otherwise please send a diagnose output.
Thanks,
Gabriele
--
Gabriele Bartolini - 2ndQuadrant Italia - Director
PostgreSQL Training, Services and Support
gabriele....@2ndQuadrant.it | www.2ndQuadrant.it
Gabriele Bartolini - 2ndQuadrant Italia - Director
PostgreSQL Training, Services and Support
gabriele....@2ndQuadrant.it | www.2ndQuadrant.it
--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB
---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
col...@plevium.ca
Oct 2, 2016, 1:58:03 PM10/2/16
to Barman, Backup and Recovery Manager for PostgreSQL
Didn't seem to help. I ran a 'barman switch-xlog testdb` which resulted in a log message: 2016-10-02 17:39:57,836 [13] barman.command_wrappers INFO: testdb: pg_receivexlog: finished segment at 0/2000000 (timeline 1)
I waited for over 5 minutes and nothing else happened (as far as I recall, 5 minutes is the default wal checkpoint length). I went to my test db, created a table and did a generate_series which resulted in another: 2016-10-02 17:51:21,239 [13] barman.command_wrappers INFO: testdb: pg_receivexlog: finished segment at 0/3000000 (timeline 1)
The 'check' command still shows: "WAL archive: FAILED"
barman diagnose:
root@a9fe2193f7d1:/# barman diagnose
{
"global": {
"config": {
"barman_home": "/var/lib/barman",
"barman_user": "barman",
"compression": "gzip",
"configuration_files_directory": "/etc/barman.d",
"errors_list": [],
"log_file": "/var/log/barman/barman.log",
"log_level": "INFO"
},
"system_info": {
"barman_ver": "2.0",
"kernel_ver": "Linux a9fe2193f7d1 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 GNU/Linux",
"python_ver": "Python 2.7.9",
"release": "Distributor ID:\tDebian\nDescription:\tDebian GNU/Linux 8.6 (jessie)\nRelease:\t8.6\nCodename:\tjessie",
"rsync_ver": "rsync version 3.1.1 protocol version 31",
"ssh_ver": "OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t 3 May 2016"
}
},
"servers": { [40/236]
"testdb": {
"backups": {},
"config": {
"active": true,
"archiver": false,
"archiver_batch_size": 0,
"backup_directory": "/var/lib/barman/testdb",
"backup_method": "postgres",
"backup_options": "concurrent_backup",
"bandwidth_limit": null,
"barman_home": "/var/lib/barman",
"barman_lock_directory": "/var/lib/barman",
"basebackup_retry_sleep": 30,
"basebackup_retry_times": 0,
"basebackups_directory": "/var/lib/barman/testdb/base",
"check_timeout": 30,
"compression": "gzip",
"conninfo": "host=postgres user=barman dbname=postgres",
"custom_compression_filter": null,
"custom_decompression_filter": null,
"description": "Test",
"disabled": false,
"errors_directory": "/var/lib/barman/testdb/errors",
"immediate_checkpoint": false,
"incoming_wals_directory": "/var/lib/barman/testdb/incoming",
"last_backup_maximum_age": null,
"minimum_redundancy": 0,
"msg_list": [],
"name": "testdb",
"network_compression": false,
"path_prefix": null,
"post_archive_retry_script": null,
"post_archive_script": null,
"post_backup_retry_script": null,
"post_backup_script": null,
"pre_archive_retry_script": null,
"pre_archive_script": null,
"pre_backup_retry_script": null,
"pre_backup_script": null,
"recovery_options": "",
"retention_policy": null,
"retention_policy_mode": "auto",
"reuse_backup": null,
"slot_name": "barman",
"ssh_command": null,
"streaming_archiver": true,
"streaming_archiver_batch_size": 0,
"streaming_archiver_name": "barman_receive_wal",
"streaming_backup_name": "barman_streaming_backup",
"streaming_conninfo": "host=postgres user=streaming_barman",
"streaming_wals_directory": "/var/lib/barman/testdb/streaming",
"tablespace_bandwidth_limit": null,
"wal_retention_policy": "main",
"wals_directory": "/var/lib/barman/testdb/wals"
},
"status": {
"config_file": "/var/lib/postgresql/data/postgresql.conf",
"connection_error": null,
"current_size": 40169784.0,
"current_xlog": "000000010000000000000003",
"data_directory": "/var/lib/postgresql/data",
"hba_file": "/var/lib/postgresql/data/pg_hba.conf",
"ident_file": "/var/lib/postgresql/data/pg_ident.conf",
"is_superuser": true,
"pg_basebackup_bwlimit": true,
"pg_basebackup_compatible": true,
"pg_basebackup_installed": true,
"pg_basebackup_path": "/usr/bin/pg_basebackup",
"pg_basebackup_tbls_mapping": true,
"pg_basebackup_version": "9.6.0",
"pg_receivexlog_compatible": true,
"pg_receivexlog_installed": true,
"pg_receivexlog_path": "/usr/bin/pg_receivexlog",
"pg_receivexlog_supports_slots": true,
"pg_receivexlog_synchronous": true,
"pg_receivexlog_version": "9.6.0",
"pgespresso_installed": false,
"replication_slot": [
"barman",
true,
"0/3EB4A70"
],
"replication_slot_support": true,
"server_txt_version": "9.6.0",
"streaming": true,
"streaming_supported": true,
"synchronous_standby_names": [
"barman_receive_wal"
],
"systemid": "6336703731832320025",
"timeline": 1,
"wal_level": "replica",
"xlogpos": "0/3EB4A70"
}
}
}
}
On Sunday, October 2, 2016 at 1:47:31 AM UTC-7, Gabriele Bartolini wrote:
Hi Colin,I just spot a bug in the documentation. Look at this section:That section should be at a higher level (not a subsection of WAL archiving). That check fails until a WAL file has been shipped. Run a switch-xlog command and then wait for cron to kick in.Let us know if this fixes it. Otherwise please send a diagnose output.Thanks,Gabriele
--
Gabriele Bartolini - 2ndQuadrant Italia - Director
PostgreSQL Training, Services and Support
gabriele.bartolini@2ndQuadrant.it | www.2ndQuadrant.it
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB
---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
Gabriele Bartolini
Oct 2, 2016, 2:31:51 PM10/2/16
to pgba...@googlegroups.com
Hi Collin,
can you paste the output of the following command please?
barman archive-wal testdb
Also, can you check what the cron command is regularly doing?
Can you also report this?
ls -la /var/lib/barman/testdb/wals
Thanks,
Gabriele
--
Gabriele Bartolini - 2ndQuadrant Italia - Director
PostgreSQL Training, Services and Support
Gabriele Bartolini - 2ndQuadrant Italia - Director
PostgreSQL Training, Services and Support
gabriele....@2ndQuadrant.it | www.2ndQuadrant.it
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB
---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+unsubscribe@googlegroups.com.
col...@plevium.ca
Oct 2, 2016, 4:40:00 PM10/2/16
to Barman, Backup and Recovery Manager for PostgreSQL
root@a9fe2193f7d1:/# barman archive-wal testdb
Processing xlog segments from streaming for testdb
000000010000000000000001
000000010000000000000002
I did a 'ps aufx' to ensure cron was running but it wasn't. I've suspended my laptop a few times during this so I'm not 100% when that stopped. I'll repeat the whole test from scratch later today and report back.
After 'barman cron' the log file had the following and the 'check' command reported everything ok:
2016-10-02 20:34:53,535 [72] barman.wal_archiver INFO: No xlog segments found from streaming for testdb.
2016-10-02 20:34:53,543 [73] barman.server INFO: Starting receive-wal for server testdb
2016-10-02 20:34:53,598 [73] barman.wal_archiver INFO: Synchronous WAL streaming for barman_receive_wal: True
2016-10-02 20:34:53,598 [73] barman.wal_archiver INFO: Activating WAL archiving through streaming protocol
2016-10-02 20:34:53,641 [73] barman.command_wrappers INFO: testdb: pg_receivexlog: starting log streaming at 0/3000000 (timeline 1)
On Sunday, October 2, 2016 at 11:31:51 AM UTC-7, Gabriele Bartolini wrote:
Hi Collin,can you paste the output of the following command please?barman archive-wal testdbAlso, can you check what the cron command is regularly doing?Can you also report this?ls -la /var/lib/barman/testdb/walsThanks,Gabriele
--
Gabriele Bartolini - 2ndQuadrant Italia - Director
PostgreSQL Training, Services and Support
gabriele.bartolini@2ndQuadrant.it | www.2ndQuadrant.it
Collin Peters
Oct 2, 2016, 8:03:16 PM10/2/16
to pgba...@googlegroups.com
Ok, I believe I know what the problem is... which is rather obvious in hindsight. When I start the barman container, it is a Docker container and PID 1 = bash. So that means no cron daemon is actually running.
So my order is this
- Start postgres container
- Start barman container. This is just a container based on Debian Jessie with Barman installed. Command is: docker run -it --rm --name barman --link postgres-barman:postgres barman /bin/bash
- Execute 'barman receive-wal --create-slot testdb'
- Execute 'barman cron' which starts pg_xreceivelog
- At this point 'barman check' still returns a failure
- Running 'barman cron' results in the log entry: barman.wal_archiver INFO: No xlog segments found from streaming for testdb.
- Now I do a new table and generate_series into it on the pg server
- In the barman log I immediately see pg_xreceivelog statements. Awesome.
- 'barman check' still returns the failure
- Now if I run 'barman cron' again it will archive the logs
- Now 'barman check' works
You received this message because you are subscribed to a topic in the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pgbarman/RFQFgLIm2LY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pgbarman+unsubscribe@googlegroups.com.
Collin Peters
Oct 2, 2016, 8:07:12 PM10/2/16
to pgba...@googlegroups.com
(sorry, hit send too soon)
The only thing I find odd about this is that traffic must be generated, the wal received, and processed, before 'barman check' or 'barman backup' work. Is this correct behaviour? Seems a bit odd that I can't run a backup immediately.
Are there any recommended best practices for running barman in a Docker container? I'm thinking either a Docker container that runs cron itself as the main process... or probably better yet a simpler script or scheduler that executes it.
Reply all
Reply to author
Forward
0 new messages