rsync - help!

[Angelo Onofri]

Hello all

I have some issues with a server  postgresql 12 and barman 3.12.1 running on ubuntu  24

barman check

Server pgintermedia:
        PostgreSQL: OK
        superuser or standard user with backup privileges: OK
        PostgreSQL streaming: OK
        wal_level: OK
        replication slot: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        backup minimum size: OK (259.1 GiB)
        wal maximum age: OK (no last_wal_maximum_age provided)
        wal size: OK (39.4 GiB)
        compression settings: OK
        failed backups: FAILED (there are 4 failed backups)
        minimum redundancy requirements: OK (have 1 backups, expected at least 0)
        ssh: OK (PostgreSQL server)
        not in recovery: OK
        exclusive backup supported: OK
        systemid coherence: OK
        pg_receivexlog: OK
        pg_receivexlog compatible: OK
        receive-wal running: OK
        archive_mode: OK
        archive_command: OK
        continuous archiving: OK
        archiver errors: FAILED (duplicates: 1)

barman show-server

Server pgintermedia:
        active: True
        archive_command: rsync -a %p bar...@x.x.x.x:/barman-pool/barman1/pgintermedia/incoming/%f
        archive_mode: on
        archive_timeout: 0
        archived_count: 4204
        archiver: True
        archiver_batch_size: 0
        autogenerate_manifest: False
        aws_await_snapshots_timeout: 3600
        aws_profile: None
        aws_region: None
        aws_snapshot_lock_cool_off_period: None
        aws_snapshot_lock_duration: None
        aws_snapshot_lock_expiration_date: None
        aws_snapshot_lock_mode: None
        azure_credential: None
        azure_resource_group: None
        azure_subscription_id: None
        backup_compression: None
        backup_compression_format: None
        backup_compression_level: None
        backup_compression_location: None
        backup_compression_workers: None
        backup_directory: /barman-pool/barman1/pgintermedia
        backup_method: rsync
        backup_options: BackupOptions({'exclusive_backup'})
        bandwidth_limit: 204800
        barman_home: /barman-pool/barman1
        barman_lock_directory: /barman-pool/barman1
        basebackup_retry_sleep: 30
        basebackup_retry_times: 0
        basebackups_directory: /barman-pool/barman1/pgintermedia/base
        check_timeout: 30
        checkpoint_timeout: 300
        cluster: pgintermedia
        compression: None
        config_changes_queue: /barman-pool/barman1/cfg_changes.queue
        config_file: /dbdata/pgclus01/postgresql.conf
        connection_error: None
        conninfo: host=10.101.11.196 port=5000 user=barman dbname=postgres
        create_slot: auto
        current_archived_wals_per_second: 0.0001694662084273012
        current_lsn: 8F3/24AFEBD8
        current_size: 277051543234
        current_xlog: 00000030000008F300000024
        custom_compression_filter: None
        custom_compression_magic: None
        custom_decompression_filter: None
        data_checksums: on
        data_directory: /dbdata/pgclus01
        description: Cluster Regione Intermendia
        disabled: False
        errors_directory: /barman-pool/barman1/pgintermedia/errors
        failed_count: 0
        forward_config_path: False
        gcp_project: None
        gcp_zone: None
        has_backup_privileges: True
        has_monitoring_privileges: True
        hba_file: /dbdata/pgclus01/pg_hba.conf
        hot_standby: on
        ident_file: /dbdata/pgclus01/pg_ident.conf
        immediate_checkpoint: False
        included_files: ['/dbdata/pgclus01/postgresql.auto.conf', '/dbdata/pgclus01/postgresql.base.conf']
        incoming_wals_directory: /barman-pool/barman1/pgintermedia/incoming
        is_archiving: True
        is_in_recovery: False
        is_superuser: True
        keepalive_interval: 60
        last_archived_time: 2025-02-13 16:56:33.620414+01:00
        last_archived_wal: 00000030000008E90000005A
        last_backup_maximum_age: None
        last_backup_minimum_size: None
        last_failed_time: None
        last_failed_wal: None
        last_wal_maximum_age: None
        local_staging_path: None
        lock_directory_cleanup: True
        max_incoming_wals_queue: None
        max_replication_slots: 10
        max_wal_senders: 10
        minimum_redundancy: 0
        msg_list: []
        name: pgintermedia
        network_compression: False
        parallel_jobs: 1
        parallel_jobs_start_batch_period: 1
        parallel_jobs_start_batch_size: 10
        passive_node: False
        path_prefix: /usr/lib/postgresql/12/bin
        pg_receivexlog_compatible: True
        pg_receivexlog_installed: True
        pg_receivexlog_path: /usr/lib/postgresql/12/bin/pg_receivewal
        pg_receivexlog_supports_slots: True
        pg_receivexlog_synchronous: False
        pg_receivexlog_version: 12.22
        post_archive_retry_script: None
        post_archive_script: None
        post_backup_retry_script: None
        post_backup_script: None
        post_delete_retry_script: None
        post_delete_script: None
        post_recovery_retry_script: None
        post_recovery_script: None
        post_wal_delete_retry_script: None
        post_wal_delete_script: None
        postgres_systemid: 6805196430123649895
        pre_archive_retry_script: None
        pre_archive_script: None
        pre_backup_retry_script: None
        pre_backup_script: None
        pre_delete_retry_script: None
        pre_delete_script: None
        pre_recovery_retry_script: None
        pre_recovery_script: None
        pre_wal_delete_retry_script: None
        pre_wal_delete_script: None
        primary_checkpoint_timeout: 0
        primary_conninfo: None
        primary_ssh_command: None
        recovery_options: RecoveryOptions()
        recovery_staging_path: None
        replication_slot: Record(slot_name='barman1234', active=True, restart_lsn='8F3/24000000')
        replication_slot_support: True
        retention_policy: RECOVERY WINDOW OF 8 DAYS
        retention_policy_mode: auto
        reuse_backup: None
        server_txt_version: 12.4
        slot_name: barman1234
        snapshot_disks: None
        snapshot_gcp_project: None
        snapshot_instance: None
        snapshot_provider: None
        snapshot_zone: None
        ssh_command: ssh -p 5021 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no post...@x.x.x.x -q
        stats_reset: 2024-04-30 14:56:08.092884+02:00
        streaming: True
        streaming_archiver: True
        streaming_archiver_batch_size: 0
        streaming_archiver_name: barman_receive_wal
        streaming_backup_name: barman_streaming_backup
        streaming_conninfo: host=10.101.11.196 port=5000 user=stream_barman dbname=postgres
        streaming_supported: True
        streaming_systemid: 6805196430123649895
        streaming_wals_directory: /barman-pool/barman1/pgintermedia/streaming
        synchronous_standby_names: ['pgintermedia-n3']
        tablespace_bandwidth_limit: None
        timeline: 48
        version_supported: True
        wal_compression: off
        wal_conninfo: None
        wal_keep_segments: 10
        wal_level: replica
        wal_retention_policy: MAIN
        wal_streaming_conninfo: None
        wals_directory: /barman-pool/barman1/pgintermedia/wals
        xlog_segment_size: 16777216
        xlogpos: 8F3/24AFEBD8

I was able to finish a backup ( the first one) but I am really struggling from that moment ( week ago)

Any idea?

many thanks

Angelo

[Angelo Onofri]

sorry  I haven't add the error

pg_basebackup: checkpoint completed
NOTIFICA:  base backup done, waiting for required WAL segments to be archived
ATTENZIONE:  still waiting for all required WAL segments to be archived (60 seconds elapsed)
HINT:  Check that your archive_command is executing properly.  You can safely cancel this backup, but the database backup will not be usable without all the WAL segments.

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to a topic in the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pgbarman/n7iGt35UiMo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pgbarman+u...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/pgbarman/55009c81-cf4b-4ae0-b11e-e3657aaf84cfn%40googlegroups.com.

[Angelo Onofri]

hello 

I  am adding more details:

2025-02-18 11:43:22.226 CET [4922] ATTENZIONE:  still waiting for all required WAL segments to be archived (480 seconds elapsed)
2025-02-18 11:43:22.226 CET [4922] SUGGERIMENTO:  Check that your archive_command is executing properly.  You can safely cancel this backup, but the database backup will not be usable without all the WAL segments.

Starting backup using rsync-exclusive method for server pgintermedia in /barman-pool/barman1/pgintermedia/base/20250218T110305
Backup start at LSN: 8F5/CB536330 (00000030000008F5000000CB, 00536330)
Starting backup copy via rsync/SSH for 20250218T110305
Copy done (time: 29 minutes, 51 seconds)
Asking PostgreSQL server to finalize the backup.

is it possible that barman 3.12.1 is too new? Is it a stable version?

[Angelo Onofri]

Hello 

barman 3.12

backup using rsync

I still have errors when I try to run the backup for an istance postgres 12

I see from the log this query below but if I run it manually a have an error like " .. not exclusive backup running"

 SELECT location, (pg_walfile_name_offset(location)).*, now() AS timestamp FROM pg_stop_backup() AS location 

base backup always ok

What could it be?

Thanks

[Angelo Onofri]

Hello

I managed to solve this "barman's misbehave" using the template barman provides for rsync and restarting the postgresql service with systectl restart postgres

Very weird situation thought

I hope this post might be useful for the group

Thanks,

Angelo

To view this discussion, visit https://groups.google.com/d/msgid/pgbarman/369b95c0-bfde-45b7-8079-49b7550319den%40googlegroups.com.

[Israel Barth]

Hello Angelo,

>       backup_method: rsync

>
> I was able to finish a backup ( the first one) but I am really struggling from that moment ( week ago)
>
> Any idea?

In the first message you sent a barman show-server output which shows the backup_method is rsync.

> sorry  I haven't add the error
>
> pg_basebackup: checkpoint completed
> NOTIFICA:  base backup done, waiting for required WAL segments to be archived
> ATTENZIONE:  still waiting for all required WAL segments to be archived (60 seconds elapsed)
> HINT:  Check that your archive_command is executing properly.  You can safely cancel this backup, but the database backup will not be usable without all the WAL > segments.

Then, the output messages that you shared in the second reply shows the output of pg_basebackup, which is only used by Barman when backup_method is postgres.
Also, keep in mind that the output you shared doesn't necessarily indicate an error. The pg_basebackup application is simply issuing warnings from time to time to inform that the WALs required for consistency of the backup were not yet archived by the archive_command.

That may be caused by different factors, to name a few:

• archive_command is failing.
• There is no load in the Postgres server, so it never switches the WAL, thus the WAL cannot be archived.
• There is too much load in the Postgres and/or Barman server, in such a way Postgres is not able to send the WAL archive as fast as one would expect.

> 2025-02-18 11:43:22.226 CET [4922] ATTENZIONE:  still waiting for all required WAL segments to be archived (480 seconds elapsed)
> 2025-02-18 11:43:22.226 CET [4922] SUGGERIMENTO:  Check that your archive_command is executing properly.  You can safely cancel this backup, but the database backup will not be usable without all the WAL segments.
>
> Starting backup using rsync-exclusive method for server pgintermedia in /barman-pool/barman1/pgintermedia/base/20250218T110305
> Backup start at LSN: 8F5/CB536330 (00000030000008F5000000CB, 00536330)
> Starting backup copy via rsync/SSH for 20250218T110305
> Copy done (time: 29 minutes, 51 seconds)
> Asking PostgreSQL server to finalize the backup.
>
> is it possible that barman 3.12.1 is too new? Is it a stable version?

Then you shared this output, which has a mix of output messages: some messages coming from a rsync-based backup, and some coming from a pg_basebackup-based backup.
With that in mind, we guess you have more than one Barman server configured in your Barman installation, each one using a different backup method.

> I still have errors when I try to run the backup for an istance postgres 12
> I see from the log this query below but if I run it manually a have an error like " .. not exclusive backup running"
>
> SELECT location, (pg_walfile_name_offset(location)).*, now() AS timestamp FROM pg_stop_backup() AS location
>
> base backup always ok
>
> What could it be?

Hard to say what could be causing the issue, the information shared along this mail thread is not clear enough to troubleshoot the issue.

About the manual execution of pg_stop_backup, that's expected to fail.
That function will only succeed if executed in the very same session where pg_start_backup was run.
Also, that's not expected to be executed by you, but by Barman as part of the base backup copy process.

> I managed to solve this "barman's misbehave" using the template barman provides for rsync and restarting the postgresql service with systectl restart postgres
>
> Very weird situation thought
> I hope this post might be useful for the group

In order to understand what was the issue that you faced, we would need a better tracking of the situation, understanding for example what was present in the logs and what exact actions you took between the failed and successful attempts.

The information that we have in this mail thread is not enough for troubleshooting the issue.

In any case, we are happy to know your backups are now functional.

Best regards,
Israel.

[Angelo Onofri]

Thanks Israel for your reply

I tried many configurations during my different posts so I apologise if I caused confusion to you all.

From my recent experience on this case I can say :

1)Barman Rsync might work but wal_level fails until when a new wal is delivered.

It is important to run barman switch-wal --archive  before barman check

2) if there is a proxy (  in my case haproxy) in front of a cluster base on 2 or more nodes ( for example with patroni) , I might happen that the proxy cut off the connection so again there is an error on barman check.

I solved this issue running a barman check my_postgres on the schedule in /etc/cron.d/barman before the command barman backup my_postgre

I don't see on haproxy something so solve this issue with the  timeout.

Thanks,

Angelo

--

--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to a topic in the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pgbarman/n7iGt35UiMo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pgbarman+u...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/pgbarman/656e607e-58ee-4195-a085-6828ee359bb8n%40googlegroups.com.