Anyone implemented barman cli to backup on Azure Storage Blob

[Ravi Chauhan]

HI All,

I am trying to configure barman with Azure storage blob.

Error - Missing required python module: azure-storage-blob

I already have azure-storage-blob install as checked in pip3 list.

Can anyone suggest something or share their expreience.

Regards,
Ravi

[Michael Wallace]

Hi Ravi,

It's possible there is an incompatibility between barman and the azure-storage-blob version on your system - which version of barman are you using? Did you install it via an OS package or through some other means?

Also, can you share the versions of azure-storage-blob and its transitive dependency azure-core as reported by `pip3 list`?

Thanks,

Mike

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/3fddbdac-efff-48a3-8362-4ec203211667n%40googlegroups.com.

[Ravi Chauhan]

Thanks, Mike for the response.

Installed packages using YUM- RHEL 7

postgresql-server

barman

barman-cli

Postgres - 14 (latest)

Barman- 2.19 Barman by EnterpriseDB (www.enterprisedb.com)

PIP output - 

Package            Version

------------------ -----------

azure-core         1.24.1

azure-storage-blob 12.12.0

certifi            2022.5.18.1

cffi               1.15.0

charset-normalizer 2.0.12

cryptography       37.0.2

idna               3.3

isodate            0.6.1

msrest             0.6.21

oauthlib           3.2.0

pip                21.3.1

pycparser          2.21

requests           2.27.1

requests-oauthlib  1.3.1

setuptools         39.2.0

six                1.16.0

typing_extensions  4.1.1

urllib3            1.26.9

[Michael Wallace]

Hi Ravi,

The RHEL 7 packages for barman currently use python2 rather than python3 so if you installed azure-blob-storage with pip3 then barman won't be able to use it.

Try installing azure-storage-blob with pip instead of pip3 and see if that helps. If you haven't already you will probably need to upgrade pip from the OS-packaged version - the most recent version which still supports python2 is 20.1 so something like `pip install --upgrade "pip==20.1"` should do the trick.

Hope this helps,

Mike

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/c6e38464-cb52-4a62-9a0d-b6368634f782n%40googlegroups.com.

[Michael Wallace]

Hi Ravi,

Just to follow up on this, a new version of msrest (0.7.0) was released recently which appears to be broken on older python versions - I can see from your pip3 output that you have 0.6.21 but if you are installing azure-storage-blob on python2 it will probably pull in 0.7.0.

Not sure if that's related to the problem you have but thought it was worth mentioning.

Best,

Mike

[Ravi Chauhan]

Hi Mike,

Yes, I observed the same.

Now I have pip 20.1.

When I try to install azure-storage-blob, it looks for azure-core>1.24.0.

Error -msrest 0.7.0 has requirement azure-core>=1.24.0, but you'll have azure-core 1.21.1 which is incompatible.

And there is no azure-core >1.21.1 available

Error-

ERROR: Could not find a version that satisfies the requirement azure-core==1.24.0 (from versions: 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0b4, 1.0.0, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.11.0, 1.12.0b1, 1.12.0, 1.13.0b1, 1.13.0, 1.14.0b1, 1.14.0, 1.15.0b1, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.19.1, 1.20.0, 1.20.1, 1.21.0, 1.21.1)

ERROR: No matching distribution found for azure-core==1.24.0

The Latest supported PIP for RHEL 7 is 20.1 and in this azure required packages are not available. 

What else we can do to barman cli work on RHEL7.

Package                          Version

-------------------------------- ---------

argcomplete                      1.7.0

azure-nspkg                      3.0.2

azure-storage-nspkg              3.1.0

backports.ssl-match-hostname     3.5.0.1

barman                           2.19

certifi                          2021.10.8

cffi                             1.15.0

chardet                          4.0.0

configobj                        4.7.2

cryptography                     3.3.2

decorator                        3.4.0

enum34                           1.0.4

ethtool                          0.8

futures                          3.1.1

idna                             2.10

iniparse                         0.4

ipaddr                           2.1.11

ipaddress                        1.0.16

IPy                              0.75

kitchen                          1.1.1

langtable                        0.0.31

lxml                             3.2.1

M2Crypto                         0.21.1

Magic-file-extensions            0.2

pciutils                         1.7.3

perf                             0.1

pexpect                          2.3

pip                              20.1

policycoreutils-default-encoding 0.1

psycopg2                         2.8.6

pyasn1                           0.1.9

pycparser                        2.21

pycurl                           7.19.0

pygobject                        3.22.0

pygpgme                          0.3

pyinotify                        0.9.4

pyliblzma                        0.5.3

pyOpenSSL                        0.13.1

python-augeas                    0.5.0

python-dateutil                  1.5

python-dmidecode                 3.10.13

python-linux-procfs              0.4.9

pyudev                           0.15

pyxattr                          0.5.1

PyYAML                           3.10

requests                         2.6.0

rhnlib                           2.5.65

schedutils                       0.4

seobject                         0.1

sepolicy                         1.1

setuptools                       0.9.8

six                              1.11.0

slip                             0.4.0

slip.dbus                        0.4.0

subscription-manager             1.24.51

syspurpose                       1.24.51

urlgrabber                       3.10

urllib3                          1.26.9

WALinuxAgent                     2.2.46

wheel                            0.37.1

yum-langpacks                    0.4.2

yum-metadata-parser              1.1.4

Regards,

Ravi

[Michael Wallace]

If you install azure-storage-blob while also pinning msrest it should work, e.g.:

    pip install azure-storage-blob 'msrest<0.7.0'

The requirement for azure-core>=1.24 comes from msrest 0.7.0 but azure-storage-blob itself does not need it so it should work as long as you force pip to install an older version of msrest.

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/844e17be-f912-468e-9d92-0212a3c1a4a3n%40googlegroups.com.

[Ravi Chauhan]

Hi Mike,

By restricting the msrest, azure-storage-blob installed.

Now while running BARMAN (manually)- "cannot import name Enum". I have enum and enum34 in pip.

-bash-4.2$ barman-cloud-wal-archive --cloud-provider azure-blob-storage --credential azure-cli  https://*********.blob.core.windows.net/postgres server-22 /opt/postgres/backup/server-22/wals/0000000100000045/0000000100000045000000E7 

Traceback (most recent call last):

  File "/bin/barman-cloud-wal-archive", line 9, in <module>

    load_entry_point('barman==2.19', 'console_scripts', 'barman-cloud-wal-archive')()

  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point

    return get_distribution(dist).load_entry_point(group, name)

  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point

    return ep.load()

  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load

    entry = __import__(self.module_name, globals(),globals(), ['__name__'])

  File "/usr/lib/python2.7/site-packages/barman/clients/cloud_walarchive.py", line 24, in <module>

    from barman.clients.cloud_cli import (

  File "/usr/lib/python2.7/site-packages/barman/clients/cloud_cli.py", line 23, in <module>

    from enum import Enum

ImportError: cannot import name Enum

[Ravi Chauhan]

Mike, 

there might be something missing here. As azure says to use Enum, use it in the below manner by importing CaseInsensitiveEnumMeta.

https://docs.microsoft.com/en-us/python/api/azure-core/azure.core.caseinsensitiveenummeta?view=azure-python

[Michael Wallace]

Hi Ravi,

The failed import is happening in one of the barman modules - this in itself is a bug since barman is relying on a transitive dependency when it should be specifying it directly. That said, if enum34 is available then the failing import should not fail.

Which version of enum34 do you have installed?

For reference, the CentOS 7 I'm using here has these python libraries installed: https://gist.github.com/mikewallace1979/24f616e536569f48d84435f136ed24c8

Best,

Mike

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/6617938f-8c3f-4023-a3a7-cdc7edcb9936n%40googlegroups.com.

[Ravi Chauhan]

Hi Mike,

In pip2(20.1) I have below enum and enum34

enum                             0.4.7

enum34                           1.1.10

Best Regards,

Ravi

[Ravi Chauhan]

I tried using the same enum34 version as yours enum34==1.0.4. Still the same issue.

[Michael Wallace]

That's odd - I have updated to 1.1.10 and it still works so perhaps this is not as related as I thought.

What happens if you just run a python repl and try `from enum import Enum`?

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/1f418d5a-574e-4c0d-9a30-19001afb116en%40googlegroups.com.

[Ravi Chauhan]

I ran a simple program with python2 that imports enum. Attached program code.

Traceback (most recent call last):

  File "repl.py", line 3, in <module>

    from enum import Enum

ImportError: cannot import name Enum

BR,

Ravi