Problems Importing Keys
460 views
Skip to first unread message
Connor Doonan
Jun 14, 2023, 3:20:12 PM6/14/23
to python-gnupg
Hi guys. I'm working on a PGP implementation that will hopefully pull keys from a Vault secret, or AWS Secrets Manager, rather than a file. This means that the keys will be imported as a string. I've tried to use the import_keys option, and it doesn't seem to work, as listing the keys afterwards gives nothing.
Code snippet:
gpg.import_keys(pri_key)
gpg.import_keys(pub_key)
print('Keys:')
print(gpg.list_keys())
The result:
Keys:
[]
For testing purposes, pri_key and pub_key are the secret keys I've copy-pasted directly from my local .asc files.
Am I missing something, or does anybody have experience with importing keys this way? Thank you!
Code snippet:
gpg.import_keys(pri_key)
gpg.import_keys(pub_key)
print('Keys:')
print(gpg.list_keys())
The result:
Keys:
[]
For testing purposes, pri_key and pub_key are the secret keys I've copy-pasted directly from my local .asc files.
Am I missing something, or does anybody have experience with importing keys this way? Thank you!
Vinay Sajip
Jun 14, 2023, 3:22:04 PM6/14/23
to python-gnupg
Turn logging on for `python-gnupg` and you might find the cause of the problem. If not, post the logs here.
Connor Doonan
Jun 14, 2023, 3:41:42 PM6/14/23
to python-gnupg
Hi,
Sorry to ask, but can you give me an example of how to enable logging for it? Thank you.
Sorry to ask, but can you give me an example of how to enable logging for it? Thank you.
Vinay Sajip
Jun 15, 2023, 2:46:29 AM6/15/23
to python-gnupg
Sid Bose
Jun 15, 2023, 5:22:10 AM6/15/23
to python...@googlegroups.com
Hi Connor,
I am using vault api to fetch key pairs, then I concatenate the string values,
key_data = vault_gpg_public() + ‘\n’ + vault_gpg_private()
And the use import_keys works fine.
gpg.import_keys(key_data)
Thanks
Sid
--
---
You received this message because you are subscribed to the Google Groups "python-gnupg" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-gnupg...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/python-gnupg/7ef16748-1715-49e5-b091-f770251b694dn%40googlegroups.com.
Sid
Sent from iPhone
Sent from iPhone
Connor Doonan
Jun 15, 2023, 9:36:32 AM6/15/23
to python-gnupg
Hi guys,
Sid, I saw that you added + \n, which I had not done on my end, and figured that might be the issue, but unfortunately that didn't resolve it. But thank you for the suggestion!
Vinay, I did some reading and managed to get the logger working. Here's the output:
initial daemon)>, <_io.BytesIO object at 0x7f1a51566bd0>, <_io.BufferedWriter name=7>
stderr reader: <Thread(Thread-4, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.798Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stderr reader: <Thread(Thread-4, initial daemon)>
closed output, 1422 bytes sent
[DEBUG] 2023-06-15T13:21:06.799Z 814e2cf0-75d6-46d1-ac51-b330f306b685 closed output, 1422 bytes sent
stdout reader: <Thread(Thread-5, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.799Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stdout reader: <Thread(Thread-5, initial daemon)>
gpg: keyring `/tmp/secring.gpg' created
[DEBUG] 2023-06-15T13:21:06.800Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: keyring `/tmp/secring.gpg' created
gpg: keyring `/tmp/pubring.gpg' created
[DEBUG] 2023-06-15T13:21:06.800Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: keyring `/tmp/pubring.gpg' created
gpg: no valid OpenPGP data found.
[DEBUG] 2023-06-15T13:21:06.818Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: no valid OpenPGP data found.
[GNUPG:] NODATA 1
[DEBUG] 2023-06-15T13:21:06.819Z 814e2cf0-75d6-46d1-ac51-b330f306b685 [GNUPG:] NODATA 1
gpg: Total number processed: 0
[DEBUG] 2023-06-15T13:21:06.819Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: Total number processed: 0
[GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[DEBUG] 2023-06-15T13:21:06.819Z 814e2cf0-75d6-46d1-ac51-b330f306b685 [GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0
gpg returned a non-zero error code: 2
[WARNING] 2023-06-15T13:21:06.838Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg returned a non-zero error code: 2
import_keys result: {'gpg': <gnupg.GPG object at 0x7f1a564dbca0>, 'results': [{'fingerprint': None, 'problem': '0', 'text': 'No valid data found'}], 'fingerprints': [], 'count': 0, 'no_user_id': 0, 'imported': 0, 'imported_rsa': 0, 'unchanged': 0, 'n_uids': 0, 'n_subk': 0, 'n_sigs': 0, 'n_revoc': 0, 'sec_read': 0, 'sec_imported': 0, 'sec_dups': 0, 'not_imported': 0, 'stderr': "gpg: keyring `/tmp/secring.gpg' created\ngpg: keyring `/tmp/pubring.gpg' created\ngpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\ngpg: Total number processed: 0\n[GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 'data': b'', 'returncode': 2}
[DEBUG] 2023-06-15T13:21:06.839Z 814e2cf0-75d6-46d1-ac51-b330f306b685 import_keys result: {'gpg': <gnupg.GPG object at 0x7f1a564dbca0>, 'results': [{'fingerprint': None, 'problem': '0', 'text': 'No valid data found'}], 'fingerprints': [], 'count': 0, 'no_user_id': 0, 'imported': 0, 'imported_rsa': 0, 'unchanged': 0, 'n_uids': 0, 'n_subk': 0, 'n_sigs': 0, 'n_revoc': 0, 'sec_read': 0, 'sec_imported': 0, 'sec_dups': 0, 'not_imported': 0, 'stderr': "gpg: keyring `/tmp/secring.gpg' created\ngpg: keyring `/tmp/pubring.gpg' created\ngpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\ngpg: Total number processed: 0\n[GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 'data': b'', 'returncode': 2}
Keys:
20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
[DEBUG] 2023-06-15T13:21:06.878Z 814e2cf0-75d6-46d1-ac51-b330f306b685 20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
stderr reader: <Thread(Thread-6, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.879Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stderr reader: <Thread(Thread-6, initial daemon)>
stdout reader: <Thread(Thread-7, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.898Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stdout reader: <Thread(Thread-7, initial daemon)>
gpg: /tmp/trustdb.gpg: trustdb created
[DEBUG] 2023-06-15T13:21:06.919Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: /tmp/trustdb.gpg: trustdb created
[]
2023-06-15T13:21:06.980Z 814e2cf0-75d6-46d1-ac51-b330f306b685 Task timed out after 3.04 seconds
Sid, I saw that you added + \n, which I had not done on my end, and figured that might be the issue, but unfortunately that didn't resolve it. But thank you for the suggestion!
Vinay, I did some reading and managed to get the logger working. Here's the output:
initial daemon)>, <_io.BytesIO object at 0x7f1a51566bd0>, <_io.BufferedWriter name=7>
stderr reader: <Thread(Thread-4, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.798Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stderr reader: <Thread(Thread-4, initial daemon)>
closed output, 1422 bytes sent
[DEBUG] 2023-06-15T13:21:06.799Z 814e2cf0-75d6-46d1-ac51-b330f306b685 closed output, 1422 bytes sent
stdout reader: <Thread(Thread-5, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.799Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stdout reader: <Thread(Thread-5, initial daemon)>
gpg: keyring `/tmp/secring.gpg' created
[DEBUG] 2023-06-15T13:21:06.800Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: keyring `/tmp/secring.gpg' created
gpg: keyring `/tmp/pubring.gpg' created
[DEBUG] 2023-06-15T13:21:06.800Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: keyring `/tmp/pubring.gpg' created
gpg: no valid OpenPGP data found.
[DEBUG] 2023-06-15T13:21:06.818Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: no valid OpenPGP data found.
[GNUPG:] NODATA 1
[DEBUG] 2023-06-15T13:21:06.819Z 814e2cf0-75d6-46d1-ac51-b330f306b685 [GNUPG:] NODATA 1
gpg: Total number processed: 0
[DEBUG] 2023-06-15T13:21:06.819Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: Total number processed: 0
[GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[DEBUG] 2023-06-15T13:21:06.819Z 814e2cf0-75d6-46d1-ac51-b330f306b685 [GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0
gpg returned a non-zero error code: 2
[WARNING] 2023-06-15T13:21:06.838Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg returned a non-zero error code: 2
import_keys result: {'gpg': <gnupg.GPG object at 0x7f1a564dbca0>, 'results': [{'fingerprint': None, 'problem': '0', 'text': 'No valid data found'}], 'fingerprints': [], 'count': 0, 'no_user_id': 0, 'imported': 0, 'imported_rsa': 0, 'unchanged': 0, 'n_uids': 0, 'n_subk': 0, 'n_sigs': 0, 'n_revoc': 0, 'sec_read': 0, 'sec_imported': 0, 'sec_dups': 0, 'not_imported': 0, 'stderr': "gpg: keyring `/tmp/secring.gpg' created\ngpg: keyring `/tmp/pubring.gpg' created\ngpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\ngpg: Total number processed: 0\n[GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 'data': b'', 'returncode': 2}
[DEBUG] 2023-06-15T13:21:06.839Z 814e2cf0-75d6-46d1-ac51-b330f306b685 import_keys result: {'gpg': <gnupg.GPG object at 0x7f1a564dbca0>, 'results': [{'fingerprint': None, 'problem': '0', 'text': 'No valid data found'}], 'fingerprints': [], 'count': 0, 'no_user_id': 0, 'imported': 0, 'imported_rsa': 0, 'unchanged': 0, 'n_uids': 0, 'n_subk': 0, 'n_sigs': 0, 'n_revoc': 0, 'sec_read': 0, 'sec_imported': 0, 'sec_dups': 0, 'not_imported': 0, 'stderr': "gpg: keyring `/tmp/secring.gpg' created\ngpg: keyring `/tmp/pubring.gpg' created\ngpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\ngpg: Total number processed: 0\n[GNUPG:] IMPORT_RES 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 'data': b'', 'returncode': 2}
Keys:
20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
[DEBUG] 2023-06-15T13:21:06.878Z 814e2cf0-75d6-46d1-ac51-b330f306b685 20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
stderr reader: <Thread(Thread-6, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.879Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stderr reader: <Thread(Thread-6, initial daemon)>
stdout reader: <Thread(Thread-7, initial daemon)>
[DEBUG] 2023-06-15T13:21:06.898Z 814e2cf0-75d6-46d1-ac51-b330f306b685 stdout reader: <Thread(Thread-7, initial daemon)>
gpg: /tmp/trustdb.gpg: trustdb created
[DEBUG] 2023-06-15T13:21:06.919Z 814e2cf0-75d6-46d1-ac51-b330f306b685 gpg: /tmp/trustdb.gpg: trustdb created
[]
2023-06-15T13:21:06.980Z 814e2cf0-75d6-46d1-ac51-b330f306b685 Task timed out after 3.04 seconds
I bolded the most interesting part. It seems like there's an issue with the keys, or at least their format, that I'm trying to import. I've generated the keypair in Kleopatra and copy/pasted them directly from their .asc files after exporting, for testing purposes. Correct me if I'm wrong, but should the format not be:
-----BEGIN PGP PUBLIC KEY BLOCK-----
XXXXXXXXXXX
-----BEGIN PGP PUBLIC KEY BLOCK-----
XXXXXXXXXXX
XXXXXXXXXXX
XXXXXXXXXXX
XY==
=ZABC
-----END PGP PUBLIC KEY BLOCK-----
XY==
=ZABC
-----END PGP PUBLIC KEY BLOCK-----
(And the private key block equivalent.)
Vinay Sajip
Jun 15, 2023, 9:48:57 AM6/15/23
to python-gnupg
That can't be all the output, there's a bit missing at the top. Anyway, you will see that the log output contains the commands passed to gpg. Select the one for importing keys and call that command line from a terminal, specifying the .asc files as input. What happens? If that doesn't work, there's your problem right there - there's some problem with the .asc files according to gpg.
Connor Doonan
Jun 15, 2023, 10:00:58 AM6/15/23
to python-gnupg
I'm programming this on AWS Lambda, and I suspect that it automatically truncates some of the logs, because I'm getting different parts of the logs on different runs.
So, I realized I wasn't adding any '\n' to my keys, and had it all as one long string. I tried adding \n at the end of each line, and it seems that had some impact. However, I'm still running into errors. Do these logs tell you anything?
000000000000000000000000000000', 'ok': '17', 'text': 'Not actually changed\nEntirely new key\nContains private key\n'}], 'fingerprints': ['0000000000000000000000000000000000000000'], 'count': 2, 'no_user_id': 1, 'imported': 0, 'imported_rsa': 0, 'unchanged': 0, 'n_uids': 0, 'n_subk': 0, 'n_sigs': 0, 'n_revoc': 0, 'sec_read': 1, 'sec_imported': 1, 'sec_dups': 0, 'not_imported': 0, 'data': b'', 'stderr': "gpg: keyring `/tmp/secring.gpg' created\ngpg: keyring `/tmp/pubring.gpg' created\ngpg: key 2BF13E1D: no valid user IDs\ngpg: this may be caused by a missing self-signature\ngpg: key FFFFFFFF: secret key imported\n[GNUPG:] IMPORT_OK 17 0000000000000000000000000000000000000000\ngpg: Total number processed: 2\ngpg: w/o user IDs: 1\ngpg: secret keys read: 1\ngpg: secret keys imported: 1\n[GNUPG:] IMPORT_RES 2 1 0 0 0 0 0 0 0 1 1 0 0 0\n", 'returncode': 2}
Keys:
20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
[DEBUG] 2023-06-15T13:57:11.639Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
stderr reader: <Thread(Thread-6, initial daemon)>
[DEBUG] 2023-06-15T13:57:11.640Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 stderr reader: <Thread(Thread-6, initial daemon)>
stdout reader: <Thread(Thread-7, initial daemon)>
gpg: /tmp/trustdb.gpg: trustdb created
[DEBUG] 2023-06-15T13:57:11.699Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 stdout reader: <Thread(Thread-7, initial daemon)>
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 gpg: /tmp/trustdb.gpg: trustdb created
line: 'sec::0:22:FFFFFFFFFFFFFFFF:1686759611::::::::::'
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'sec::0:22:FFFFFFFFFFFFFFFF:1686759611::::::::::'
line: 'fpr:::::::::0000000000000000000000000000000000000000:'
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'fpr:::::::::0000000000000000000000000000000000000000:'
line: 'uid:::::::16D3F23DB610B9DF185590667C65C580E023184C::Doonan, Connor <*******@***.ca>:'
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'uid:::::::16D3F23DB610B9DF185590667C65C580E023184C::Doonan, Connor <*****@***.ca>:'
line: 'ssb::0:18:FFFFFFFFFFFFFFFF:1686759611::::::::::'
[DEBUG] 2023-06-15T13:57:11.701Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'ssb::0:18:FFFFFFFFFFFFFFFF:1686759611::::::::::'
line: 'fpr:::::::::0000000000000000000000000000000000000000:'
[DEBUG] 2023-06-15T13:57:11.701Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'fpr:::::::::0000000000000000000000000000000000000000:'
[{'type': 'sec', 'trust': '', 'length': '0', 'algo': '22', 'keyid': 'FFFFFFFFFFFFFFFF', 'date': '1686759611', 'expires': '', 'dummy': '', 'ownertrust': '', 'sig': '', 'cap': '', 'issuer': '', 'flag': '', 'token': '', 'hash': '', 'curve': 'unavailable', 'compliance': 'unavailable', 'updated': 'unavailable', 'origin': 'unavailable', 'keygrip': 'unavailable', 'uids': ['Doonan, Connor <****@***.ca>'], 'sigs': [], 'subkeys': [['FFFFFFFFFFFFFFFF', None, '0000000000000000000000000000000000000000', None]], 'fingerprint': '0000000000000000000000000000000000000000', 'subkey_info': {'FFFFFFFFFFFFFFFF': {'type': 'ssb', 'trust': '', 'length': '0', 'algo': '18', 'keyid': 'FFFFFFFFFFFFFFFF', 'date': '1686759611', 'expires': '', 'dummy': '', 'ownertrust': '', 'uid': '', 'sig': '', 'cap': '', 'issuer': '', 'flag': '', 'token': '', 'hash': '', 'curve': 'unavailable', 'compliance': 'unavailable', 'updated': 'unavailable', 'origin': 'unavailable', 'keygrip': 'unavailable'}}}]
2023-06-15T13:57:11.741Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 Task timed out after 3.02 seconds
END RequestId: bd3dba14-9086-4b1a-91db-9b27fc3c7462
REPORT RequestId: bd3dba14-9086-4b1a-91db-9b27fc3c7462 Duration: 3022.38 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 80 MB Init Duration: 321.31 ms
So, I realized I wasn't adding any '\n' to my keys, and had it all as one long string. I tried adding \n at the end of each line, and it seems that had some impact. However, I'm still running into errors. Do these logs tell you anything?
000000000000000000000000000000', 'ok': '17', 'text': 'Not actually changed\nEntirely new key\nContains private key\n'}], 'fingerprints': ['0000000000000000000000000000000000000000'], 'count': 2, 'no_user_id': 1, 'imported': 0, 'imported_rsa': 0, 'unchanged': 0, 'n_uids': 0, 'n_subk': 0, 'n_sigs': 0, 'n_revoc': 0, 'sec_read': 1, 'sec_imported': 1, 'sec_dups': 0, 'not_imported': 0, 'data': b'', 'stderr': "gpg: keyring `/tmp/secring.gpg' created\ngpg: keyring `/tmp/pubring.gpg' created\ngpg: key 2BF13E1D: no valid user IDs\ngpg: this may be caused by a missing self-signature\ngpg: key FFFFFFFF: secret key imported\n[GNUPG:] IMPORT_OK 17 0000000000000000000000000000000000000000\ngpg: Total number processed: 2\ngpg: w/o user IDs: 1\ngpg: secret keys read: 1\ngpg: secret keys imported: 1\n[GNUPG:] IMPORT_RES 2 1 0 0 0 0 0 0 0 1 1 0 0 0\n", 'returncode': 2}
Keys:
20: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --list-secret-keys --fingerprint --fingerprint
stderr reader: <Thread(Thread-6, initial daemon)>
stdout reader: <Thread(Thread-7, initial daemon)>
[DEBUG] 2023-06-15T13:57:11.699Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 stdout reader: <Thread(Thread-7, initial daemon)>
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 gpg: /tmp/trustdb.gpg: trustdb created
line: 'sec::0:22:FFFFFFFFFFFFFFFF:1686759611::::::::::'
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'sec::0:22:FFFFFFFFFFFFFFFF:1686759611::::::::::'
line: 'fpr:::::::::0000000000000000000000000000000000000000:'
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'fpr:::::::::0000000000000000000000000000000000000000:'
line: 'uid:::::::16D3F23DB610B9DF185590667C65C580E023184C::Doonan, Connor <*******@***.ca>:'
[DEBUG] 2023-06-15T13:57:11.700Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'uid:::::::16D3F23DB610B9DF185590667C65C580E023184C::Doonan, Connor <*****@***.ca>:'
line: 'ssb::0:18:FFFFFFFFFFFFFFFF:1686759611::::::::::'
[DEBUG] 2023-06-15T13:57:11.701Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'ssb::0:18:FFFFFFFFFFFFFFFF:1686759611::::::::::'
line: 'fpr:::::::::0000000000000000000000000000000000000000:'
[DEBUG] 2023-06-15T13:57:11.701Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 line: 'fpr:::::::::0000000000000000000000000000000000000000:'
[{'type': 'sec', 'trust': '', 'length': '0', 'algo': '22', 'keyid': 'FFFFFFFFFFFFFFFF', 'date': '1686759611', 'expires': '', 'dummy': '', 'ownertrust': '', 'sig': '', 'cap': '', 'issuer': '', 'flag': '', 'token': '', 'hash': '', 'curve': 'unavailable', 'compliance': 'unavailable', 'updated': 'unavailable', 'origin': 'unavailable', 'keygrip': 'unavailable', 'uids': ['Doonan, Connor <****@***.ca>'], 'sigs': [], 'subkeys': [['FFFFFFFFFFFFFFFF', None, '0000000000000000000000000000000000000000', None]], 'fingerprint': '0000000000000000000000000000000000000000', 'subkey_info': {'FFFFFFFFFFFFFFFF': {'type': 'ssb', 'trust': '', 'length': '0', 'algo': '18', 'keyid': 'FFFFFFFFFFFFFFFF', 'date': '1686759611', 'expires': '', 'dummy': '', 'ownertrust': '', 'uid': '', 'sig': '', 'cap': '', 'issuer': '', 'flag': '', 'token': '', 'hash': '', 'curve': 'unavailable', 'compliance': 'unavailable', 'updated': 'unavailable', 'origin': 'unavailable', 'keygrip': 'unavailable'}}}]
2023-06-15T13:57:11.741Z bd3dba14-9086-4b1a-91db-9b27fc3c7462 Task timed out after 3.02 seconds
END RequestId: bd3dba14-9086-4b1a-91db-9b27fc3c7462
REPORT RequestId: bd3dba14-9086-4b1a-91db-9b27fc3c7462 Duration: 3022.38 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 80 MB Init Duration: 321.31 ms
Connor Doonan
Jun 15, 2023, 10:02:31 AM6/15/23
to python-gnupg
Logs from a 2nd run (no timeout):
able', 'compliance': 'unavailable', 'updated': 'unavailable', 'origin': 'unavailable', 'keygrip': 'unavailable'}}}]
23: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --decrypt --skip-verify
[DEBUG] 2023-06-15T14:01:44.580Z e8d6d5d1-690f-4537-927a-c502a0255eb2 23: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --decrypt --skip-verify
data copier: <Thread(Thread-8, initial daemon)>, <_io.BufferedReader name='/tmp/encryptionTest.gpg'>, <_io.BufferedWriter name=8>
[DEBUG] 2023-06-15T14:01:44.600Z e8d6d5d1-690f-4537-927a-c502a0255eb2 data copier: <Thread(Thread-8, initial daemon)>, <_io.BufferedReader name='/tmp/encryptionTest.gpg'>, <_io.BufferedWriter name=8>
stderr reader: <Thread(Thread-9, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.601Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stderr reader: <Thread(Thread-9, initial daemon)>
closed output, 332 bytes sent
[DEBUG] 2023-06-15T14:01:44.601Z e8d6d5d1-690f-4537-927a-c502a0255eb2 closed output, 332 bytes sent
stdout reader: <Thread(Thread-10, initial daemon)>
[GNUPG:] ENC_TO 4FB37A789DC75D9F 18 0
[DEBUG] 2023-06-15T14:01:44.622Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stdout reader: <Thread(Thread-10, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.640Z e8d6d5d1-690f-4537-927a-c502a0255eb2 [GNUPG:] ENC_TO 4FB37A789DC75D9F 18 0
decrypt result[:100]: b''
[DEBUG] 2023-06-15T14:01:44.641Z e8d6d5d1-690f-4537-927a-c502a0255eb2 decrypt result[:100]: b''
verify_file: <_io.BytesIO object at 0x7f76fc0374f0>, None
[DEBUG] 2023-06-15T14:01:44.660Z e8d6d5d1-690f-4537-927a-c502a0255eb2 verify_file: <_io.BytesIO object at 0x7f76fc0374f0>, None
27: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --verify
[DEBUG] 2023-06-15T14:01:44.680Z e8d6d5d1-690f-4537-927a-c502a0255eb2 27: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --verify
data copier: <Thread(Thread-11, initial daemon)>, <_io.BytesIO object at 0x7f76fc0374f0>, <_io.BufferedWriter name=7>
[DEBUG] 2023-06-15T14:01:44.681Z e8d6d5d1-690f-4537-927a-c502a0255eb2 data copier: <Thread(Thread-11, initial daemon)>, <_io.BytesIO object at 0x7f76fc0374f0>, <_io.BufferedWriter name=7>
closed output, 23 bytes sent
[DEBUG] 2023-06-15T14:01:44.700Z e8d6d5d1-690f-4537-927a-c502a0255eb2 closed output, 23 bytes sent
stderr reader: <Thread(Thread-12, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.720Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stderr reader: <Thread(Thread-12, initial daemon)>
stdout reader: <Thread(Thread-13, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.721Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stdout reader: <Thread(Thread-13, initial daemon)>
gpg: no valid OpenPGP data found.
[DEBUG] 2023-06-15T14:01:44.721Z e8d6d5d1-690f-4537-927a-c502a0255eb2 gpg: no valid OpenPGP data found.
[GNUPG:] NODATA 1
[DEBUG] 2023-06-15T14:01:44.740Z e8d6d5d1-690f-4537-927a-c502a0255eb2 [GNUPG:] NODATA 1
[GNUPG:] NODATA 2
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 [GNUPG:] NODATA 2
gpg: the signature could not be verified.
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 gpg: the signature could not be verified.
Please remember that the signature file (.sig or .asc)
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 Please remember that the signature file (.sig or .asc)
should be the first file given on the command line.
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 should be the first file given on the command line.
gpg returned a non-zero error code: 2
[WARNING] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 gpg returned a non-zero error code: 2
False
[GNUPG:] ENC_TO 4FB37A789DC75D9F 18 0
Decrypted Data:
END RequestId: e8d6d5d1-690f-4537-927a-c502a0255eb2
REPORT RequestId: e8d6d5d1-690f-4537-927a-c502a0255eb2 Duration: 2581.51 ms Billed Duration: 2582 ms Memory Size: 128 MB Max Memory Used: 39 MB
able', 'compliance': 'unavailable', 'updated': 'unavailable', 'origin': 'unavailable', 'keygrip': 'unavailable'}}}]
23: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --decrypt --skip-verify
[DEBUG] 2023-06-15T14:01:44.580Z e8d6d5d1-690f-4537-927a-c502a0255eb2 23: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --decrypt --skip-verify
data copier: <Thread(Thread-8, initial daemon)>, <_io.BufferedReader name='/tmp/encryptionTest.gpg'>, <_io.BufferedWriter name=8>
[DEBUG] 2023-06-15T14:01:44.600Z e8d6d5d1-690f-4537-927a-c502a0255eb2 data copier: <Thread(Thread-8, initial daemon)>, <_io.BufferedReader name='/tmp/encryptionTest.gpg'>, <_io.BufferedWriter name=8>
stderr reader: <Thread(Thread-9, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.601Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stderr reader: <Thread(Thread-9, initial daemon)>
closed output, 332 bytes sent
[DEBUG] 2023-06-15T14:01:44.601Z e8d6d5d1-690f-4537-927a-c502a0255eb2 closed output, 332 bytes sent
stdout reader: <Thread(Thread-10, initial daemon)>
[GNUPG:] ENC_TO 4FB37A789DC75D9F 18 0
[DEBUG] 2023-06-15T14:01:44.622Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stdout reader: <Thread(Thread-10, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.640Z e8d6d5d1-690f-4537-927a-c502a0255eb2 [GNUPG:] ENC_TO 4FB37A789DC75D9F 18 0
decrypt result[:100]: b''
[DEBUG] 2023-06-15T14:01:44.641Z e8d6d5d1-690f-4537-927a-c502a0255eb2 decrypt result[:100]: b''
verify_file: <_io.BytesIO object at 0x7f76fc0374f0>, None
[DEBUG] 2023-06-15T14:01:44.660Z e8d6d5d1-690f-4537-927a-c502a0255eb2 verify_file: <_io.BytesIO object at 0x7f76fc0374f0>, None
27: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --verify
[DEBUG] 2023-06-15T14:01:44.680Z e8d6d5d1-690f-4537-927a-c502a0255eb2 27: /opt/python/gpg/g10/gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --homedir /tmp --verify
data copier: <Thread(Thread-11, initial daemon)>, <_io.BytesIO object at 0x7f76fc0374f0>, <_io.BufferedWriter name=7>
[DEBUG] 2023-06-15T14:01:44.681Z e8d6d5d1-690f-4537-927a-c502a0255eb2 data copier: <Thread(Thread-11, initial daemon)>, <_io.BytesIO object at 0x7f76fc0374f0>, <_io.BufferedWriter name=7>
closed output, 23 bytes sent
[DEBUG] 2023-06-15T14:01:44.700Z e8d6d5d1-690f-4537-927a-c502a0255eb2 closed output, 23 bytes sent
stderr reader: <Thread(Thread-12, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.720Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stderr reader: <Thread(Thread-12, initial daemon)>
stdout reader: <Thread(Thread-13, initial daemon)>
[DEBUG] 2023-06-15T14:01:44.721Z e8d6d5d1-690f-4537-927a-c502a0255eb2 stdout reader: <Thread(Thread-13, initial daemon)>
gpg: no valid OpenPGP data found.
[GNUPG:] NODATA 1
[DEBUG] 2023-06-15T14:01:44.740Z e8d6d5d1-690f-4537-927a-c502a0255eb2 [GNUPG:] NODATA 1
[GNUPG:] NODATA 2
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 [GNUPG:] NODATA 2
gpg: the signature could not be verified.
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 gpg: the signature could not be verified.
Please remember that the signature file (.sig or .asc)
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 Please remember that the signature file (.sig or .asc)
should be the first file given on the command line.
[DEBUG] 2023-06-15T14:01:44.741Z e8d6d5d1-690f-4537-927a-c502a0255eb2 should be the first file given on the command line.
gpg returned a non-zero error code: 2
False
[GNUPG:] ENC_TO 4FB37A789DC75D9F 18 0
Decrypted Data:
END RequestId: e8d6d5d1-690f-4537-927a-c502a0255eb2
REPORT RequestId: e8d6d5d1-690f-4537-927a-c502a0255eb2 Duration: 2581.51 ms Billed Duration: 2582 ms Memory Size: 128 MB Max Memory Used: 39 MB
Connor Doonan
Jun 15, 2023, 10:16:06 AM6/15/23
to python-gnupg
Also, to be completely clear:
[DEBUG] 2023-06-15T14:01:44.641Z e8d6d5d1-690f-4537-927a-c502a0255eb2 decrypt result[:100]: b''
This is not a successful decryption, this doesn't resemble the data that was actually encrypted. (Decrypting the same file locally on Kleopatra works.)
[DEBUG] 2023-06-15T14:01:44.641Z e8d6d5d1-690f-4537-927a-c502a0255eb2 decrypt result[:100]: b''
Vinay Sajip
Jun 15, 2023, 10:39:35 AM6/15/23
to python-gnupg
You don't appear to have tried my other suggestion: "Select the [command line] for importing keys and call that command line from a terminal, specifying the .asc files as input. What happens?"
Connor Doonan
Jun 15, 2023, 11:51:03 AM6/15/23
to python-gnupg
Out of curiosity, since the keys (and .asc files) are working on my laptop, I uploaded them directly to our s3 bucket, and imported them directly with the import_keys_file() function. And, incredibly, I'm getting the same results. This seems to imply an inherent issue on python-gnupg's side...
Vinay Sajip
Jun 15, 2023, 11:55:40 AM6/15/23
to python-gnupg
>
This seems to imply an inherent issue on python-gnupg's side
Why would that be your inference? You could (with no evidence either way) blame some aspect of AWS' Lambda environment - e.g. the gpg version provided.
Connor Doonan
Jun 15, 2023, 11:58:49 AM6/15/23
to python-gnupg
Lambda doesn't provide gpg by default, I had to download and upload it myself.
I managed to find the full logs of the run, I've removed the info about the keys and the email and saved it as a .txt file which I'll attach here. I noticed that in the 'import keys:' lines, there was no '-----END public/private key block----', not sure if that's relevant.
I managed to find the full logs of the run, I've removed the info about the keys and the email and saved it as a .txt file which I'll attach here. I noticed that in the 'import keys:' lines, there was no '-----END public/private key block----', not sure if that's relevant.
Connor Doonan
Jun 15, 2023, 2:06:14 PM6/15/23
to python-gnupg
I've also just confirmed that I can decrypt the same file using 'gpg --decrypt' in terminal on my laptop.
Sid Bose
Jun 15, 2023, 8:53:28 PM6/15/23
to python...@googlegroups.com
Hi Conor,
I am using the lambda deployment for gpg encryption/decryption, I initially created it when it was python 3.6. and used secret manager to fetch the keys then api from vault to fetch the keys. It all worked fine until python 3.6 was deprecated and from python 3.9 lambda os image doesn’t have gpg binary any more so you can’t use python-gnupg. So for the moment I switched to 3.7 and working on container image which contains gpg rather than uploading it myself (which are doing) which had its own issues. I would suggest look closely the hog version and then python-gnupg version when using in lambda, all the testing you are doing is it on local laptop because it will have different env than lambda.
In the past I try to create same virtual env as a lambda and then test on it before deploying the package.
Hope this helps.
Thanks
Sid
To view this discussion on the web, visit https://groups.google.com/d/msgid/python-gnupg/9feaa526-8cd4-4da0-ab64-343e6f7c85f6n%40googlegroups.com.
Connor Doonan
Jun 16, 2023, 9:34:19 AM6/16/23
to python-gnupg
Hi Sid, thanks for the suggestions.
I'm using Python 3.8 on Lambda and I built the GPG binary and pushed it to a layer. python-gnupg is able to find the gpg executable successfully. I'm only encountering problems upon trying to import the keys and decrypt the file for some reason.
Here's my full lambda code currently, maybe you can spot something I'm doing wrong:
import json
import base64
import sys
import os
sys.path.insert(0, '/opt')
import gpg
import gnupg
import boto3
from botocore.exceptions import ClientError
import botocore
import boto3
import io
from aws_lambda_powertools.utilities import parameters
import logging
def lambda_handler(event, context):
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logger = logging.getLogger("gnupg")
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel('DEBUG')
logger.addHandler(handler)
logging.info('Started')
gpg = gnupg.GPG(gpgbinary='/opt/python/gpg/g10/gpg', gnupghome='/tmp')
region_name = "ca-central-1"
s3 = boto3.client(
service_name='s3'
)
file = s3.download_file('wdef-7300-s3b-npr-raw-dooc', 'dooc/test/encryptionTest.txt.gpg', '/tmp/encryptionTest.gpg')
pub_file = s3.download_file('wdef-7300-s3b-npr-raw-dooc', 'dooc/test/*****_public.asc', '/tmp/public.asc')
priv_file = s3.download_file('wdef-7300-s3b-npr-raw-dooc', 'dooc/test/*****_SECRET.asc', '/tmp/private.asc')
gpg.import_keys_file("/tmp/public.asc")
gpg.import_keys_file("/tmp/private.asc")
gpg.trust_keys("*****", 'TRUST_ULTIMATE')
print('Keys:')
print(gpg.list_keys(True))
try:
file_contents = s3.get_object(Bucket='wdef-7300-s3b-npr-raw-dooc', Key='dooc/test/encryptionTest.txt.gpg')["Body"].read()
print(file_contents)
decrypted_data = gpg.decrypt(file_contents, always_trust = "true")
print(decrypted_data.ok)
print(decrypted_data.status)
print(decrypted_data.stderr)
print('Decrypted Data: ')
print(str(decrypted_data))
except ClientError as e:
raise e
logging.info('Finished')
return
I'm using Python 3.8 on Lambda and I built the GPG binary and pushed it to a layer. python-gnupg is able to find the gpg executable successfully. I'm only encountering problems upon trying to import the keys and decrypt the file for some reason.
Here's my full lambda code currently, maybe you can spot something I'm doing wrong:
import json
import base64
import sys
import os
sys.path.insert(0, '/opt')
import gpg
import gnupg
import boto3
from botocore.exceptions import ClientError
import botocore
import boto3
import io
from aws_lambda_powertools.utilities import parameters
import logging
def lambda_handler(event, context):
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logger = logging.getLogger("gnupg")
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel('DEBUG')
logger.addHandler(handler)
logging.info('Started')
gpg = gnupg.GPG(gpgbinary='/opt/python/gpg/g10/gpg', gnupghome='/tmp')
region_name = "ca-central-1"
s3 = boto3.client(
service_name='s3'
)
file = s3.download_file('wdef-7300-s3b-npr-raw-dooc', 'dooc/test/encryptionTest.txt.gpg', '/tmp/encryptionTest.gpg')
pub_file = s3.download_file('wdef-7300-s3b-npr-raw-dooc', 'dooc/test/*****_public.asc', '/tmp/public.asc')
priv_file = s3.download_file('wdef-7300-s3b-npr-raw-dooc', 'dooc/test/*****_SECRET.asc', '/tmp/private.asc')
gpg.import_keys_file("/tmp/public.asc")
gpg.import_keys_file("/tmp/private.asc")
gpg.trust_keys("*****", 'TRUST_ULTIMATE')
print('Keys:')
print(gpg.list_keys(True))
try:
file_contents = s3.get_object(Bucket='wdef-7300-s3b-npr-raw-dooc', Key='dooc/test/encryptionTest.txt.gpg')["Body"].read()
print(file_contents)
decrypted_data = gpg.decrypt(file_contents, always_trust = "true")
print(decrypted_data.ok)
print(decrypted_data.status)
print(decrypted_data.stderr)
print('Decrypted Data: ')
print(str(decrypted_data))
except ClientError as e:
raise e
logging.info('Finished')
return
Note that I've tried both gpg.decrypt and decrypt_file and gotten the same results with both.
I've also found a possibly relevant Stackoverflow article that seems to imply that python-gnupg just doesn't work with python 3.8...Although this was posted 3 years ago, so I'm unsure if it's still relevant or not: gnupg - GPG not working with python 3.8 but ok with 3.6 - Stack Overflow
I've also found a possibly relevant Stackoverflow article that seems to imply that python-gnupg just doesn't work with python 3.8...Although this was posted 3 years ago, so I'm unsure if it's still relevant or not: gnupg - GPG not working with python 3.8 but ok with 3.6 - Stack Overflow
Reply all
Reply to author
Forward
0 new messages