Issue with invalid SSL certificate

[Richard Cheesmar]

I'm getting the following error on the local dev machine when using googlecloudstorage to write a file:

  File "/usr/local/google_appengine/google/appengine/api/urlfetch.py", line 467, in _get_fetch_result
    'Invalid and/or missing SSL certificate for URL: ' + url)
SSLCertificateError: Invalid and/or missing SSL certificate for URL: https://accounts.google.com/o/oauth2/token

import cloudstorage as gcs

with gcs.open(filename, 'w', content_type=mime, options={b'x-goog-acl': b'public-read'}) as f:
    f.write(img)
    f.close()

This was working perfectly and there have not been any code changes.
It also works on the live version. 


Has anyone noticed the same?

[George (Cloud Platform Support)]

Hello Richard, 

This is a known issue, and Developers are currently working towards a fix. You can refer to the same issue in the public tracker; further progress will get reflected in that thread, where a temporary workaround is proposed:  

"As a temporary workaround, you can replace the expired urlfetch_cacerts.txt with https://curl.haxx.se/ca/cacert.pem 

On linux you can use the following command: 

wget https://curl.haxx.se/ca/cacert.pem -O {Root of Python SDK}/lib/cacerts/urlfetch_cacerts.txt  ".

[Richard Cheesmar]

Thanks, George, I seem to recall the same error last year. I'll swap the certs as suggested.

[George (Cloud Platform Support)]

If the proposed solution works, it were helpful for many if you confirm here; it would help those in the same situation. 

[Richard Cheesmar]

Yes, it did work, but if you are using app engine you need to find the path for urlfetch_cacerts.txt for the app engine sdk and not the gloud sdk. The rest is as stated, thanks.

Just a quick not that since I had this problem last year, it seems a persistent yet low priority issue, which is more an inconvenience than a problem, but should be dealt with instead of left lingering.

[George (Cloud Platform Support)]

One should not readily assume that the issue is left lingering: in fact, people do work on this issue, which, as you mention, is more of an inconvenience. The fact is that there are more stringent issues, and priorities are assigned accordingly. 

[Richard Cheesmar]

With all due respect George, I know that issues have priorities, but a year is an awful long time for such an issue as a certs with expiry dates, wouldn't you say. The fact is that this issue can cost us little people time and money in our daily development lives.

[George (Cloud Platform Support)]

There should be no doubt that this issue enjoys active attention from side of Developers. You name it an inconvenience, maybe Developers tend to think similarly. However, this issue is accepted as a real issue that needs to get solved. 

[George (Cloud Platform Support)]

Hi Richard, 

Upon closer examination, I have noticed that the main tracker issue, the one covering the situation you described, was in fact closed. I just re-opened this issue with adequate priority, as well as all other related ones, so Developers may now take a closer look and offer a satisfying solution. You are perfectly right, a year is a long time for such an error to still be present. My previous replies were based on the idea that the issues were still open and actively worked upon. I hope my last correction will start things going and progressing at an appropriate pace. 

[Richard Cheesmar]

Thanks, George,

I'll look forward to not having to swap certs next new year.

[George (Cloud Platform Support)]

Hi Richard, 

We do hope that fixing this issue won't take till next year. I'll keep an eye on developments and update the thread accordingly. 

[Kai Wang]

Hi Richard,

Could you share more details to help us look into this issue?

We wonder:

1-  Before swapping to  https://curl.haxx.se/ca/cacert.pem,

     - What SDK (cloud SDK or app engine SDK), and version, were you using?  Asking this because chances are that a antique version of cert cannot connect to Google (yet the chance is low and the cert has to be very antique).

          - you can download arbitrary version of app engine SDK in: https://pantheon.corp.google.com/storage/browser/appengine-sdks/featured/?pli=1

          - you can download install arbitrary version of cloud SDK by:

                   export CLOUDSDK_COMPONENT_MANAGER_FIXED_SDK_VERSION=0.9.57

                   curl https://sdk.cloud.google.com | bash

     - Make sure the urlfetch_cacerts.txt was intact from the SDK you were using? 

     - What was Operating System and version? What was the system openssl version? 

2-  Following this sample: https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/app-engine-cloud-storage-sample

   I succeeded to run a tweaked the cloudstroage demo on dev_appserver 

   Attached is the demo app tweaked from  python-docs-samples/appengine/standard/storage/appengine-client

   NOTE, in main.py line 88-90, I changed arguments to open() to match your report.

 88         with cloudstorage.open(filename, 'w', content_type='binary/octet-stream', options={b'x-goog-acl': b'public-read'}) as f:

 89           f.write('a\n')

 90           f.close()

   Also, injecting debug statements in urlfetch api (located in {<cloud sdk root>/platform, <appengine sdk root>}/google_appengine/google/appengine/api/urlfetch.py)  I found validate_certificate is False when above demo ran.

   Are you trying to connect to real service or just local fake?

   Could you attach a minimum demo, which should be a GAE App similar to I attached, to reproduce the SSL validation error?

I am looking forward to your feedback :)

[Kai Wang]

Hi Richard,

 error on the local dev machine when using googlecloudstorage to write a file:

Which development tool were you using?   Is it GAE's dev_appserver ?  Is it gsutil ? Or else?

I have some experience working on dev_appserver. AFAIK, apps running on dev_appserver have cloudstroage calls backed by a local fake. 

I found two docs mentioning dev_appserver with cloud storage [ 1, 2]

Following the doc 1, I ran the attached demo app using dev_appserver.py;

And hit "localhost:8080" returns:

Demo GCS Application running from Version: None.873839430855777998

Using bucket name: app_default_bucket

Creating file /app_default_bucket/demo-testfile

To be close to your code snippet, I especially changed main.py line 88 to:

with cloudstorage.open(filename, 'w', content_type='binary/octet-stream', options={b'x-goog-acl': b'public-read'}) as f:

FYI, Digging into the urlfetch.py in our SDK, I found validate_certificate defaultly disabled.

For trouble shooting, please DO update this thread with as much input as possible. 

Thanks.

On Thursday, January 4, 2018 at 5:04:40 AM UTC-8, Richard Cheesmar wrote:

[Richard Cheesmar]

Hi, kai,

Chezs-MacBook-Air:google_appengine chez$ gcloud version
Google Cloud SDK 183.0.0
app-engine-python 1.9.64
beta 2017.09.15
bq 2.0.27
core 2017.12.08
gcloud
gsutil 4.28


Chezs-MacBook-Air:google_appengine chez$ openssl version
OpenSSL 0.9.8zh 14 Jan 2016

[Richard Cheesmar]

I'm using Pycharm Pro - latest version

[Kai Wang]

The gcloud version is pretty new. But these information is not enough for reproducing.

If you are running the app with dev_appserver, please ATTACH A DEMO APP for reproducing the issue.

If not, please also give a guide of reproducing it.

Otherwise, without even a stacktrace, it is hard to understand how the following is triggered:

  File "/usr/local/google_appengine/google/appengine/api/urlfetch.py", line 467, in _get_fetch_result
    'Invalid and/or missing SSL certificate for URL: ' + url)
SSLCertificateError: Invalid and/or missing SSL certificate for URL: https://accounts.google.com/o/oauth2/token

Your help will be sincerely appreciated.

[Richard Cheesmar]

kai,

I will have to revert back to the old certs to reproduce it, I'll try to provide some more information sometime this week if possible.

[Kai Wang]

Cool.

By saying "have to roll revert back", do you mean this issue is already solved in your current version of cloud sdk, without swapping urlfetch_cacerts.txt ?

Thanks.

[Richard Cheesmar]

Kai, I'll look into this again soon as I get a moment, please bear with me

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/TU3LwvRClCU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6139c297-a900-4055-a083-a4185a789594%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[James Lumb]

Hi Richard, any update herre? 

To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

[Ray Toal]

When you say "the path for urlfetch_cacerts.txt for the app engine sdk and not the gcloud sdk" what do you mean?

The ONLY thing on my machine is the gcloud sdk. There is no ***separate*** app engine sdk.

Within the gcloud sdk I have this folder

    ~/google-cloud-sdk/platform/google_appengine/lib/cacerts/

And in that folder I replaced urlfetch_cacerts.txt but no luck. So I'm wondering if your mention that the GAE SDK is somehow different from the gcloud sdk is a problem. I scored the entire gcloud installation and as far as I can tell there is ***only one*** urlfetch_cacerts.txt in the whole installation. If I should be using a separate app engine installation I would love to know.

I'm teaching high schoolers GAE this summer with a Google-provided curriculum which uses dev_appserver extensively and using external APIs is part of the course. Unfortunately it is unusable. The thought of saying "hey kids you can't test this locally bc there is a two year-old bug is *not* my first choice. I was thrilled to see this thread but in investigating it and related threads it seems a lot of people have this problem. Replacing the certs for me did not work, so I'm hoping someone could shed light on

    "the path for urlfetch_cacerts.txt for the app engine sdk and not the gcloud sdk"

because I have no idea about a separate app engine sdk, given the fact that app engine is bundled inside gcloud!

Thanks