Error 429 when using cBioPortal API

[matteo....@gmail.com]

Hi everyone.

I am having some issues with the cBioPortal API, querying it using Python results in error 429 (too many requests), even from IP addresses from which I've done no requests before:

In [1]: from bravado.client import SwaggerClient
   ...: cbioportal = SwaggerClient.from_url('https://www.cbioportal.org/api/v3/api-docs',
   ...:                                     config={"validate_requests":False,"validate_responses":False,"validate_swagger_spec": False})

...

HTTPTooManyRequests: 429 Too Many Requests

From a cursory look at the mailing list, it looks like I'm not the only one.

Can you please advise? This is a problem for us, as we rely on cBioPortal API as part of a longer data collection pipeline, which we cannot use at the moment

thank you in advance,

[Aaron L]

HI Matteo, 

Thanks for reaching out.  Last week programmatic api usage was bringing down our servers and the only the way for us to recover was to disable all activity.   We are actively looking for a solution.  If it's very time dependnet, one possibility is that we could give you a different url to point to.  Would that work?

--Aaron

--
You received this message because you are subscribed to the Google Groups "cBioPortal for Cancer Genomics Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cbioportal+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cbioportal/240c0af1-b686-41fe-ab42-889e32c74ef3n%40googlegroups.com.

[Matteo Tiberti]

Hi Aaron,

That would be amazing, thank you. We would like to restart our data collection as soon as possible.

An alternative would be to be able to run a local instance of the API server (which would also include the same data that is included in cBioPortal). I saw instructions on the cBioPortal github on how to have a local instance of the website, but it’s unclear to me whether it would also include a local API server. Would this be a viable long term alternative?

Thanks again 

_____________________________

Matteo Tiberti, PhD

Staff Scientist at the Cancer Structural Biology group
Danish Cancer Society Research Center

Strandboulevarden 49

2100, Copenhagen, Denmark

[Aaron L]

HI Matt, 

Yes but it would be a shame for you to do all the work to stand this up and keep it up to date.   We want to see if we can stabilize this for everyone and we ought to be able to.   We should have api.cbioportal.org set up today and you can start using that.  

What institution do you work for?

[Matteo Tiberti]

Sounds good thank you. We will try your new endpoint today.

I work in the Cancer structural biology group at the Danish Cancer Society in Copenhagen, Denmark

Cheers

_____________________________

Matteo Tiberti, PhD

Staff Scientist at the Cancer Structural Biology group
Danish Cancer Society Research Center

Strandboulevarden 49

2100, Copenhagen, Denmark

[Matteo Tiberti]

Hi again,

I'm testing connecting to the new endpoint using the bravado package, but it still fails (differently):

In [2]:     cbioportal = SwaggerClient.from_url('https://api.cbioportal.org',

   ...:                                         config={"validate_requests":False,"validate_responses":False,"validate_swagger_spec": False})

...

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'api.cbioportal.org'. (_ssl.c:1032)

During handling of the above exception, another exception occurred:

...

MaxRetryError: HTTPSConnectionPool(host='api.cbioportal.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'api.cbioportal.org'. (_ssl.c:1032)")))

...

RequestsFutureAdapterConnectionError:

I guess there still are some issues with SSL. I can open the https://api.cbioportal.org with no issues.

Thanks again,

_____________________________

Matteo Tiberti, PhD

Staff Scientist at the Cancer Structural Biology group
Danish Cancer Society Research Center

Strandboulevarden 49

2100, Copenhagen, Denmark

[Matteo Tiberti]

Sorry - just a clarification on my last email - I can open the https://api.cbioportal.org link via browser with no issues (it loads the APi documentation page).

Best,

_____________________________

Matteo Tiberti, PhD

Staff Scientist at the Cancer Structural Biology group
Danish Cancer Society Research Center

Strandboulevarden 49

2100, Copenhagen, Denmark

[Aaron L]

Ok, thanks for the heads up.  Working on it now.  

[Aaron L]

We're a little behind in setting this up.

Btw, are you by any chance making a lot calls for mutation data, such as this, using a python client?

/api/molecular-profiles/nsclc_tcga_broad_2016_mutations/mutations

[Matteo Tiberti]

Normally we would do some calls for mutation data or data in general, things like

Studies.getAllStudiesUsingGET()

getAllCancerTypesUsingGET()

Molecular_Profiles.fetchMolecularProfilesUsingPOST

fetchMutationsInMultipleMolecularProfilesUsingPOST

our final objective is to fetch all available mutations for a given gene. Last week we didn't run this pipeline much, as far as I'm aware - mostly connected to running tests for some changes we are making to our code

if you mean right now - no, we're not doing any request as far as I'm aware

_____________________________

Matteo Tiberti, PhD

Staff Scientist at the Cancer Structural Biology group
Danish Cancer Society Research Center

Strandboulevarden 49

2100, Copenhagen, Denmark

[Aaron L]

Ok, we have restored access to the api at the original address (www.cbioportal.org).  

Someone was calling GET mutations endpoint with an erroneous gene parameter that was causing it to return mutations for ALL genes.  This was bringing down the server.  We put proper validation in place so now that would result in an error. 

Let us know how it goes.

[Matteo Tiberti]

I see - glad you found out what the problem was!

I just ran a test with the usual pipeline and it worked fine. We'll be running it in the next few days and we'll let you know if anything comes up.

Thank you very much for your help with this. Looks like I won't be needing to set up a local instance after all :)

Best,

_____________________________

Matteo Tiberti, PhD

Staff Scientist at the Cancer Structural Biology group
Danish Cancer Society Research Center

Strandboulevarden 49

2100, Copenhagen, Denmark

[Ivan Grubisic]

Hi,

Can we expect the python client documentation to be updated once the move to https://api.cbioportal.org is live? 

Thanks,

Ivan

[Aaron L]

Hello, 

For now we have restored access for api users at the usual address www.cbioportal.org.  We found the source of the problem that arose last week.  We are still deciding whether to adopt api.cbioportal.org.

--Aaron

To view this discussion visit https://groups.google.com/d/msgid/cbioportal/100fab86-4150-4a4a-8538-380d8d9714f6n%40googlegroups.com.