Service Account Multiple Private Keys not working as expected

[arvsi]

Hi,

 

I recently generated a new public-private key pair for a Service Account that we are using in our application. My intent was that I would roll out the new key to the rest of our developers, but leave the old one enabled for a few days and then delete it to mimic the roll out of a new key in a production environment. Based on the comments found in the Google documentation(https://developers.google.com/console/help/#service_accounts):

 

"You can generate multiple public-private key pairs for a single service account. This makes it easier to update or rollover credentials without application downtime. Note, however, that you cannot delete a key pair if it is the only one created for that service account."

 

I expected that both key sets should work equally for authentication of the Service Account. I tested the new key pair and everything seemed to work as expected. Upon running further tests a few hours later I noticed odd behavior where using the new key pair a functional test that uploaded a file to Google Storage worked about 50% of the time. The other 50% of the time the GoogleCredential object was unable to recieve an authentication token. Switching back to the old token on my local environment yielded the same results.

 

Is there an expected timeframe that it takes for a new authentication key pair to be fully deployed on Google's Authentication systems? Has anyone else had similar experiences in this regard?

 

Thank you for any assistance you can provide regarding this matter.

 

-arvsi