Refresh IAM authentication token used for database authentication

[Sunil Parte]

Hello,

I am using Quarkus 2.5.1.Final to build one microservice. I am using hibernate reactive with postgres as database.  At my organization we use AWS IAM auth tokens for database authentication. AWS IAM token is generated using amazon awssdk and passed as password to database.  To generate IAM auth token I have created custom CredentialsProvider. 

...

@Named("iam-ds-credential-provider")
public class IamDataSourceCredentialProvider implements CredentialsProvider {

...

@Override
public Map<String, String> getCredentials(String credentialsProviderName) {

...

    return Map.of(PASSWORD_PROPERTY_NAME, generatedAWSToken,

USER_PROPERTY_NAME, datasourceUsername);

}

application.properties

...

quarkus.datasource.db-kind=postgresql
quarkus.datasource.credentials-provider=custom
quarkus.datasource.credentials-provider-name=iam-ds-credential-provider

....

Our IAM auth tokens expire every 10 minutes. When server boots up,  password/IAM auth token is cached somewhere in Quarkus framework. Connections created within first 10 minute work. After 10 minutes, new connections created try to use expired IAM auth token. Hence password authentication fails for these connections.

How can I refresh cached database credentials? Or how can i force postgres connection pool to get fresh password(IAM auth token) everytime? Or is there a way to intercept failed connection and then refresh password and retry connection?

Any help is much appreciated. 

Thanks,

Sunil.

[Jaikiran Pai]

Hello Sunil,

I'm not an expert in this area, but looking at the Quarkus code, it
appears to me that this might be an issue within the Agroal (datasource)
framework itself. A quick look seems to indicate that the
CredentialsProvider bean's getCredentials(...) method only gets called
during the Agroal configuration creation and the returned credentials
gets stored for the lifetime of the application within the Agroal
configuration. New connections created in the pool use this in-memory
held credentials. That's just my understanding of what I could find in
the Quarkus code, so what I am saying may not be accurate. You can
confirm this by probably adding something like Thread.dumpStack() within
the implementaiton of the getCredentials method of your
IamDataSourceCredentialProvider. That should tell you when and how many
times that method gets called (I'm guessing it will be called just once).

I don't know if there's a way to get past this and if this is a
limitation in the API.

-Jaikiran

[Luis Barreiro]

Hi Jaikiran,

I want to correct that Agroal does call getCredentials(...) each time a new connection is established.

Password rotation is tested on the library and I believe there are tests in the Vault extension as well.

What can be happening here in Sunil's case is that Agroal is not being used, since it's mentioned that the application is on hibernate reactive.

Unfortunately, I not familiar enough with the vert.x postgres driver to know if it caches credentials or not.

Regards,

--

Luis Barreiro

Middleware Performance Team

[Sunil Parte]

Hi Jai, Luis,

Thanks for looking into this. And nice to see you Jai after a long time :)

Postgres reactive datasource caches credentials.  io.vertx.pgclient.impl.PgConnectionFactory caches the username and password and uses the same one for every new connection. 

What I ended up doing is this - 

1. Override io.vertx.pgclient.impl.PgConnectionFactory.doConnectInternal to fetch IAM authentication for every new connection.

2. On startup of Quarkus app, set io.vertx.pgclient.PgPool.connectionProvider to the new implementation done in step 1.

Thanks,

Sunil.

[Jaikiran Pai]

Hello Luis,

You are right indeed. I now paid a bit more closer attention to the
connection establishment code and it does call the getCredentials(...)
each time. I misread it the first time, sorry about that.

-Jaikiran