Druid 0.9.1.1: Enabling datasource level authorization in Druid

[Shinesun]

Hi guys,

As stated in Druid 0.9.1.1 release notes one new feature is listed among others: Enabling datasource level authorization in Druid, but couldn't find any documentation regarding it and struggling in setting it up. Is it going to be available any time soon?

Asking because have to enable basic auth at Druid level in order to secure Broker queries from outside of the application.

Best regards,
Shinesun

[Parag Jain]

Hello Shinesun,

Sorry, this feature is not documented mostly because it was experimental and we thought it is going to change. Anyways, I can briefly describe here how it works so that you can work with it.

**** This is what is already there and how it works  ****

For the purpose of authorization, Druid HTTP endpoints will extract two information namely a `Resource` and an `Action`. As of now, there are three types of Resources in Druid - `DATASOURCE`, `CONFIG` and `STATE`. For any type of resource, possible actions are  - `READ` or `WRITE`. Here are some examples of what it means in practice -
If a query is posted to Druid then that means the `Resource` is the 'DATASOURCE' being queried and the `Action` is a `READ` action
If an indexing task is submitted then the `Resource` is the `DATASOURCE ` being indexed and the `Action` is `WRITE`
If a GET is done on coordinator dynamic config then the `Resource` is `CONIFG` and the `Action` is `READ`, vice-versa if a POST is done then `Action` is `WRITE`
If a GET is done on /status endpoint or /leader endpoint then the  `Resource` is `STATE` and the `Action` is `READ`

Thus, POST or DELETE always corresponds to the `WRITE` action and GET to `READ`, the `Resource` type is pre-defined for each endpoint.  `DATASOURCE` resource type contains more information about datasource name, `CONFIG` and `STATE` resource type does not carry any information about the exact configuration or state being accessed or modified.

For more implementation details read the master comment on PR  - https://github.com/druid-io/druid/pull/2424

**** This is what you will have to implement ****

If the `druid.auth.enabled` is set to `true` then for each HTTP call Druid will extract the appropriate `Resource` and `Action` information depending on the endpoint. After this, Druid code will look for an implementation of `AuthorizationInfo` interface in the request attribute of the HTTP request with attribute name as `AuthConfig.DRUID_AUTH_TOKEN` (which is currently set to "Druid-Auth-Token").

Therefore, you will need to implement a servlet Filter which will inject appropriate implementation of `AuthorizationInfo` interface as a request attribute ("Druid-Auth-Token"). After getting this object, Druid will then call `isAuthorized(Resource resource, Action action)` method of your implementation. Thus, you will write your authorization logic inside this method, you can use the Resource object to figure out what is being accessed. For example, for an indexing task POST, resource.getType() will return `ResourceType.DATASOURCE` and resource.getName() will return the datasource name. However, for all `CONFIG` and `STATE` ResourceType, return value of resource.getName() is same and does not matter.

Now the question is how to inject the servlet Filter in Druid. The answer is you will need to create a Druid extension. I am assuming you would already be knowing how to create and load custom extensions with Druid. So, I will just give some pointers here - http://druid.io/docs/latest/development/modules.html. Also, there are numerous extensions in the Druid repo here https://github.com/druid-io/druid/tree/master/extensions-core.

In your extension, you will need to implement `ServletFilterHolder` interface which will return your implementation of  `AuthorizationInfo`. You can follow example of `QosFilterHolder` here - https://github.com/druid-io/druid/blob/master/server/src/main/java/io/druid/server/initialization/jetty/JettyBindings.java#L53 to write logic that can return your own implementation of `AuthorizationInfo`. Finally, bind your implementation of `ServletFilterHolder` in the DruidModule of your extension like this - https://github.com/druid-io/druid/blob/master/server/src/main/java/io/druid/server/initialization/jetty/JettyBindings.java#L48

CONCLUSION -
 - Implement AuthorizationInfo interface and write your authorization logic in the isAuthorized method
 - Inject this implementation in every request sent to Druid as a request attribute with name "Druid-Auth-Token" using servlet filter
 - Enable security by setting druid.auth.enabled=true

As noted this is an experimental feature and is not perfect. If you have any more questions, feel free to ask. Thanks

- Parag

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/de5db44e-9331-4538-a707-88506bb93c45%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Parag Jain]

A minor correction - I said that "In your extension, you will need to implement `ServletFilterHolder` interface which will return your implementation of  `AuthorizationInfo`." but actually I meant "In your extension, you will need to implement `ServletFilterHolder` interface which will return your implementation of servlet Filter."

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/1763962590.2660799.1474506806217%40mail.yahoo.com.

[sy...@mz.com]

Hi Parag, 

I'm trying to implement what you've mentioned in your post, however, I don't know where "druid.auth.enabled" is at? 

Thanks.

Best

Shirley 

[Nishant Bangarwa]

Hi shirley, 

AFAIK, druid.auth.enabled is the config you will need to set in druid runtime.props

the related code reference is https://github.com/nishantmonu51/druid/blob/master/server/src/main/java/io/druid/server/security/AuthConfig.java#L38-L38

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d8473a89-6eb5-41dc-b25e-4aeddd25a056%40googlegroups.com.

[sy...@mz.com]

Hi Nishant, 

I have implemented authorization based on the instructions. It has the basic functionalities. However, I have one question, how to distinguish user input and druid itself's communications. Currently the filter's path is set to "/*", and authorization is using basic authorization from request header. 

This means it will check every request received. If Druid itself is sending request, since there is no user credentials, it will not pass the authorization. Is there a way to bypass this issue? Thanks!

Best

Shirley

[sy...@mz.com]

In more detail, say if I have a path "/druid/v1/*" in the filter. Then whenever I want to query coordinator to get the metadata for druid, the filter will require user credentials. However, druid overlord also uses this path to do lots of operations, for example, getting task status in overlord console. I'm wondering if there is a way to allow druid perform those queries while restricting outside user for it? 

Thanks. 

Best

Shirley

[pja...@yahoo-inc.com]

UPDATED POST -

Hello Shirley,

This is how we have solved this problem - Druid uses HttpClient created here - https://github.com/druid-io/druid/blob/master/server/src/main/java/io/druid/guice/http/HttpClientModule.java#L103 for internal communication with the nodes. Therefore, one way to solve the problem you mentioned is to implement your own HttpClient by extending com.metamx.http.client.AbstractHttpClient.

In your implementation of com.metamx.http.client.AbstractHttpClient, you will have to override the go method. In the go method, you get the Request object and thus you can set appropriate header in the request object using setHeader method such that your authorization module understands this header and puts appropriate implementation of AuthorizationInfo in the request attribute that enables Druid nodes communicate with each other. In our implementation, we set actual security tokens that authorize your Druid nodes for admin access to all actions on the resources but you can set it to whatever you want.

For gluing all these together you will need to extend HttpClientModule.HttpClientProvider and return an instance of your HttpClient in the get method of this class. So this class looks like this -

public class YourHttpClientProvider extends HttpClientModule.HttpClientProvider
{

  // class members...@Inject works here.

  public YourHttpClientProvider()
  {
  }

  public YourHttpClientProvider(Annotation annotation)
  {
    super(annotation);
  }

  public YourHttpClientProvider(Class<? extends Annotation> annotationClazz)
  {
    super(annotationClazz);
  }

  @Override
  public HttpClient get()
  {
    return new YourHttpClient(super.get(), ...other instantiating variables for this class);
  }
}

Finally you will need to bind this provider in the configure method of your DruidModule like this -

binder.bind(HttpClient.class)
      .annotatedWith(Global.class)
      .toProvider(new YourHttpClientProvider(Global.class))
      .in(LazySingleton.class);
binder.bind(HttpClient.class)
      .annotatedWith(Client.class)
      .toProvider(new YourHttpClientProvider(Client.class))
      .in(LazySingleton.class);

For reference YourHttpClient looks like this -

public class YourHttpClient extends AbstractHttpClient
{
  
  private final HttpClient delegate;
  // other class members

  public YourHttpClient(HttpClient delegate, ..other variables)
  {
    this.delegate = delegate;
    ...
  }

  @Override
  public <Intermediate, Final> ListenableFuture<Final> go(
      Request request,
      HttpResponseHandler<Intermediate, Final> handler,
      Duration readTimeout
  )
  {
    // some logic
    // manipulate request object..set headers...do something else so that this request has appropriate things 
    // for your authorization module to identify that this an internal request and 
    // put in appropriate instance of AuthorizationInfo in the request attribute
    return delegate.go(request, handler, readTimeout);
  }
}

All these implementations will go into your module. My bad, I should have included this information in the earlier posts. Let me know if you have any further questions.

Thanks,
Parag

[Priyanka Parekh]

Hi Parag

Thanks for all the info... we did implement everything as suggested by you but don't see our http client being called at all hence all internal requests are getting rejected..

Can you please help?

[pja...@yahoo-inc.com]

Are you sure that your authentication extension is being loaded by druid. If the module is loaded then you should see some log statements like -

INFO [main] io.druid.initialization.Initialization - Loading extension [<your authentication extension>] for class [...]
OR
INFO [main] io.druid.initialization.Initialization - Adding classpath extension module [<your authentication extension>] for class [...]

depending on how whether you load the extension using loadList or put the extension on classpath respectively. Please read "Loading community and third-party extensions (contrib extensions)" and  "Loading extensions from classpath", here - http://druid.io/docs/latest/operations/including-extensions.html

It would help to put some log statements in the configure method of your authentication module (that you are trying to load) to check if bindings are getting executed.