Using Dynamic Configuration Lookups in Pivot

[anu...@getblueshift.com]

Hi,

I am trying to use Dynamic Configuration experimental Lookup feature. As documented here, I had posted the lookup mapping to druid coordinator. My mapping looks like:

{

    "__default": {

        "account_uuid": {

          "type": "map",

          "map": {

            "gwew58ac-ea50-43a2-aaa0-54aca67412de": "example.com"

          }

        }

    }

}

Using coordinator's lookup API curl http://druid-master.blueshift.vpc:8081/druid/coordinator/v1/lookups/__default/account_uuid, I can verify that the mapping was successfully updated.

I am trying to make this mapping available in Pivot. 

Pivot's documentation(http://pivot.imply.io/app-settings) has a section on Lookups but I can't make sense of the formula. 

The example listed in the document is:

      - name: correctValue
        formula: $lookupKey.lookup('my_awesome_lookup')

What is 'my_awesome_lookup'  in the above example?

Can I use lookup tier and id in Pivot? If yes, how? If I am missing something please let me know.

Thanks

Anuraj

[Anuraj Pandey]

FYI: Using imply 2.0.0 which bundles Druid 0.9.2 and Pivot 0.10.27.

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/JEuE7hlhEA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/adac6a70-4ad2-4d1f-9530-49c71f63ce7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rohit kochar]

In your case it should be "account_uuid"

On Saturday, December 17, 2016 at 5:23:26 AM UTC+5:30, Anuraj Pandey wrote:

FYI: Using imply 2.0.0 which bundles Druid 0.9.2 and Pivot 0.10.27.

On Fri, Dec 16, 2016 at 3:50 PM, <anu...@getblueshift.com> wrote:

Hi,

I am trying to use Dynamic Configuration experimental Lookup feature. As documented here, I had posted the lookup mapping to druid coordinator. My mapping looks like:

{

    "__default": {

        "account_uuid": {

          "type": "map",

          "map": {

            "gwew58ac-ea50-43a2-aaa0-54aca67412de": "example.com"

          }

        }

    }

}

Using coordinator's lookup API curl http://druid-master.blueshift.vpc:8081/druid/coordinator/v1/lookups/__default/account_uuid, I can verify that the mapping was successfully updated.

I am trying to make this mapping available in Pivot. 

Pivot's documentation(http://pivot.imply.io/app-settings) has a section on Lookups but I can't make sense of the formula. 

The example listed in the document is:

      - name: correctValue
        formula: $lookupKey.lookup('my_awesome_lookup')

What is 'my_awesome_lookup'  in the above example?

Can I use lookup tier and id in Pivot? If yes, how? If I am missing something please let me know.

Thanks

Anuraj

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/JEuE7hlhEA4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.

[anu...@getblueshift.com]

I have tried $account_uuid.lookup('account_uuid') but didn't work. Realized that lookup mapping is not propagated from coordinator to broker and historic due to following exceptions. Do I need to add some extension to druid.extensions.loadList?

I see the following exception in overlord log:

2016-12-17T23:14:37,695 ERROR [LookupCoordinatorManager--8] io.druid.server.lookup.cache.LookupCoordinatorManager - Error submitting to [http://ip-172-31-103-190.us-west-2.compute.internal:8100/druid/listen/v1/lookups]: {class=io.druid.server.lookup.cache.LookupCoordinatorManager, exceptionType=class java.io.IOException, exceptionMessage=Bad update request to [http://ip-172-31-103-190.us-west-2.compute.internal:8100/druid/listen/v1/lookups] : [500] : [Internal Server Error]  Response: [:)

��error�Could not resolve type id 'account_uuid' into a subtype of [simple type, class io.druid.query.lookup.LookupExtractorFactory]

 at [Source: N/A; line: -1, column: -1]��]}

java.io.IOException: Bad update request to [http://ip-172-31-103-190.us-west-2.compute.internal:8100/druid/listen/v1/lookups] : [500] : [Internal Server Error]  Response: [:)

��error�Could not resolve type id 'account_uuid' into a subtype of [simple type, class io.druid.query.lookup.LookupExtractorFactory]

 at [Source: N/A; line: -1, column: -1]��]

Historical and Broker node shows the same message:

2016-12-17T23:38:16,779 ERROR [qtp208437930-58] io.druid.server.listener.resource.ListenerResource - Exception in handling POSTAll request

java.lang.IllegalArgumentException: Could not resolve type id 'account_uuid' into a subtype of [simple type, class io.druid.query.lookup.LookupExtractorFactory]

 at [Source: N/A; line: -1, column: -1]

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2716) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2707) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.server.listener.resource.AbstractListenerHandler$2.apply(AbstractListenerHandler.java:96) ~[druid-server-0.9.2.jar:0.9.2]

at com.google.common.collect.Maps$7.transformEntry(Maps.java:1810) ~[guava-16.0.1.jar:?]

at com.google.common.collect.Maps$10.getValue(Maps.java:1855) ~[guava-16.0.1.jar:?]

at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:87) ~[guava-16.0.1.jar:?]

at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:294) ~[guava-16.0.1.jar:?]

at io.druid.server.listener.resource.AbstractListenerHandler.handlePOSTAll(AbstractListenerHandler.java:89) ~[druid-server-0.9.2.jar:0.9.2]

at io.druid.server.listener.resource.ListenerResource.serviceAnnouncementPOSTAll(ListenerResource.java:92)

[anu...@getblueshift.com]

Resolved. 

It's because a while back I had POST'ed a bad mapping (see below). I was not able to delete it via /druid/coordinator/v1/lookups/ because you know id was "". Nuked the entry from druid metadata store DELETE FROM druid.druid_config where name='lookups'. Uploaded the correct mapping and it worked.

{

    "__default": {

        "": {

          "type": "account_uuid",

          "map": {

            "gwew58ac-ea50-43a2-aaa0-54aca67412de": "example.com"