realtime data from rabbitmq to Druid

120 views
Skip to first unread message

Thuy Le

unread,
Aug 17, 2018, 8:02:50 AM8/17/18
to Druid User
Hi all,

I need a help to ingest realtime data from rabbitmq to Druid 

[root@dmse02lx1567c ~]# curl -X POST -H 'Content-Type: application/json' -d @Druid_Ingestion/rabbitmq-index.json http://localhost:8081/druid/indexer/v1/supervisor


{"error":"Could not resolve type id 'realtime' into a subtype of [simple type, class io.druid.indexing.overlord.supervisor.SupervisorSpec]\n at [Source: HttpInputOverHTTP@4ff1a0dd[c=1749,q=1,[0]=EOF,s=STREAM]; line: 1, column: 4]"} 



{
  "type": "realtime", 
  "spec": {
    "dataSchema" : {
      "dataSource" : "rabbitmq_test",
      "parser" : {
        "type" : "base64",
        "parseSpec" : {
          "format" : "json",
          "timestampSpec" : {
            "column" : "submit_date",
            "format" : "auto"
          },
          "dimensionsSpec" : {
            "dimensions": [ "request_type","origin","description" ],
            "dimensionExclusions" : [],
            "spatialDimensions" : []
          }
        }
      },
      "metricsSpec" : [ {
        "type" : "count",
        "name" : "count"
      } ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "hour",
        "queryGranularity" : "none"
      }
    },
    "ioConfig" : {
      "type" : "realtime",
      "firehose": {
        "type": "rabbitmq",
        "connection" : {
          "host": "172.29.242.173",
          "port": "5672",
          "username": "guest",
          "password": "guest",
          "virtualHost": "/",
          "uri": "amqp://172.29.242.173/"
        },
        "config" : {
          "exchange": "NiFi_Exchanges_test_1",
          "queue" : "NiFi_Queues_test_1",
          "routingKey": "NiFi_Routing_key_test_1",
          "durable": "true",
          "exclusive": "false",
          "autoDelete": "false",
          "maxRetries": "10",
          "retryIntervalSeconds": "1",
          "maxDurationSeconds": "300"
        },
        "plumber": {
          "type": "realtime"
        } 
      }
    },
    "tuningConfig": {
      "type" : "realtime",
      "maxRowsInMemory": 7500000,
      "intermediatePersistPeriod": "PT10m",
      "windowPeriod": "PT10m",
      "basePersistDirectory": "\/tmp\/realtime\/basePersist",
      "rejectionPolicy": {
        "type": "serverTime"
      }
    }
  }
}





Nishant Bangarwa

unread,
Aug 20, 2018, 12:55:40 PM8/20/18
to Druid User
you need to submit a task instead of hitting the supervisor end point. 
Try - 

curl -X POST -H 'Content-Type: application/json' -d @Druid_Ingestion/rabbitmq-index.json http://localhost:8081/druid/indexer/v1/task

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/5f977a1c-15ca-42d3-8c95-421261a2a8e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thuy Le

unread,
Aug 20, 2018, 2:48:54 PM8/20/18
to druid...@googlegroups.com
Hi Bangarwa,


For batch files. This is Streaming Data from rabbitmq. We want to see, use and analysis data during streaming time


Sent from my iPhone
Reply all
Reply to author
Forward
0 new messages