I want to remove attachment_body and attachment_mime type field to not go to solr via Morphline

70 views

Skip to first unread message

ixit shah

unread,

Jun 2, 2016, 5:37:13 AM6/2/16

to CDK Development

Please see the below conf file of Morphline.

How can i do the same.

# Specify server locations in a SOLR_LOCATOR variable; used later in

# variable substitutions:

SOLR_LOCATOR : {

# Name of solr collection

collection : gettingstarted

# ZooKeeper ensemble

zkHost : "localhost:2181"

}

morphlines : [

{

# Name used to identify a morphline. E.g. used if there are multiple

# morphlines in a morphline config file

id : morphline1

# Import all morphline commands in these java packages and their

# subpackages. Other commands that may be present on the classpath are

# not visible to this morphline.

importCommands : ["org.kitesdk.**", "org.apache.solr.**"]

#importCommands : ["org.**", "com.**"]

commands : [

{ readJson: {} }

{ extractJsonPaths { flatten: false, paths: {

"componentType" : /componentType,

"id" : /id,

"jobName" : /jobName,

"jobKey" : /jobKey,

"jobGroup" : /jobGroup,

"endTime" : /endTime,

"jobName" : /jobName,

"jobStatus" : /jobStatus,

"agentName" : /agentName

} } }

{ logError { format : "record: {}", args : ["@{}"] } }

# Parse input attachment and emit a record for each input line

{ logDebug { format : "output record: {}", args : ["@{}"] } }

# This command deletes record fields that are unknown to Solr

# schema.xml.

# Recall that Solr throws an exception on any attempt to load a document

# that contains a field that is not specified in schema.xml.

{

sanitizeUnknownSolrFields {

# Location from which to fetch Solr schema

solrLocator : ${SOLR_LOCATOR}

}

# log the record at INFO level to SLF4J

{ logInfo { format : "output record: {}", args : ["@{}"] } }

# load the record into a Solr server or MapReduce Reducer

{

loadSolr {

solrLocator : ${SOLR_LOCATOR}

}

]

}

]

=========================================

I am getting exception:

2016-06-02 14:39:48,040 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: Failed to send events

at org.apache.flume.sink.solr.morphline.MorphlineSink.process(MorphlineSink.java:186)

at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.solr.client.solrj.impl.CloudSolrServer$RouteException: ERROR: [doc=ixits-ux_job1_1464858313005] unknown field '_attachment_body'

at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:360)

at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:533)

at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)

at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)

at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)

at org.kitesdk.morphline.solr.SolrServerDocumentLoader.sendLoads(SolrServerDocumentLoader.java:140)

at org.kitesdk.morphline.solr.SolrServerDocumentLoader.sendBatch(SolrServerDocumentLoader.java:131)

at org.kitesdk.morphline.solr.SolrServerDocumentLoader.commitTransaction(SolrServerDocumentLoader.java:94)

at org.kitesdk.morphline.solr.LoadSolrBuilder$LoadSolr.doNotify(LoadSolrBuilder.java:104)