Flume -> Solr Sink loadSolr Error attempting partial update: missing required field

[Joshua Ostrom]

Attempting a simple partial document update as show on the documentation page

http://kitesdk.org/docs/1.0.0/morphlines/morphlines-reference-guide.html#loadSolrPartialUpdate

Seeing an error when attempting a partial document update

```
        java {
        imports : "import java.util.*;import org.json.*;"

        code : """
                    ...
              String item_id = "foo";
              String multi_val = "bar";
              record.put("id", item_id);
              record.put("multi_val_field", Collections.singletonMap("add", multi_val ));
              return child.process(record);
          """
       }

```
Returns an error     ... 3 more
Caused by: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server: [doc=foo] missing required field: title

I have defined stored="true" on each field def for this schema.  Simply trying to append a new value to whatever is present in the solr index via Flume (i.e. partial update).  What am I missing here?

[Wolfgang Hoschek]

Sounds like you need this fix: https://issues.cloudera.org/browse/KITE-1074 (which is included in CDH 5.4.8 and higher).

Wolfgang

--
You received this message because you are subscribed to the Google Groups "CDK Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdk-dev+u...@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

[Joshua Ostrom]

Thank you.  I'm running vanilla Solr 5.1 / Flume 1.6.  Kite 1.x doesn't support solr5.x AFIAK, rather Solr provides upstream versions of solr-morphlines-cell and solr-morphlines-core.  I've downloaded Solr 5.1 source and added the patch to LoadSolrBuilder.java (which looked quite dated compared to what is slated for kite 1.2


    private SolrInputDocument convert(Record record) {
      Map<String, Collection<Object>> map = record.getFields().asMap();
      SolrInputDocument doc = new SolrInputDocument(new HashMap(2 * map.size()));
      for (Map.Entry<String, Collection<Object>> entry : map.entrySet()) {
        String key = entry.getKey();
       
        //doc.setField(key, entry.getValue(), getBoost(key));
       
        // Patch to fix solr updating entire document
        // https://github.com/kite-sdk/kite/commit/f77b7bf8b124bcb6154b3ea7f3e47826c2046eac#diff-51ec825471148c657183a2bf0a9ae06b
           Collection<Object> values = entry.getValue();
          if (values.size() == 1 && values.iterator().next() instanceof Map) {
            doc.setField(key, values.iterator().next(), getBoost(key)); // it is an atomic update
          } else {
            doc.setField(key, values, getBoost(key));
          }      
      }
      return doc;
    }

I compiled that and unfortunately it's still erring:

Caused by: org.kitesdk.morphline.api.MorphlineRuntimeException: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server  [doc=3_29342115] missing required field: title
    at org.apache.solr.morphlines.solr.LoadSolrBuilder$LoadSolr.doNotify(LoadSolrBuilder.java:95)

Any thoughts?

[Wolfgang Hoschek]

Maybe the new jar somehow isn't getting picked up by your environment?

[Joshua Ostrom]

The new jar is being picked up, I've added a  System.out.println("Atomic Update for " + key); so I can verify the the new logic is, in fact, being reached, which it is.  But solrcould is still complaining about required fields not present

2016-04-19 15:30:23,114 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:872)] Request to collection retail_feeds failed due to (400) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at  [doc=3_29342115] missing required field: title, retry? 0

Is there a simple way to test solrcloud support for atomic updates?  (I'm assuming they were present in 5.1).

Thanks!
Joshua