Hadoop with Datastore connector

[Andrea Minari]

Hi all,

I'm trying to run a MapReduce job fetching data from the Datastore and outputting into BigQuery following the steps here: https://developers.google.com/hadoop/datastore-to-bigquery-sample#samplecode

I've got the Datastore and project all set up, as well as Bigquery, the Hadoop cluster running, but while running both the job I wrote and the sample "test-mr-datastore-to-bigquery.sh", I'm getting a DatastoreException error:

hadoop-m:tCaused by: com.google.api.services.datastore.client.DatastoreException: a composite filter must have at least one sub-filter

hadoop-m:t at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:114)

hadoop-m:t at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:80)

hadoop-m:t at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109)

hadoop-m:t at com.google.cloud.hadoop.io.datastore.DatastoreIterator.getIteratorAndMoveCursor(DatastoreIterator.java:67)

hadoop-m:t ... 12 more

hadoop-m:tCaused by: com.google.api.client.http.HttpResponseException: 400 Bad Request

hadoop-m:ta composite filter must have at least one sub-filter

hadoop-m:t at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1054)

hadoop-m:t at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:77)

hadoop-m:t ... 14 more

Full log is attached.

On my job, the Exception specify the full query:

14/09/04 23:46:56 INFO mapred.JobClient: Task Id : attempt_201409042325_0001_m_000000_2, Status : FAILED

java.io.IOException: Error connecting to the Datastore to run query: kind {

  name: "FacebookProfile"

}

filter {

  composite_filter {

    operator: AND

  }

}

 

ERROR:a composite filter must have at least one sub-filter

but the query I'm generating doesn't have any filtering:

        Query.Builder q = Query.newBuilder();

        KindExpression.Builder kind = KindExpression.newBuilder();

        kind.setName(inputKindName);

        q.addKind(kind);

        String query = TextFormat.printToString(q);

        DatastoreHadoopInputFormat.setInput(job, query, datastoreDatasetId);

Extra info:

Bdutil version: bdutil-0.35.1

For the sample, I'm using everything that is contained within the samples folder, and for my Job I also used these two jars:

bigquery-connector-0.4.3

datastore-connector-0.14.6

is there any Maven repository where to access them?

Thanks and regards,

Andrea