Load data from Google Cloud Storage to BigQuery using Java

45 views
Skip to first unread message

user3737020 via StackOverflow

unread,
Jun 17, 2014, 1:05:02 AM6/17/14
to google-appengin...@googlegroups.com

I am using code such as
Job job = new Job(); JobConfiguration config = new JobConfiguration(); JobConfigurationLoad loadConfig = new JobConfigurationLoad(); config.setLoad(loadConfig);

    job.setConfiguration(config);

    // Set where you are importing from (i.e. the Google Cloud Storage paths).
    List<String> sources = new ArrayList<String>();
    sources.add("gs://gms_cloud_project/bigquery_data/06_13_2014/namesbystate.csv");
    loadConfig.setSourceUris(sources);
    //state:STRING,sex:STRING,year:INTEGER,name:STRING,occurrence:INTEGER
    // Describe the resulting table you are importing to:
    TableReference tableRef = new TableReference();
    tableRef.setDatasetId("database_test");
    tableRef.setTableId("myTable");
    tableRef.setProjectId(projectId);
    loadConfig.setDestinationTable(tableRef);

    List<TableFieldSchema> fields = new ArrayList<TableFieldSchema>();
    TableFieldSchema fieldState = new TableFieldSchema();
    fieldState.setName("state");
    fieldState.setType("string");
    TableFieldSchema fieldSex = new TableFieldSchema();
    fieldSex.setName("sex");
    fieldSex.setType("string");
    TableFieldSchema fieldName = new TableFieldSchema();
    fieldName.setName("name");
    fieldName.setType("string");
    TableFieldSchema fieldYear = new TableFieldSchema();
    fieldYear.setName("name");
    fieldYear.setType("integer");
    TableFieldSchema fieldOccur = new TableFieldSchema();
    fieldOccur.setName("occurrence");
    fieldOccur.setType("integer");
    fields.add(fieldState);
    fields.add(fieldSex);
    fields.add(fieldName);
    fields.add(fieldYear);
    fields.add(fieldOccur);
    TableSchema schema = new TableSchema();
    schema.setFields(fields);
    loadConfig.setSchema(schema);

    // Also set custom delimiter or header rows to skip here....
    // [not shown].

    Insert insert = bigquery.jobs().insert(projectId, job);
    insert.setProjectId(projectId);
    JobReference jobRef =  insert.execute().getJobReference();

    // ... see rest of codelab for waiting for job to complete.
    return jobRef;

But the message when running is 400 error Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Field name already exists in schema", "reason" : "invalid" } ], "message" : "Field name already exists in schema" } at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:159) at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:203) at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:237) at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:207) at com.google.api.services.bigquery.Bigquery$Jobs$Insert.executeUnparsed(Bigquery.java:1978) at com.google.api.services.bigquery.Bigquery$Jobs$Insert.execute(Bigquery.java:2002) at com.google.gms.bq.BigQueryJavaGettingStarted.startQuery(BigQueryJavaGettingStarted.java:259) at com.google.gms.bq.BigQueryJavaGettingStarted.main(BigQueryJavaGettingStarted.java:99)

I am also trying to solve this case. I will be very happy if I can have some suggestions in this case...:) Regards,



Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/10671266/load-data-from-google-cloud-storage-to-bigquery-using-java/24256011#24256011

user3737020 via StackOverflow

unread,
Jun 17, 2014, 4:15:09 AM6/17/14
to google-appengin...@googlegroups.com

But the message when running is 400 error Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Job gimasys.com:compute:job_EQ1gKuqtE3QbKaQ6zribl7vBQCY is not a query job", "reason" : "invalid" } ], "message" : "Job gimasys.com:compute:job_EQ1gKuqtE3QbKaQ6zribl7vBQCY is not a query job" } at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:159) at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:203) at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:237) at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:207) at com.google.api.services.bigquery.Bigquery$Jobs$GetQueryResults.execute(Bigquery.java:2421) at com.google.gms.bq.BigQueryJavaGettingStarted.displayQueryResults(BigQueryJavaGettingStarted.java:379) at com.google.gms.bq.BigQueryJavaGettingStarted.main(BigQueryJavaGettingStarted.java:157)

Reply all
Reply to author
Forward
0 new messages