Loading CSV from GCS to BigQuery

0 views
Skip to first unread message

user4032667 via StackOverflow

unread,
Apr 16, 2017, 10:52:05 AM4/16/17
to google-appengin...@googlegroups.com

I'm a Python beginner trying to load CSV from my bucket to BigQuery through App Engine with Flask. But although the job.state shows that the job is DONE, I don't see the job in my BigQuery WebUI.

I'm not sure what is the issue at this point.

Here's my code:

@app.route('/bqload')
def bqload():
    job_name = str(uuid.uuid4())
    client = bigquery.Client(project='sturdy-mode-163801')
    dataset = client.dataset('test')
    table = dataset.table(name='testtable')
    table.schema = [
        SchemaField('dt', 'TIMESTAMP', mode='NULLABLE'),
        SchemaField('fans', 'INTEGER', mode='NULLABLE')]
    job = client.load_table_from_storage(job_name, table, 'gs://bucket-
name/*')
    job.field_delimiter = ','
    job.source_format = 'CSV'
    job.skip_leading_rows = 1
    job.write_disposition = 'WRITE_TRUNCATE'
    job.begin()

    retry_count = 100
    while retry_count > 0 and job.state != 'DONE':
        retry_count -= 1
        time.sleep(10)
        job.reload()  
        return job.state


Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/43438445/loading-csv-from-gcs-to-bigquery
Reply all
Reply to author
Forward
0 new messages