Downloading large CSV report files with Python

[Marcel Zemp]

Hi

I've successfully implemented a Python workflow to download scheduled report files (CSV) on the cloud (download to Linux on AWS). Following the samples, I use the get_media method:

response = service.files().get_media(reportId=report_id, fileId=file_id).execute()

with open(file_name, 'wb') as file:

    file.write(response)

So far, so good. This works perfectly for nearly all reports.

Unfortunately, for large report files I have trouble and get the following message:

Traceback (most recent call last):

  File "/home/ec2-user/work/bin/doubleclick-get/doubleclick-campaign-manager-get.py", line 88, in <module>

    .get_media(reportId=args.report, fileId=report_file_id).execute())

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper

    return wrapped(*args, **kwargs)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/googleapiclient/http.py", line 835, in execute

    method=str(self.method), body=self.body, headers=self.headers)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/googleapiclient/http.py", line 162, in _retry_request

    resp, content = http.request(uri, method, *args, **kwargs)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/oauth2client/transport.py", line 175, in new_request

    redirections, connection_type)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/oauth2client/transport.py", line 282, in request

    connection_type=connection_type)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/httplib2/__init__.py", line 1322, in request

    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/httplib2/__init__.py", line 1124, in _request

    headers=headers, redirections=redirections - 1)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/oauth2client/transport.py", line 175, in new_request

    redirections, connection_type)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/oauth2client/transport.py", line 282, in request

    connection_type=connection_type)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/httplib2/__init__.py", line 1322, in request

    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/httplib2/__init__.py", line 1072, in _request

    (response, content) = self._conn_request(conn, request_uri, method, body, headers)

  File "/home/ec2-user/python-env/doubleclick-get/local/lib/python3.5/site-packages/httplib2/__init__.py", line 1054, in _conn_request

    content = response.read()

  File "/usr/lib64/python3.5/http/client.py", line 446, in read

    s = self._safe_read(self.length)

  File "/usr/lib64/python3.5/http/client.py", line 597, in _safe_read

    return b"".join(s)

MemoryError

The problematic file is 2.3 GB. The RAM on this Linux machine is 4 GB. So that should not be the problem, I think.

Questions:

1) What is the best way to download large report files? Are there other ways than the get_media method? BTW where is this method documented? I can only find the get method.

2) Is it possible to download the report files in gzipped format? Those files compress pretty well, e.g. 2.3 GB *.csv compresses to 37 MB *.csv.gz How can I do that? This would also save quite some bandwidth.

3) Can I somehow stream directly into a file? How would I do that?

Thanks for your help!

Marcel

[Marcel Zemp]

See solution in this thread: https://groups.google.com/forum/#!topic/google-doubleclick-for-advertisers-api/oT8noi2S_Aw