Work around for 'The read operation timed out'?

[brend...@hivemedia.com]

Hi,

I'm getting an SSLError: ('The read operation timed out',) DownloadReportToFile step in my reporting. The job ID is "1738120645". I confirmed that the report status is "COMPLETED". Here are the dimensions:

report_job = {
  'reportQuery': {
      'dimensions': ['DATE', 'COUNTRY_NAME','CUSTOM_CRITERIA','AD_UNIT_NAME'],
          'columns': ['TOTAL_LINE_ITEM_LEVEL_CPM_AND_CPC_REVENUE'
                  ,'TOTAL_LINE_ITEM_LEVEL_IMPRESSIONS'
                  ,'TOTAL_LINE_ITEM_LEVEL_CLICKS'
                  ,'AD_SERVER_IMPRESSIONS'
                  ,'AD_SERVER_CLICKS'
                  ,'AD_SERVER_CPM_AND_CPC_REVENUE'
                  ,'AD_EXCHANGE_LINE_ITEM_LEVEL_REVENUE'
                  ,'AD_EXCHANGE_LINE_ITEM_LEVEL_TARGETED_IMPRESSIONS'
                  ,'AD_EXCHANGE_LINE_ITEM_LEVEL_TARGETED_CLICKS'
                  ,'ADSENSE_LINE_ITEM_LEVEL_REVENUE'
                  ,'ADSENSE_LINE_ITEM_LEVEL_IMPRESSIONS'
                  ,'ADSENSE_LINE_ITEM_LEVEL_CLICKS'
                     ],
      'dateRangeType': 'CUSTOM_DATE',
      'startDate': {'year': single_date.year,
                    'month': single_date.month,
                    'day': single_date.day},
      'endDate': {'year': single_date.year,
                  'month': single_date.month,
                  'day': single_date.day}
  }

This is only for ONE day.

I have no doubt that this is a lot of data. The issue is that I need all of these dimensions to pivot off each other and I can't easily filter on the largest dimensions (CustumCriteria can't be filtered?).

This issue happens irregularly depending on traffic volume.

I am curious if there are any simple ways to extend the timeout period?

I'm also seeing talk about the ability to filter rows before downloading? How could I add that to my code?

report_downloader = dfp.DfpClient.LoadFromStorage().GetDataDownloader(version='v201608')

try:
# Run the report and wait for it to finish.
    report_job_id = report_downloader.WaitForReport(report_job)
except errors.DfpReportError, e:
    print 'Failed to generate report. Error was: %s' % e
print 'report generated, downloading'
export_format = 'CSV_DUMP'
report_file = tempfile.NamedTemporaryFile(suffix='.csv.gz', delete=False)


report_downloader.DownloadReportToFile(report_job_id, export_format, report_file)
report_file.close()

Thanks!

[Tesfamichael Hailegeorgis(DFP API Team)]

Hi,

Please try to increase the system default timeout value in your code according to this document and let us know how this goes.

If you still have this unresolved after implementing the suggestion, please provide your network code so we can take a look from our side.

Thanks,

Tesfamichael Hailegeorgis, DFP API Team

[brend...@hivemedia.com]

Hi Tesfamichael. Thanks for the quick reply!

I have always used the following code to import googleads and set the socket timeout:

from datetime import datetime
from datetime import timedelta
import csv
import gzip
from googleads import dfp
from googleads import errors
import tempfile
import socket
socket.setdefaulttimeout(15 * 60)
print socket.getdefaulttimeout()

The query does seem to time out in 90 seconds even with this timeout tip.

Does the socket defaults change my timeout duration or DFPs? 

My network code is 21257525

Thanks,

Brendan

[Tesfamichael Hailegeorgis(DFP API Team)]

Hi Brendan,

Thanks for your reply.

This is for the network code 21287525, right? I am asking this as the network code 21257525 doesn't exist.

I am able to run this API report with the Java client library for just one day date range value (March 14) and it runs successfully within a span of 7 minutes. 

It sounds like this report times out due to the size of the report data (104.5 MB).

Since you are using the Python client library and the requests sent by this library are sent via urllib, which is where the timeout is set as described in the document. So, please try to override/increase the timeout with about 8 minutes so that you can be able to avoid this timeout error.

Feel free to let us know how this goes.

Thanks,

Tesfamichael Hailegeorgis, DFP API Team

[brend...@hivemedia.com]

Hi Tesfamichael, sorry for the late reply. Here is a screenshot of our network code...

So I guess the main question I have is that i'm using the socket timeout hint described in the link, but the report downloader is still timing out after 90 seconds.

Here is the full code:

from datetime import datetime
from datetime import timedelta
import csv
import gzip
from googleads import dfp
from googleads import errors
import tempfile
import socket
socket.setdefaulttimeout(15 * 60)

createddatetime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
end_date = datetime.now()
start_date = end_date - timedelta(days=1)
single_date = start_date


# Create report job.
report_job = {
  'reportQuery': {
      #'dimensions': ['DATE', 'COUNTRY_NAME','CUSTOM_CRITERIA','AD_UNIT_NAME'],
  'dimensions': ['DATE', 'CUSTOM_CRITERIA','AD_UNIT_NAME'],


#       'dimensionAttributes': ['ORDER_TRAFFICKER', 'ORDER_START_DATE_TIME',
#                               'ORDER_END_DATE_TIME'],
#           'statement': filter_statement,

          'columns': ['TOTAL_LINE_ITEM_LEVEL_CPM_AND_CPC_REVENUE'
                  ,'TOTAL_LINE_ITEM_LEVEL_IMPRESSIONS'
                  ,'TOTAL_LINE_ITEM_LEVEL_CLICKS'
                  ,'AD_SERVER_IMPRESSIONS'
                  ,'AD_SERVER_CLICKS'
                  ,'AD_SERVER_CPM_AND_CPC_REVENUE'
                  ,'AD_EXCHANGE_LINE_ITEM_LEVEL_REVENUE'
                  ,'AD_EXCHANGE_LINE_ITEM_LEVEL_TARGETED_IMPRESSIONS'
                  ,'AD_EXCHANGE_LINE_ITEM_LEVEL_TARGETED_CLICKS'
                  ,'ADSENSE_LINE_ITEM_LEVEL_REVENUE'
                  ,'ADSENSE_LINE_ITEM_LEVEL_IMPRESSIONS'
                  ,'ADSENSE_LINE_ITEM_LEVEL_CLICKS'
                     ],
      'dateRangeType': 'CUSTOM_DATE',
      'startDate': {'year': single_date.year,
                    'month': single_date.month,
                    'day': single_date.day},
      'endDate': {'year': single_date.year,
                  'month': single_date.month,
                  'day': single_date.day}
  }
}

# Initialize a DataDownloader.

report_downloader = dfp.DfpClient.LoadFromStorage().GetDataDownloader(version='v201608')


try:
# Run the report and wait for it to finish.
    report_job_id = report_downloader.WaitForReport(report_job)
except errors.DfpReportError, e:
    print 'Failed to generate report. Error was: %s' % e
print 'report generated, downloading'
export_format = 'CSV_DUMP'
report_file = tempfile.NamedTemporaryFile(suffix='.csv.gz', delete=False)

# Download report data.
attempts = 0
while attempts < 10:
    try:
        report_downloader.DownloadReportToFile(report_job_id, export_format, report_file)
        report_file.close()
        print 'report downloaded'
        attempts = 10
    except:
        attempts +=1
        print 'failed download, retry attempt:' + str(attempts)

It is failing on the last step

report_downloader.DownloadReportToFile(report_job_id, export_format, report_file)

Thoughts?

Thanks,

Brendan