TCP connection left in CLOSE_WAIT state after downloading report

[Chris]

Hi Guys,

I have a script which goes through lots of AdWords accounts, and pulls around 30 reports for each. Recently (around 10 days ago) we started having some problems. Sometimes the process will die, and looking at the errors, the operating system is reporting 'too many open files'. Running the lsof utility I see lots of tcp connections in a CLOSE_WAIT state

e.g. java    17555 root 1332u  IPv6 563454454      0t0        TCP myserver.net:56586->vc-in-f147.1e100.net:https (CLOSE_WAIT)

I am still using the 'old' java client library for the AdWords API, version 201306

To download the reports I am doing the following:

ReportDownloadResponse response = ReportUtils.downloadReport(user, reportDefinition, fos);

The annoying thing is that sometimes the problem with 'too many files' occurs, sometimes it doesn't (and only an expected number of ESTABLISHED connections showing via lsof). Typically the scheduled early morning runs fail, and then when I do a manual run later in the day it will usually succeed (though sometimes the same problem still appears).

There isn't anything in the code that I can/should be doing to close the connection following the report download is there?

If there are more details that would be useful in figuring this out, please let me know. For now I just wanted to present my general problem, and add any relevant info as dialogue hopefully progresses.

Any thoughts / information are very gratefully received! I've been struggling with this issue for a while now, but due to the sporadic nature of whether the problem does / does not occur, it's proving difficult.

Thanks

[Chris]

I should also point out that each account runs in a separate thread, with a thread pool size of 6. An AdWordsUser object is created once per account, and re-used for each call to ReportUtils.downloadReport for that account.

[Josh Radcliff (AdWords API Team)]

Hi Chris,

A few things that might help me track down the source of this issue are:

1. Do you have the stack trace from every exception that your process threw before you encountered the 'too many files' error?  That should help us pinpoint which input or output stream is not getting closed.
   
2. Do you start a new report download process every day, or does your process stay up for several days?  The reason I ask is that if you encounter exceptions on several days in a row on a long running process and those exceptions mean some input or output stream isn't closed, then eventually you will run out of file handles.
3. Have you considered moving to the current Java library?  We made many improvements in the current library, including reworking how we download reports.

Thanks,

Josh, AdWords API Team