UrlFetchApp.fetch(http://example.com) Is throwing "Unexpected error: http://example.com" Error

[Greg]

I am running an ad link check very similar to the google link checker v1.  As of Feb17th I started receiving the error "Unexpected error: http://example.com" in the logs of my daily link checker.

It doesn't happen for every url or even the same url consistently. Some times a url will work and some times it wont work a second later. And ever time I am able to load the page in a web browser with out issues.

Is there anyway to determine what is causing this issue? Is it a bug in UrlFetchApp.fetch(), timeout issue, a server issue or a network issue? Some of the URLs are on different servers. The error message isn't very helpful so any insight would be greatly appreciated.

Here is a simplified version of my script that dies while checking the ad links.(I have verified this code throws the error as well)

var URLFETCH_OPTIONS = { "followRedirects" : false, muteHttpExceptions: false };
function main() {
  urls=[

"http://culliganevansville.com/culligan-water-softener-systems/",

"http://culliganevansville.com/water-test-mt-vernon-princeton-boonville-newburgh-chandler-in/",

"http://GulfCoastCulligan.com/drinking-water-made-by-reverse-osmosis-st-petersburg-clearwater-largo-fl/",

"http://culliganevansville.com/",

"http://gulfcoastculligan.com/culligan-water-softener-benefits/",
"http://culliganwestpalm.com/culligan-bottleless-water-coolers/",
"http://culliganuppercumberland.com/"

]
  for( i in urls){
  try{
      Utilities.sleep(1000);
      var response = UrlFetchApp.fetch(urls[i], URLFETCH_OPTIONS);
    } catch(e){
      Logger.log("Error Fetching: "+urls[i]+" with error: "+ e.message)
    }
    Logger.log(response.getResponseCode()+" : "+urls[i])
  }
}

It might fail the first time you run it or you might have to run it a few times to get it to fail.

Here is the error you would get: 

Unexpected error: http://culliganevansville.com/

Let me know if you need anything else.

Greg

[Joyce Lava (AdWords Scripts Team)]

Hi Greg,

I've tried to check the responseCode returned for the URL and it's returning OK to me. The Apps Script team might be the proper channel to check this with regards the UrlFetchApp. You may contact them through here.

Regards,

Joyce Lava

AdWords Scripts Team

[Naman Jindal]

Set muteHttpExceptions to true and then it should not throw that error.

Cheers,

Naman

[Kilian]

I´m experiencing exact the same error as described by Greg. Not always but sometimes.  
Setting "muteHttpExceptions" to true does not solve the randomly occuring error answer. Code to do the fetch call is this: 

answer = UrlFetchApp.fetch(url, {muteHttpExceptions: true, followRedirects:false});

In addition: That just started to happen like one week ago.

Cheers, 

Kilian  

[Greg]

Thanks for the responses.

Naman, my original script used muteHttpExceptions: true, I just changed it for this example to try to get it to return exceptions. 

The issue isn't that the error is being thrown, the issue is that the error is happening to begin with for no reason. As Killian stated setting muteHttpExceptions doesn't actually stop the error from happening it just makes handling it a little better. Which my original script used muteHttpExceptions: true.

I have a work around in mind with some error catching and multiple fetch attempts, but I will contact the Apps Script team to see if they have any insight related to this issue.

Thanks

Greg

[Kilian]

Hey Greg, 

thx for reaching out to the Apps Script Team. Please let me/us know if you find any answers there!
Sidenote: At first I thought the error is somehow related with the daily url fetch quota imposed by google. But if you hit the limit an according error message is returned: "Service invoked too many times for one day: urlfetch". And this is not the case here. The error happens randomly - as far as I can see. 

Kind regards, 

Kilian  

[Greg]

So I added more information to this App Script issue: https://code.google.com/p/google-apps-script-issues/issues/detail?id=3515 

My work around is to add a while loop that stops after fetching the url a max of 3 times(with a sleep) or if there is a valid response and then add a try-catch inside that to handle the error better so it can continue to run the script.