URLFetchApp returns HTML instead of JSON, maybe need to fake different user agent?

[Melissa Belvadi]

I have a problem involving UrlFetchApp.fetch.

I have a URL that should retrieve JSON data in response. In my web browser, it does. But in my Google Apps Script, I get back an HTML response instead of the JSON data. I suspect that the server is doing a "user agent" check and won't send the JSON data to the Google Apps user agent. And I can't figure out how to tell it a different user agent, as I can do in python, for example.

This is actually just one example URL of literally hundreds that I have this problem for - the context of this is library subscription vendors providing monthly usage data on their products for the librarians according to a standard called COUNTER SUSHI. This is a small but critical part of a larger project that uses that data, and if I can't solve this, I'm going to have to convert the entire project to python or javascript on my desktop.

Here's one sample URL:

https://avmajournals.avma.org/reports/tr_j1?customer_id=175152&requestor_id=colle...@upei.ca&begin_date=2019-01&end_date=2019-04

Here's the function I'm using to test it and looking at the View-Logs to see the output:

function avmatest()

{

   var testurl = 'https://avmajournals.avma.org/reports/tr_j1?customer_id=175152&requestor_id=colle...@upei.ca&begin_date=2019-01&end_date=2019-04';

   var response = UrlFetchApp.fetch(encodeURI(testurl, { muteHttpExceptions: true }));

   var content = response.getContentText(); 

   Logger.log(testurl);

   Logger.log(content.substring(0,500));  

}

You can paste the URL directly into your browser and see the JSON. Then put the function in a script file and see that you get back something else entirely.

[Adam Morris]

Just do JSON.parse on the text content

JSON.parse(response.getContentText());

Your python libraries that you use already do this for you. What you're using in this stack is just a bit more "bare metal" so to speak.

————————————————————————————

Adam Morris | IT Systems & English Teacher | IGB International School
Jalan Sierramas Utama, Sierramas,
47000 Sungai Buloh, Selangor DE, Malaysia

t    +60 3 6145 4688
f    +60 3 6145 4600
w   www.igbis.edu.my
e    adam....@igbis.edu.my

————————————————————————————

--
You received this message because you are subscribed to the Google Groups "Google Apps Script Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-script-community.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/c1ced315-2feb-445f-ac9b-15e09f130a0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Melissa Belvadi]

Thanks for the reply, but maybe I wasn't clear about the problem. The urlfetchapp is not returning JSON formatted data at all.  Instead it's returning some HTML garbage. Sometimes it returns a generic website, sometimes a page that's talking about how it needs to set cookies or some such. But the json data just isn't in the response at all.

And again, this only happens if the URL hits the server from the Google Script's URLFetchApp. If you put the URL directly into a browser manually, you get the JSON data. If you use the python equivalent of URLFetchApp, you get the JSON data. It's ONLY the URLFetchApp function that the server is refusing to send the JSON data to.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-community+unsub...@googlegroups.com.

[Martin Hawksey]

Looks like the server needs to see a cookie before it will server JSON. The modified code below makes a call to get the cookie from the header response and then uses it in the next call:

function avmatest()

{

   var testurl = 'https://avmajournals.avma.org/reports/tr_j1?customer_id=175152&requestor_id=colle...@upei.ca&begin_date=2019-01&end_date=2019-04&';

   

   // get the cookie

   var cookie = UrlFetchApp.fetch(testurl).getAllHeaders()['Set-Cookie'];

   

   // use the cookie provided

   var options = {"headers":{ cookie: cookie}};

   var response = UrlFetchApp.fetch(encodeURI(testurl), options);

   

   var content = response.getContentText(); 

   Logger.log(testurl);

   Logger.log(JSON.parse(response.getContentText()));  

}

[Melissa Belvadi]

That solved it, thank you very much!

[Данил Евдокимов]

Можно на Русском языке ?

07 июня 2019 г. 19:42 пользователь "Melissa Belvadi" <mbel...@upei.ca> написал:

That solved it, thank you very much!

On Thursday, June 6, 2019 at 4:21:48 PM UTC-3, Martin Hawksey wrote:

Looks like the server needs to see a cookie before it will server JSON. The modified code below makes a call to get the cookie from the header response and then uses it in the next call:

function avmatest()

{

   var testurl = 'https://avmajournals.avma.org/reports/tr_j1?customer_id=175152&requestor_id=collections@upei.ca&begin_date=2019-01&end_date=2019-04&';

https://avmajournals.avma.org/reports/tr_j1?customer_id=175152&requestor_id=collections@upei.ca&begin_date=2019-01&end_date=2019-04

Here's the function I'm using to test it and looking at the View-Logs to see the output:

function avmatest()

{

   var testurl = 'https://avmajournals.avma.org/reports/tr_j1?customer_id=175152&requestor_id=collections@upei.ca&begin_date=2019-01&end_date=2019-04';

   var response = UrlFetchApp.fetch(encodeURI(testurl, { muteHttpExceptions: true }));

   var content = response.getContentText(); 

   Logger.log(testurl);

   Logger.log(content.substring(0,500));  

}

You can paste the URL directly into your browser and see the JSON. Then put the function in a script file and see that you get back something else entirely.

--
You received this message because you are subscribed to the Google Groups "Google Apps Script Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-community+unsubs...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-apps-script-community.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/c1ced315-2feb-445f-ac9b-15e09f130a0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Apps Script Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-community+unsub...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-script-community.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/dde1e54c-ec74-4854-b8f3-d903aa5a65dd%40googlegroups.com.

[Mateusz Krajewski]

Great option to use retrieved cookie in fetch() calls but what if I get an error like this one:

SyntaxError: Unexpected token < in JSON at position 0

Thanks in advance!