Attitude towards crawling activities from GCE infrastructure

[Or Ricon]

Hi there,

I would like to inquire about the attitude of Google Compute Platform regarding customers performing crawling, scraping and other kinds of similar activities from GCE infrastructure.

I have gone through the Terms of Service but failed to find a mention of this, which is why I decided to post here.

I can provide further context if required.

Thank you,

Or

[Faizan (Google Cloud Support)]

Hello Or,

Crawling and scraping on GCE shouldn't be an issue, as far as you know how and what your crawl could be, for example:

- Don't ignore robots.txt.

- Don't open multiple simultaneous connections to a given domain.

- Don't crawl the entire web.

- Do put a web page up on your IP explaining what you're up to and how to contact you.

- Make sure crawler's user agent contains a link to the web page explaining the crawler.

- Also you need to abide by our AUP[1].

I hope that helps.

Faizan

[1] https://cloud.google.com/terms/aup