Make appspot URL unsearchable

[Himanshu Kapoor]

Hi, 

I have my staging environment running on appspot URL but it is searchable on google search though it is conflicting with my production URL.

Can i remove staging appspot URL for search or make it private?

Thanks

Himanshu Kapoor

[Barry Hunter]

Probably the easiest way is to use robots.txt http://www.robotstxt.org/

But its only honorary, engines choose to honour it. Google Search should with time. 

Could also add [ login: required ] to app.yaml to make need logging into view (that prevents crawlers accessing it!) 

https://cloud.google.com/appengine/docs/python/config/appref

Or can implment some custom logic in the application itself. Check for some cookie, or even a white listed list of IPs etc, and only show the site if match. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a611b03f-7086-4cca-a9e7-c9266a84aaa2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

As detailed above, assuming that you need some pages present in your website and accessible, while in the same time keeping them out of reach of the search engine indexing process, the first step is disallowing the crawling of these pages in robots.txt. In this way, your pages will meet page removal criteria, and Googlebot will remove them from index when re-crawling.

In addition, it is also advisable to submit a removal request, as detailed below.  Without a specific removal request, this automatic process will take time, usually 60 to 90 days; so to speed up removal, you need to submit a removal request to the Google webmaster tools. More details about page removal criteria on the “Remove URLs Tool” documentation page.  In any case,  once  the removal request submitted, pages will stop being returned in search results quite a while prior to being finally removed from index.

If you need extra fast removal from the GSS index, within 24 hours, you may remove URLs on-demand (based on your quota). To remove multiple URLs, you may submit the sitemap by adding the <expires> tag to the those specific URLs that you want  removed from index. Removing outdated contents is covered in the “Remove outdated content” documentation page.

Independently of the above, in case you want to stop serving certain pages, and to additionally speed up the removal process, you can submit these URL patterns for registration with the GSS search engine by navigating to control panel->Setup->Basics->Advanced->Exclude sites section on the “Exclude sites” documentation page. Registering the URLs will immediately stop serving the corresponding pages.

[Himanshu Kapoor]

Thanks @barry

Can i add [login: admin] in nodeJS?

Added below config in my app.yaml and when i try to view appspot URL it throws 403 : forbidden

runtime: nodejs

handlers:

- url: /.*

  script: IGNORED

  login: admin

On Friday, February 3, 2017 at 4:31:44 PM UTC+5:30, barryhunter wrote:

Probably the easiest way is to use robots.txt http://www.robotstxt.org/

But its only honorary, engines choose to honour it. Google Search should with time. 

Could also add [ login: required ] to app.yaml to make need logging into view (that prevents crawlers accessing it!) 

https://cloud.google.com/appengine/docs/python/config/appref

Or can implment some custom logic in the application itself. Check for some cookie, or even a white listed list of IPs etc, and only show the site if match. 

On 3 February 2017 at 10:24, himanshu kapoor <himanshu....@gmail.com> wrote:

Hi, 

I have my staging environment running on appspot URL but it is searchable on google search though it is conflicting with my production URL.

Can i remove staging appspot URL for search or make it private?

Thanks

Himanshu Kapoor

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

[Attila-Mihaly Balazs]

Yes, "login: admin" is independent of the runtime you use. Are you using a browser to access the URL or some kind of tool (like cURL?). In the browser it should redirect you to the google login page to ensure that you're logged in with a google user that has "owner" level permissions on the project.

Attila

[Himanshu Kapoor]

I am using the browser to access the URL but it does not redirect me to google login page. Mostly I am already logged In with user ID of the project owner.