doc.sagemath.org gone from google

[William Stein]

I'm googling for links to the sage reference manual, e.g.,

https://www.google.com/search?q=sage+Elements+of+Quotients+of+Univariate+Polynomial+Rings&rlz=1C5CHFA_enUS691US691&oq=sage+Elements+of+Quotients+of+Univariate+Polynomial+Rings&aqs=chrome..69i57j69i64.1153j0j7&sourceid=chrome&ie=UTF-8

and they now **all** go to combinat.sagemath.org. I had expected to
find links such as

http://doc.sagemath.org/html/en/reference/polynomial_rings/sage/rings/polynomial/polynomial_quotient_ring_element.html

but nope.

--
William (http://wstein.org)

[Harald Schilly]

Does anyone know who is administrating the combinat pages? Hiding those pages could help. Besides that, I'm working with Paul Masson for months now to fix these indices. It's a stubborn problem, though...

-- h

--

Harald Schilly -- SageMath, Inc.
https://cloud.sagemath.com

[Harald Schilly]

Btw, try

http://www.bing.com/search?q=sage+Elements+of+Quotients+of+Univariate+Polynomial+Rings

https://yandex.ru/search/?text=sage%20Elements%20of%20Quotients%20of%20Univariate%20Polynomial%20Rings

https://search.yahoo.com/search;_ylc=?p=sage+Elements+of+Quotients+of+Univariate+Polynomial+Rings

-- h

[Volker Braun]

Presumably Google decided that combinat is the canonical url, right? Hiding combinat is only kicking the problem down the road, there are presumably many copies of the Sage docs hosted somewhere. The correct solution would be to include a <link rel="canonical" href="http://doc.sagemath.org/path/to/help.html"/> in our docs to disambiguate. For example like in https://github.com/Pylons/pylons_sphinx_theme/pull/8/files

[Harald Schilly]

Yes, you are right, this sounds like a really good idea!

-- h

[Paul Masson]

As I understand them canonical URLs distinguish between the same content accessed through different URL forms. Currently Google thinks the documentation is mostly located at www.sagemath.org/doc/ while we want it to appear at doc.sagemath.org/html/en, so the canonical links should help with that distinction. It's unclear to me that it will have any effect on combinat.sagemath.org/doc/, since that is much older content and Google should be able to tell the difference.

The biggest problem right now, however, is that Google is taking a very long time to index doc.sagemath.org at all. At the beginning of June I created sitemaps listing every HTML and PDF document in the Sage documentation and Harald submitted them to Google promptly. After about a month most of the PDF links were indexed and those documents appear at the new location. At about the same time less than a third of the HTML links were listed as indexed by Google. That figure has been steady for the last month, and is currently just a bit over one third.

In order to prompt Google to index the sitemap, over the last month I have manually removed links to documents at www.sagemath.org/doc/ in Google's webmaster tools and added them to the index at doc.sagemath.org/html/en. This has had little effect: even links I've added manually several times still will not appear in a search including site:doc.sagemath.org. Google is just taking it's time to index the new location, even with an explicit sitemap of all HTML files.

If someone knows a better way to prompt Google to index a sitemap I'm all ears, but I don't think it's possible. If we want the documentation to appear at doc.sagemath.org we simply need to wait until Google indexes things. Adding canonical links isn't likely to have an effect if the documents aren't getting indexed.

Regarding combinat.sagemath.org/doc/ I don't think removing it is a displacement of the problem. This is the only other copy of the documentation in the sagemath.org domain. Presumably Google thinks it's relevant even though old because it is located in the same /doc/ directory where Google thinks the newer documentation is located. Removing it from the Internet and pointing to the newer version at doc.sagemath.org would be in the best interest of the entire Sage community, if anyone knows who has access to it.

TL;DR waiting for Google to index...

[leif]

Shouldn't it be *docs*.sagemath.org anyway?

Like docs.python.org, docs.cython.org, etc. (The former has an alias
though, redirecting to docs.)


-leif

[Paul Masson]

Harald got access to the server hosting combinat.sagemath.org and added a server-side redirect to the corresponding documents on doc.sagemath.org. Hopefully that will speed up the indexing by Google of the new documentation location.

[Paul Masson]

In case no one else has noticed, doc.sagemath.org is now showing up in Google searches, including the link that initiated this thread.

Google taketh away, and Google giveth back.

On Sunday, August 7, 2016 at 8:25:56 AM UTC-7, William wrote:

[Samuel Lelievre]

2016-10-20 00:24:17 UTC+2, Paul Masson:

> doc.sagemath.org is now showing up in Google searches,

> including the link that initiated this thread.

Hoorray! This is great news for all SageMath users.

Thanks Paul and Harald for your efforts to make it happen.

And thanks to anyone else involved (eg in the combinat team).

[Johan S. H. Rosenkilde]

> In case no one else has noticed, doc.sagemath.org is now showing up in
> Google searches, including the link that initiated this thread.

Indeed I have noticed! It's really good news for us and - especially -
for new Sage users :-)

Best,
Johan

--

[kcrisman]

On Wednesday, October 19, 2016 at 6:24:17 PM UTC-4, Paul Masson wrote:

In case no one else has noticed, doc.sagemath.org is now showing up in Google searches, including the link that initiated this thread.

Google taketh away, and Google giveth back.

Boy, I sure hope you are right - even yesterday I tried to get a canonical link for something in a thematic tutorial and ended up getting sent to all kinds of old versions living in random places on the web.  Good!

[Paul Masson]

Searching for Sage on Google was hit-and-miss for me yesterday but much more consistent today. It apparently takes time for all Google servers to sync their search results and may take a couple more days to appear consistently.