Firebase blocking Googlebot?

[David Notik]

Hi!

So it seems Google's getting much better at indexing SPAs.

It seems it grabs all of my page except the Firebase stuff, which is of course the meat:

http://d.pr/i/17VhN

Any clues, tips?

Thank you!

--D

[Jacob Wenger]

Hey David,

We currently don't allow the any robots to crawl subdomains of firebaseio.com. You can see that in one of our robots.txt files. We made this decision before the Googlebot could actually handle running JavaScript and have not revisited it since. I can't guarantee we will change our stance on this, but we've added it to our weekly tech discussion to see if we should make any changes here.

Just an FYI that we have heard a lot of people asking for us to add SEO capabilities to Firebase Hosting. This is on our longterm roadmap but I don't have any details to share at the moment. Once we have SEO for Firebase Hosting, you'll be all set with your single-page app.

Jacob

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Notik]

Thanks Jacob! I have no intention to use Firebase Hosting, and so I'm hoping whatever the solution, that it would be generalized?

Why not let the bot in? Because of the resources/data transfer it eats?

SEO is a HUGE void for SPAs still. Yes, there's Phantom.js type solutions, but come on.

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/GJPWJu25nJo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.

[Jacob Wenger]

Hey David,

As I said, I've added this item to our weekly tech discussion. The entire engineering team will discuss it and figure out the best course of action for all of our customers. This did not used to be an issue since Googlebot never executed our JavaScript, so this is still a new thing. Googlebots (and other robots) will count as extra concurrent users and will download data, both of which are metrics we charge on. Please be patient while we figure out what we can do here to make your life easier and the lives of the rest of our developers.

Thanks,

Jacob

[David Notik]

Thank you!

--D

On Fri Nov 14 2014 at 1:46:58 PM David Notik <da...@woven.org> wrote:

Thank you!

--D

On Fri Nov 14 2014 at 1:33:21 PM Jacob Wenger <ja...@firebase.com> wrote:

Hey David,

As I said, I've added this item to our weekly tech discussion. The entire engineering team will discuss it and figure out the best course of action for all of our customers. This did not used to be an issue since Googlebot never executed our JavaScript, so this is still a new thing. Googlebots (and other robots) will count as extra concurrent users and will download data, both of which are metrics we charge on. Please be patient while we figure out what we can do here to make your life easier and the lives of the rest of our developers.

Thanks,

Jacob

On Fri, Nov 14, 2014 at 10:22 AM, David Notik <dave...@gmail.com> wrote:

Thanks Jacob! I have no intention to use Firebase Hosting, and so I'm hoping whatever the solution, that it would be generalized?

Why not let the bot in? Because of the resources/data transfer it eats?

SEO is a HUGE void for SPAs still. Yes, there's Phantom.js type solutions, but come on.

On Fri Nov 14 2014 at 1:10:30 PM Jacob Wenger <ja...@firebase.com> wrote:

Hey David,

We currently don't allow the any robots to crawl subdomains of firebaseio.com. You can see that in one of our robots.txt files. We made this decision before the Googlebot could actually handle running JavaScript and have not revisited it since. I can't guarantee we will change our stance on this, but we've added it to our weekly tech discussion to see if we should make any changes here.

Just an FYI that we have heard a lot of people asking for us to add SEO capabilities to Firebase Hosting. This is on our longterm roadmap but I don't have any details to share at the moment. Once we have SEO for Firebase Hosting, you'll be all set with your single-page app.

Jacob

On Thu, Nov 13, 2014 at 10:20 PM, David Notik <dave...@gmail.com> wrote:

Hi!

So it seems Google's getting much better at indexing SPAs.

It seems it grabs all of my page except the Firebase stuff, which is of course the meat:

http://d.pr/i/17VhN

Any clues, tips?

Thank you!

--D

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/GJPWJu25nJo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/GJPWJu25nJo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to firebase-talk+unsubscribe@googlegroups.com.

[David Notik]

Any word? SEO can be mission critical for some of us.

Thank you!

[Jacob Wenger]

Hi David,

We discussed this and are going to do some due diligence and see what we can do here. I imagine that we will do those investigations next week. It is a medium to high priority task, but not top of mind at the moment. We appreciate your patience while we figure everything out.

Jacob

On Mon, Dec 1, 2014 at 10:16 PM, David Notik <dave...@gmail.com> wrote:

Any word? SEO can be mission critical for some of us.

Thank you!

--

You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/a5253ee7-6cad-4dd8-9f99-8f47a7de21df%40googlegroups.com.

[Niko Nyman]

Hi Jacob,

Does this mean that even if I set up a custom domain on Firebase hosting, the site will be hidden from Google and other search results?

-Niko

[Kato Richardson]

Niko,

Firebase apps are hosting on firebaseapp.com, not firebaseio.com--which refers to the data store, not the web site content (i.e. static assets).

So the answer is no, this does not affect your site, just dynamically served data within the pages (which is generally not going to provide SEO boosts anyways, since it changes frequently). That is a very complicated topic and what Jacob was discussing in his last reply is only one tiny facet of the whole. I'd recommend reading up on this extensively before you have any sleepless nights about SEO.

Cheers,

Kato

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/3670de90-3f2a-403c-bd30-27c9532da49d%40googlegroups.com.

[David Notik]

"this does not affect your site, just dynamically served data within the pages (which is generally not going to provide SEO boosts anyways, since it changes frequently)"

How frequently it changes varies depending on your use case. And for many sites, this dynamically served data is the meat. I understand that Google may devalue content that changes very often (like chat), but it also values fresh content. I also have permanent pages with dynamically served content (http://woven.co/item/LUpmVEhmTnZ2ZktTYmV1SWVrakk=).

We just need some guidance. I'm building an SPA with all its content dynamically served, and the advice was to wait for Google to start understanding JS. Now that it does, I discover Firebase is blocking Google. What are we supposed to do if we care about SEO, or even consider it mission-critical? Ditch Firebase?

Thank you!

--D

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/GJPWJu25nJo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEa%2B2bVe_HCYr-pjn-7nF72eg0GHFjv82ogpvRnWbmLGVQ%40mail.gmail.com.

[Kato Richardson]

David,

This is still a top priority, as Jacob mentioned. Hang in there.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CACekCDnRxuAbc2dayV%2BKQvYs3hnojTwjUxemh1e6enfrb%2B4SBw%40mail.gmail.com.

[Jim Bosveld]

Hi Kato,

Any news on this subject?

Thanks!

[Jacob Wenger]

Hey Jim,

We do have good news! At ng-conf 2015 a few months back, we announced full SEO support for Googlebot. See this video (at the 16:30 mark) for the announcement and this other Google Groups thread for more details.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/72dfb76a-a1b0-47b3-940e-b9cb2f870dd2%40googlegroups.com.

[David Notik]

Great news! Had missed that.

Alas, though I have 2.2.2 (which has the update according to the changelog), Googlebot still doesn't seem to render my site: http://d.pr/i/126Sl

Ideas?

--D

[Michael Lehenbauer]

Hey David, any chance you can ping me (directly if you want, mic...@firebase.com) with your site URL so I can take a look and see if I can figure out what's going on?

Thanks,

-Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/9b165340-a3d6-4aba-ba5d-aa2935817f2a%40googlegroups.com.

[Michael Lehenbauer]

Oh, sorry.  I see the URL is in the screenshot.  Nevermind. :-)

[Jim Bosveld]

I've upgraded in my test envorinment to 2.2.4 and it seems to work. 'Fetch as Google' from the Webmaster Tools now returns the full page including Firebase data. Thanks for the reply!

[Michael Lehenbauer]

Great to hear.  Thanks for following up!

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/79ac3091-7a2c-448d-b4ed-09bd83571d10%40googlegroups.com.