SEO for dynamically-generated URLs in AngularJS?

[Keith Chima]

TL;DR: My AngularJS app (newsbrute.com, for your reference) dynamically generates URLs for articles like so: "website.com/posts/<article-id>", which are opened by clicking on the post title. Can these dynamic posts be indexed?

These will be permanent posts, but there's one template "post" page with content fetched dynamically from the article-id. Think like Reddit's sub-post page. Obviously I'd want users to be able to google the article title and land directly on the article on my site, rather than my home page. For them to not be able to do that is almost a death knell for the site as a whole.

I understand that Google can now execute javascript for indexing dynamic content. Is it possible for it to index these dynamic URLs, even though they are the same "page" in terms of the routing?

app.config(function ($routeProvider) {

  $routeProvider

    .when('/', {

      templateUrl: 'views/posts.html',

      controller: 'PostsCtrl',

      reloadOnSearch: false

    })

})

Thank you!

[o_O Bille]

Hi Keith,

According to Google, they should right on top of the crawling. They write, that they have made adjustments to the indexing engine, so it will automatically see javascript and css if you have not blocked it out with robots.txt or noindex.

http://googlewebmastercentral.blogspot.dk/2014/05/understanding-web-pages-better.html

I can see, that Google has already found you primary URL: https://www.google.dk/search?q=site%3Anewsbrute.com

Since the description shows, that it has also found some of the articles and their text, I suggest you try to make a sitemap with all the URLs incl. the dyn-URL and send it to Google via Search Console.

But you should remove the hashtag, make pretty "readable URL's" and follow this guide: https://scotch.io/quick-tips/pretty-urls-in-angularjs-removing-the-hashtag

o_O Bille

www.elitenet.dk

[Keith Chima]

Thanks so much for responding, you are a big help so far. So I have a few questions:

1) So these "pages" will be modal-popups with the URL changing "on-click", so I'm concerned about google's ability to index them, as they are currently not hrefs. To get them automatically-indexed, should I just have links to them on the main page? I want users to be able to navigate to the articles directly from Google. What if, for each of those elements, I had a <a href="post-url"> hidden there, like display: none or height: 0px or something? Do you think they would be indexed properly?

2) Since they will be modal popups, the main page will still be shadowed out in the background, but the content will still be in the html. I hear that Google likes to ignore pages with duplicate content, even though it will be hidden in the background. Is there a way, using javascript or something, that I can dynamically tell google not to crawl that content in the background, and just focus on the content of the article for indexing purposes? Another, less-desirable option would be to create a separate link for the article separate from the modal. I basically want the user experience of never having to leave the main page if they don't want to, but still want to enjoy the SEO benefits of having all the content on my site as separate "pages".

3) I don't think I would be able to create a sitemap.xml from the dynamic URLs, as they will be adding new ones every day. Is there some alternative, or way to do this easily?

4) Based on this article, it sounds like not only do they want the # to indicate that I'm using javascript, but they even want me to use #! instead:
http://googlewebmastercentral.blogspot.com/2015/10/deprecating-our-ajax-crawling-scheme.html
Can they detect that my pages are generated using javascript, even if I don't use #! in the URL?


Thank you very much for your time!
-Keith

--
You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/pojLllMG6x8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/d/optout.

[o_O Bille]

Hi Keith,
First I have to tell you, that any angularJS app is a javascript app, not static in it's form. It's a web-app, made for millions of variations.
You have to SEO the concept, not the content. If you absolutely want to SEO the content, you can use ambassadors on SOMA or influencers. Organic search for AngularJS is based on graceful degredation. Not all search engines are as complex as Google's, and Google's javascript reading technology, is IMO far from ready to go competition with the ordinary SEO techniques.
You would need static landingpages for each article, made from a CMS. When the user lands on the page, your AngularJS executes and find the exact "state" in the App, and roll on from there in javascript.
As far as I can see, theres a remote possibilty that Google will index your site, using either pretty urls or the hashbangmethod, that Google mentions in the text "now deprecated".
Both methods are made in desperation for a rising demand. The key is graceful degredation and landing pages.(unique urls).


Answers to your numbered questions:

1. No, you should never use black hat SEO as hidden links or text. If you make display none or height 0 on a link, Google will most likely give you a penguin penalty.
2. Google does not dislike duplicate content, problem is that Google wont (clearly) show the same result to a user on 800 results. Chances are that you "real" content will drown in the duplication.

You need to think out of the box, and generate some unique hook for the search engine to hang on to. IE. a plain template on a unique URL.
3. No, no easy way, you will need to serve the content for Google on a unique hook for it to be optimal. Else you can put your trust in the "javascript reading Google-bot which cannot figure out arcane scripting".
4.You could try to old hashbang method and then force Google to crawl to site with Search Console. The result will be on Google search quite fast.

I hope I did'nt ruin your mood, but you have read my opinion on the matter. The problem with SEO is not in the AngularJS, it's the way you use AngularJS.
You can easely make an application where ppl can still be on the same page. But you will have to throw some serious hours in the project. 
The way I see it, you have made a cool newsticker, you could make another page telling about the concept and SEO that page. Then make an App, a website app. Later you can develop the project so people can customize the news, subscribe to different premade categories.. and so on. 
Hope all that makes sense, I am sure you will find other opinions. Here is a link to some folks who have used months to force some SEO out of webapps. http://www.ng-newsletter.com/posts/serious-angular-seo.html IMO, they have been on the wrong track from the beginning. They propose prerendered pages 301 redirects, external services making prerender and redirects. 


o_O Bille

www.elitenet.dk

[Keith Chima]

Hey Bille,

Thanks for your response, it is very helpful.

The difficult part is that they clearly intended at least some sort of accommodation for dynamically-generated pages, as they say in the deprecated AJAX link, and as they now hope to achieve with their execution of javascript. I thought my method was hacky, sure, but I wouldn't think of it as black-hat SEO; it's not like cramming in a bunch of hidden keyword metatags or something. Heck, it seems less hacky than their original solution of serving up pre-rendered pages to crawlers only :). It's sort of just circumventing the issue that Google is unable to trigger/index dynamic content reachable by users. As long as the content matches up, it seems pretty fair to me. However, that's just my opinion, and you're right that they may have some automated system to detect black hat SEO, and that solution would probably be flagged. You're probably right about the hacky strategies not working or flagging blackhat SEO detection, and considering how difficult it is to get support on any non-paid Google service, it would probably be difficult to recover from that, so I won't do those.

Based on what they've said, however, I believe their intention is to be able to index dynamic javascript pages. I believe the main issue, as it stands right now, is that their crawler can't see my pages. If I were to activate them with a traditional href, I bet Google would crawl it just fine. This is definitely a feature that could wait until after I officially "launch" it, but I worry about suffering in the search results until then. Maybe what I'll do in the meantime as an experiment is set up a way for users to land directly on the article "pages" themselves through a unique URL, add a single direct link to a specific article on the homepage, and see how the search console handles it.

Thanks again,
-Keith

[o_O Bille]

I think thats the right step for now, and also, mb you know that already, but as soon as you use "send to index" in console, its on the net almost just after.
You can check link appearance in Google with the site: command in Google search also. site:www.yousite.com

Looking forward to check out progression :-)

//o_O Bille