Main problem of TiddlyWiki site - is bad Google indexation.

[Siniy-Kit]

For example open Google and write "This plugin adds the ability to display mathematical notation written in LaTeX."  

I see tiddlywiki.com only on 7 position (not too good) in my national search it  is on first position.  if we click this position we will see this https://tiddlywiki.com/static/KaTeX%2520Plugin.html  very nice static page with 2 links. First link to tiddlywiki.com and second link to 404 "file not find"  

TiddlyWiki is great, but "static" mechanism is very old. 

First of all this page https://tiddlywiki.com/static/KaTeX%2520Plugin.html must have right menu.

And the  second -  all links in "static" pages must look like  <a href="https://tiddlywiki.com/static/KaTeX%2520Plugin.html" onclick="window.open('https://tiddlywiki.com/#KaTeX%20Plugin')"">KaTeX Plugin</a>

Google will index this link in a proper way, and people will see main tiddlywiki.com after click on any link.

[Siniy-Kit]

Nobody understand me, and i continue talking with myself :) 

I want to see new webDAV (or ftp auto synchronization by TiddlyDesktop) multipage Tiddlywiki CMS 6.0.0  so we need all Tiddlywiki plugins, TOC, $tw.modal.display, $tw.wiki.addTiddler, $tw.notifier.display and SEARCH in ALL tiddlers work in multipage Tiddlywiki.

I can modify TOC macro to pure CSS view.   I can "duplicate" TW js modules, so  they start work in multipage. But...

Long ago I write javascript Cart  for tiddlywiki, so we can use it for creating on-line store. This module can work both in onepage and multipage Tiddlywiki.

For example Here my multipage Tiddlywiki javascript Cart and here the same view one page Tiddlywiki and zip
 Google can index only  multipage .....

We have https://tiddlywiki.com/#Saving%20on%20TiddlyDesktop  it will be great, if TiddlyDesktop can generate many static pages and upload them to ftp hosting  using login and password.  Now  I use Node.js to generate pages and filezilla to upload them.

[@TiddlyTweeter]

Not me.

I hear you. 

Its part of the general problem of The Great Solution NO ONE CAN FIND in a working form :-)

[TonyM]

Siniy,

You are not talking to yourself. I belive I understand your concerns. Something similar was raised recently. I have not worked with static sites which I believe are the key to improving tiddlywiki appearence in search results.

It seems to me nessasary for implementations that demand a searchable presence on the internet.

Is this your concern?

I am confident there is a solution and in fact believe I have a good idea how to, and believe tiddlywiku has what we need.

Could you state what you need, not a solution or a possible solution just what you want to happen?

I think this will help us proceed.

Regards
Tony

[@TiddlyTweeter]

TonyM wrote:

... static sites which I believe are the key to improving tiddlywiki appearence in search results.

It would be far better to put effort into getting JS rich TW "sites" (single page) better indexed. The idea you make a TW and then have to make another another "static" version SUCKS.

Google can  obviously traverse pages with JS otherwise most of the net would not be indexed.

There is likely a way we are missing. 

I think we need better understanding of G. indexing.

Just my guess

Josiah

[TonyM]

Josiah,

Imagin you have a fantastic tiddlywiki, all its bells and whistles, it is served or single file and you put it up on the internet. The search engins dont load your wiki just the files they can see. They missout on seeing what an interactive user sees.

A bach export of all html that an interactive user may see into to html pages stored along side your wiki online and ensure the search engins can see this content and index it.

Now if as a result of a search engin some one arrives at such a static page, any attempt to interact with that page opens the interactive tiddlywiki transparently would work.

Add a feature to generate new static pages for updates, and I belive the problem will be solved. It just a matter of putting the right files and links online for the search engins to use.

Regards
Tony

[@TiddlyTweeter]

Tony M.

Great reply. Thanks. Complex but doable.

The problem is ...

1 -- HOW will Siniy-Kit get his dynamic shops known? 

2 -- The route of having to go static first does NOT look optimal between one-more-click and "don't bother".

Just impressions

Josiah

[Siniy-Kit]

I see TW 6.0.0  in this way.  I open  TiddlyDesktop , press "add new tiddler" and write "Tiddlywiki is the best CMS for static sites!"  and  then close  TiddlyDesktop.

Next day I open Google and write "Tiddlywiki is the best CMS for static sites!"  and see my site heeg.ru on the first position, I click this link and see my tiddler  with this text (not 404 page).

It is simple, no magic, no node.js, filezilla and server script. Will be it one page or many....  It doesn't matter. 

The main idea - if we publish something (idea, plugin, image) people must have possibility to find it.  Nothing  was done in tiddlywiki in this direction. 

 

понедельник, 7 января 2019 г., 15:06:24 UTC+3 пользователь TonyM написал:

[h0p3]

* https://groups.google.com/forum/#!topic/tiddlywiki/MhVsFURHpoM
* https://philosopher.life/#2019.01.01%20-%20TWGGF%3A%20Google's%20Incentives

I've waited a while to respond to this. I might as well join.

Imho, Google's indexing isn't just poor for https://tiddlywiki.com/ but for all TWs. I grant SEO can be improved for TW, but I am not convinced this problem is going to be solved anytime soon (and most likely not at all).

I'm fairly cynical about the matter. Google has enormous incentives to punish us for not channeling and packaging content into mobile/cache-friendly formats, static tooling, oversimplified atoms of content, and models which maximize their profits. Given their dominance, they can force others to play by their rules, and you won't defeat them on their own turf. They aim to optimize how we build their walled-garden for them,<<ref "l">> track and model users efficiently, and control the masses through a world-class advertisement machine.<<ref "t">> They don't want you to own your identities except insofar as it benefits them. They don't want you to have dynamic control over your data because it's not price efficient for them (static is cheaper to handle) and they also want you to pay (directly or indirectly) for the privilege of using services on or built in virtue of their infrastructure instead.<<ref "a">>

TW embodies the hacker ethic: it is antithetical to Google's ends (and the means to those ends). They have no incentive to enable the decentralization of information power except insofar as it has asymmetrically disrupted their competition. I expect we will continue to be punished for trying to own our data with a tool like TW. Perhaps that will change one day; TW would need to become mainstream enough for them to find it worth specifically parsing, mapping, etc.<<ref "p">>

If SEO matters to you, you are likely relegated to using TW as a CMS or development environment but never as the complete final product. It is possible that TW still isn't the best pick there either.

I hate to say it, but I think TW is the wrong tool for the job even if that's not TW's fault.


---
<<footnotes "l" "They want to be the sole lense through which you see and use the internet (and more).">>

<<footnotes "t" "~85% of their revenue is generated through advertisement. I'm tinfoil-hat enough to believe that's only most of the equation.">>

<<footnotes "a" "Dynamic control over your data, which they are fighting against, is a necessary condition to achieving several forms of digital autonomy.">>

<<footnotes "p" "PDFs are a fine example, but TW is radically more dynamic.">>

[TonyM]

h0p3,

Your way may be one way to look at Google, but when I put myself in their shoes, especially with search engine Optimisation, I just cant subscribe to your cynical view, although I do understand how the profit motive impacts what they do. The key problem for TiddlyWiki in this space is something that always has being. If I have a WebSite Driven by a database that an interactive user must login to access, and the result is dependent on the Queries made to that database not only cant Google index it, but it has no right to. If we want to make something searchable there is the robots file and practices you can follow, The WordPress Yoast plugin for WordPress is a great way to get to know how to improve search-ability even in already public websites.

TiddlyWiki is by design somewhat personal in nature, interactive and dynamic in what it displays, and all its content is displayed in response to the users activity., and can be deployed as a single large file, how does a robot know how to extract from tiddlywiki.html the search strings?, all which can only point to the file, not within the file unless google understood how tiddlers are addressed.

If we want the world and google to index our tiddlywiki content, It is simply a matter of supplying it in a form the robots can consume, this includes not hiding it behind authentication or within interactive front ends that a robot can not and will not try and search.

I believe it should be easy to "publish" static html in independent and crosslinked files in batch, or on every change, that contains a direct link into the interactive TiddlyWiki for user interaction. This is no different to adding a Post to WordPress, as they look like a html address even although the are served from a database.

At first this suggestion may look like breaking the single file model, but it is not, the searchable content  can always be generated again from the single file. All it would be is a smart facsimile the robots can read, and while we are at it lets generate a siteindex, and some metadata in the generated html.

Personally, I think we just have not done this yet.

Regards

Tony

[TonyM]

Post Script,

Actually I think we need to also Generate generate a  XML Sitemap from our Wiki to point to the static pages.

XML sitemaps

Main article: Sitemaps

Google introduced the Sitemaps protocol so web developers can publish lists of links from across their sites. The basic premise is that some sites have a large number of dynamic pages that are only available through the use of forms and user entries. The Sitemap files contains URLs to these pages so that web crawlers can find them. Bing, Google, Yahoo and Ask now jointly support the Sitemaps protocol.

Since the major search engines use the same protocol,^[2] having a Sitemap lets them have the updated page information. Sitemaps do not guarantee all links will be crawled, and being crawled does not guarantee indexing.^[3] Google Webmaster Tools allow a website owner to upload a sitemap that Google will crawl, or they can accomplish the same thing with the robots.txt file.^[4]

XML Sitemaps have replaced the older method of "submitting to search engines" by filling out a form on the search engine's submission page. Now web developers submit a Sitemap directly, or wait for search engines to find it. Regularly submitting an updated sitemap when new pages are published may allow search engines to find and index those pages more quickly than it would by finding the pages on its own.^[5] 

Regards

Tony

[Siniy-Kit]

Tony, Tiddlywiki is AJAX site (if I understand, what I say:) Like all AJAX sites, It generates its content by javascript on clients side. Google and all other searching sites can index only final html code (WordPress generates it on server side and gives  result to client)  , they are not able to run js  so file https://tiddlywiki.com/index.html will be not indexed.  BUT we have very primitive "static" mechanism in TW5 . This page https://tiddlywiki.com/static/KaTeX%2520Plugin.html Google see very good, but this page has a very low quality. 

So we need to improve it = and right menu, add normal links, put static folder content to root directory AND make https://tiddlywiki.com/index.html "static" file too...  it will look like now, but it will be indexed very good . It is not difficult, but for my sites I do it manually by node.js and filezilla.

I dont know how to write desktop application for it :(

All my sites have sitemap.xml 

http://heeg.ru/sitemap.xml

 http://novye-podarki.ru/6_7_0/sitemap.xml

Tiddliwiki generates it automatically by this tiddler  http://heeg.ru/heeg.html#%24%3A%2F_sitemap

среда, 9 января 2019 г., 3:58:51 UTC+3 пользователь TonyM написал:

[Thomas Elmiger]

Google (better than other search engines) is very well able to index JS generated content. [1]

It would be interesting to test different settings for TW URL generation and browser history manipulation. My guess is, that appending #tiddlernames to internal links and actually use/insert them in the URL could help.

Maybe other optimisations would be possible with little effort, e.g.
* integration of description meta info in the head
* text on preload page
* generation of XML sitemap with all possible deep links

Cheers,
Thomas

1 https://www.google.com/search?q=does+Google+index+Javascript&oq=does+Google+index+Javascript

[@TiddlyTweeter]

Testing this out. Right. What to add on meta etc? interests me a lot. 

I do want to transfer www.weyersberg.ws to TW and update it. But it has v. good ranking on Google. I'm nervous about going fully TW and losing the ranking. 

How can we best assist Google robots index TW correctly?

Josiah

[Archizona V]

Thomas, can we change #tiddlername to ?tiddlername view? Google don't like # in Url?

[Thomas Elmiger]

Well, here’s a simple test with https://tid.li/tw5/plugins

A search for "site:tid.li/tw5/plugins" gives one single result at the moment (my plugin site). [1]
https://www.google.com/search?q=site%3Atid.li%2Ftw5%2Fplugins

Now I changed the settings (from use always the same URL) to [2]:

• Include the target tiddler and the current story sequence
• Update history
  

I tested in Google Search Console (https://www.google.com/webmasters/) and the bot renders the pages [3] the same way a human would see it ... the bot is just not patient enough and stops rendering before the last part of the page is ready (partial rendering, the last plugin in the long list of the control panel is not entirely visible). The mobile bot is even less patient [4].

LEARNING 1: Don’t make your pages too long.

Then I asked Google to index the pages including internal links. Now let’s see how many pages I have in the index after some time and if my plugins can be found better via Google.

All the best,

Thomas

[1]

To be exact, 3 times the same page with different URLs (1 result, expandable to this):

[2]

[3]

https://tid.li/tw5/plugins.html#Get%20More%20Plugins:%5B%5BGet%20More%20Plugins%5D%5D%20ToDoNow%20%5B%5BToDoNow%20Installation%5D%5D%20%24%3A%2FControlPanel – Desktop Version

... Google sees more than this.

[4]

https://tid.li/tw5/plugins.html – Mobile (Smartphone)

Google sees from top of page to here:

Am Mi., 9. Jan. 2019 um 19:45 Uhr schrieb Archizona V <tiva...@gmail.com>:

Thomas, can we change #tiddlername to ?tiddlername view?  Google don't like # in Url?

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/MhVsFURHpoM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f968685e-b168-4a82-ac2a-6e9fe263472f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Elmiger]

Archizona, let’s see what Google does in my test. Personally I don’t think this should make a difference, both types of parmeters can change what a user sees – and this is what matters for Google too.

[TonyM]

Thomas

Thanks so much for undertaking this analysis. I look forward to the outcomes.

Tony

[Thomas Elmiger]

Hello TW-Googlers,

My first experiment failed, just activating URL changes in the TW settings did not improve the situation.

Next step: I tried to activate the parameter # manually, so Google knows that it can change page content. I let the bot decide, which URLs it wants to crawl.

Let’s see if that changes something.

Cheers,

Thomas

[@TiddlyTweeter]

Thank you for doing these tests Thomas. I appreciate it a lot.

Its important to know the optimal Google orientated strategy.

Tschüss

Josiah

[Arlen Beiler]

Having the static landing pages is an awesome idea if TiddlyWiki doesn't play well with Google, because it has the added benefit of being able to async'ly load the core javascript (if served as a separate file) putting it in the browser cache before the user gets to the page where it's needed making load times much faster. The rest of the wiki could be loaded this way as well, possibly if a short (10 minute) cache value was set on it.

I'm just going to put a plug in here for having one centrally located core and plugin repo that all publicly-accessible wikis can load, allowing the browser to cache the plugins and use them across sites. Since the core code is not used until the page is entirely loaded and then is used as text rather than active Javascript, there are a lot of options available. Of course, you could say this is a security risk, but for public facing sites, the risk is negligible since the entire web is worried about that and they take steps to make sure that doesn't happen. 

I recommend compiling the plugins into the format I use on TiddlyServer, then publishing that to npm with the correct versioning, and using jsdelivr to serve it.

Just my thoughts. 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f41ca853-30ab-446e-8d1b-7721802d05c2%40googlegroups.com.

[TonyM]

Folks,

Without trawling through the thread again. I was wondering if the following feature of a Served TiddlyWiki would overcome the SEO issue 

see https://tiddlywiki.com/#Using%20the%20read-only%20single%20tiddler%20view copied below

The thing is if key read-only single tiddler views was added to the site index would they not be trawled by the search engines?

Regards

Tony

TiddlyWiki's experimental single tiddler per page, read-only view uses a simplified page layout, and implements links between tiddlers, but there are no other interactive features. Compared to a full TiddlyWiki user interface, it is very lightweight and usable even over very slow connections.

Alongside serving the full interactive wiki at the path / (e.g. http://127.0.0.1:8080/), TiddlyWiki serves each tiddler at the path /<url-encoded-tiddler-title>. For example:

• http://127.0.0.1:8080/HelloThere
• http://127.0.0.1:8080/Philosophy%20of%20Tiddlers

Ordinary, non-system tiddlers are rendered through a special view template while system tiddlers are rendered through a template that returns the raw text of the rendered output. In this way ordinary tiddlers can be browsed by end users while system tiddlers can be included in their raw form to use them as JS, HTML or CSS templates. Additionally these defaults can be overwritten on a per tiddler basis by specifying the _render_type and _render_template fields accordingly.