Program to Split Single-Page HTML Documentation

[Computer Nerd Kev]

I don't like browsing huge single HTML pages of documentation. Does
anyone know of a program or script (preferably for Linux) that can
scan a big software manual's single HTML page and automatically
break it up according to the contents section and the corresponding
anchor links?

Basically I want something to turn this:
http://www.gnu.org/software/coreutils/manual/coreutils.html

into this:
http://www.gnu.org/software/coreutils/manual/html_node/index.html

But without the Texinfo source like GNU software (usually) uses.
Just from the HTML itself. I also want it to output static HTML, so
no solutions using Javascript or browser add-ons.

One option might be to use csplit to break it up at common section
separator patterns, then a simple script renames the new files
according to their heading text. But I'd like to have HTML
navigation links, ideally including converting existing anchor
links inside the document.

A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
single-page layout:
https://www.raspberrypi.com/documentation/computers/configuration.html

--
__ __
#_ < |\| |< _#

[Computer Nerd Kev]

In comp.os.linux.misc Bud Frede <fr...@mouse-potato.com> wrote:
>> A prime target would be the Raspberry Pi configuration
>> documentation, which has convinced me of the merit of multi-page
>> docs by how confusing it has become for me since they switched to a
>> single-page layout:
>> https://www.raspberrypi.com/documentation/computers/configuration.html
>

> I've never used it myself, but I seem to remember talking to someone in
> the past who used htmldoc to do this kind of thing?
>
> https://www.msweet.org/htmldoc/

Thanks for that. I've been to that website before but it's not
until you look in the README that it mentions the option of HTML as
an output format as well as for input. In fact the "htmlsep"
format option does exactly what I wanted:

mkdir rpi_config
htmldoc -t htmlsep -d ./rpi_config 'https://www.raspberrypi.com/documentation/computers/configuration.html'

It did pass through a few broken relative links that pointed to
other pages at the website, but I think I can live with that.

[Carlos E.R.]

Interesting. I find multi-page docs confusing, I prefer single page. I
can do searches on the whole thing.

No, sorry, I do not know a tool to do the splitting automatically. I
would simply use an html editor and save chunks manually. LO could
possibly do it. Otherwise, I would try Composer in Seamonkey.

--
Cheers, Carlos.

[Helmut Richter]

On Tue, 1 Aug 2023, Carlos E.R. wrote:

> No, sorry, I do not know a tool to do the splitting automatically. I would
> simply use an html editor and save chunks manually. LO could possibly do it.
> Otherwise, I would try Composer in Seamonkey.

Web pages are made by lots of tools which yield readable or unreadable
HTML code. Things that make code unreadable are

– CSS for distributing text over the screen (this has decreased a
little since Web designers reckon with smartphones that have no
screen in the classic sense, i.e. at least 25 cm wide). CSS that works
locally (fonts, sizes, ...) is less of a problem, you only have to make
sure to repeat the declarations in each portion,

– a converter from something else, e.g. Word or PDF, to HTML, especially if
it is expected to convert also vice versa,

– many tools working simultaneously (a CMS, a WYSIWYG editor, a
converter of text formats, macro techniques like SSI, and – most
prominently – server side scripting like PHP and others) not
knowing which other tools are underway as well.

If you look into the HTML code without understanding anything you
cannot expect any tool to understand more. In simple cases, a tool
like HTML Tidy (https://www.w3.org/People/Raggett/tidy/) might improve
the situation but a tool working on everything is hardly conceivable.

Consequently, the proposed tool HTMLdoc excludes virtually everything:

While it currently does not support many things in “the modern web”
such as Cascading Style Sheets (CSS), forms, full Unicode, and Emoji
characters, ...

--
Helmut Richter

[Computer Nerd Kev]

In comp.infosystems.www.misc Helmut Richter <hr.u...@email.de> wrote:
> Consequently, the proposed tool HTMLdoc excludes virtually everything:
>
> While it currently does not support many things in "the modern web"
> such as Cascading Style Sheets (CSS), forms, full Unicode, and Emoji
> characters, ...

Well no, it doesn't exclude most software documentation because for
whatever reason the HTML in much of that has remained relatively
sane. Some CSS is creeping in, but a tool that ignores it still
produces clear text with some formatting. So far I've tested it
with two documentation pages published in 2023 and it's understood
the HTML fine (except for that out-of-page link problem in the RPi
doc). Plus often I'm looking at docs published 10-20 years ago
anyway.

[Computer Nerd Kev]

In comp.infosystems.www.misc Carlos E.R. <robin_...@es.invalid> wrote:
> Interesting. I find multi-page docs confusing, I prefer single page. I
> can do searches on the whole thing.

Having both is really the ideal for me. But I read the multi-page
version and resort to the single-page one if I'm reduced to text
searching the whole document. On the other hand often it's useful
to do text searches limited to paticular sections/pages too.

If I set out to read a manual from start to finish then the
single-page version is good. But for reference use it's too easy
to lose my place in them. I usually open different parts in
multiple tabs. Five minutes after I've learnt something important
from a manual I've usually forgotten it again, but I remember the
tab or what the layout of the page looked like, so I can get back
to it quickly. That doesn't work so well on a massive single
webpage.

[Helmut Richter]

Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
It is known to which subpage each link belongs, at least if you go over
the text in two passes. This is a procedure which I apply to all my web
pages, which are written as one document, and split into pieces later.

Another automatic change is to close and reopen each open tag:

<tag1>
a...
<tag2>
b...
---------------------------- cut here --------------
c...
</tag2>
d...
</tag1>

becomes

<tag1>
a...
<tag2>
b...
</tag2>
</tag1>
--------------------------was cut here ---------------
<tag1>
<tag2>
c...
</tag2>
d...
</tag1>

It is not obvious which attributes should be repeated in the second copy.
Mostly it is safer to repeat them, at least "class" and "style" attributes
and "id" attributes if they are referred to in style sheets.

--
Helmut Richter

[Helmut Richter]

On Wed, 2 Aug 2023, Helmut Richter wrote:

> Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
> It is known to which subpage each link belongs, at least if you go over
> the text in two passes. This is a procedure which I apply to all my web
> pages, which are written as one document, and split into pieces later.

It might be interesting to see an example of the TOC (table of contents)
of such a split article (https://hhr-m.de/sw-fibel/contents.html). It
contains all anchors in the whole article, which are possible but not
necessarily used (except from the TOC, of course) link targets. The link
structure might be still better visible if you look into the source code
of that web page which is fairly readable.

Sorry, the article itself is in German, a tutorial for learning Swahili.

--
Helmut Richter

[Computer Nerd Kev]

In comp.os.linux.misc Helmut Richter <hr.u...@email.de> wrote:
> On Wed, 2 Aug 2023, Helmut Richter wrote:
>> Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
>> It is known to which subpage each link belongs, at least if you go over
>> the text in two passes. This is a procedure which I apply to all my web
>> pages, which are written as one document, and split into pieces later.
>
> It might be interesting to see an example of the TOC (table of contents)
> of such a split article (https://hhr-m.de/sw-fibel/contents.html). It
> contains all anchors in the whole article, which are possible but not
> necessarily used (except from the TOC, of course) link targets. The link
> structure might be still better visible if you look into the source code
> of that web page which is fairly readable.

I think you misunderstood the problem. Perhaps I should have
explained that I would prefer it to rewrite relative links to other
webpages as absolute links.

As it is, a link like this:
<a href="/documentation/computers/processors.html#bcm2835">BCM2835</a>

From here:
https://www.raspberrypi.com/documentation/computers/configuration.html

Doesn't work when conveted unless the processors.html page is also
saved locally. Seeing as the program saw the source URL, I would
have liked it to be smart enough to turn such relative links into
absolute links when the link distination is another webpage.

This has fixed many of those relative links which had a directory
path:
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done

Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.

Such as this:
<a href="config_txt.html#video-options">

It's not a major complaint.

[23k.304]

Ummm ... are you trying to do this STATICALLY, on
pre-existing HTML, or DYNAMICALLY, as users actually
access the pages ???

For the first case, a little Python will do wonders.
Identify tags, where you need to insert the absolute
parts of the paths, do it. Python is great with text
strings.

The second case ... not as easy. There you'd be best
off using PHP. Yea, JavaScript will do it too, but
it's always something<dot>something<dot>something<dot>
something<dot>something<dot> ... functional but it
gets UGLY/OPAQUE real quick :-)

[Computer Nerd Kev]

In comp.os.linux.misc 23k.304 <23k...@bfxw9.net> wrote:
> On 8/4/23 8:07 AM, Computer Nerd Kev wrote:
>> This has fixed many of those relative links which had a directory
>> path:
>> for page in *.html; do sed -i \
>> 's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
>>
>> Pre-processing the page to rewrite relative links to other pages in
>> the same directory when the path isn't in the href, before running
>> HTMLDOC, would fix the rest.
>>
>> Such as this:
>> <a href="config_txt.html#video-options">
>>
>> It's not a major complaint.
>
> Ummm ... are you trying to do this STATICALLY, on
> pre-existing HTML, or DYNAMICALLY, as users actually
> access the pages ???

Statically, as stated in the first post.

Also no users besides me, browsing locally with file:// URLs.

> For the first case, a little Python will do wonders.
> Identify tags, where you need to insert the absolute
> parts of the paths, do it. Python is great with text
> strings.

Well Sed will do that too with a smarter regex (using selection
brackets), and I don't like Python for reasons we've already
argued about. As it happens there are so few links without an
absolute path in that doc (I thought there were none until browsing
around after that first try) that I figured it wasn't worth
bothering with. Other docs may sufficiently motivate me eventually.

[Helmut Richter]

On Sat, 5 Aug 2023, Computer Nerd Kev wrote:

> In comp.os.linux.misc 23k.304 <23k...@bfxw9.net> wrote:
> > On 8/4/23 8:07 AM, Computer Nerd Kev wrote:
> >> This has fixed many of those relative links which had a directory
> >> path:
> >> for page in *.html; do sed -i \
> >> 's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
> >>
> >> Pre-processing the page to rewrite relative links to other pages in
> >> the same directory when the path isn't in the href, before running
> >> HTMLDOC, would fix the rest.
> >>
> >> Such as this:
> >> <a href="config_txt.html#video-options">
> >>
> >> It's not a major complaint.
> >
> > Ummm ... are you trying to do this STATICALLY, on
> > pre-existing HTML, or DYNAMICALLY, as users actually
> > access the pages ???
>
> Statically, as stated in the first post.

Yes, there is nothing dynamic in it.

I had still another idea: leave the links as is, and map them to the
correct URLs by a rewrite in the server. The table of necessary rewrite
statements is static but can be modified if necessary. I discarded that
because it is too complex for too little benefit, if any.

> Also no users besides me, browsing locally with file:// URLs.

Of course, the rewrite is not an option in this case as there is no
server.

--
Helmut Richter

[23k.304]

It's also more "hidden" ... which can getcha when you,
or somebody, has to debug/rewrite in a couple of years.

[23k.304]

I absolutely hate regex. To me a few clear lines of Python
are much more agreeable and obvious to all.

But, for static, existing, pages either WILL get you there
without much drama.

[Theo]

In comp.infosystems.www.misc Computer Nerd Kev <n...@telling.you.invalid> wrote:
> A prime target would be the Raspberry Pi configuration
> documentation, which has convinced me of the merit of multi-page
> docs by how confusing it has become for me since they switched to a
> single-page layout:
> https://www.raspberrypi.com/documentation/computers/configuration.html

It doesn't answer the general question, but just to note that documentation
is generated from Asciidoc, and the source files can be found in their repo:
https://github.com/raspberrypi/documentation

Presumably there's a way to build Asciidoc to generate multi-page HTML, but
if not you could just read the .adoc files - github makes a fair stab at
rendering them.

Theo

[Computer Nerd Kev]

In comp.os.linux.misc Theo <theom...@chiark.greenend.org.uk> wrote:
> In comp.infosystems.www.misc Computer Nerd Kev <n...@telling.you.invalid> wrote:
>> A prime target would be the Raspberry Pi configuration
>> documentation, which has convinced me of the merit of multi-page
>> docs by how confusing it has become for me since they switched to a
>> single-page layout:
>> https://www.raspberrypi.com/documentation/computers/configuration.html
>
> It doesn't answer the general question, but just to note that documentation
> is generated from Asciidoc, and the source files can be found in their repo:
> https://github.com/raspberrypi/documentation

That's a good point. But I'm doubtful that Asciidoc converters will
easily give me the "previous contents next" links at the top and
bottom of pages, like published page-split HTML docs usually have.
The suggested HTMLDOC program generates those navigation links, so
I'll stick with it even for the RPi docs.

> Presumably there's a way to build Asciidoc to generate multi-page HTML, but
> if not you could just read the .adoc files - github makes a fair stab at
> rendering them.

GitHub unfortunately seems to have become a Javascript-only
solution now. Files and directories in repos can no longer be
viewed without Javascript (except for the top directory and the
readme, on the first page). Very disappointing.