How can I create html only from the modified .rst files after changing the project directory?

[ycproject]

How can I create html only from modified .rst files?

I copied the whole project to another disk and only modified a few .rst files, but after running make html, all the .rst files are regenerated into html files. (There are hundreds of .rst files in the whole project)

I want to generate html only for those .rst files that have been modified, and for the unmodified .rst files, don't generate new html files.

Translated with www.DeepL.com/Translator (free version)

[Wol]

On 19/04/2023 23:18, ycproject wrote:
> I want to generate html only for those .rst files that have been
> modified, and for the unmodified .rst files, don't generate new html files.

Not knowing my way around make that well but ... did you just do a cp,
or a "cp -a".

The fact you copied the files has probably changed all the timestamps,
and that will have upset the "files changed" calculation ...

Cheers,
Wol

[ycproject]

Do you mean that there is no way to deal with this problem unless I can change the "file changed" calculation in sphinx?

[Wol]

On 20/04/2023 23:26, ycproject wrote:
>
> Do you mean that there is no way to deal with this problem unless I can
> change the "file changed" calculation in sphinx?

No, not at all.

The problem is you have COPIED the files, and changed the DATE MODIFIED
flag. That has confused make. This problem is NOTHING to do with Sphinx,
and EVERYTHING to do with the copy command (and with make).

Forget Sphinx, you need to learn about timestamps and make. There's no
such thing as "the file changed calculation in Sphinx".

Basically, you need to either (1) stop the copy command changing the
"date modified" time by using something like "cp -a", or (2) you need to
fudge the copy by making sure it copies the build directory before the
source directory, or (3) you fool make by running something like "touch"
over the build files so make things they're later than the source.

IT'S NOTHING TO DO WITH SPHINX.

Cheers,
Wol

> 在2023年4月21日星期五 UTC+8 02:22:38<antl...@youngman.org.uk> 写道：
>
> On 19/04/2023 23:18, ycproject wrote:
> > I want to generate html only for those .rst files that have been
> > modified, and for the unmodified .rst files, don't generate new
> html files.
>
> Not knowing my way around make that well but ... did you just do a cp,
> or a "cp -a".
>
> The fact you copied the files has probably changed all the timestamps,
> and that will have upset the "files changed" calculation ...
>
> Cheers,
> Wol
>

> --
> You received this message because you are subscribed to the Google
> Groups "sphinx-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sphinx-users...@googlegroups.com
> <mailto:sphinx-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sphinx-users/13dee6ce-6092-4d00-8e98-dbde980b8d35n%40googlegroups.com <https://groups.google.com/d/msgid/sphinx-users/13dee6ce-6092-4d00-8e98-dbde980b8d35n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[ycproject]

Thanks for your reply.
I just did a test：

1. I used the commands cp -a and cp -r respectively, and the result was that when I ran make html, both regenerated all the html.

2. I created a new empty directory (newproject) and copied the _build directory from the original project directory to the new directory (newproject) first; after three minutes, I copied all the directories except the _build directory from the original project directory to the new directory (newproject). After that, I ran make html in the new directory (newproject) and it still regenerated all the html.

3. I am not quite sure what kind of touchwg command should be used to do this.

[Wols Lists]

On 21/04/2023 00:39, ycproject wrote:
> Thanks for your reply.
> I just did a test：
>
> 1. I used the commands cp -a and cp -r respectively, and the result was
> that when I ran make html, both regenerated all the html.
>
> 2. I created a new empty directory (newproject) and copied the _build
> directory from the original project directory to the new directory
> (newproject) first; after three minutes, I copied all the directories
> except the _build directory from the original project directory to the
> new directory (newproject). After that, I ran make html in the new
> directory (newproject) and it still regenerated all the html.
>
> 3. I am not quite sure what kind of touchwg command should be used to do
> this.
>

But if you run make a second time? It shouldn't then remake everything
again.

If that's the case, you'll just have to accept that copying the Sphinx
directory will trigger a complete rebuild the first time you run make.

It's a pain, but not the end of the world, until you find out why make
thinks the source files are newer than the target.

But at the end of the day, it's nothing to do with Sphinx.

Cheers,
Wol

[Kevin Cole]

Did you read my message explaining how to take care of the problem?

[Kevin Cole]

Sent April 19...

---------- Forwarded message ---------
From: Kevin Cole <dc....@gmail.com>
Date: Wed, Apr 19, 2023 at 7:22 PM
Subject: Re: [sphinx-users] How can I create html only from the modified .rst files after changing the project directory?
To: ycproject <class...@gmail.com>

I am not 100% confident, but I suspect the problem is that you copied without keeping the original creation dates on the files.

Sphinx is comparing the new dates of the RST files to the new dates of the HTML files: If the date of an RST file is newer than the date a generated HTML, Sphinx will assume that the HTML file older than the RST and needs to be regenerated.

If you are on a Linux system, you could issue a command from the root directory of your project like:

find . -name "*.html" -exec touch {} +

This would search all subdirectories for HTML files and "touch" them without changing anything, which forces the dates to be set to "now".

Then:

touch filename.rst

for each of the "few modified files" so that their date is newer than the HTML files. Now, MOST of the RST files will be "older" than the HTML files and the system will see no need to regenerate them.

[Wol]

This is basically what I've been saying, and ycproject appears to have
tried that. It hasn't worked.

It's a bummer, but just accept that if you copy the repository, make
will get confused.

Cheers,
Wol

On 21/04/2023 15:15, Kevin Cole wrote:
> Sent April 19...
>
> ---------- Forwarded message ---------
> From: *Kevin Cole* <dc....@gmail.com <mailto:dc....@gmail.com>>
> Date: Wed, Apr 19, 2023 at 7:22 PM
> Subject: Re: [sphinx-users] How can I create html only from the modified
> .rst files after changing the project directory?
> To: ycproject <class...@gmail.com <mailto:class...@gmail.com>>
>
> On Wed, Apr 19, 2023 at 6:18 PM ycproject <class...@gmail.com
> <mailto:class...@gmail.com>> wrote:
>
> How can I create html only from modified .rst files?
>
> I copied the whole project to another disk and only modified a few
> .rst files, but after running make html, all the .rst files are
> regenerated into html files. (There are hundreds of .rst files in
> the whole project)
>
> I want to generate html only for those .rst files that have been
> modified, and for the unmodified .rst files, don't generate new html
> files.
>
> Translated with www.DeepL.com/Translator

> <http://www.DeepL.com/Translator> (free version)

>
>
> I am not 100% confident, but I suspect the problem is that you copied
> without keeping the original creation dates on the files.
>
> Sphinx is comparing the new dates of the RST files to the new dates of
> the HTML files: If the date of an RST file is newer than the date a
> generated HTML, Sphinx will assume that the HTML file older than the RST
> and needs to be regenerated.
>
> If you are on a Linux system, you could issue a command from the root
> directory of your project like:
>
> find . -name "*.html" -exec touch {} +
>
> This would search all subdirectories for HTML files and "touch" them
> without changing anything, which forces the dates to be set to "now".
>
> Then:
>
> touch filename.rst
>
> for each of the "few modified files" so that their date is newer than
> the HTML files. Now, MOST of the RST files will be "older" than the HTML
> files and the system will see no need to regenerate them.
>

> --
> You received this message because you are subscribed to the Google
> Groups "sphinx-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sphinx-users...@googlegroups.com
> <mailto:sphinx-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sphinx-users/CA%2BpvXRcTvtB5003%2BSDLWGX4qeRdZQ-40RDxDNrF23w7D8RFMZQ%40mail.gmail.com <https://groups.google.com/d/msgid/sphinx-users/CA%2BpvXRcTvtB5003%2BSDLWGX4qeRdZQ-40RDxDNrF23w7D8RFMZQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[ycproject]

1. Execute make html in the project directory oldproject (this directory contains the cony.py file and index.rst file, as well as the Makefile file), making sure that no new html files are generated.
2. Copy the entire project directory oldproject to another disk and rename it to test.
3. Check the time attribute of the html file in the test/_build/html directory: stat index.html, and the output will be the time when the project directory was copied.
4. Three minutes later, in the test directory (this directory contains the cony.py file and the index.rst file, as well as the Makefile file), execute find . -name "*.html" -exec touch {} + command.
5. Check the time attribute of the html file in the test/_build/html directory: stat index.html, the output is the current time, which is about 4 minutes later than the project directory copy time.
6. Execute make html in the test/ directory.
7. Result: All html files are regenerated.

In the above process, the touch index.rst file was not executed in the test/ directory.

After that, I deleted the whole test directory, then repeated the above 7 steps, but added one step before step 6: touch index.rst. Finally when make html, it still generated all the html files.

After that, I deleted the whole new project directory and repeated the 7 steps above again, but this time instead of using the touch command before the step6, I modified the content of the index.rst file in the test/ directory: added some text. Finally when make html, it still generated all the html files.

So, modifying the time property of the html file (atime mtime ctime) seems to be invalid for now.

==========================
ps:
I am not sure if it is related to the following environmental situation:

OS: debian11
sphinx-bulider --version : 5.3.0.
The installation method for sphinx is: sudo apt install python3-sphinx python3-sphinx-rtd-theme.

[Kevin Cole]

On Fri, Apr 21, 2023 at 8:09 PM ycproject <class...@gmail.com> wrote:

> 1. Execute make html in the project directory oldproject (this directory contains the cony.py file and index.rst file, as well as the Makefile file), making sure that no new html files are generated.
> 2. Copy the entire project directory oldproject to another disk and rename it to test.
> 3. Check the time attribute of the html file in the test/_build/html directory: stat index.html, and the output will be the time when the project directory was copied.
> 4. Three minutes later, in the test directory (this directory contains the cony.py file and the index.rst file, as well as the Makefile file), execute find . -name "*.html" -exec touch {} + command.
> 5. Check the time attribute of the html file in the test/_build/html directory: stat index.html, the output is the current time, which is about 4 minutes later than the project directory copy time.
> 6. Execute make html in the test/ directory.
> 7. Result: All html files are regenerated.
>
> In the above process, the touch index.rst file was not executed in the test/ directory.
>
> After that, I deleted the whole test directory, then repeated the above 7 steps, but added one step before step 6: touch index.rst. Finally when make html, it still generated all the html files.
>
> After that, I deleted the whole new project directory and repeated the 7 steps above again, but this time instead of using the touch command before the step6, I modified the content of the index.rst file in the test/ directory: added some text. Finally when make html, it still generated all the html files.
>
> So, modifying the time property of the html file (atime mtime ctime) seems to be invalid for now.

Well, in my defense, I did say I wasn't certain. ;-)

I did see that Wol had suggested the same source for the trouble.

I cannot see how Sphinx and / or make would be able to determine which
files need regeneration without comparing dates... I don't think
there's any hidden cache or tmp directory...

[ycproject]

Thanks~

I added html_last_updated_fmt to the conf.py file, so the generated html files will show the latest update date, and the above issue causes the update date to be inaccurate.

I will comment out the parameter html_last_updated_fmt for now and not use it.

[jfbu]

I have not read all the details, but the pickled environment contains the full path to the project.

And at https://www.sphinx-doc.org/en/master/_modules/sphinx/environment.html you can find the lines

def setup(self, app: Sphinx) -> None:
"""Set up BuildEnvironment object."""
if self.version and self.version != app.registry.get_envversion(app):
raise BuildEnvironmentError(__('build environment version not current'))
if self.srcdir and self.srcdir != app.srcdir:
raise BuildEnvironmentError(__('source directory has changed'))

This BuildEnvironment.setup() is called from sphinx.application:
(see https://www.sphinx-doc.org/en/master/_modules/sphinx/application.html)

def _load_existing_env(self, filename: str) -> BuildEnvironment:
try:
with progress_message(__('loading pickled environment')):
with open(filename, 'rb') as f:
env = pickle.load(f)
env.setup(self)
self._fresh_env_used = False
except Exception as err:
logger.info(__('failed: %s'), err)
env = self._create_fresh_env()
return env

The environment "env" loaded from the pickled file carries the absolute path
to the former location. On the other hand the app.srcdir above contains the
absolute path to the new location.

Hence after doing "cp -a originalproject copiedproject" one gets
on next build attempt in new directory:

loading pickled environment... failed
failed: source directory has changed

Also other files under _build/doctrees such as index.doctree contain
absolute paths

[Wols Lists]

In other words, if I understand you correctly, you are saying that
actually you NEED to do a complete rebuild on copy (or more especially,
move)?

Because all the absolute paths in the output are no longer valid?

In which case, ycproject should not be worried about this, make is
working as designed, and "fixing" his project will actually break it ...

Cheers,
Wol

On 22/04/2023 08:11, jfbu wrote:
> I have not read all the details, but the pickled environment contains
> the full path to the project.
>
> And at
> https://www.sphinx-doc.org/en/master/_modules/sphinx/environment.html
> you can find the lines
>
>     def setup(self, app: Sphinx) -> None:
>         """Set up BuildEnvironment object."""
>         if self.version and self.version !=
> app.registry.get_envversion(app):
>             raise BuildEnvironmentError(__('build environment version
> not current'))
>         if self.srcdir and self.srcdir != app.srcdir:
>             raise BuildEnvironmentError(__('source directory has
> changed'))
>
> This BuildEnvironment.setup() is called from sphinx.application:
> (see https://www.sphinx-doc.org/en/master/_modules/sphinx/application.html)
>
>     def _load_existing_env(self, filename: str) -> BuildEnvironment:

>         try:Facts

[jfbu]

> In other words, if I understand you correctly, you are saying that actually you NEED to do a complete rebuild on copy (or more especially, move)?
>

Yes, after copying or moving to a new location or renaming the project directory, the build process (if triggered anew) will start afresh and ignore the pickled data.

> Because all the absolute paths in the output are no longer valid?
>

I was referring to pickled build environment, not output. The HTML output has relative paths (afaik, and simply because otherwise it would be bad), and you can move it (else how would people deploy their docs...). But if you have moved the project source yes any rebuild attempt even without any change to rst files will ignore saved build environment.

I tried this with minimal project having index.rst and a toctree including 3 files and after a "cp -a" which keeps mtimes:

loading pickled environment... failed
failed: source directory has changed

building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 4 source files that are out of date
updating environment: [new config] 4 added, 0 changed, 0 removed

There is only one location in Sphinx code base where "source directory has changed" originates and it is the one I pointed to.

> In which case, ycproject should not be worried about this, make is working as designed, and "fixing" his project will actually break it ...

I did not read carefully the thread, but yes I tend to agree !

Best,

Jean-François
PS: copy paste from Firefox introduced a spurious "Facts" after "try:" in this code snippet
It is not (of course) in original on https://www.sphinx-doc.org/en/master/_modules/sphinx/application.html (maybe I should have linked to github repo)

[ycproject]

Since I need to go to different cities, I occasionally change computers, and when I do, I copy the project to my laptop; when I come back, I copy the project back to my desktop computer.

Now, I think one solution I can use is to store the project in a virtual machine and execute make html in the virtual machine to generate the final html.

When I change computers, instead of copying the project directory, I copy the virtual machine. In this case, no matter what disk or directory is copied to, it will not affect the time attribute of the rst/html file.

Now, I can use the "html_last_updated_fmt" option again.

[Wol]

On 23/04/2023 02:03, ycproject wrote:
> Since I need to go to different cities, I occasionally change computers,
> and when I do, I copy the project to my laptop; when I come back, I copy
> the project back to my desktop computer.
>
> Now, I think one solution I can use is to store the project in a virtual
> machine and execute make html in the virtual machine to generate the
> final html.
>
> When I change computers, instead of copying the project directory, I
> copy the virtual machine. In this case, no matter what disk or directory
> is copied to, it will not affect the time attribute of the rst/html file.
>
> Now, I can use the "html_last_updated_fmt" option again.
>

Or just make your desktop into a git repository, and clone it to your
laptop. When you go away, pull the changes on to your laptop, "make" for
anything that's changed in the interim, and you're good to go.

When you get home, you just push any changes back to the laptop.

A virtual machine feels like "a sledgehammer to crack a nut" here.

Cheers,
Wol

[ycproject]

Sounds like a very good solution, I'll test it out and see how it works.

[ycproject]

I've taken your advice, but instead of using git, I use rsync.