GitLab continuous integration/deployment (CD/CI) and sphinx-doc's change detection
Denis Bitouzé
That works well, except that even if only a single source file is changed, the HTML pages of all the source files are regenerated and, since they are more than 1200 source files, that takes too much time (more than 15 minutes).
- with mainly default Sphinx settings,
- with a minimal conf.py config file,
- but with many (100) test files in order to highlight the issue I'm facing.
I then got help outside this forum from a guy (but who has no clue about sphinx-doc). Here is what he said:
[Y]ou are shooting yourself in your foot because you use make. make is utterly ineffective in CI pipelines because at the beginning of each pipeline the repo is cloned afresh, meaning the file modification dates of the source files are usually newer than the cached output files even if nothing did change. Because make only considers file modification dates/timestamps, make is definitely not helpful for the first invocation. The second invocation obviously behaves correctly because the first make invocation rebuilds all outputs.
So you can reduce this minimal example by removing make and simply calling sphinx-build. And the pipelines for this new repository also show that a cache is pulled at the beginning and uploaded at the end of the pipeline. So GitLab’s caching is working. It’s now a question of what sphinx-build needs to determine whether to rebuild. If it works like make, you are out of luck because of file timestamps. If it works with another mechanism, you should look up where that is stored and check that all information needed for the rebuild are cached (maybe more is needed than just the doctrees directory).
I told him that:
- I used to try with sphinx-build instead of make but I didn’t work either,
- I’m afraid sphinx-build doesn’t work with another mechanism and, regarding the cache, only the doctrees directory is involved.
He answers then:
So you need to use one of the solutions modifying the file timestamps to match the git commits to have an effective solution here. There are a variety of tools out there that do this. Just search the internet for git checkouts with mtime preserved.
Wols Lists
> [Y]ou are shooting yourself in your foot because you use make. make is
> utterly ineffective in CI pipelines because at the beginning of each
> pipeline the repo is cloned afresh, meaning the file modification dates
> of the source files are usually newer than the cached output files even
> if nothing did change.
work, as it's supposed to copy everything.
The other approach to try is "cp -lR", because it copies the directory
structure, but links to and does not change any files. That also might
work, and actually will be a lot faster than cp -a.
That's assuming you can control how the repo is copied, of course. I
suspect they often do a "git clone" which will lose all the target files
on the spot ...
Cheers,
Wol
Denis Bitouzé
> I’m building a (French LaTeX) FAQ
> with Sphinx-doc <https://www.sphinx-doc.org/>. In order to let other people
> the changed source files thanks to:
>
> - CD/CI
> - GitLab pages
>
> on a gilab.com instance.
>
> That works well, except that even if only a single source file is changed,
> the HTML pages of all the source files are regenerated and, since they are
> more than 1200 source files, that takes too much time (more than 15
> minutes).
“Tutorial: Build your first project”:
┌────
│ https://www.sphinx-doc.org/en/master/tutorial/index.html
└────
including “Appendix: Deploying a Sphinx project online”, following the
“GitHub Pages” route in order to see whether my problem is
GitLab-specific or not:
┌────
│ https://github.com/dbitouze/lumache
└────
As you can see here:
┌────
│ https://github.com/dbitouze/lumache/commit/c4bcbd9c5fc239edb603e80a31b15e22ff574768
└────
only the `index.rst` file was changed but the build performed by the CI/CD:
┌────
│ https://github.com/dbitouze/lumache/actions/runs/5691725103/job/15427441621#step:4:25
└────
writes all the other source files:
- `api`
- `generated/lumache`
- `index`
- `usage`
(The deploy steps fails but that's not the point.)
Hence my problem is not a GitLab-specific one and seems to affect all
CI/CD ways of deploying a Sphinx project online :(
--
Denis
Denis Bitouzé
> On 24/07/2023 22:19, Denis Bitouzé wrote:
>> [Y]ou are shooting yourself in your foot because you use make. make is utterly
>> ineffective in CI pipelines because at the beginning of each pipeline the repo
>> is cloned afresh, meaning the file modification dates of the source files are
>> usually newer than the cached output files even if nothing did change.
>
> How does it clone the repo? If you can clone using "cp -a", that MIGHT work, as
> it's supposed to copy everything.
guess) “Getting source from Git repository”. See e.g.:
┌────
│ https://gitlab.com/gutenberg1/minimal-sphinx-minimal/-/jobs/4747563014#L14
└────
> The other approach to try is "cp -lR", because it copies the directory
> structure, but links to and does not change any files. That also might
> work, and actually will be a lot faster than cp -a.
>
> That's assuming you can control how the repo is copied, of course. I suspect
> they often do a "git clone" which will lose all the target files on the spot ...
Many thanks for your answer!
I hope I could find a way to fix this issue!
Cheers,
--
Denis
Tuncay Güzel
Wols Lists
>> That's assuming you can control how the repo is copied, of course. I suspect
>> they often do a "git clone" which will lose all the target files on the spot ...
>
> Many thanks for your answer!
>
> I hope I could find a way to fix this issue!
git clone.
Isn't the idea of CI/CD that you start afresh for every update? In which
case you want a bare repository, which means you have no target files,
which means you need a full rebuild as your first step.
Sorry.
Cheers,
Wol