Cloning submodules on github actions very slow

[विश्वासो वासुकिजः (Vishvas Vasuki)]

( Using github actions to build a sanskrit related site. )

Any ideas regarding https://stackoverflow.com/questions/75318595/github-actions-cloning-submodules-very-slow ?

--

--
Vishvas /विश्वासः

[Shreevatsa R]

AFAICT the issue has nothing to do with submodules or GitHub actions; the repository in question is just massive.

Just ran git clone g...@github.com:vishvAsa/purANam.git and it (still running) seems to be about 8 GiB in size!

See this Gitlab help page for some ideas on reducing repository size (basically rewrite history). (If you want to understand what's going on, this blog post I wrote a few years ago (needs to be updated for Python 3) may be helpful though it's been a while and I haven't thought about how relevant it is in this case...)

--
You received this message because you are subscribed to the Google Groups "sanskrit-programmers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sanskrit-program...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sanskrit-programmers/CAFY6qgHNvXQq1Fjg4K3E5aPwr5JMxegjSJymgcGus-iw8L4riA%40mail.gmail.com.

[विश्वासो वासुकिजः (Vishvas Vasuki)]

On Thu, 2 Feb 2023 at 11:39, Shreevatsa R <shree...@gmail.com> wrote:

AFAICT the issue has nothing to do with submodules or GitHub actions; the repository in question is just massive.

I thought so too initially, but it doesn't seem so. Observe the timestamps on the following lines

2023-02-01T17:34:32.3793039Z Cloning into '/home/runner/work/purANam/purANam/content'...
2023-02-01T17:47:40.0953104Z Cloning into '/home/runner/work/purANam/purANam/static'...
2023-02-01T18:00:36.8478598Z Cloning into '/home/runner/work/purANam/purANam/themes/sanskrit-documentation-theme-hugo'...

Now, you see it took 13 minutes for the static_files branch. And, that branch has - just a few files, and just two commits - https://github.com/vishvAsa/purANam/tree/static_files .

To add to this - This is in contrast with 27s for purANam_vaiShNavam, where the number of files is quite large.

 

Just ran git clone g...@github.com:vishvAsa/purANam.git and it (still running) seems to be about 8 GiB in size!

See this Gitlab help page for some ideas on reducing repository size (basically rewrite history). (If you want to understand what's going on, this blog post I wrote a few years ago (needs to be updated for Python 3) may be helpful though it's been a while and I haven't thought about how relevant it is in this case...)

I already tried squashing commits ( https://github.com/vishvAsa/purANam/issues/5#issuecomment-1412653883 ) and force pushing - did not help.

(Something like git rebase -i 4d7ca0eaf2583d8454b36671fed09b7ab2501680  followed by squashing all but a few commits, followed by force-push)

 

On Wed, 1 Feb 2023 at 20:13, विश्वासो वासुकिजः (Vishvas Vasuki) <vishvas...@gmail.com> wrote:

( Using github actions to build a sanskrit related site. )

Any ideas regarding https://stackoverflow.com/questions/75318595/github-actions-cloning-submodules-very-slow ?

--

--
Vishvas /विश्वासः

--
You received this message because you are subscribed to the Google Groups "sanskrit-programmers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sanskrit-program...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sanskrit-programmers/CAFY6qgHNvXQq1Fjg4K3E5aPwr5JMxegjSJymgcGus-iw8L4riA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "sanskrit-programmers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sanskrit-program...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sanskrit-programmers/CAKEM%3DPMnBYqW-0TkB_P6Tav5JonEkefs71fOXr9s7hVpJ0uK6A%40mail.gmail.com.

[विश्वासो वासुकिजः (Vishvas Vasuki)]

On Thu, 2 Feb 2023 at 13:39, विश्वासो वासुकिजः (Vishvas Vasuki) <vishvas...@gmail.com> wrote:

On Thu, 2 Feb 2023 at 11:39, Shreevatsa R <shree...@gmail.com> wrote:

AFAICT the issue has nothing to do with submodules or GitHub actions; the repository in question is just massive.

I thought so too initially, but it doesn't seem so. Observe the timestamps on the following lines

2023-02-01T17:34:32.3793039Z Cloning into '/home/runner/work/purANam/purANam/content'...
2023-02-01T17:47:40.0953104Z Cloning into '/home/runner/work/purANam/purANam/static'...
2023-02-01T18:00:36.8478598Z Cloning into '/home/runner/work/purANam/purANam/themes/sanskrit-documentation-theme-hugo'...

Now, you see it took 13 minutes for the static_files branch. And, that branch has - just a few files, and just two commits - https://github.com/vishvAsa/purANam/tree/static_files .

Oh - It seems like I had to squash ALL branches (and ensure that the deploy-branch history does not keep growing over time).

I could not figure out a way to make sure that submodule initialization/ updation works without peeking into unrelated branches.

[Shreevatsa R]

Yes, the size of the repo has gone from ~8 GB to ~120MB now, so it's faster now.

The size of the repository depends on all the objects in it, which includes the history of all branches. It need not have any direct relationship with the number of files.

To view this discussion on the web visit https://groups.google.com/d/msgid/sanskrit-programmers/CAFY6qgG%2BYTX63ND68zdF98SqTK87%2B4kKu3aQUVCX-dsAVwqGAg%40mail.gmail.com.

[विश्वासो वासुकिजः (Vishvas Vasuki)]

 + g...@vger.kernel.org

(A bug report - kindly reply-all/ cc me if I am mistaken, since I am not subscribed to g...@vger.kernel.org .)


Why does submodule cloning pull in unrelated branches? Trying the below did not work (the second command fails with " Unable to find refs/remotes/origin/content revision in submodule path 'content'"):

        git submodule update  --single-branch --init --recursive
        git submodule update --remote --merge --recursive

For background info, please see https://github.com/vishvAsa/purANam/issues/5 or the thread below.

To view this discussion on the web visit https://groups.google.com/d/msgid/sanskrit-programmers/CAKEM%3DPOA6L4pZCeWgYiqgamaAt4OG6Fg50QSRnXBbFEekmOXSQ%40mail.gmail.com.

[विश्वासो वासुकिजः (Vishvas Vasuki)]

(A bug report - resending as plain text as the html one bounces.