Get a list of added repos when updating the admin repo at end of POST_COMPILE trigger.

[y]

Hello all,

What is the best way to get a list of newly created repos at the end of the POST_COMPILE trigger when the admin repo is updated?

I need this in the end of the POST_COMPILE hook because I set some config values (origin url and some fetch refspecs) on the newly created repos.

When I check the date modified on the repo folders they are all updated so this seems to not be reliable. The goal is to be able to run a `git fetch --all` on all newly initialized repos automatically.

For the curious the relevant config bits are below. This setup allows me to mirror various git repos and maintain my own branches locally in the same repo. My branches are contained by specific ref_specs. I cannot overwrite upstream work. But I have full access to my refspecs.

The multi-valued fetch refspecs are from https://groups.google.com/g/gitolite/c/9G2oz9eeomE/m/bcHTkuSgAgAJ

```

macro _mirror_aur @mirror = mirror/aur/%1
repo mirror/aur/%1
   config remote.origin.url = https://aur.archlinux.org/%1.git
end

_mirror_aur ntopng

repo @mirror
  config remote.origin.mirror = true
  option multi-config.remote.origin.fetch.1 = "^refs/heads/my_ref_spec/*"
  option multi-config.remote.origin.fetch.2 = "^refs/heads/my_ref_spec_*"
  option multi-config.remote.origin.fetch.3 = "+refs/*:refs/*"
  R = @all
  RW+ refs/(tags|heads)/my_ref_spec_.* = my_user

```

Thanks,

Yevgeniy

[Sitaram Chamarty]

On Mon, Feb 07, 2022 at 03:41:53AM -0800, y wrote:
> Hello all,
>
> What is the best way to get a list of newly created repos at the end of the
> POST_COMPILE trigger when the admin repo is updated?
>
> I need this in the end of the POST_COMPILE hook because I set some config
> values (origin url and some fetch refspecs) on the newly created repos.
> When I check the date modified on the repo folders they are all updated so
> this seems to not be reliable. The goal is to be able to run a `git fetch
> --all` on all newly initialized repos automatically.

Do you actually want that list? Seems to me adding a
POST_CREATE trigger containing the appropriate commands would
just work. You may have to `cd $GL_REPO_BASE/$2` first though;
I'm not sure what PWD is when that trigger is invoked.

regards
sitaram

[y]

I tried using POST_CREATE initially but it ran too early in the process. The config files were not updated with remotes yet. That happens late in POST_COMPILE for me. I am now marking the new repo with a file in a 'mirror-mark' script during POST_CREATE. Then process the fetch in a separate POST_COMPILE 'mirror-fetch' script after searching all repos with marker files. (I know I can improve this a bit...)

I solved my immediate issue but noticed some unexpected behavior. Initially I used a single script and had it in both POST_CREATE and POST_COMPILE triggers. After your reply I noticed that the script only runs in POST_COMPILE and ignores POST_CREATE. The repo does get created on disk successfully. Is this working as intended?

Here are the two relevant vars for the 'input-er' test script.

```

$ gitolite query-rc POST_COMPILE
update-multi-git-configs
input-er    <- *** only this one runs ***
post-compile/ssh-authkeys --key-file-name
post-compile/ssh-authkeys-shell-users xxx
post-compile/update-git-configs
repo-specific-hooks
mirror-fetch
$ gitolite query-rc POST_CREATE  
update-multi-git-configs
mirror-mark
input-er <- *** this one does not get executed according to the logs and I do not see its side effects ***
post-compile/update-git-configs
repo-specific-hooks
```

I am using gitolite3 in fedora: `gitolite3.noarch                         1:3.6.12-1.fc33                 @fedora`

Thanks

Yevgeniy

[y]

I retested the script running only on the POST_COMPILE trigger and not POST_CREATE on a fresh gitolite install on a fresh fedora install. Here is the full config:

```

$ gitolite query-rc -d
$VAR1 = {
         'ACCESS_1' => [
                         'Writable::access_1'
                       ],
         'COMMAND' => [],
         'COMMANDS' => {
                         'desc' => 1,
                         'help' => 1,
                         'info' => 1,
                         'perms' => 1,
                         'writable' => 1
                       },
         'ENABLE' => [
                       'help',
                       'desc',
                       'info',
                       'perms',
                       'writable',
                       'ssh-authkeys',
                       'git-config',
                       'daemon',
                       'gitweb'
                     ],
         'GIT_CONFIG_KEYS' => '',
         'GL_ADMIN_BASE' => '/var/lib/gitolite3/.gitolite',
         'GL_BINDIR' => '/usr/share/gitolite3',
         'GL_LIBDIR' => '/usr/share/gitolite3/lib',
         'GL_LOGFILE' => '/var/lib/gitolite3/.gitolite/logs/gitolite-2022-02.log',
         'GL_REPO_BASE' => '/var/lib/gitolite3/repositories',
         'GL_TID' => '1851',
         'LOCAL_CODE' => '/var/lib/gitolite3/local',
         'LOG_EXTRA' => 1,
         'LOG_TEMPLATE' => '/var/lib/gitolite3/.gitolite/logs/gitolite-%y-%m.log',
         'POST_COMPILE' => [
                             'input',
                             'post-compile/ssh-authkeys',
                             'post-compile/update-git-configs',
                             'post-compile/update-gitweb-access-list',
                             'post-compile/update-git-daemon-access-list'
                           ],
         'POST_CREATE' => [
                            'input',
                            'post-compile/update-git-configs',
                            'post-compile/update-gitweb-access-list',
                            'post-compile/update-git-daemon-access-list'
                          ],
         'ROLES' => {
                      'READERS' => 1,
                      'WRITERS' => 1
                    },
         'UMASK' => 63
       };
```

Here is the test script itself

```

$ ls -la local/triggers/input  
-rwxr-xr-x. 1 gitolite3 gitolite3 504 Feb  9 04:28 local/triggers/input

$ cat local/triggers/input
#!/bin/bash
out=/var/lib/gitolite3/input.dump
echo "###########################################################" >> $out 2>&1
date >> $out 2>&1
echo "##################### PRINT STDIN ###" >> $out 2>&1
timeout 1 cat /dev/stdin >> $out 2>&1
echo "##################### PRINT ENVIRONMENT ###" >> $out 2>&1
env >> $out 2>&1
echo "##################### PRINT CWD ###" >> $out 2>&1
pwd >> $out 2>&1
echo "##################### PRINT ARGS ###" >> $out 2>&1
echo "${@}" >> $out 2>&1
echo -e '\n\n' >> $out 2>&1

```

[Sitaram Chamarty]

Hi,

Apologies once again for the delay :(

On Tue, Feb 08, 2022 at 02:34:51PM -0800, y wrote:
> I tried using POST_CREATE initially but it ran too early in the process.
> The config files were not updated with remotes yet. That happens late in
> POST_COMPILE for me. I am now marking the new repo with a file in a
> 'mirror-mark' script during POST_CREATE. Then process the fetch in a
> separate POST_COMPILE 'mirror-fetch' script after searching all repos with
> marker files. (I know I can improve this a bit...)

Although I have not "officially" documented it, there *is* a
supported way in which to add your own POST_CREATE trigger and
make it run *after* the built-in trigger scripts.

If you don't mind, take a look at
https://groups.google.com/g/gitolite/c/2kZaqLohSz0/m/LsIo_W8B2I8J

I think the reason I did not document it is that it's very
rarely needed (you're only the second person to have come across
such a situation in all these years, and the last one was in
2013!)

But it *is* supported, so now I think I will document it as soon
as I find some time.

Meanwhile *please let me know if that works for you!*

> I solved my immediate issue but noticed some unexpected behavior. Initially
> I used a single script and had it in both POST_CREATE and POST_COMPILE
> triggers. After your reply I noticed that the script only runs in
> POST_COMPILE and ignores POST_CREATE. The repo does get created on disk
> successfully. Is this working as intended?

yes. That was an optimisation that went in at some point (see
https://github.com/sitaramc/gitolite/commit/cf423a6a74b398eaa1dd67966ab2e02ff422274a)

If the solution in the earlier part of this email does not work
for you, and you *need* your trigger script ('input') to run for
both, you'll just have to `ln input input2` and use `input2` in
one of them to defeat the optimisation.

(side note, "input" is not a good name, because gitolite itself
has "INPUT" triggers so there is the possibility of confusion!)

hope this helps!

sitaram