Jobs Failing with git.exc.GitCommandError: Cmd('git') failed due to: exit code(-9)

[Wajid Baig]

Hi AWX team, 

We have k8s deployment of AWX on EKS cluster. Issue I'm facing is Jobs fail intermittently with below error. Next second I run same job it runs fine. I tried google around but could not find any solution. 

Is this a known issue? What is fix/workaround of this issue?

Stdout of failed job: 

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 457, in run
    self.pre_run_hook(self.instance, private_data_dir)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 970, in pre_run_hook
    RunProjectUpdate.make_local_copy(job.project, private_data_dir, scm_revision=job_revision)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 1374, in make_local_copy
    git.Repo.clone_from(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/repo/base.py", line 1019, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/repo/base.py", line 960, in _clone
    finalize_process(proc, stderr=stderr)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/util.py", line 328, in finalize_process
    proc.wait(**kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/cmd.py", line 408, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(-9)
  cmdline: git clone --branch=awx_internal/de9a0c84-3209-4c3b-8222-182440e579d4 --depth=1 --single-branch -v file:///var/lib/awx/projects/_493__kc1_azure_automations /tmp/awx_34605_50jrcqp5/project
  stderr: 'Cloning into '/tmp/awx_34605_50jrcqp5/project'...
Updating files:   4% (173/3704)
Updating files:   5% (186/3704)
Updating files:   6% (223/3704)
Updating files:   7% (260/3704)
Updating files:   8% (297/3704)
Updating files:   9% (334/3704)
Updating files:  10% (371/3704)
Updating files:  11% (408/3704)
Updating files:  12% (445/3704)
Updating files:  13% (482/3704)
Updating files:  14% (519/3704)
Updating files:  15% (556/3704)
Updating files:  16% (593/3704)
Updating files:  17% (630/3704)
Updating files:  18% (667/3704)
Updating files:  19% (704/3704)
Updating files:  20% (741/3704)
Updating files:  21% (778/3704)
Updating files:  22% (815/3704)
Updating files:  23% (852/3704)
Updating files:  24% (889/3704)
Updating files:  25% (926/3704)
Updating files:  26% (964/3704)
Updating files:  27% (1001/3704)
Updating files:  28% (1038/3704)
Updating files:  29% (1075/3704)
Updating files:  30% (1112/3704)
Updating files:  31% (1149/3704)
Updating files:  32% (1186/3704)
Updating files:  33% (1223/3704)
Updating files:  34% (1260/3704)
Updating files:  35% (1297/3704)
Updating files:  36% (1334/3704)
Updating files:  37% (1371/3704)
Updating files:  38% (1408/3704)
Updating files:  39% (1445/3704)
Updating files:  40% (1482/3704)
Updating files:  41% (1519/3704)
Updating files:  42% (1556/3704)
Updating files:  43% (1593/3704)
Updating files:  44% (1630/3704)
Updating files:  45% (1667/3704)
Updating files:  46% (1704/3704)
Updating files:  47% (1741/3704)
Updating files:  48% (1778/3704)
Updating files:  48% (1796/3704)
Updating files:  49% (1815/3704)

' 

[AWX Project]

Could be a bug in the gitpython module, lots of hits on google around this error e.g https://github.com/gitpython-developers/GitPython/discussions/1432

[Wajid Baig]

Is there any workaround/fix for this? 

Thanks & Regards,

Ahmed

--
You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/raYIV0G3Yas/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/b7a32ff4-8f44-4730-82ac-90328c2c1493n%40googlegroups.com.

--

Sincerely,
Baig
9160800772

[Wajid Baig]

Hi,

   Is there any workaround for this issue?

Thanks in advance,

Ahmed

[AWX Project]

Hi,

Wondering if this error is specific to the EKS cluster deployment, have you tried testing AWX on something like Kind, k3s, or minikube to see if the problem exists there? might be a useful exercise to narrow down the problem. Let us know what you find

AWX Team

[AWX Project]

Also, which version of AWX are you on? we addressed an issue around this in 21.5.0 https://github.com/ansible/awx/releases/tag/21.5.0

particularly this PR https://github.com/ansible/awx/pull/12356