Patchbot/CI improvements

[Matthias Koeppe]

Patchbot experts:

1) Currently the patchbot does not run on certain tickets that make changes to build/pkgs, so we do not have automatic testing for these tickets. Is this something that can be changed? 

2) Can "make doc-pdf" please be added? 

I don't know if Volker's integration scripts look at the patchbot status at all, but nevertheless these two changes could improve our workflow.

[Vincent Delecroix]

Le 23/01/2022 à 18:35, Matthias Koeppe a écrit :
> Patchbot experts:
> 1) Currently the patchbot does not run on certain tickets that make changes
> to build/pkgs, so we do not have automatic testing for these tickets. Is
> this something that can be changed?

I would say that this is doable (since if I remember correctly, packages
can be downloaded from their upstream urls). Please open an issue on
sage-patchbot.

> 2) Can "make doc-pdf" please be added?

It is (sort of) already there

https://github.com/sagemath/sage-patchbot/blob/master/sage_patchbot/plugins.py#L179

That could be activated by default if the experimental status of
this plugin is removed. You can also open an issue.

[Matthias Koeppe]

Thanks, I've opened 

- https://github.com/sagemath/sage-patchbot/issues/148

- https://github.com/sagemath/sage-patchbot/issues/149

[Matthias Koeppe]

I have also prepared a PR for the "git releasemgr" scripts to make it easier to merge only tickets in the current milestone.

https://github.com/sagemath/git-trac-command/pull/54

[Matthias Koeppe]

And here is one so that on merge conflicts, there is no manual work for the release managers, and developers get some information about the conflict other than "merge conflict". https://github.com/sagemath/git-trac-command/pull/57

[Kwankyu Lee]

On Monday, January 24, 2022 at 2:35:54 AM UTC+9 Matthias Koeppe wrote:

Patchbot experts:

1) Currently the patchbot does not run on certain tickets that make changes to build/pkgs, so we do not have automatic testing for these tickets. Is this something that can be changed? 

I think this is related with "safe_only" concept of sage patchbot. Patchbot only runs on "safe" tickets by default. If I remember correctly, a ticket that makes changes out of sage library is considered "unsafe".

If a ticket is unsafe, then patchbot uses "ccache" by default. If I again remember correctly, this "ccache" also caused some problem. I remember I tried to turn off "ccache" to fix that.

The following is the relevant part of "sage_patchbot/trac.py". 

-----------------------------------------------

def pull_from_trac(sage_root, ticket_id, branch=None, force=None,
                   use_ccache=False,
                   safe_only=False):
    """
    Create four branches from base and ticket.

    If ticket deemed unsafe then clone git repo to temp directory. ?!

    Additionally, if ``use_ccache`` then install ccache. Set some global
    and environment variables.

    There are four branches at play here:

    - patchbot/base -- the latest release that all tickets are merged into
      for testing
    - patchbot/base_upstream -- temporary staging area for patchbot/base
    - patchbot/ticket_upstream -- pristine clone of the ticket on trac
    - patchbot/ticket_merged -- merge of patchbot/ticket_upstream into
      patchbot/base
    """
    merge_failure = False
    is_safe = False
    temp_dir = None
    try:
        os.chdir(sage_root)
        info = scrape(ticket_id)
        ensure_free_space(sage_root)
        do_or_die("git checkout patchbot/base")
        if ticket_id == 0:
            do_or_die("git branch -f patchbot/ticket_upstream patchbot/base")
            do_or_die("git branch -f patchbot/ticket_merged patchbot/base")
            return
        branch = info['git_branch']
        repo = info['git_repo']
        do_or_die("git fetch %s +%s:patchbot/ticket_upstream" % (repo, branch))
        base = describe_branch('patchbot/ticket_upstream', tag_only=True)
        do_or_die("git rev-list --left-right --count %s..patchbot/ticket_upstream" % base)
        do_or_die("git branch -f patchbot/ticket_merged patchbot/base")
        do_or_die("git checkout patchbot/ticket_merged")
        try:
            do_or_die("git merge -X patience patchbot/ticket_upstream")
        except Exception:
            do_or_die("git merge --abort")
            merge_failure = True
            raise

        is_safe = inplace_safe()
        if not is_safe:
            if safe_only:
                raise SkipTicket("unsafe")
            # create temporary dir
            temp_dir = tempfile.mkdtemp(temp_build_suffix + str(ticket_id))
            ensure_free_space(temp_dir)
            do_or_die("git clone . '{}'".format(temp_dir))
            os.chdir(temp_dir)
            os.symlink(os.path.join(sage_root, "upstream"), "upstream")
            os.environ['SAGE_ROOT'] = temp_dir
            do_or_die("git branch -f patchbot/base remotes/origin/patchbot/base")
            do_or_die("git branch -f patchbot/ticket_upstream remotes/origin/patchbot/ticket_upstream")
            do_or_die("make configure")
            do_or_die("./configure")
            if use_ccache:
                if not os.path.exists('logs'):
                    os.mkdir('logs')
                do_or_die("./sage -i ccache")
    except Exception as exn:
        if not is_safe and not safe_only:
            if temp_dir and os.path.exists(temp_dir):
                # Reset to the original sage_root
                os.chdir(sage_root)
                os.environ['SAGE_ROOT'] = sage_root
                shutil.rmtree(temp_dir)  # delete temporary dir

        if merge_failure or (not is_safe):
            raise
        else:
            raise ConfigException(str(exn)) 

[Matthias Koeppe]

On Sunday, January 23, 2022 at 9:35:54 AM UTC-8 Matthias Koeppe wrote:

I don't know if Volker's integration scripts look at the patchbot status at all

Turns out they don't.

https://github.com/sagemath/git-trac-command/pull/58 fixes this