Persistent issues with Copybara picking the wrong base

[Rylie Pavlik]

Hi folks. I'm using Copybara to import changes merged to "mostly-downstream" public repos back into an upstream monorepo. However, it's not working how I expected it to, and I thought it worked better in the past. Apparently I'm not actually pushing the copybara files publicly with each release, but I can share that if it matters. The workflow looks broadly like:

core.workflow(
    name="import-sdk-source",
    custom_rev_id="SDK-Source-RevId",
    origin=git.github_origin(
        url="https://github.com/KhronosGroup/OpenXR-SDK-Source.git",
        ref="main",
        first_parent=False,
    ),
    destination=git.destination(
        url=gitlab_url,
        fetch="refs/tags/" + last_release,
        push="import-sdk-source",
    ),
    mode="ITERATIVE",
    origin_files=sdk_source_origin_files,
    destination_files=sdk_source_dest_files,
    authoring=author,
    transformations=sdk_source_transforms,
)

(We also vendor dependencies with Copybara, but that part works fine.)

Below my signoff is my writeup on how stuff works with the repositories, how I mentally expected Copybara to work (but suspect now it isn't, at least due to the custom rev id), and how I think it's working. The SDK-Source import works better than the CTS import, for some reason, even though the workflow is basically identical.

I would appreciate any help that you can offer in trying to make it work more like "Desired Behavior", or at least avoid the hacky step needed right now. Thanks!

Rylie Pavlik

• General repo interaction
  • Internal development happens on repo A, a private GitLab monorepo.
  • Periodically, releases are performed that update several public GitHub repos (broadly "repo B") from this monorepo A, in a single commit. This step is performed with manual scripts, rather than Copybara, though we have been adding commit trailers for tool assistance. When a public repo B is updated from the monorepo A, the commit message on B includes a GitOrigin-RevId: trailer indicating the corresponding repo A commit (in addition to release notes, etc.)
  • At least once per release cycle, if any changes have been merged to the downstream public repos (B), we want to import those (commit by commit) into the monorepo A, using Copybara. We apply some transformations with Copybara but maintain nearly-unmodified the commit metadata. We are using a "custom rev ID" in the configuration for these steps, so each commit imported from repo B into repo A contains a CTS-RevId:, SDK-Source-RevId:, or Docs-RevId: trailer.
    • We would also like to be able to mirror pull requests too, have experimented with mixed success.
  • We once considered/tried out adding an empty commit to repo A at releases containing the downstream RevId trailers for all repo B. Unsure if this is needed or useful. This is easy enough to automate if it is required.
• Desired behavior
  • Repo A: Monorepo
  • Repo B: CTS (or SDK-Source, or...) with specified source ref tipB​ . When periodically updated from A, the commit message includes a GitOrigin-RevId: trailer indicating the corresponding repo A commit
  • Starting at a ref tipB​ on repo B, iterate until a commit is found with GitOrigin-RevId: trailer. Call this commit on B the baseB​ , and the hash mentioned in the trailer baseA​ .
  • Create a branch targetA​ for repo A at baseA​ .
  • Iteratively process commits between the baseB​ and tipB​ of repo B, transforming them and applying them to targetA​ on repo A. In the process, each is marked with a CTS-RevId: trailer indicating their original hash in repo B.
  • We would then rebase this, fix anything needed to pass the internal CI, and merge.
• Possibly current behavior?
  • Repo A: Monorepo with main branch at tipA​
  • Repo B: CTS (or SDK-Source, or...) with specified source ref tipB​
  • Starting at tipA​ on repo A, iterate back through the log until a commit is found with a CTS-RevId: trailer (or other specific trailer as applicable). Call this commit on A the baseA​ , and the hash mentioned in the trailer baseB​ . If nothing is found close enough to the tip, which is usually the case, use tipA​ as baseA​ , and not sure how baseB​ is computed, might be in the config. (This is usually the wrong behavior, and results in the first commit imported also undoing all changes since release.)
  • Create a branch targetA​ for repo A at baseA​ .
  • Iteratively process commits between the baseB​ and tipB​ of repo B, transforming them and applying them to targetA​ on repo A.
  • In practice, I have to then manually git rebase --onto from main to whatever the nearest of the last release or last imported change from that repo was, to clean up that first commit. Often some release commits are imported initially too, and need to be manually dropped to avoid conflicts. Needs careful inspection and is error prone. We then rebase back to main, fix CI errors, and merge.