Open sourcing from an internal (bazel built) monorepo

[Matthew Mackay]

After some workflow advice here really. Currently we use Copybara to migrate and keep our internal repos in sync with our internal monorepo while we transition more projects to it. I'm looking into open sourcing a number of projects that now live in the internal monorepo onto GitHub.

Our projects are mostly built with Bazel, and we have one WORKSPACE file at the root of the repo, and BUILD files scattered around as required.

My question here is that I'd like to know about the workflow around open sourcing a project that builds with Bazel, I don't want to open our entire workspace file, and the paths within the BUILD files will be wrong the external GitHub repo. Would I keep a separate WORKSPACE file perhaps in the projects folder and use Copybara to move it to the correct place on migration (and same with BUILD files)? What's the general approach here if anyone has done something similar.

Thanks in advance.

- Matt

[Miguel Alcon]

For WORKSPACE, yes probably having one for OSS or generating one with Copybara is the best option.

For BUILD files, internally we have a transformation that changes the AST or the BUILD files (Uses https://github.com/bazelbuild/buildtools/tree/master/buildozer), we want to put it in the OSS tree at some point, but we haven't had time for that. But in your case, I would say that a core.replace is probably going to be more effective.

--
You received this message because you are subscribed to the Google Groups "Copybara OSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copybara-discu...@googlegroups.com.
To post to this group, send email to copybara...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/copybara-discuss/2e174e4e-9470-42c1-a0ff-e524d0b85c5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Mackay]

Thanks Miguel, that worked for the migration to GH from internal. 

What I'm struggling to understand now is importing changes workflow. I have a simple change request workflow going, but it breaks down where I apply transforms to BUILD files, or remove internal parts of readme files. 

For example, we transform `//some/internal/label/src` to `//src`, and I'm wondering how the change request workflow knows to transform these back. Or to ignore parts of a readme with `// BEGIN-INTERNAL` comment blocks. This wasn't clear to me from reading the docs. 

Any insight would be helpful. Thanks!

[Miguel Alcon]

Answering bellow: 

On Wed, May 8, 2019 at 8:42 AM Matthew Mackay <mat...@gmail.com> wrote:

Thanks Miguel, that worked for the migration to GH from internal. 

What I'm struggling to understand now is importing changes workflow. I have a simple change request workflow going, but it breaks down where I apply transforms to BUILD files, or remove internal parts of readme files. 

For example, we transform `//some/internal/label/src` to `//src`, and I'm wondering how the change request workflow knows to transform these back.

One problem that we've seen with core.replace is that if you transform "//something/very/specific" --> "something_generic" then the mapping cannot be done back properly, as many things matches "something_generic". The trick that sometimes our users use is to map it to  "something_generic" # some specific comment. In that way you can do the mapping back in a safe way.

We'll see if we can move our buildozer transformations to OSS soon, as it is much easier with AST transformations. 

 

Or to ignore parts of a readme with `// BEGIN-INTERNAL` comment blocks. This wasn't clear to me from reading the docs. 

Scrubbing is tricky to reverse. We have some support for detecting it in affected files (so we discard not affected files) but sadly it is still not available for git*destinations.  Would you mind creating an issue in GitHub for this? I think we might be able to do something here.

 

Any insight would be helpful. Thanks!

On Mon, Apr 22, 2019, 3:59 PM Miguel Alcon <mal...@google.com> wrote:

For WORKSPACE, yes probably having one for OSS or generating one with Copybara is the best option.

For BUILD files, internally we have a transformation that changes the AST or the BUILD files (Uses https://github.com/bazelbuild/buildtools/tree/master/buildozer), we want to put it in the OSS tree at some point, but we haven't had time for that. But in your case, I would say that a core.replace is probably going to be more effective.

On Mon, Apr 22, 2019 at 1:20 PM Matthew Mackay <mat...@gmail.com> wrote:

After some workflow advice here really. Currently we use Copybara to migrate and keep our internal repos in sync with our internal monorepo while we transition more projects to it. I'm looking into open sourcing a number of projects that now live in the internal monorepo onto GitHub.

Our projects are mostly built with Bazel, and we have one WORKSPACE file at the root of the repo, and BUILD files scattered around as required.

My question here is that I'd like to know about the workflow around open sourcing a project that builds with Bazel, I don't want to open our entire workspace file, and the paths within the BUILD files will be wrong the external GitHub repo. Would I keep a separate WORKSPACE file perhaps in the projects folder and use Copybara to move it to the correct place on migration (and same with BUILD files)? What's the general approach here if anyone has done something similar.

Thanks in advance.

- Matt

--
You received this message because you are subscribed to the Google Groups "Copybara OSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copybara-discu...@googlegroups.com.
To post to this group, send email to copybara...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/copybara-discuss/2e174e4e-9470-42c1-a0ff-e524d0b85c5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Copybara OSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to copybara-discu...@googlegroups.com.
To post to this group, send email to copybara...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/copybara-discuss/CAEtOHk-0VOH442c2Lp4kT%3DQDJZghfE72vugOPeKBcG_obgFJ_w%40mail.gmail.com.