[RFC] serving sub-projects out of a parent projects (on-demand filter-branch?)

[Kenny Ho]

Hi,

Has there been any discussions about the possibility of serving sub-projects out of a parent project?  In a way, these subprojects could be serving an on-demand, filter-branch-ed version of the parent project.

Use case:

* Given a large 'monolithic' upstream code base that does not use submodule or repo, you want to replace some modules or subdirectories while everything else continues to track upstream.

* Simplify transition from or inter-op with other SCM systems that have a single 'layout'/'view'

* Directory level access control can be enforced by setting the permission in the sub-projects and then 'recombine' them into a bigger project using other means.  Submissions to the sub-projects are effectively submitting to the parent project.

I am aware of mechanisms such as repo or submodule and I am also familiar with git-filter-branch where you can manually modify the content of a git repository to remove/keep only certain files or directories as needed but none of those really serve the use cases mentioned above.

Thoughts?  I am actually not sure if this is a good idea myself.  It would be great to hear your experiences if you have encountered these use cases before.

Regards,

Kenny

[Doug Kelly]

I think you could try doing something like this, but the results would be horribly, horribly painful.  Years ago, our site ran a custom version of Gerrit that used subtree merges internally to provided "view"-like behaviors from previous systems.  The result was a version of Gerrit that was difficult to maintain, frequently broken (had all sorts of strange quirks), and generally everything we needed could be replicated by submodules anyway.

You may be able to get some space savings by reusing objects, but are the costs worth it? (The Git LFS backend plugins do something like this, but those are for very large objects.)

Something else to consider: there's the submodule subscription feature that may be beneficial here (note: it has some caveats of its own, specifically when using it with CI systems), and you can also "fork" a repo using "git replace" to substitute a previous tree, instead of importing all past history (if that's a concern: I generally don't have an issue with importing the history).

--Doug

[Kenny Ho]

Hey Doug,

Thanks for the comments.

Space saving may be a side effect of this feature but it's not really the driving aspect in my view.  The main drive is really the social/political/administrative aspects.  I am sure you are right about things being painful but I am not sure how else to crack the social problems.

To be more concrete, my first use case is really about the linux kernel.  We don't control the upstream kernel tree but there are modules that we contribute to that we want to be able to support both in-tree and out-of-tree build while tracking upstream as closely as possible.  We can certainly have an internal fork and do whatever we want but then we loses the tracking upstream part.  May be I am asking too much of the SCM system but I was hoping to find a solution without going into a full blown yocto or buildroot type of thing.

The third use case is also more of a social issue than a technical one.  It's mostly about things like crypto code and export control rules.  Some can view the entire repo but not others.  The perfect solution may be to split those parts into their own repos and enforce different permission on them but things sometimes can't be separated so cleanly.  And when the repo is something like the linux kernel then splitting it is also not possible.

'git replace' looks interesting.  I will definitely check it out.

Regards,

Kenny