Are subprojects the right thing for my team?

[oyvind]

Hello Meson community,

In my team we are considering to update our project structure to use Meson subprojects (+wrap) instead of git submodules for sharing libraries between our projects. We use Meson already. I know there is a FAQ on subdir vs subproject but I'm still unsure if subproject is intended for our use case.

We have a distributed code base and use git submodules to bring in shared code across projects. There are dependencies between some of the submodules, including diamonds. For the time being this is manageable, but we expect to see a growth in both team members and code base in the near future, and want to keep things as simple as possible.

The number one thing that appeals to us with subprojects is that we can use meson for dependency management. Furthermore it seems to make it easier to develop modules in isolation and bring them into other projects (my impression at least).

So much background info but I hope you are still with me. I guess my questions are:

- At a glance, can subprojects be a better alternative to git submodules for us?

- Are the two options even mutually exclusive?

- Are there any pitfalls, limitations or gotchas to look out for with subprojects?

- Are there any recommendations or best practices for using Meson subprojects?

Best regards,

Øyvind

[Jussi Pakkanen]

On Sat, 6 Nov 2021 at 19:16, oyvind <oyvind...@gmail.com> wrote:

> So much background info but I hope you are still with me. I guess my questions are:
> - At a glance, can subprojects be a better alternative to git submodules for us?
> - Are the two options even mutually exclusive?

They are not. In fact you can provide subprojects as Git submodules if
you want to. I personally prefer wrap files for this, but both are
supported and it is mostly a question of which one is better suited to
your setup and workflow.

Meson subprojects are basically a logical thing meaning "stuff in the
top level subprojects directory". It does not care how things get
there, only that they can be found there. Git submodules are
conceptually about "put repository X in directory Y".

> - Are there any pitfalls, limitations or gotchas to look out for with subprojects?
> - Are there any recommendations or best practices for using Meson subprojects?

Try to think of things in terms of "dependencies" rather than
subprojects. That is, try to use `dep = dependency('foo')` rather than
`dep = subproject('foo')...`. Try to keep different subprojects as
loosely coupled as possible. Avoid poking the insides of other
subprojects as much as possible. Instead try to be declarative.

[Eli Schwartz]

On 11/6/21 1:16 PM, oyvind wrote:
> So much background info but I hope you are still with me. I guess my
> questions are:
> - At a glance, can subprojects be a better alternative to git submodules
> for us?
> - Are the two options even mutually exclusive?

As Jussi said, they are not exclusive and you can use them together. I'd
like to elaborate on this:

subprojects attempt to configure a directory as an isolated namespace.
You don't get to (or want to) share context between a subproject and a
superproject -- you cannot poke directly at files in a subproject e.g.
add them as source files for build targets in the superproject.

What you are supposed to do instead is provide exported interfaces,
built programs and dependency objects created by declare_dependency,
which you can access via foo_subproject.get_variable() or by
override_find_program() and override_dependency().

That's all that a subproject means. Logical dependency isolation. So, I
think your impression is right and using them will appeal to you. :)


There are a couple ways to *get* the source files of a subproject. Either:

- add them as submodules, then `git submodule update --init --recursive`

- check them directly into the superproject repo

- use wrap files and meson will download (clone or http tarball) them
for you

- use wrap files *and* submodules, and meson will init the submodule for
you


This last option is highly convenient, as you get the best of both worlds.

> - Are there any pitfalls, limitations or gotchas to look out for with
> subprojects?
> - Are there any recommendations or best practices for using Meson
> subprojects?

As you mention diamond dependencies, keep in mind that meson resolves
all dependencies by hoisting them from subprojects into the superproject
and using a single version of each one. Hopefully that is a good
thing... if you have nested component dependencies, ideally you want to
only have one version of that dependency and share it between the
subprojects that need it.

In meson, the rule is that the first time meson encounters a
subproject('name') or a dependency('name') that is resolved using wrap
provides, it is hoisted to the toplevel, and thereafter every attempt to
resolve that dependency will resolve to the originally resolved version.

(For cases where two different versions of a dependency are maintained
in parallel, e.g. Gtk 3 and Gtk 4, they simply get different names.)


--
Eli Schwartz

[oyvind]

Jussi and Eli

Thank you for the feedback, you have given me a lot to think about :)

I will let this sink in and test a bit more to gain some experience with different work flows, especially testing mixing submodules and wrap.

I drew a dependency map for my self and realized we have more diamonds than I initially thought, so being able to reason about a

compatible version across dependencies will be important. Typically we have something like this going on, where A is the super project and the rest are dependencies:
A->B->C
A->D->C
A->C

...

I think the way meson would resolve this by hoisting C to the top level makes sense for us. We want to have only 1 version of a common dependency.

Regarding dependency versions and use of wrap-git.

How should we go about using "revision" and project version effectively?

It would be nice if Meson could fetch a compatible version when calling subproject(..., version:'x').

Instead it seems like "revision" serves this purpose and that the project version specified in subproject() is used to check for compatibility.

- Would it be possible to have Meson automatically fetch a compatible version when calling subproject()?

Best regards,

Øyvind

[oyvind]

Hi, it's me again with a follow up question:

- Can the top-level subprojects directory be part of the include path by default? If not, then why?

The reason I ask is because up until now I (oops) used to do include_directories('..') to have it on the path when a project detects that it's a subproject.

I did that because in our code base we typically do #include "some_module/header.h". Since some_module is hoisted to top-level when building as subproject, I've had to include the illegal path. I've found another way to make it work that won't violate sandbox rules, but it required me to alter the source tree like so: 

some_module/
└── some_module
    └── header.h

but I would like to avoid that.

I've attached 2 examples that I hope clarifies my concern.

Best regards,

Øyvind

[Jussi Pakkanen]

On Tue, 11 Jan 2022 at 13:50, oyvind <oyvind...@gmail.com> wrote:

> - Can the top-level subprojects directory be part of the include path by default? If not, then why?
>
> The reason I ask is because up until now I (oops) used to do include_directories('..') to have it on the path when a project detects that it's a subproject.
> I did that because in our code base we typically do #include "some_module/header.h". Since some_module is hoisted to top-level when building as subproject, I've had to include the illegal path. I've found another way to make it work that won't violate sandbox rules, but it required me to alter the source tree like so:

This is not really supported by design, because that would expose too
much. You could do #include<othersubproject(foo.h> either
intentionally, which is bad, or unintentionally which is even worse.

This is actually a common issue. There are several projects that keep
their headers in a directory called "include" but install them so that
end users do #include<foobar/header.h>. This is not really
recommended, instead the header tree inside a project should exactly
match the public installed one. Not only is this consistent, but also
means that you can install the entire header subdirectory with a
single `install_subdir` (not all projects need this, of course, but
many do).

This does have the unfortunate side effect that projects that do not
follow this layout have to do some extra work.

There are awful workarounds, as there always area, such as doing the
following at the beginning of the top level project.

hack_dep = declare_dependency(include_directories: 'subprojects')
meson.override_dependency('includehack', hack_dep)

and then this in the subproject

hack_dep = dependency('includehack')
build_target(..., dependencies: hack_dep)

But i _highly_ recommend that you don't do this. In addition to the
issues mentioned above this also ties these two together, you can only
build them together in this very specific setup. Whenever you make
subprojects "peek out" from their own isolated directory tree you are
going to have trouble.

[oyvind]

Thanks for replying,

I decided to not use the tricks mentioned but instead spent some time this week changing the folder structure of a couple of our projects. The change was as mentioned above, just adding a new directory of the same name and put the files there. 

Although the idea did not appeal to me at first, I have to say that the result far outweighs this "sacrifice". Actually being able to work on modules in isolation and bring them into a project has really

become plug and play. Seems to work really nice with our use case which is cross-building for bare metal embedded systems. I think this can really elevate our team and speed up development. Great stuff!

Best regards

Øyvind