Some way to group variant dirs in subdirectory? (have 36+ variants)

[Pascal]

Hi,
I have a C project with 36 (and growing) variants where each can be build with debug and release configuration. It is an embedded project where the firmware for a controller device is compiled in multiple variants for different sets of tools attached to the controller.

If I understand it correctly, with tup, I would need a variant directory for configuration and build outputs for each variant in the root directory of the project. This would translate to 36 debug variant directories plus 36 release variant directories. That is ok but is there a way to not put them in the root directory of the project without completely restructuring my soure code?

The project is basically structured like this

project_root/
    .git
    core_sources
    lib_sources
    tool_driver_sources
    makefile  # currently
    .other-
    .dev_tool-
    .config_files

So for navigating all sources plus the variant configurations in an IDE, it would be nice to not have 72 variant directories directly under the project root but in some subdirectory. Is there a way to achieve this? After studying the tup manual it looks like there is no straight forward way.

I could think of something like

project_root/
    .git
    build/
        .tup
        variant_a
        variant_b
        ...
        variant_xy
        core_link -> ../core_sources
        lib_link -> ../lib_sources
        drv_link -> ../tool_driver_sources
    core_sources
    lib_sources
    tool_driver_sources
    .other-
    .dev_tool-
    .config_files

But I have to use this project on windows and somehow file system links feel wonky on windows. Would encourage me to do it like this anyways?

[Mike Shal]

On Mon, Apr 1, 2024 at 3:45 PM Pascal <p.ro...@gmail.com> wrote:

Hi,
I have a C project with 36 (and growing) variants where each can be build with debug and release configuration. It is an embedded project where the firmware for a controller device is compiled in multiple variants for different sets of tools attached to the controller.

If I understand it correctly, with tup, I would need a variant directory for configuration and build outputs for each variant in the root directory of the project. This would translate to 36 debug variant directories plus 36 release variant directories. That is ok but is there a way to not put them in the root directory of the project without completely restructuring my soure code?

Unfortunately at this time, tup only supports variants at the root of the project (where 'tup init' is run). (There's been an issue open for a long time, but I don't think anyone has tried to address it: https://github.com/gittup/tup/issues/232 - it might be easier now with the explicit variants merge, as previously the structure was tied to the FUSE overlay but now it is just internal path manipulations). One thing to note, is that what you (and git) consider the "root of the project" doesn't have to match tup - your git directory could be a subdirectory of where tup is run. I'm not sure if this helps, and it isn't entirely what you are asking for, but you could try this structure:

somedir/

   .tup

   variant_a/

   variant_b/

   variant_.../

   project_root/

       .git

       core_sources

       lib_sources

       etc.

Basically, run 'tup init' at a level above where you are checking out your project code. Tupfiles should all be compositable by default, such that this approach should work. (ie: the use of things like $(TUP_CWD) and the like are all relative paths, so they should be able to be at a higher directory). At the very least, this keeps all of the variants outside of your project_root/ directory, which I think is your primary goal?

The downside obviously is that to get to your sources you'd have to cd/browse into somedir/project_root/ instead of just project_root/

I wouldn't recommend trying this - by moving the 'tup init' down into the build directory, tup would not be saving dependencies on files outside of that tree by default. You could set the tup option updater.full_deps to 1 to track those now "external" dependencies on your source code (where "external" means they are somewhere outside of the subtree where .tup exists), but in addition to the system links it adds another level of wonkyness.

Another option is to try to roll your own variants. Depending on how complicated your project is and your familiarity with Lua, this could be pretty easy or very difficult. Here's a tiny example to see what I mean:

project_root/

   Tupdefault.lua

   Tuprules.lua

   core_sources/

        foo.c

        bar.c

        sub/

             coresub.c

Tuprules.lua:

-- PROJECT_ROOT is a relative directory always pointing to the root of the project

PROJECT_ROOT = tup.getcwd()

-- BUILD_ROOT is similar, it points to where variants should go

BUILD_ROOT = PROJECT_ROOT .. '/build/'

Tupdefault.lua

-- Our list of variants (presumably each one would have different CCFLAGS or whatever)

variants = {'build1', 'build2', 'build3'}

-- This variable lets us keep the source structure in the builddir (ie: core_sources/foo.c becomes build/variant/core_sources/foo.o)

subdir = tup.getrelativedir(PROJECT_ROOT)

-- Build each variant - most of the wonkyness is in the path construction of the output file, since normally tup expects you to write to an output alongside the Tupfile.

local k, v
for k, v in ipairs(variants) do
        tup.foreach_rule('*.c', 'gcc -c %f -o %o', BUILD_ROOT .. v .. '/' .. subdir .. '/%B.o')
end

This results in the following structure in build:

build/build3
build/build3/core_sources
build/build3/core_sources/foo.o
build/build3/core_sources/bar.o
build/build3/core_sources/sub
build/build3/core_sources/sub/subcore.o
build/build2
build/build2/core_sources
build/build2/core_sources/foo.o
build/build2/core_sources/bar.o
build/build2/core_sources/sub
build/build2/core_sources/sub/subcore.o
build/build1
build/build1/core_sources
build/build1/core_sources/foo.o
build/build1/core_sources/bar.o
build/build1/core_sources/sub
build/build1/core_sources/sub/subcore.o

Hope that helps,

-Mike

[Mike Shal]

On Mon, Apr 1, 2024 at 10:34 PM Mike Shal <mar...@gmail.com> wrote:

On Mon, Apr 1, 2024 at 3:45 PM Pascal <p.ro...@gmail.com> wrote:

Hi,
I have a C project with 36 (and growing) variants where each can be build with debug and release configuration. It is an embedded project where the firmware for a controller device is compiled in multiple variants for different sets of tools attached to the controller.

If I understand it correctly, with tup, I would need a variant directory for configuration and build outputs for each variant in the root directory of the project. This would translate to 36 debug variant directories plus 36 release variant directories. That is ok but is there a way to not put them in the root directory of the project without completely restructuring my soure code?

Unfortunately at this time, tup only supports variants at the root of the project (where 'tup init' is run). (There's been an issue open for a long time, but I don't think anyone has tried to address it: https://github.com/gittup/tup/issues/232 - it might be easier now with the explicit variants merge, as previously the structure was tied to the FUSE overlay but now it is just internal path manipulations).

Actually I just gave this a try and it seems to be pretty straightforward to do now. There are a few small details to work out, but if I can set aside some time to work on it I might be able to add support for "variants anywhere" later this week.

-Mike

[Mike Shal]

Well I didn't have time that week apparently :). But I did push a commit to master which enables variants in subdirectories now. You can't nest a variant inside a variant, but otherwise they should work at any level in the tup hierarchy. Let me know if you have any issues with it.

-Mike