buildworld in NFS /usr/src/ vs. automount /usr/src path

[Steve Rikli]

I have small VM's for testing, and recently have been trying builds with
/usr/src/ mounted from my NFS server, read-only. /usr/obj/ is local on
the virtual system disk.

I'm simply using buildworld, nothing fancy; same routine I use on my
other -STABLE systems, with a local /usr/src/ read-write on their own
sysdisk.

Depending on how the NFS mount is done, this works fine; that is, I've
noticed an apparent difference in buildworld behavior, and it seems to
cause a failure. Here are the 2 different setups:

1) NFS mount the remote server's export directly at the VM's /usr/src/

E.g. it essentially looks like:

$ df -h /usr/src/
Filesystem Size Used Avail Capacity Mounted on
srv:/os/freebsd/src/stable-15 14T 9.5T 4.2T 70% /usr/src

This works fine, aside from some warnings about read-only /usr/src/, it
behaves just as if /usr/src/ was on the local system disk, and results
in successful buildworld completion.

2) /usr/src is a symlink pointing to an automount path of the remote
NFS server's export

This setup looks like:

$ ls -la /usr/src
lrwxr-xr-x 1 root wheel 35 Oct 17 11:51 /usr/src -> /net/srv/os/freebsd/src/stable-15

While the data itself in each case's /usr/src is the same (i.e. from
the same export on the same NFS server), the 2nd scenario does not work;
it gives a buildworld failure like this:

...
awk -f /net/srv/os/freebsd/src/stable-15/sys/conf/kmod_syms.awk irdma.ko.full export_syms | xargs -J % objcopy % irdma.ko.full
objcopy --only-keep-debug irdma.ko.full irdma.ko.debug
objcopy --strip-debug --add-gnu-debuglink=irdma.ko.debug irdma.ko.full irdma.ko
===> isci (all)
[Creating objdir /usr/obj/net/srv/os/freebsd/src/stable-15/amd64.amd64/sys/GENERIC/modules/net/srv/os/freebsd/src/stable-15/sys/modules/isci...]
machine -> /net/srv/os/freebsd/src/stable-15/sys/amd64/include
x86 -> /net/srv/os/freebsd/src/stable-15/sys/x86/include
i386 -> /net/srv/os/freebsd/src/stable-15/sys/i386/include
make[4]: don't know how to make sci_base_memory_descriptor_list.c. Stop

make[4]: stopped making "all" in /net/srv/os/freebsd/src/stable-15/sys/modules/isci
*** Error code 2

Stop.
make[3]: stopped making "all" in /net/srv/os/freebsd/src/stable-15/sys/modules
*** Error code 1

Stop.
make[2]: stopped making "all" in /usr/obj/net/srv/os/freebsd/src/stable-15/amd64.amd64/sys/GENERIC
3039.85 real 2180.19 user 241.05 sys
*** Error code 1

Stop.
make[1]: stopped making "buildkernel" in /net/srv/os/freebsd/src/stable-15
*** Error code 1

Stop.
make: stopped making "buildkernel" in /net/srv/os/freebsd/src/stable-15

----

I've tried a couple different symlink automount paths but it doesn't
seem to matter -- I didn't really expect it would.

For clarity, that same git src tree runs buildworld to completion OK
when it is mounted directly as /usr/src/ on the same VM.

I'm guessing that buildworld is getting confused because something
like /bin/pwd is not "/usr/src/" in the symlink automount path setup,
i.e. it will instead be something like "/net/nfsserver/path/..." and
that seems to be confusing buildworld during creation of files in the
local /usr/obj/ directory.

I can workaround the mounting setup well enough, I'm just wondering if
buildworld is expected to work okay with building in paths other than
"/usr/src". Until recently I've never tried this sort of thing. :-)

Cheers,
sr.

[void]

On Thu, Nov 13, 2025 at 02:03:40PM -0800, Steve Rikli wrote:

>Depending on how the NFS mount is done, this works fine; that is, I've
>noticed an apparent difference in buildworld behavior, and it seems to
>cause a failure. Here are the 2 different setups:

You may find https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275030
interesting
--

[Steve Rikli]

Somewhat. My /usr/obj/ today is local to the buildworld's sysdisk so no
NFS is involved in writing to /usr/obj/ -- only /usr/src/ is from NFS,
and it is read-only.

But some of the notes in the bug make me wonder if moving /usr/obj/ to
NFS, as well as /usr/src/, might somehow produce different results.

The NFS /usr/obj/ export would have to be writeable, of course, but I'd
prefer to keep /usr/src/ read-only, assuming that's a supported setup
for buildworld.

That aside, I still wonder about buildworld having some problem with
"/usr/src/" vs. a /usr/src symlink which points to "/net/srv/...".

Thanks,
sr.

[void]

On Thu, Nov 13, 2025 at 04:13:04PM -0800, Steve Rikli wrote:

>Somewhat. My /usr/obj/ today is local to the buildworld's sysdisk so no
>NFS is involved in writing to /usr/obj/ -- only /usr/src/ is from NFS,
>and it is read-only.
>
>But some of the notes in the bug make me wonder if moving /usr/obj/ to
>NFS, as well as /usr/src/, might somehow produce different results.
>
>The NFS /usr/obj/ export would have to be writeable, of course, but I'd
>prefer to keep /usr/src/ read-only, assuming that's a supported setup
>for buildworld.
>
>That aside, I still wonder about buildworld having some problem with
>"/usr/src/" vs. a /usr/src symlink which points to "/net/srv/...".

I don't see why /usr/obj or /usr/src over nfs wouldn't be supported
I mean the /usr/obj over nfs not working properly is the basis for
that ticket after all. I'm not the OP of that PR though.

Having /usr/obj /usr/src and /var/cache/ccache over nfs is very useful
in situations where the client machine might be low on resources like
disk space or have low performance disk media like mmcsd, but might
have acceptable network throughput.

I think a symlink (particularly a soft link) might produce unexpected
results though. The last time I tried that in a buildworld context was
many years ago and although I don't remember the exact outcome, I didn't
persist with that configuration.

--

[Steve Rikli]

On Fri, Nov 14, 2025 at 12:54:41AM +0000, void wrote:
> On Thu, Nov 13, 2025 at 04:13:04PM -0800, Steve Rikli wrote:
>
> > That aside, I still wonder about buildworld having some problem with
> > "/usr/src/" vs. a /usr/src symlink which points to "/net/srv/...".
>
> I don't see why /usr/obj or /usr/src over nfs wouldn't be supported
> I mean the /usr/obj over nfs not working properly is the basis for that
> ticket after all. I'm not the OP of that PR though.
>
> Having /usr/obj /usr/src and /var/cache/ccache over nfs is very useful
> in situations where the client machine might be low on resources like
> disk space or have low performance disk media like mmcsd, but might
> have acceptable network throughput.

Agreed -- that's essentially my situation with these test systems.

> I think a symlink (particularly a soft link) might produce unexpected
> results though. The last time I tried that in a buildworld context was many
> years ago and although I don't remember the exact outcome, I didn't persist
> with that configuration.

I suspect that's what I'm running into. Another experiment occurred to
me, e.g. symlink /usr/src to somewhere else temporarily on the local
disk, thereby taking NFS out of the picture, and see if the problem
replicates.

I'm expecting it will. In that case, it would seem like a reasonble
conclusion that you're right about /usr/src/ links and buildworld; I'll
just use a different NFS mount config if that's how it turns out.

cheers,
sr