Unfortunate dynamic linking for everything
dy...@iquest.net
Please revisit the dynamic linking for everything. The
cost for using shared libs in cases like shells actually
is higher than statically linking (both in memory and
in time.) It appears that there is a loss of VM understanding
over time. Don't confuse the advantages of using shared
libs on X windows (and environments like that) with the
same advantages being non-existant when using shared
libs on normal layout programs (shells are perhaps the
worst general case, but not necessarily the worst in all ways.)
Another issue: if you continue to support the dynamic
linking mechanisms for special shared libs, but otherwise
link statically, much of the high overhead of shared libs
for-everything will still be mitigated. Just because there
might be a need for a special shared lib, that doesn't justify
using shared libs for everything (and add the cost of
sparse memory allocation and significantly higher fork/exec
times than necessary.) For a 'fun time', take a look at
the process map in the deprecated /proc filesystem. Compare
the size (complexity) of a shared program with a static
program... It doesn't show all of the internal differences,
but is an externally accessable (and benchmark free) exemplar.
The only real reason for building various programs dynamic
in order to gain the advantage of specific shared object
would be an all or nothing type argument (a typical fallacy
in most religious discussions.)
It really doesn't make sense to arbitrarily cut-off a
discussion especially when a decision might be incorrect.
Perhaps the all-dynamic scheme had been decided upon so
as to give a competitive performance advantage to those who
rebuild everything (appropriate) static? :-).
If there hadn't been a noticed increase in cost by using
all-shared-libs, then the measurements were done
incorrectly. If the decision is made based upon allowing
for 1.5X (at least) times increase in fork/exec times, and
larger memory usage (due to sparse allocations), then it
would be best to have decided that performance isn't as
important as it used to be (which it just might not be
anymore.) Last time that I heard, disk space is still
much much less expensive than main memory :-).
Remember: the cost for disk space is nil nowadays, space
only made obvious by an insanely small default root filesystem
size allocation. (An 'insanely' small root filesystem is
okay when that root is a mini-root recovery system, but
the cost of an extra 500MB is rather small on a non-specialty
system is nil compared to other resources.)
I do use an all-dynamic configuration for certain embedded
applications (but also a case where there are no seperated
filesystems, and the memory usage isn't quite as important
because of the fact that AVAILABLE features is more important
than lots of concurrently running programs.) For the best
sharing and quickest system response (both due to memory and
raw program/image invocation times), at least make the shells
static.
(Sorry for any misspellings or grammar errors -- it is early
in the morning... I'll probably not participate in any further
discussion on this matter either, but it would be good to
generally avoid loosing 'ancient', but still 'accurate'
technical history.)
John
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
M. Warner Losh
dy...@iquest.net writes:
: It really doesn't make sense to arbitrarily cut-off a
: discussion especially when a decision might be incorrect.
I'd say that good technical discussion about why this is wrong would
be good. However, emotional ones should be left behind. Except for
John's message, most of the earlier messages have been more emotional
than technical.
John, do you have any good set of benchmarks that people can run to
illustrate your point?
Warner
Dan Nelson
--HcAYCG3uE/tztfnV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In the last episode (Nov 18), M. Warner Losh said:
> In message: <200311181307....@dyson.jdyson.com>
> dy...@iquest.net writes:
> : It really doesn't make sense to arbitrarily cut-off a discussion
> : especially when a decision might be incorrect.
>
> I'd say that good technical discussion about why this is wrong would
> be good. However, emotional ones should be left behind. Except for
> John's message, most of the earlier messages have been more emotional
> than technical.
>
> John, do you have any good set of benchmarks that people can run to
> illustrate your point?
Pretty much any benchmark that you can build statically and dynamically
should suffice. I've attached a simple one that fills an array with
random numbers then qsorts them. "make compare" will generate three
graphs at the end: time spent loading the executable, time spent within
the loops, and total time. Note that both load and loop timings are
higher for the dynamic binary. I ran it on a busy system, which is why
there are so many outliers. Make sure you have
src/tools/tools/ministat installed someplace in your path.
Also see
http://lists.freebsd.org/pipermail/freebsd-current/2003-April/001106.html ,
where I had posted proc/pid/maps for a static and dynamic ls.
--
Dan Nelson
dne...@allantgroup.com
--HcAYCG3uE/tztfnV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=Makefile
all: compare
REPS=50
STATICLOGS=static.total.log static.loop.log static.load.log
DYNAMICLOGS=dynamic.total.log dynamic.loop.log dynamic.load.log
LOGS=${STATICLOGS} ${DYNAMICLOGS}
CFLAGS+=-Wall
static: svd.o
${CC} -static ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${.ALLSRC} ${LDADD}
dynamic: svd.o
${CC} ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${.ALLSRC} ${LDADD}
clean:
rm -f static dynamic ${LOGS} *.o
.PHONY: run
run ${LOGS}: static dynamic
@rm -f ${LOGS}
@reps=0; while [ $$reps -lt ${REPS} ] ; do \
time -p ./static 2>&1 >> static.loop.log | sed -ne '/real/s/real //p' >> static.total.log ; \
time -p ./dynamic 2>&1 >> dynamic.loop.log | sed -ne '/real/s/real //p' >> dynamic.total.log ; \
echo -n . ; \
reps=$$(($$reps+1)) ; \
done
@echo
@paste static.total.log static.loop.log | awk '{print $$1 - $$2}' > static.load.log
@paste dynamic.total.log dynamic.loop.log | awk '{print $$1 - $$2}' > dynamic.load.log
compare: ${LOGS}
ministat -s static.load.log dynamic.load.log
ministat -s static.loop.log dynamic.loop.log
ministat -s static.total.log dynamic.total.log
--HcAYCG3uE/tztfnV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="svd.c"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#define SIZE 1024000
int comp(const void *a, const void *b)
{
return memcmp(a, b, sizeof(int));
}
int main(int argc, char *argv[])
{
int numbers[SIZE];
int i;
struct timeval tv1, tv2;
gettimeofday(&tv1, NULL);
srand(5);
for (i = 0; i < SIZE; i++)
{
numbers[i] = rand();
}
qsort(numbers, SIZE, sizeof(*numbers), comp);
gettimeofday(&tv2, NULL);
timersub(&tv2, &tv1, &tv1);
printf("%ld.%06ld\n", tv1.tv_sec, tv1.tv_usec);
return 0;
}
--HcAYCG3uE/tztfnV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="stats.txt"
ministat -s static.load.log dynamic.load.log
x static.load.log
+ dynamic.load.log
+--------------------------------------------------------------------------+
| xx + |
| xx ++ |
| xx ++ |
| xxxx ++ |
| xxxx ++ |
| xxxx ++ |
| xxxx ++ |
| xxxx +++ |
| xxxxx+++ |
| xxxxx*++++ |
| xxxxx*+++++ + |
| xxxxx**+++++ + ++ |
| xxxxx**+++++ ++ ++ + |
| xxxx*****+++++++**+ ++ |
| xxx******+*++++***+*++ ++ * x ++ + +|
||___M_A____| |
| |______M__A_________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 96 0.001561 0.027646 0.003969 0.0049661042 0.0040137086
+ 96 0.003765 0.05356 0.00824 0.010588323 0.0072195082
Difference at 95.0% confidence
0.00562222 +/- 0.00165239
113.212% +/- 33.2733%
(Student's t, pooled s = 0.00584085)
ministat -s static.loop.log dynamic.loop.log
x static.loop.log
+ dynamic.loop.log
+--------------------------------------------------------------------------+
| + |
| x + |
| x + |
| xxx+ + |
| xxx+ + + |
| xx xxx+ ++ + |
| xx xxx+x *+ + + |
| xx xxx*x **x + ++ |
| xx xxx*x +*** + ++ + |
| xxxx xx*** +***+++++ ++x |
| xxxxxxx****++***+*+++++++x + + |
|xxxxxxx*****+*******+**+*+x++++ + + x + + x + x+ *|
||________M__A___________| |
| |________M__A__________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 96 2.117437 2.946891 2.214953 2.2526589 0.13604786
+ 96 2.196508 2.945064 2.305462 2.3386929 0.13174233
Difference at 95.0% confidence
0.086034 +/- 0.037884
3.81922% +/- 1.68175%
(Student's t, pooled s = 0.133912)
ministat -s static.total.log dynamic.total.log
x static.total.log
+ dynamic.total.log
+--------------------------------------------------------------------------+
| x + |
| x x ++ |
| x x ++ |
| x x ++ |
| x x + ++ |
| x xx* ++ + |
| x xxx* x+* + ++ |
| xx xxx*x +*** ++++ + |
| xxxx x***x++***++++++ ++ |
| xxxxxxx*********+**+++ ++x ++ + |
|xxxxxxxx************+***+**++++ ++ x + + x + * +x|
||________M__A___________| |
| |________M__A___________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 96 2.119 2.961 2.218 2.257625 0.13686538
+ 96 2.208 2.952 2.313 2.3492813 0.13316888
Difference at 95.0% confidence
0.0916563 +/- 0.0382001
4.05985% +/- 1.69205%
(Student's t, pooled s = 0.13503)
--HcAYCG3uE/tztfnV
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
--HcAYCG3uE/tztfnV--
dy...@iquest.net
> <<On Tue, 18 Nov 2003 18:07:32 -0500 (EST), dy...@iquest.net said:
>
> > If the object is to maximally 'share',
>
> The object, AIUI, is for ~username expansion to work in the shells
> when the user stored somewhere defined by an external NSS module. I
> don't believe that there is anything else in a (sane) shell that
> cares.
>
It is a fallacy that EVERYTHING (e.g. libc) needs to be dynamically
linked to support that feature. A single, self contained library
that might even be sparse WRT memory isn't going to hurt much.
A compromise that supports both a 'feature' and performance
seems to be appropriate. Making everything shared (with
the sparse allocation of .data, and .text somewhat) to implement
a specific feature is certainly wasteful.
All or nothing isn't the optimal answer here.
John
Scott Long
> M. Warner Losh said:
> > In message: <200311181307....@dyson.jdyson.com>
> > dy...@iquest.net writes:
> > : It really doesn't make sense to arbitrarily cut-off a
> > : discussion especially when a decision might be incorrect.
> >
> > I'd say that good technical discussion about why this is wrong would
> > be good. However, emotional ones should be left behind. Except for
> > John's message, most of the earlier messages have been more emotional
> > than technical.
> >
> > John, do you have any good set of benchmarks that people can run to
> > illustrate your point?
> >
> not disk -- lots of stuff has rotted away.). I had hoped
> to avoid doing much in the way of proof. Pointing to the
> much more complex process map along with the implication
> for significantly higher overhead :-). (The VM code generally
> gets slower with more complex process maps and more memory used.
> For smallish programs -- including those with a few shared libs,
> changing the VM code won't help much, because the cost is built
> in to the complexity of the process image.)
I'm glad that real arguments are finally starting to surface =-)
Our rationale for encouraging Gordon is as follows:
1. 4.x upgrade path: As we approach 5-STABLE, a lot of users might want
to upgrade from 4-STABLE. Historically in 4.x, the / partition has
been very modest in size. One just simply cannot cram the bloat that
has grown in 5.x into a 4.x partition scheme. Of course there is the
venerable 'dump - clean install - restore' scheme, but we were looking
for something a little more user-friendly.
2. NSS support out-of-the-box: Again, this is a user-experience issue
in that forcing NSS users to recompile world was seen as a potential
roadblock to them.
3. Binary security updates: there is a lot of interest in providing a
binary update mechanism for doing security updates. Having a dynamic
root means that vulnerable libraries can be updated without having to
update all of the static binaries that might use them.
4. I've probably forgotten the other factors...
As for performance, we felt that the typical use of FreeBSD did not fall
into the category of doing forkbomb performance tests. While there might
certainly be users that do continuously create lots of short-lived
processes, we felt that the above benefits outweighed this, but left the
door open for tailoring to this need (recompiling world with
NO_DYNAMICROOT).
If people really are concerned with the VM microbenchmarks associated with
complex process maps, then hopefully this is a challenge to look for
optimizations there.
Scott
dy...@iquest.net
featured /rescue should also help to counterbalance this (but maybe
the upgrade would be selective?) There are reasons for /usr/bin
and /usr/sbin (along with /usr/local, /usr/share.)
Disk space is CHEAP and grandfathering 20MB hard drives (or the
yr2000 equivalent of 4GB hard drives) is nice -- but repartitioning
them with more reasonable root allocations seems like a good
idea anyway :-). 50MB root was silly when 2-4GB disks were common.
Continuing to pay for a poorly made partitioning decision in the past
seems to be unworthy (in my opinion.)
>
> 2. NSS support out-of-the-box: Again, this is a user-experience issue
> in that forcing NSS users to recompile world was seen as a potential
> roadblock to them.
>
The cool thing about properly implemented shared libs is that not everything
has to be shared. Even the kernel supports runtime loading/addition
of modules, and the NSS sounds like a good candidate for a library
feature. Burdening everything with the sparse .data associated with
libc seems to be a waste (but, perhaps the performance that was carefully
crafted into the codebase is no longer important?) Really -- it is
possible that performance isnt' important anymore, and fully accepting
the ongoing loss of performance for features (without careful tradeoffs)
might be an appropriate strategy. (In the case of the NSS stuff, it
shouldn't require everything to be built dynamic.)
>
> 3. Binary security updates: there is a lot of interest in providing a
> binary update mechanism for doing security updates. Having a dynamic
> root means that vulnerable libraries can be updated without having to
> update all of the static binaries that might use them.
>
The additional hole of exploiting the system through the shared libs
is a negative tradeoff.
>
> 4. I've probably forgotten the other factors...
>
> As for performance, we felt that the typical use of FreeBSD did not fall
> into the category of doing forkbomb performance tests.
>
Nope - even shell execution times (e.g. long shell scripts) will see a
difference. Prejudicial comments like 'fork-bombs' and
'microbenchmarks' aren't really fair. Proper benchmarking is
time consuming and analysis is tricky. System loading conditions
are difficult to control (for experiments.)
Note that alot of the lost performance is hidden by the VM code (e.g. one
of the purposes for prezeroing on SMP box -- before SMP was fine grained.)
The cost of sparse data and needless inheritance of modified pages
has performance hits that are incredibly difficult to accurately (and
honestly) measure. Anyone can create 'preordained' benchmark results.
This is one reason why I am loathe to get into the FreeBSD performance
game again -- it is much more difficult than a few micro-benchmarks :-)
that simply measure a fork exec time.
John
Colin Percival
>Our rationale for encouraging Gordon is as follows:
>
>1. 4.x upgrade path: As we approach 5-STABLE, a lot of users might want
> to upgrade from 4-STABLE. Historically in 4.x, the / partition has
> been very modest in size. One just simply cannot cram the bloat that
> has grown in 5.x into a 4.x partition scheme. Of course there is the
> venerable 'dump - clean install - restore' scheme, but we were looking
> for something a little more user-friendly.
Of course, making / dynamic results in added complication of removing
old libraries from /usr/lib, now that some of them have moved to /lib...
>3. Binary security updates: there is a lot of interest in providing a
> binary update mechanism for doing security updates. Having a dynamic
> root means that vulnerable libraries can be updated without having to
> update all of the static binaries that might use them.
As far as I'm concerned, this is a non-issue. Identifying which static
binaries need to be replaced is now a solved problem, replacing them is
easy, and if binary patches are used, there is effectively no impact on
bandwidth usage either.
On the issue of performance, however: I know people have benchmarked
fork-bombs, but has anyone done benchmarks with moderate numbers of
long-lived, library-intensive, processes? It seems to me that dynamic
linking could have caching advantages.
Colin Percival
masta
dy...@iquest.net wrote:
> Scott Long said:
>
>>On Tue, 18 Nov 2003 dy...@iquest.net wrote:
>>
>>>M. Warner Losh said:
>>>
>>>>In message: <200311181307....@dyson.jdyson.com>
>>>> dy...@iquest.net writes:
>>>>: It really doesn't make sense to arbitrarily cut-off a
>>>>: discussion especially when a decision might be incorrect.
>>>>
>>>>I'd say that good technical discussion about why this is wrong would
>>>>be good. However, emotional ones should be left behind. Except for
>>>>John's message, most of the earlier messages have been more emotional
>>>>than technical.
>>>>
>>>>John, do you have any good set of benchmarks that people can run to
>>>>illustrate your point?
>>>>
>>>
>>>I'll see if I can find some of my old benchmarks (from memory,
>>>not disk -- lots of stuff has rotted away.). I had hoped
>>>to avoid doing much in the way of proof. Pointing to the
>>>much more complex process map along with the implication
>>>for significantly higher overhead :-). (The VM code generally
>>>gets slower with more complex process maps and more memory used.
>>>For smallish programs -- including those with a few shared libs,
>>>changing the VM code won't help much, because the cost is built
>>>in to the complexity of the process image.)
>>
>>I'm glad that real arguments are finally starting to surface =-)
>>
>>Our rationale for encouraging Gordon is as follows:
>>
>>1. 4.x upgrade path: As we approach 5-STABLE, a lot of users might want
>> to upgrade from 4-STABLE. Historically in 4.x, the / partition has
>> been very modest in size. One just simply cannot cram the bloat that
>> has grown in 5.x into a 4.x partition scheme. Of course there is the
>> venerable 'dump - clean install - restore' scheme, but we were looking
>> for something a little more user-friendly.
>>
>
[snip]
>
>>2. NSS support out-of-the-box: Again, this is a user-experience issue
>> in that forcing NSS users to recompile world was seen as a potential
>> roadblock to them.
>>
>
[snip]
>
>>3. Binary security updates: there is a lot of interest in providing a
>> binary update mechanism for doing security updates. Having a dynamic
>> root means that vulnerable libraries can be updated without having to
>> update all of the static binaries that might use them.
>>
>
[snip]
>
>
>>4. I've probably forgotten the other factors...
>>
>>As for performance, we felt that the typical use of FreeBSD did not fall
>>into the category of doing forkbomb performance tests.
>>
>
One of ther things he might have "forgot" to mention is dynamic tricks
releated to PAM, which sorta falls in the same league as NSS working out
of the box. It was worth mentioning IMHO.
Besides, I see nothing preventing anybody from building their system with
static worlds, and there is nothing stopping anybody from putting /rescue
in the PATH before /bin or /sbin.
__ __ _
| \/ | __ _ ___| |_ __ _
| |\/| |/ _` / __| __/ _` |
| | | | (_| \__ \ || (_| |
|_| |_|\__,_|___/\__\__,_|
unzip ; strip ; touch ; finger ; mount ; fsck ; more ; yes ; umount ; sleep
ma...@wifibsd.org
http://wifibsd.org
Scott Long
> featured /rescue should also help to counterbalance this (but maybe
> the upgrade would be selective?) There are reasons for /usr/bin
> and /usr/sbin (along with /usr/local, /usr/share.)
>
> Disk space is CHEAP and grandfathering 20MB hard drives (or the
> yr2000 equivalent of 4GB hard drives) is nice -- but repartitioning
> them with more reasonable root allocations seems like a good
> idea anyway :-). 50MB root was silly when 2-4GB disks were common.
>
> Continuing to pay for a poorly made partitioning decision in the past
> seems to be unworthy (in my opinion.)
/boot has grown quite large too and threatens to be unbounded in size as
times go on. Shaving off the 30-40MB of size in /bin and /sbin can
help alleviate this, even on system formatted in 5.x partition sized.
This argument wasn't the most compelling one by itself, but it played a
part in the decision so I included it here.
>
> >
> > 2. NSS support out-of-the-box: Again, this is a user-experience issue
> > in that forcing NSS users to recompile world was seen as a potential
> > roadblock to them.
> >
> The cool thing about properly implemented shared libs is that not everything
> has to be shared. Even the kernel supports runtime loading/addition
> of modules, and the NSS sounds like a good candidate for a library
> feature. Burdening everything with the sparse .data associated with
> libc seems to be a waste (but, perhaps the performance that was carefully
> crafted into the codebase is no longer important?) Really -- it is
> possible that performance isnt' important anymore, and fully accepting
> the ongoing loss of performance for features (without careful tradeoffs)
> might be an appropriate strategy. (In the case of the NSS stuff, it
> shouldn't require everything to be built dynamic.)
Unfortunately, our dynamic linker prevents dlopen() and friends from
working in a static binary. There was some talk about having NSS
lookups happen through a proxy process, but our implementation just
plain wasn't done that way, and no one has stepped forward with
alternate code.
NSS modules can affect a surprisingly large number of utilities, including
the shells. Selectively choosing which utilities to link static vs.
shared would have resulted in quite a patchy result that probably would
not have had a significant benefit.
>
> >
> > 3. Binary security updates: there is a lot of interest in providing a
> > binary update mechanism for doing security updates. Having a dynamic
> > root means that vulnerable libraries can be updated without having to
> > update all of the static binaries that might use them.
> >
> The additional hole of exploiting the system through the shared libs
> is a negative tradeoff.
Exploits in libraries happen though. The LD_LIBRARY_PATH attack is an old
one that most Unixes are hopefully hardened against.
>
> >
> > 4. I've probably forgotten the other factors...
> >
> > As for performance, we felt that the typical use of FreeBSD did not fall
> > into the category of doing forkbomb performance tests.
> >
> Nope - even shell execution times (e.g. long shell scripts) will see a
> difference. Prejudicial comments like 'fork-bombs' and
> 'microbenchmarks' aren't really fair. Proper benchmarking is
> time consuming and analysis is tricky. System loading conditions
> are difficult to control (for experiments.)
I apologize for the poor choice of words, I certainly did not mean to
insult you or anyone else.
>
> Note that alot of the lost performance is hidden by the VM code (e.g. one
> of the purposes for prezeroing on SMP box -- before SMP was fine grained.)
> The cost of sparse data and needless inheritance of modified pages
> has performance hits that are incredibly difficult to accurately (and
> honestly) measure. Anyone can create 'preordained' benchmark results.
>
> This is one reason why I am loathe to get into the FreeBSD performance
> game again -- it is much more difficult than a few micro-benchmarks :-)
> that simply measure a fork exec time.
>
> John
>
>
>
I agree that reaching for improved performance is a beneficial goal, and
that tradeoffs of performance vs. features must be carefully considered
and balanced. With FreeBSD's popularity as a server platform, things like
integrated NSS were too long in coming and were very compelling reasons to
go this route. So it looks like the performance vs. features see-saw has
tipped from one side to the other. Backing out dynamic root will not help
rebalance it though. Again, I hope that this will inspire some graduate
student somewhere to experiment with optimizing the VM system in this
direction.
Scott
Garance A Drosihn
>
> It really doesn't make sense to arbitrarily cut-off a
> discussion especially when a decision might be incorrect.
All I wanted to cut off was the claim that this decision had
not been discussed publicly before. It was also annoying that
most the recent complaints (before your message) were issues
that had been explicitly discussed and addressed before the
decision had been reached. Many people seem to be complaining
on what they think we did, as opposed to what we actually did.
> If there hadn't been a noticed increase in cost by using
> all-shared-libs, then the measurements were done
> incorrectly. If the decision is made based upon allowing
> for 1.5X (at least) times increase in fork/exec times, and
> larger memory usage (due to sparse allocations), ...
I do remember some comments about benchmarks, and it was
true that the all-dynamic bin/sbin does come out slower. I
don't remember if the benchmarks were ours or from NetBSD's
investigation. However, I think we measured increase in
overall time for some set of commands, instead of "increase
in the fork() routine". Thus, the penalty observed was much
less than 50%. I think it was under 2%, but I don't remember
the exact number. When we're dealing with a 100% increase
in the cost of compiling something with the newer gcc, the
increase due to this change seemed pretty insignificant...
It is probably true that there are several commands where we
would see a worthwhile performance benefit by static linking,
and which have no need for things like dynamic PAM modules.
But commands which care about userids or group information
would need to stay dynamic (IMO), and I think that means all
shells.
Also, when it comes to security fixes, it would be nice for
binary upgrades if all we had to do was replace one library,
instead of replacing every command which is statically-linked
with the problem routine.
The announcement of this change may have played up disk-savings
more than it should have, because we weren't doing this to save
disk space. It was just that the disk savings were a very nice
side-effect. It's pretty rare that we can talk about the system
getting smaller! :-)
--
Garance Alistair Drosehn = g...@gilead.netel.rpi.edu
Senior Systems Programmer or g...@freebsd.org
Rensselaer Polytechnic Institute or dro...@rpi.edu
Garance A Drosihn
>
>Besides, I see nothing preventing anybody from building
>their system with static worlds,
true...
>and there is nothing stopping anybody from putting /rescue
>in the PATH before /bin or /sbin.
Note that this will not have the same performance as the
older all-static setup, because each binary in /rescue
is a crunchgen'ed combination of *all* the commands there.
They're all hardlinked together. Eg:
(332) # ls -l /bin/cp /rescue/cp
-r-xr-xr-x 1 root wheel 113372 Nov 16 02:29 /bin/cp*
-r-xr-xr-x 131 root wheel 3807836 Nov 16 02:30 /rescue/cp*
I have no idea what kind of a difference that would make,
I'm just saying that it is different... :-)
Scott Long
> At 17:06 18/11/2003 -0700, Scott Long wrote:
> >
> >1. 4.x upgrade path: As we approach 5-STABLE, a lot of users might want
> > to upgrade from 4-STABLE. Historically in 4.x, the / partition has
> > been very modest in size. One just simply cannot cram the bloat that
> > has grown in 5.x into a 4.x partition scheme. Of course there is the
> > venerable 'dump - clean install - restore' scheme, but we were looking
> > for something a little more user-friendly.
>
> old libraries from /usr/lib, now that some of them have moved to /lib...
The magic for solving this exact problem already exists in the source
upgrade path. Fixing this for the binary upgrade path (via sysinstall) is
still under investigation. It is mitigated by the fact the systems that
are binary-upgraded have their library search path set to look at /lib
first. Leaving around stale libraries is of course bad, and we will
hopefully come up with a solution soon.
>
> >3. Binary security updates: there is a lot of interest in providing a
> > binary update mechanism for doing security updates. Having a dynamic
> > root means that vulnerable libraries can be updated without having to
> > update all of the static binaries that might use them.
>
> As far as I'm concerned, this is a non-issue. Identifying which static
> binaries need to be replaced is now a solved problem, replacing them is
> easy, and if binary patches are used, there is effectively no impact on
> bandwidth usage either.
Bandwidth is still a concern for a lot of people, and this has the
potential to save significant bandwidth in many situations.
>
> On the issue of performance, however: I know people have benchmarked
> fork-bombs, but has anyone done benchmarks with moderate numbers of
> long-lived, library-intensive, processes? It seems to me that dynamic
> linking could have caching advantages.
>
> Colin Percival
>
>
>
I can't comment well here. To my untrained mind, I would think that more
sharing of libc would help with memory pressure, but I'm just making
simple assumptions.
Scott
dy...@iquest.net
> On Tue, 18 Nov 2003 dy...@iquest.net wrote:
> > The cool thing about properly implemented shared libs is that not everything
> > has to be shared. Even the kernel supports runtime loading/addition
> > of modules, and the NSS sounds like a good candidate for a library
> > feature. Burdening everything with the sparse .data associated with
> > libc seems to be a waste (but, perhaps the performance that was carefully
> > crafted into the codebase is no longer important?) Really -- it is
> > possible that performance isnt' important anymore, and fully accepting
> > the ongoing loss of performance for features (without careful tradeoffs)
> > might be an appropriate strategy. (In the case of the NSS stuff, it
> > shouldn't require everything to be built dynamic.)
>
> Unfortunately, our dynamic linker prevents dlopen() and friends from
> working in a static binary. There was some talk about having NSS
> lookups happen through a proxy process, but our implementation just
> plain wasn't done that way, and no one has stepped forward with
> alternate code.
>
things then. (That limitation has been operative for years.)
Making that decision 6months ago (or whenever the decision was
made) seems to result in sidestepping the best solution (esp
when 6months is alot of time to fix such limitations.)
Again, I cannot take the responsibility, and mostly trying to
keep 'improvements' from undoing previous work. I am worried
(really) that there'll be a progressive loss of performance in
order to gain features that could have been implemented with
less impact. There was ALOT of work in trying to make FreeBSD
very performant, and throwing that away with incrementalism
seems to be wasteful. I am NOT meaning to negatively criticize,
because that is quite seductive, and I am NOT suggesting that
FreeBSD should have the Plan9 mimalist ethic. :-).
Favorite 'tricky' features will come and go -- but basic system
performance gets harder and hard to recover as time goes on!!!
I guess that all OSes have a size/performance curve where they
get bigger and fatter, SOMETIMES (not always) losing previous
attributes for new features. Just don't let the performance loss
go too far. IMO, I wouldn't have supported everything be built
shared if there was a POTENTIAL mitigating solution of a proper
dlopen implementation. DG would likely have had a 'fit.'
Perhaps there should be a FreeBSD performance (and footprint)
responsibility officer before it becomes too bloated to recover? :-).
John
dy...@iquest.net
> At 8:07 AM -0500 11/18/03, dy...@iquest.net wrote:
> >
> > It really doesn't make sense to arbitrarily cut-off a
> > discussion especially when a decision might be incorrect.
>
> All I wanted to cut off was the claim that this decision had
> not been discussed publicly before. It was also annoying that
> most the recent complaints (before your message) were issues
> that had been explicitly discussed and addressed before the
> decision had been reached. Many people seem to be complaining
> on what they think we did, as opposed to what we actually did.
>
>
> > If there hadn't been a noticed increase in cost by using
> > all-shared-libs, then the measurements were done
> > incorrectly. If the decision is made based upon allowing
> > for 1.5X (at least) times increase in fork/exec times, and
> > larger memory usage (due to sparse allocations), ...
>
> I do remember some comments about benchmarks, and it was
> true that the all-dynamic bin/sbin does come out slower.
>
Please remember: that several people worked VERY VERY hard
to make FreeBSD performant. It would be very difficult to
recover the performance after many cascaded performance losses.
Throwing away performance is almost an irrevocable decision, and
features (once added) are difficult to remove or substantially
rearchitect (for performance.) (Once a slow, but workable
functionality is added, the cost to recover lost performance
after the fact is very very high.)
Gordon Tetlow
--Hjg+pC/5+IEur/Yc
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Tue, Nov 18, 2003 at 08:03:23PM -0500, dy...@iquest.net wrote:
>=20
> However, PAM and NSS 'tricks' really seem to be exactly that,
> and certainly worthy of special builds. However, that isn't
> necessary, yet still not building everything with a shared
> libc.
Things like nss_ldap (which is used *heavily* at my place of employment)
are some reasons that FreeBSD doesn't make it into more places. It was
the reason why FreeBSD isn't being used here. Calling them 'tricks'
(and succumbing to the name calling you wanted to avoid) doesn't change
the fact that every major contender (IRIX, Solaris, Linux to name a few)
all support this feature set.
-gordon
--Hjg+pC/5+IEur/Yc
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (FreeBSD)
iD8DBQE/us1ZRu2t9DV9ZfsRAh3CAJ9Kzz6yr65aM+dWY53p8SZ5+Q6X6wCeIeTw
8LmEZp5OzgetkGt9AWwqmwU=
=PD3F
-----END PGP SIGNATURE-----
--Hjg+pC/5+IEur/Yc--
dy...@iquest.net
> On Tue, Nov 18, 2003 at 08:03:23PM -0500, dy...@iquest.net wrote:
> >
> > However, PAM and NSS 'tricks' really seem to be exactly that,
> > and certainly worthy of special builds. However, that isn't
> > necessary, yet still not building everything with a shared
> > libc.
>
> Things like nss_ldap (which is used *heavily* at my place of employment)
> are some reasons that FreeBSD doesn't make it into more places. It was
> the reason why FreeBSD isn't being used here. Calling them 'tricks'
>
had elided :-). Please quote the message that set-up the context for
the usage.
>
> (and succumbing to the name calling you wanted to avoid) doesn't change
> the fact that every major contender (IRIX, Solaris, Linux to name a few)
> all support this feature set.
>
As discussed before, it DEFINITELY isn't necessary to dynamically link
EVERYTHING to implement your favorite feature.
Not everyone needs PAM/NSS, even though everyone needs memory management
and scarce resource allocation. Why build in the overhead for everyone,
and make it UNNCESSARILY worse (e.g. dynamically link libc when you want
NSS or PAM?) Of course, there was a
development resource limitation, but the decision (discussion) was
made approx 6months ago? (Enough time to solve the problem without
a GLOBAL performance hit.)
Matthew Dillon
:Our rationale for encouraging Gordon is as follows:
:
:1. 4.x upgrade path: As we approach 5-STABLE, a lot of users might want
: to upgrade from 4-STABLE. Historically in 4.x, the / partition has
: been very modest in size. One just simply cannot cram the bloat that
: has grown in 5.x into a 4.x partition scheme. Of course there is the
: venerable 'dump - clean install - restore' scheme, but we were looking
: for something a little more user-friendly.
This argument would apply to very old 4.x users but not to anyone who
installed it as of March 2001. I bumped the nominal size of the
root partition to 128MB in 1.98.2.7 of sysinstall/label.c.
Prior to that Jordan had bumped the root partition size to 100MB
in 1.98.2.3 in March 2001. It was 50MB before then, which is too small
even for 4.x.
:2. NSS support out-of-the-box: Again, this is a user-experience issue
: in that forcing NSS users to recompile world was seen as a potential
: roadblock to them.
Users have to recompile the world anyway, I don't think NSS is an issue.
Or to put it another way: Nobody in their right mind should be
contemplating upgrading a 4.x system to 5.x for production purposes.
There will simply be too much 4.x cruft left over. Upgrading
really requires a wipe and reinstall in this instance.
I seem to recall that the main reason 5.x went to a dynamic /bin was
to work around a terribly broken PAM design. The other issues were
just eye candy compared to the PAM problems. Personally speaking, I
think the solution is to fix PAM instead :-)
:3. Binary security updates: there is a lot of interest in providing a
: binary update mechanism for doing security updates. Having a dynamic
: root means that vulnerable libraries can be updated without having to
: update all of the static binaries that might use them.
Non-issue if you have an automatic security update mechanism or script
to do the work for the user, which we don't. Even so, /bin and /sbin are
sufficiently self-contained in the source hierarchy that recompiling and
reinstalling them is not a big deal.
I think the security argument is a red-herring because, frankly,
security problems are far more prevalient with ports and larger services
(Apache, sendmail, sshd, etc... mostly in /usr/bin and /usr/local),
and not as big an issue for /bin and /sbin.
:As for performance, we felt that the typical use of FreeBSD did not fall
:into the category of doing forkbomb performance tests. While there might
:certainly be users that do continuously create lots of short-lived
:processes, we felt that the above benefits outweighed this, but left the
:door open for tailoring to this need (recompiling world with
:NO_DYNAMICROOT).
fork() is a non-issue (forking overhead for a dynamic executable imposes
a slight time penalty but does not really impose any resource penalty).
However, I personally believe that anything run by a shell script is
an issue in regards to performance, and anything you exec multiple
separate copies of is an issue in regards to memory overhead. A system
running a large number of JAIL'd environments will wind up with a larger
number of swap-managed dirty pages.
But, all this aside, the reason I am decidedly against a dynamic /bin is
simply one of system recovery and safety. I've blown up /usr so many
times over the years that had /bin and /sbin been dynamic I would have
been in real trouble on many occassions.
If I recall correctly, some people have proposed and/or implemented some
sort of emergency fall back or safety bin in 5.x to make up for this
issue. I would humbly submit that this simply acknowledges that a dynamic
/bin and /sbin is a bad idea in the first place.
I do not intend to make /bin or /sbin dynamic in DragonFly by default.
I don't mind adding support for those people who want it, but I am not
going to make it the default.
-Matt
Matthew Dillon
<dil...@backplane.com>
Matthew Dillon
:> As far as I'm concerned, this is a non-issue. Identifying which static
:> binaries need to be replaced is now a solved problem, replacing them is
:> easy, and if binary patches are used, there is effectively no impact on
:> bandwidth usage either.
:
:Bandwidth is still a concern for a lot of people, and this has the
:potential to save significant bandwidth in many situations.
:Scott
I would not consider this a viable argument since binary downloads are
usually compressed. A compressed /bin stacks up as follows (from 4.x):
-rw-r--r-- 1 root wheel 4034560 Nov 18 18:34 /tmp/x1.tar static
-rw-r--r-- 1 root wheel 849920 Nov 18 18:34 /tmp/x2.tar dynamic
-rw-r--r-- 1 root wheel 1860215 Nov 18 18:34 /tmp/x1.tgz static
-rw-r--r-- 1 root wheel 354576 Nov 18 18:34 /tmp/x2.tgz dynamic
So you are talking about 1.5 MBytes less bandwidth, which is nothing
compared to the cost of doing a full install over the net. Yah, yah,
/sbin too... but you get the idea.
It certainly isn't enough of a difference to justify going to a
dynamic /bin and /sbin. I'm not saying there aren't other arguments,
just that this isn't one of them.
Matthew Dillon
:times go on. Shaving off the 30-40MB of size in /bin and /sbin can
:help alleviate this, even on system formatted in 5.x partition sized.
:This argument wasn't the most compelling one by itself, but it played a
:part in the decision so I included it here.
You aren't saving that much. I don't have a 5.x box handy but on
a 4.x box a static /bin is only 4MB and /sbin is only 12MB. The dynamic
versions will save you around 12MB is my guess.
-Matt
dy...@iquest.net
I suspect that my position has been expressed
adequately.
Further discussion might become divisive, but
a decision that incurs the overhead of performance
or a rebuild on the default user base seems
wrong (JUST MY OPINION.) It took ALOT of WORK
(person years) to make FreeBSD perform as well
as it does.
BOTH the add-on crew and the general user base can
have the performance and feature set without
rebuilding, but the decision was apparently made
to impose the cost of performance or rebuild and
binary maintenance on the default user base.
It makes more sense to have appropriately
upgraded the system (by the NSS project) to avoid
the performance hit by others and also provide
the feature set. Apparently (I haven't fully
analysed this) implementing the dlopen stuff for
non-dynamic programs would have helped to mitigate
this issue. (It might have put more burden on the
NSS/PAM/whatever addon projects, but those are
indeed addons that shouldn't take ANYTHING away
from the rest of the project.)
I am suggesting that the NSS crew and those who
are concerned about performance can BOTH have
the results that they wish for.
'All or nothing' creates divisiveness, and in these
discussions it is TOO EASY to fall into that trap.
I am not suggesting the loss of the new NSS stuff,
but also suggest that ANY loss of performance when
it can be avoided, is unwise.
My opinion is known, and hopefully the loss of
hard earned performance with person-years of work
won't happen as time goes on. A little loss isn't
that bad, but how much loss is too much loss (esp when
not necessary?)
<EOT>
John
Garance A Drosihn
>
> So you are talking about 1.5 MBytes less bandwidth,
> which is nothing compared to the cost of doing a full
> install over the net. Yah, yah, /sbin too... but you
> get the idea.
Many freebsd users (me for one) are still living on a modem,
where even one bump of 1.5 meg is a significant issue...
Remember that the issue we're talking about is security
updates, not full system upgrades. "Everyone" would want
the security updates, even if they're on a slow link.
--
Garance Alistair Drosehn = g...@gilead.netel.rpi.edu
Senior Systems Programmer or g...@freebsd.org
Rensselaer Polytechnic Institute or dro...@rpi.edu
Robert Watson
> There might be a certain 'coolness' WRT dynamically linking everything,
> but the overhead is certainly measurable. If the object is to maximally
> 'share', then for shells the FreeBSD VM shares maximally without using
> shared libs (perhaps there is a lost know-how about how aggressively the
> FreeBSD VM implements parent/child and exec based sharing?)
>
> I'll try to put together a few simple benchmarks -- mostly from my
> defective memory. I had recently refreshed myself on this issue (but
> lost again due to disk hardware disasters and being clobbered by vinum
> problems -- which I have given up on.) Gimme a day or so -- I have
> several other things in my queue.
I guess one of the key observations to make here is that the vast majority
of applications in FreeBSD have been dynamically linked for years. The
only thing that has changed recently is a few binaries in /bin and /sbin.
Of these binaries, the vast majority are never run for most systems, are
run once during boot, or extremely infrequently in response to an
administrative action where minor differences in latency will be
unnoticeable. This would include applications like ping, mount_*,
fsirand, newfs, swapon, kldload, chflags, rcorder, quotacheck, etc. The
"once during boot" case is interesting in the aggregate, but most of the
binaries in question should probably have been linked dynamically long ago
simply because there's no real benefit to making them statically linked.
So I think this leaves three interesting cases:
(1) Shells, which are run for extended periods of time, and which are
often run in a large number (propotional to #users logged in, or
#windows open). I'm not to interested in this argument simply because
the applications most people are interested in using FreeBSD for are
already dynamically linked: Apache, databases, perl, XWindows, KDE,
... The vast majority of long-lived processes are already dynamically
linked.
(2) Shells again, because they will be fork()d and exec()d frequently
during heavily scripted activities, such as system boot, periodic
events, large make jobs, etc. And presumably the only shell of
interest is sh, although some of the supporting non-builtin binaries
may also be of interest.
(3) Other binaries, such as mount_*, etc, which aren't run very often, but
when they are run, will be run in aggregate with many other similar
binaries, such as during system boot. The cost of one binary going
dynamic is presumably extremely small, but if you suddenly make many
binaries dynamic, it's presumably much more noticeable.
Some macrobenchmark results that would probably be interesting to see
(some of which I know have already been looked at as part of this
discussion):
- With a move to rcNG (/etc/rc.d), our boot process is now probably quite
a bit more sensitive to the net performance change from dynamic linking.
This would be at least one benchmark that would be interesting to look
at (and I understand has been looked at by those exploring dynamic
linking).
- The impact on large multi-part build tasks, such as buildworld, is also
interesting. Turning on process accounting shows that 'sh' is run by
make(1) once for each task it spawns off during the build. A
macrobenchmark would be helpful to look at whether there's a positive or
negative impact.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Network Associates Laboratories
Matthew Dillon
:Many freebsd users (me for one) are still living on a modem,
:where even one bump of 1.5 meg is a significant issue...
:
:Remember that the issue we're talking about is security
:updates, not full system upgrades. "Everyone" would want
:the security updates, even if they're on a slow link.
:
:--
:Garance Alistair Drosehn = g...@gilead.netel.rpi.edu
I really have to disagree with this comment. By your reasoning saving
a few bytes could be argued as being 'significant'. I've done net
installs over slow modems before.. hell, I ran cvsup over a slow modem
for over a year! My problem was never / or /bin. Not once. Not ever.
I really don't care about a measily few megabytes relative to the size
of the whole dist, and I doubt modem users of today care much either
when you put it in that perspective. Sure, if you could save 50% of
the bandwidth over the whole dist it would be significant. But
12 MBytes? No.
The reason your argument make little sense is that there are plenty of
OTHER ways you can make modem user's lives easier that have not
been implemented. We aren't talking about a few measily megabytes here,
we are talking about not making modem users have to wait sitting in front
of their terminal staring at a slow download for hours before they even
know whether their system will boot the dist!
A two-stage install... basic kernel and /, reboot, then install the rest,
would have a major impact on modem users. A thousand times greater impact
then the few measily megabytes. Modem users don't mind waiting as long
as they don't have to sit in front of the terminal while doing so. Once
a basic dist is up and running on a modem user's machine believe me, they
will happily go off and do something else while waiting for the rest of
the bits to download and install because they know if something blows up
they won't have to start all over again.
-Matt
Frank Mayhar
gasoline or at least a few oily rags. :-)
For what it's worth, I've been running FreeBSD literally since before its
inception, when it was merely a collection of patches to 386BSD 0.1. I'm
also a longtime kernel guy so I do have a bit of a professional opinion.
I tend to think that making the binaries in / dynamic is a bad idea. It is
one of those that is floated from time to time, and ends up being voted
down, often after someone has to recover a hosed system without having the
appropriate shared libraries available. It seems like a good idea on the
surface, but I really think it's an "if it ain't broke, don't fix it"
situation. The amount of space it will save is trivial relative to the
size of modern disks and the size of the rest of the distribution. I
really don't care that much about how fast the system boots; even if you
slowed it to half its current speed it would still be much faster than
anything Microsoft has produced.
If you're upgrading "security libraries" (or whatever), it's really as
easy to upgrade the binaries in /bin and /sbin as it is to replace the
libraries. There's just not that much there.
Both Matt and John are right. You guys are trying to solve a non-problem.
Please don't. There are much better things on which to spend your time.
--
Frank Mayhar fr...@exit.com http://www.exit.com/
Exit Consulting http://www.gpsclock.com/
http://www.exit.com/blog/frank/
David Schultz
> (2) Shells again, because they will be fork()d and exec()d frequently
> during heavily scripted activities, such as system boot, periodic
> events, large make jobs, etc. And presumably the only shell of
> interest is sh, although some of the supporting non-builtin binaries
> may also be of interest.
This is the only performance argument that truly seems compelling,
particularly since fork()/exec() performance for dynamic binaries
in FreeBSD really sucks as compared to Solaris and AIX (and
presumably other systems that have gone dynamic long ago).
I think this is mostly an aspect of the dynamic linker and the
lack of prebinding rather than performance problems in the VM system.
Most of the other arguments both for and against seem less
important. The disk space argument that people have been giving
is particularly bogus given that disk capacity is increasing 60%
per year and the size of /bin isn't. The NSS argument is only
valid because NSS was implemented specifically to require dlopen()
rather than a daemon, so that disadvantage is in no way
fundamental.
The best reason in my mind for going dynamic hasn't even been
mentioned much. In a fully dynamic system, it is possible to
change kernel ABIs without breaking anything as long as the kernel
and libc are in sync. Thus, we wouldn't have three versions of
statfs() and two versions of accept(), along with other cruft in
the kernel to support binaries going back to 4.2BSD. The
compatX.Y libraries would suffice. Also, developers would have
greater flexibility to extend or otherwise improve system
interfaces between minor releases. Unfortunately, this would mean
giving up on ABI compatibility for static binaries entirely, so I
doubt it would fly anytime soon. Moreover, the performance issues
we have with dynamic binaries now really are significant, at least
as of a few months ago. (Maybe mdodd's prebinding work addresses
some of that.)
Dragonfly has addressed the same issue in a different, perhaps
more elegant way, wherein even static binaries have forward and
reverse ABI compatibility. But it does still require the kernel
to mock up legacy fields in shared structures forever, and it
doesn't permit me to, for instance, take advantage of the fact
that mmap(), msync(), mprotect(), minherit(), madvise(), and
mincore() have essentially the same access checks and rewrite them
to use a common monitor.
David Schultz
> > The additional hole of exploiting the system through the shared libs
> > is a negative tradeoff.
>
> Exploits in libraries happen though. The LD_LIBRARY_PATH attack is an old
> one that most Unixes are hopefully hardened against.
FreeBSD had a lingering LD_LIBRARY_PATH-related vulnerability
until Sunday, actually[1]. ;-) But I don't mean to dispute your
point. Like most of the other arguments in this bikeshed, there
is nothing fundamental about the LD_LIBRARY_PATH problem---nothing
that can't be fixed easily.
[1] The bug is either that nologin(8) respected LD_LIBRARY_PATH or
that sshd(8) and login(1) allow environment poisoning, depending
on your point of view.
M. Warner Losh
Garance A Drosihn <dro...@rpi.edu> writes:
: At 9:02 PM -0500 11/18/03, dy...@iquest.net wrote:
: > Of course, there was a development resource limitation,
: >but the decision (discussion) was made approx 6months ago?
: >(Enough time to solve the problem without a GLOBAL
: >performance hit.)
: Well, yes, perhaps. But there is that issue of "development
: resource limitation". Back when we did debate this publicly,
: no one stepped forward and said "I have the time to implement
: a better solution". Thus, we went with this solution.
And it still isn't too late for someone to step forward with a better
solution. Or to develop one. The main reason we went with dynamic /
was to be able to get dynamic libary/loading support for newer
authentication and user technologies. The size advantage is one minor
added benefit.
: Speaking as to what we can do right now, I would not want to
: delay the 5.x-stable branch by adding some project right now
: to start writing an alternate PAM/NSS solution. If someone
: wants to write one for 6.0, that would be good. There is
: nothing in this solution which would cause problems for
: some later solution. Disk space will only get cheaper.
I tend to agree. If there's a performance regression with the dynamic
change, then the right solution isn't to yell 'back it out', but
rather to build a better mouse trap. FreeBSD gets better through
these tentions and people scratching an itch. FreeBSD doesn't get
better by stagnating and never changing anything.
With all things it is a balance. Sometimes it tips a little this way
or that. As our user base changes, the balance point changes. If
FreeBSD isn't willing to adapt to the new balance point, it runs the
risk of losing relevance.
Finally, I'll just observe that most of our users care most about
apache, which has been dynamically linked for a long time.
Warner
Robert Watson
On Tue, 18 Nov 2003, David Schultz wrote:
> On Tue, Nov 18, 2003, Robert Watson wrote:
> > (2) Shells again, because they will be fork()d and exec()d frequently
> > during heavily scripted activities, such as system boot, periodic
> > events, large make jobs, etc. And presumably the only shell of
> > interest is sh, although some of the supporting non-builtin binaries
> > may also be of interest.
>
> This is the only performance argument that truly seems compelling,
> particularly since fork()/exec() performance for dynamic binaries in
> FreeBSD really sucks as compared to Solaris and AIX (and presumably
> other systems that have gone dynamic long ago). I think this is mostly
> an aspect of the dynamic linker and the lack of prebinding rather than
> performance problems in the VM system.
Some of Matthew Dodd's exploration of prebinding seems to indicate it's a
big win for large applications with many libraries (XWindows, KDE,
OpenOffice, whathaveyou), but that the base cost of loading relocation
caches overwhelms the benefits of prebinding. More refinement could
correct this, perhaps. Part of the cost of excessive cost of prebinding
right now is that we almost always map libraries to different addresses
for different programs, which results in a very large relocation cache. A
quick sample on my notebook reveals libc mapped at no less than 28
different base addresses. On systems like Mac OS X, use of a "shared
region" permits not only use of prebinding, but assumptions of common load
addresses for common libraries. In addition, the "shared region" approach
allows the reuse of page table and VM meta-data across many processes.
Leaving aside special-purpose optimizations like that, using some simple
heuristics, we could probably easily reduce the number of load addressed
dramatically, making prebinding a far more feasible approach, and
something we should explore thoroughly.
My feeling is we should all go away for a day or two, and run our favorite
macro-benchmark on our favorite sensitive dynamic linking-sensitive
application. I.e., buildworld, system boot, periodic, or whatever. Attempt
to focus on the actual performance loss, both the cause, and potential
solutions. Returning to a static /bin/sh would be one potential solution.
John Dyson's suggestion of "mixed mode" linking, in which we statically
link known required parts of binaries, but permit dynamic linking of
"plug-in" functionality is another quite reasonable solution, as long as
we take care of the potential issue of "plug-ins" relying on symbols from
a library that is statically linked to the application, but not in the set
of symbols depended on by the application. Remaining with dynamic linking
is another possible "solution" if macrobenchmarks don't reveal an
interesting hit. If I had to guess, I think it probably comes down to
comparing the costs of repeated fork/exec of /bin/sh, and that the "mixed
mode" solution may be the easiest way to address a possible performance
problem while retaining the dynamic linking benefits. But I want to be
sure we address a real-world performance problem, not an anticipated one.
For big applications like KDE, Open Office, et al, prebinding probably is
the right approach, and hopefully we have plenty of time before 5.3 to
refine Matthew's work and see if we can't get things up to speed.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Network Associates Laboratories
_______________________________________________
Kirk Strauser
Content-Transfer-Encoding: quoted-printable
At 2003-11-19T04:43:23Z, "M. Warner Losh" <i...@bsdimp.com> writes:
> The main reason we went with dynamic / was to be able to get dynamic
> libary/loading support for newer authentication and user technologies.
Just a quick interjection:
As one who recently migrated a FreeBSD server from NIS to OpenLDAP
authentication, and who can now see user and group names instead of UIDs and
GIDs in directory listings, I sincerely thank everyone who worked on the
dynamic linking project.
=2D-=20
Kirk Strauser
"94 outdated ports on the box,
94 outdated ports.
Portupgrade one, an hour 'til done,
82 outdated ports on the box."
--=-=-=
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
iD8DBQA/uvg55sRg+Y0CpvERArk4AKCDVrKBYUzc2+743XcvTE8PlDU9MgCgo8eu
1lRjmlc3XS+nYm1u6FFE788=
=tYnd
-----END PGP SIGNATURE-----
--=-=-=--
David Schultz
> On systems like Mac OS X, use of a "shared
> region" permits not only use of prebinding, but assumptions of common load
> addresses for common libraries. In addition, the "shared region" approach
> allows the reuse of page table and VM meta-data across many processes.
> Leaving aside special-purpose optimizations like that, using some simple
> heuristics, we could probably easily reduce the number of load addressed
> dramatically, making prebinding a far more feasible approach, and
> something we should explore thoroughly.
IIRC, Dyson proposed something like that a while ago. It's a neat
idea, particularly if it gets us away from PIC on register-starved
architectures such as the i386.
> John Dyson's suggestion of "mixed mode" linking, in which we statically
> link known required parts of binaries, but permit dynamic linking of
> "plug-in" functionality is another quite reasonable solution, as long as
> we take care of the potential issue of "plug-ins" relying on symbols from
> a library that is statically linked to the application, but not in the set
> of symbols depended on by the application.
Wouldn't the first shared object you loaded instantly pull in
libc.so as a dependency, giving you pieces of both a static and a
dynamic libc? It seems that that would break something...
> For big applications like KDE, Open Office, et al, prebinding probably is
> the right approach, and hopefully we have plenty of time before 5.3 to
> refine Matthew's work and see if we can't get things up to speed.
For these applications, the fork/exec overhead is so far into the
noise and there are so many shared libraries that dynamic linking
wins outright, prebinding or not. Maybe for servers that fork a
lot this would be more contentious.
Garance A Drosihn
>
>My feeling is we should all go away for a day or two, and run
>our favorite macro-benchmark on our favorite sensitive dynamic
>linking-sensitive application.
I wish I had the time and background to implement one solution
that I'd like to benchmark.
have a: chflags ldcache /bin/sh
When the loader goes to execute some binary, it checks for this
ldcache flag. If set, it checks to see if it already has a
cached copy of this program (probably checking based on a key
of inode+lastchg for a match).
If not, it links and loads the file into memory. It saves away
a read-only copy of the result of that load. Then it does the
appropriate magic to create a copy-on-write image of the loaded
file, and executes that.
If it already has a cached copy of that file, well, it just
makes another copy-on-write image of the loaded file, and
executes that. No I/O, no loading, no linking. Just RAM.
Yes, disks have been getting bigger, but so has available memory.
I think we would see a MUCH bigger win by taking advantage of
that memory, than we will ever see by statically-linking system
binaries. On the other hand, I have no idea if the above idea
would be easy to implement in FreeBSD. It also needs to be a
bit smarter than what I described, to cover the dynamic-library
case (eg: the very PAM/NSS issue that we're interested in).
If that isn't workable, I'm sure there are other strategies
which could be imagined. Strategies which will give us more
of a performance boost than static-linking these few programs
will give us.
--
Garance Alistair Drosehn = g...@gilead.netel.rpi.edu
Senior Systems Programmer or g...@freebsd.org
Rensselaer Polytechnic Institute or dro...@rpi.edu
Don Lewis
> At 8:07 AM -0500 11/18/03, dy...@iquest.net wrote:
>
>> If there hadn't been a noticed increase in cost by using
>> all-shared-libs, then the measurements were done
>> incorrectly. If the decision is made based upon allowing
>> for 1.5X (at least) times increase in fork/exec times, and
>> larger memory usage (due to sparse allocations), ...
>
> I do remember some comments about benchmarks, and it was
> don't remember if the benchmarks were ours or from NetBSD's
> investigation. However, I think we measured increase in
> overall time for some set of commands, instead of "increase
> in the fork() routine". Thus, the penalty observed was much
> less than 50%. I think it was under 2%, but I don't remember
> the exact number. When we're dealing with a 100% increase
> in the cost of compiling something with the newer gcc, the
> increase due to this change seemed pretty insignificant...
I thought there were some NetBSD benchmark numbers posted, but after
digging through my mail archives, I now think the results that I'm
remembering were posted by Gordon and were run with rcNG, which is
somewhat more shell intensive than our previous rc system:
On 2 Jun, Gordon Tetlow wrote:
> I'm planning on making a dynamically-linked root partition by 5.2. To
> that end, I'm planning on doing to the following:
>
> Integrate Tim Kientzle's /rescue patches into the tree
> Create /lib and populate with all the libs needed to support dynamically
> linked binaries in /bin and /sbin
> Have a big (probably NO_DYNAMIC_ROOT) knob to switch from static to
> dynamic.
>
> There will be a performance hit associated with this. I did a quick
> measurement at boot and my boot time (from invocation of /etc/rc to
> the login prompt) went from 12 seconds with a static root to 15
> seconds with a dynamic root. I have yet to perform a worldstone on
> it.
I was thinking the difference was smaller than that.
Don Lewis
> (2) Shells again, because they will be fork()d and exec()d frequently
> during heavily scripted activities, such as system boot, periodic
> events, large make jobs, etc. And presumably the only shell of
> interest is sh, although some of the supporting non-builtin binaries
> may also be of interest.
You left out my favorite fork()/exec() intensive exmple, our ports
system. During portupgrade, visible activity can grind stop for quite a
while at the "Registering installation" stage, while top's "last pid"
field increases rapidly and system CPU time is an embarrassingly large
number, and this is with a static /bin and /sbin.
Rather than trying to re-"optimize" this by converting /bin/sh back to
being static, I think a got more could be gained by re-writing this part
of the ports infrastructure to be more efficient. I'm not volunteering
...
Gunther Nikl
>
> :Our rationale for encouraging Gordon is as follows:
> :
> :1. 4.x upgrade path: As we approach 5-STABLE, a lot of users might want
> : to upgrade from 4-STABLE. Historically in 4.x, the / partition has
> : been very modest in size. One just simply cannot cram the bloat that
> : has grown in 5.x into a 4.x partition scheme. Of course there is the
> : venerable 'dump - clean install - restore' scheme, but we were looking
> : for something a little more user-friendly.
>
> This argument would apply to very old 4.x users but not to anyone who
> installed it as of March 2001. I bumped the nominal size of the
> root partition to 128MB in 1.98.2.7 of sysinstall/label.c.
Don't you think that people are able to change defaults if they think
thats appropriate?
> Prior to that Jordan had bumped the root partition size to 100MB
> in 1.98.2.3 in March 2001. It was 50MB before then, which is too
> small even for 4.x.
Hm, then why do I have still room on my 50MB root partition?
$ df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/vinum/root 49583 29040 16577 64% /
Thats enough for installworld (if softupdates are turned off)
Gunther
Jan Grant
> Garrett Wollman said:
> > <<On Tue, 18 Nov 2003 18:07:32 -0500 (EST), dy...@iquest.net said:
> >
> > > If the object is to maximally 'share',
> >
> > The object, AIUI, is for ~username expansion to work in the shells
> > when the user stored somewhere defined by an external NSS module. I
> > don't believe that there is anything else in a (sane) shell that
> > cares.
> >
> It is a fallacy that EVERYTHING (e.g. libc) needs to be dynamically
> linked to support that feature. A single, self contained library
> that might even be sparse WRT memory isn't going to hurt much.
It's a fallacy that ANYTHING (except nscd/lookupd) needs to be
dynamically linked to support that feature :-)
--
jan grant, ILRT, University of Bristol. http://www.ilrt.bris.ac.uk/
Tel +44(0)117 9287088 Fax +44 (0)117 9287112 http://ioctl.org/jan/
User interface? I hardly know 'er!
Jacques A. Vidrine
I suppose I should comment on this thread, since I'm closely related
to at least two of the rationales mentioned for moving towards an
all-dynamically-linked system. (I would prefer to stay out of this
thread. In my mind we've had all these arguments in various
forums months ago and the issue was put to rest.)
On Tue, Nov 18, 2003 at 05:06:06PM -0700, Scott Long wrote:
> 2. NSS support out-of-the-box: Again, this is a user-experience issue
> in that forcing NSS users to recompile world was seen as a potential
> roadblock to them.
Some followups mentioned that a different implementation of NSS would
not require dynamically linked binaries. This is technically true.
Certainly, I explored that avenue. But practical considerations made
that alternative a poor choice. Supporting NSS really also means
supporting (in as much as possible) the existing base of NSS modules.
These modules were all designed to be loaded by libc and invoked
in-process. (nscd doesn't do what some seem to think it does, and in
practice it is not well-loved.)
Existing NSS modules also shaped some other decisions that I made
(such as API entry points and thread awareness or lack thereof).
Breaking with the fundamental designs of NSS as found on Solaris and
Linux means practically starting from scratch when `porting' NSS
modules.
I'm sure someone more clever and with more time could make it work
out-of-process. Based on my experience, I think the result would be
overengineered. *shrug*
Finally, if we could call `dlopen' from statically-linked binaries,
this wouldn't be an issue. I'd really like to see dlopen support for
statically-linked binaries, for NSS and many other applications.
> 3. Binary security updates: there is a lot of interest in providing a
> binary update mechanism for doing security updates. Having a dynamic
> root means that vulnerable libraries can be updated without having to
> update all of the static binaries that might use them.
Actually, not only binary security updates but any security updates,
or bug fixes for that matter. If there is a bug in a library, fixing
that bug on your system is much more straightforward if everything is
dynamically linked. It is much easier to rebuild libc or libcrypto
and restart applications then to have to go through an entire `make
world'. It can be hard for many administrators to avoid `make world',
because it is not always easy to ascertain what applications are using
what APIs and libraries.
There's been a lot of talk about performance, but for my environment
all the workhorse applications are already dynamically linked. I'd
guess that is the case for most FreeBSD users. And of course, most
FreeBSD binaries--- in the base system, in ports, and commercially
available--- are already dynamically linked. For the minority of
users that it actually has a performance impact (show of hands?), of
course they are sophisticated enough to build the affected binaries
statically. Unless we are talking about /bin/sh, they probably already
have to go through special measures to get a statically linked binary.
Cheers,
--
Jacques Vidrine NTT/Verio SME FreeBSD UNIX Heimdal
nec...@celabo.org jvid...@verio.net nec...@freebsd.org nec...@kth.se
Leo Bicknell
--zhXaljGHf11kAtnf
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In a message written on Wed, Nov 19, 2003 at 08:10:59AM -0600, Jacques A. V=
idrine wrote:
> statically. Unless we are talking about /bin/sh, they probably already
> have to go through special measures to get a statically linked binary.
Something has been bothering me about the whole /bin/sh function,
and today when fixing a box I realized what it was.
To boot a machine into single user mode you need a kernel, init,
and /bin/sh (minimally). It would seem to me that alone is a good
argument for those three things to be static. Yes, I'm one of those
people who rebuilds kernels without modules. The kernel problem
doesn't bother me as much, as you can always boot without loading
modules, so it's not really a "part of the required kernel is
dynamic" problem.
Outside of these issues I have no problem with a dynamic root and a
/rescue.
--=20
Leo Bicknell - bick...@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - tmbg-lis...@tmbg.org, www.tmbg.org
--zhXaljGHf11kAtnf
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (FreeBSD)
iD8DBQE/u3wGNh6mMG5yMTYRAstxAJ4+qMetdoilKMkmdu49jtn8U2i61ACfV2Lw
8YGO9rlb+TgrDLO4Ck4QZQo=
=jtR+
-----END PGP SIGNATURE-----
--zhXaljGHf11kAtnf--
Ken Smith
> To boot a machine into single user mode you need a kernel, init,
> and /bin/sh (minimally).
Roughly the same thing was bothering me last night. You get a chance
to specify the shell when init is in the last phase of getting you to
single-user mode so you can say /rescue/sh at that point. init is
another story and I asked someone about that, they said it either is
or will shortly be a loader option so you can override that to be
/rescue/init that way.
--
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
Leo Bicknell
--azLHFNyN32YCQGCU
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In a message written on Wed, Nov 19, 2003 at 09:25:35AM -0500, Ken Smith wr=
ote:
> Roughly the same thing was bothering me last night. You get a chance
> to specify the shell when init is in the last phase of getting you to
> single-user mode so you can say /rescue/sh at that point. init is
> another story and I asked someone about that, they said it either is
> or will shortly be a loader option so you can override that to be
> /rescue/init that way.
Perhaps /rescue/sh should be the default when booting into single user.
The more I think about init the more I don't like dynamic linking for
it. init needs to have as few failure modes as possible. I do still
think it's fine for all the other /bin and /sbin things.
--=20
Leo Bicknell - bick...@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - tmbg-lis...@tmbg.org, www.tmbg.org
--azLHFNyN32YCQGCU
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (FreeBSD)
iD8DBQE/u38nNh6mMG5yMTYRAvo5AJ9BT5GncLHb2bjJVOorAEFnmAEAVQCfSV5k
my3fTttJxcpFm0mS1JFfUFc=
=Ve2S
-----END PGP SIGNATURE-----
--azLHFNyN32YCQGCU--
Matthew Dillon
: Don't you think that people are able to change defaults if they think
: thats appropriate?
:
:> Prior to that Jordan had bumped the root partition size to 100MB
:> in 1.98.2.3 in March 2001. It was 50MB before then, which is too
:> small even for 4.x.
:
: Hm, then why do I have still room on my 50MB root partition?
:
: $ df
: Filesystem 1K-blocks Used Avail Capacity Mounted on
: /dev/vinum/root 49583 29040 16577 64% /
:
: Thats enough for installworld (if softupdates are turned off)
:
: Gunther
Try running installkernel and see what happens when it tries to
rename the old kernel and modules and install a new one. Or try
installing a kernel.debug (which is 14MB+) instead of a kernel.
The point here is that just because you can barely squeeze the system
into 50MB doesn't mean it's a good idea to. It might work for a very
narrow use pattern, but it will create terrible problems if you ever
tried to expand your use and the system defaults have to cover more
then just generic users reasonably.
This is why the default is no longer 50MB.
-Matt
Matthew Dillon
<dil...@backplane.com>
Lyndon Nerenberg
<dro...@rpi.edu> wrote:
> have a: chflags ldcache /bin/sh
Shouldn't that be 'chmod +t /bin/sh' ???
--lyndon
Bruce Evans
> set init_path=/rescue/init
If dynamic root were ready to be turned on, then /rescue/init would be
in the default init_path.
> A dynamicly linked /sbin/init just
> makes it harder to get to the rescue bits, so it makes sense to
> link init(8) staticly. Especially since there's no advantage to
> dynamic linking init(8) that compensates for the inconvience.
It obviously uses NSS. How else could it be so bloated? :
$ ls -l /sbin/init
-r-x------ 1 root wheel 453348 Nov 18 10:30 /sbin/init
(My version is linked statically of course.)
The NSS parts of init might not be needed in normal operation, but its
hard to tell.
Bruce
Ken Smith
> > set init_path=/rescue/init
>
> If dynamic root were ready to be turned on, then /rescue/init would be
> in the default init_path.
I had that explained to me too. :-)
There is a loop in sys/kern/init_main.c that "probes" for an init
to run. But it only does what you want for cases of the files
not existing or otherwise just totally not executable. It won't
handle the "started but then dumped core" case the way it would
need to if /sbin/init were to fail because of shared library
problems. So if just relying on this mechanism it would either
not work right (/sbin/init in the path before /rescue/init) or
it would always start /rescue/init (/rescue/init before /sbin/init
in the path).
--
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
Robert Watson
On Thu, 20 Nov 2003, Bruce Evans wrote:
> On Wed, 19 Nov 2003, Marcel Moolenaar wrote:
>
> > set init_path=/rescue/init
>
> If dynamic root were ready to be turned on, then /rescue/init would be
> in the default init_path.
The fallback path only works if the exec() fails cleanly without actually
starting the userspace code. Peter and I have been talking about
improving failure mode handling by having rtld return a special error code
if (getpid() == 1 && oops_shlib_problem) and having special kernel
handling, but that's complicated by the fact that presumably you'd then
need to pick up the whole init thing in exit() rather than continuing it
in start_init(). In the meantime, Gordon has committed a change to always
link init statically.
> > A dynamicly linked /sbin/init just
> > makes it harder to get to the rescue bits, so it makes sense to
> > link init(8) staticly. Especially since there's no advantage to
> > dynamic linking init(8) that compensates for the inconvience.
>
> It obviously uses NSS. How else could it be so bloated? :
>
> $ ls -l /sbin/init
> -r-x------ 1 root wheel 453348 Nov 18 10:30 /sbin/init
>
> (My version is linked statically of course.)
>
> The NSS parts of init might not be needed in normal operation, but its
> hard to tell.
There appear to be at least two dependencies there, from my reading:
(1) getpwnam("root") so that we can check the root password when the
console isn't marked as secure.
(2) Calls to setusercontext() to set up resources when running rc, also
for uid 0, which results in libutil looking up the user to find the
class.
I think that while home directory completion is useful for sh, it's safe
to have a working assumption of a local root user in the password file,
and that when prompting for a single user mode password, access to
directory services is probably going to fail anyway.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Network Associates Laboratories
_______________________________________________
Richard Coleman
> On Tue, Nov 18, 2003 at 08:03:23PM -0500, dy...@iquest.net wrote:
>
>>However, PAM and NSS 'tricks' really seem to be exactly that,
>>and certainly worthy of special builds. However, that isn't
>>necessary, yet still not building everything with a shared
>>libc.
>
>
> Things like nss_ldap (which is used *heavily* at my place of employment)
> are some reasons that FreeBSD doesn't make it into more places. It was
> the reason why FreeBSD isn't being used here. Calling them 'tricks'
> (and succumbing to the name calling you wanted to avoid) doesn't change
> the fact that every major contender (IRIX, Solaris, Linux to name a few)
> all support this feature set.
The fact that you can't easily do centralized authentication (nss_ldap
and pam_ldap) with FreeBSD is a major show stopper. At my former
employer, we built several very large systems that required centralized
authentication using ldap. We had to use Linux, since none of the BSD's
supported this correctly at the time.
I don't really care whether everything is statically or dynamically
linked. With the fast machines and huge disks these days, bloat is not
much of an issue. But nss and pam need to work correctly. If the folks
that are against dynamic linking have an alternate method to make this
work, I'm all for it. But it needs to be more than theory. We need code.
To be honest, I've never understood the (seemingly irrational)
resistance against this change. Solaris made this change 10 years ago.
Richard Coleman
richard...@mindspring.com
Dan Nelson
> I don't really care whether everything is statically or dynamically
> linked. With the fast machines and huge disks these days, bloat is not
> much of an issue. But nss and pam need to work correctly. If the folks
> that are against dynamic linking have an alternate method to make this
> work, I'm all for it. But it needs to be more than theory. We need code.
>
> To be honest, I've never understood the (seemingly irrational)
> resistance against this change. Solaris made this change 10 years ago.
Not completely:
$ uname -a
SunOS pd1 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-Enterprise
$ file /bin/sh
/sbin/sh: ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped
$ file /sbin/* | grep statically | cut -d: -f1 | fmt
/sbin/autopush /sbin/fdisk /sbin/jsh /sbin/mount /sbin/sh
/sbin/soconfig /sbin/sync /sbin/umount /sbin/uname
--
Dan Nelson
dne...@allantgroup.com
Richard Coleman
> In the last episode (Nov 19), Richard Coleman said:
>
>>I don't really care whether everything is statically or dynamically
>>linked. With the fast machines and huge disks these days, bloat is not
>>much of an issue. But nss and pam need to work correctly. If the folks
>>that are against dynamic linking have an alternate method to make this
>>work, I'm all for it. But it needs to be more than theory. We need code.
>>
>>To be honest, I've never understood the (seemingly irrational)
>>resistance against this change. Solaris made this change 10 years ago.
>
>
> Not completely:
>
> $ uname -a
> SunOS pd1 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-Enterprise
> $ file /bin/sh
> /sbin/sh: ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped
> $ file /sbin/* | grep statically | cut -d: -f1 | fmt
> /sbin/autopush /sbin/fdisk /sbin/jsh /sbin/mount /sbin/sh
> /sbin/soconfig /sbin/sync /sbin/umount /sbin/uname
I have no problem with FreeBSD doing something similar and leaving a few
binaries static. I think most of the resistance to that was due to the
increased complexity of the build system.
It seems /bin/sh is the real sticking point. But if the compromise is
to statically link /bin/sh, that would be cool with me. Other than
tilde expansion not working when using nss_ldap, I can't think of any
other problems. I consider that a minor blemish I could easily live
with. Normal users will not generally have /bin/sh has their shell
anyways. And I could always compile a dynamically linked version into
/usr/bin if necessary.
To be honest, 98% of the time that someone notices brokeness due to
nss_ldap, it comes when using /bin/ls.
Richard Coleman
richard...@mindspring.com
Robert Watson
On Wed, 19 Nov 2003, Dan Nelson wrote:
> In the last episode (Nov 19), Richard Coleman said:
> > I don't really care whether everything is statically or dynamically
> > linked. With the fast machines and huge disks these days, bloat is not
> > much of an issue. But nss and pam need to work correctly. If the folks
> > that are against dynamic linking have an alternate method to make this
> > work, I'm all for it. But it needs to be more than theory. We need code.
> >
> > To be honest, I've never understood the (seemingly irrational)
> > resistance against this change. Solaris made this change 10 years ago.
>
> Not completely:
>
> $ uname -a
> SunOS pd1 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-Enterprise
> $ file /bin/sh
> /sbin/sh: ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped
> $ file /sbin/* | grep statically | cut -d: -f1 | fmt
> /sbin/autopush /sbin/fdisk /sbin/jsh /sbin/mount /sbin/sh
> /sbin/soconfig /sbin/sync /sbin/umount /sbin/uname
Seems to depend a bit on the install, maybe...
raven:~> uname -a
SunOS raven 5.8 Generic_108528-06 sun4u sparc SUNW,Ultra-250
raven:~> file /bin/sh /sbin/sh
/bin/sh: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped
/sbin/sh: ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Network Associates Laboratories
dy...@iquest.net
> In message: <p0600202fbbe08225be4f@[128.113.24.47]>
> Garance A Drosihn <dro...@rpi.edu> writes:
> : At 9:02 PM -0500 11/18/03, dy...@iquest.net wrote:
> : > Of course, there was a development resource limitation,
> : >but the decision (discussion) was made approx 6months ago?
> : >(Enough time to solve the problem without a GLOBAL
> : >performance hit.)
> :
> : Well, yes, perhaps. But there is that issue of "development
> : resource limitation". Back when we did debate this publicly,
> : no one stepped forward and said "I have the time to implement
> : a better solution". Thus, we went with this solution.
>
> And it still isn't too late for someone to step forward with a better
> was to be able to get dynamic libary/loading support for newer
> added benefit.
>
message :-)).
My reason for bringing this shared lib issue up is indeed related
to 'performance', but also the apparent loss of performance (or
other non-obvious features) through 'incrementalism.' When
spending time on optimizing the FreeBSD VM and buffer cache
performance, I had found that the total system performance is
very difficult to recover when several (new or pre-existing)
impedements are consipiring against best possible performance.
VERY OFTEN, you'll not be able to accurately measure each of the
incremental performance lossages, but eventually the system
will disappoint.
If someone very strongly wants XYZ feature that has been impeding
the product adoption, then the solution SHOULD NOT be at the cost
of other positive (and hard earned) attributes of the project. Just
because a certain part of the project isn't being actively worked on,
that doesn't mean that the previous work should be slowly and surely
degraded by EITHER valid new features, or creeping featurism.
If the XYZ feature was strongly desired, then it is important
to develop a 'best' solution rather than an expedient answer. It
is very clear that building all binaries as fully 'shared' is NOT a
necessary prerequisite for the new (and apparently necessary for
some applications) features. This doesn't mean that building
everything shared (in the way it is done today) shouldn't be
a temporary work-around. (*My biggest concern about building
programs shared is about shells, but other -- already well
shared and performance sensitive processes should also
be carefully considered.) I am NOT religious about this issue,
but definitely religious about the QUALITY of system wide
performance and administrative behavior of a system.,
Hopefully, when an expedient solution is developed, then that
should be remembered, and those who changed the system behavior should
work (advocate, program, redesign, etc -- or whatever) to
recover the previous desirable attribute. It is almost impossible
for someone to follow along trying to clean up 'undesirable'
side effects, unless there is a specific assignment to do that.
It does seem appropriate that those who had benefitted most for
the adoption of the new features should contribute towards an
effort to make the 'features' better (which might include assisting
in the stewardship of performance, or other negatively changed
attributes.)
Quality of an OS isn't just dependent on the number of features on
a dot item list, but also on how those dot-items avoid accumulating
negative effects on other attributes of the system. When I was
working on FreeBSD, one of my goals was to avoid making the same
mistakes as other OSes had made. Bloat and loss of performance
from 'uncareful' addition of features is something that should
be avoided -- it has already been perfected by other OS vendors.
(Please don't reduce my argument to the absurd by mentioning Plan9 :-)).
The 'bloat' attribute is something that I'd like to see FreeBSD avoid.
Stagnation also needs to be avoided :-).
John
Tim Kientzle
> It seems /bin/sh is the real sticking point.
There is a problem here: Unix systems have historically used
/bin/sh for two somewhat contradictory purposes:
* the system script interpreter
* as a user shell
The user shell must be dynamically linked in order
to support centralized administration. I personally
see no way around that. Given that many users do
rely on /bin/sh, it seems that /bin/sh must be
dynamically linked.
There are good reasons to want the system script
interpreter statically linked.
Maybe it's time to separate these two functions?
I would be content to have a static /sbin/sh
that is used as the system script interpreter for
rc scripts, etc.
Tim Kientzle
dy...@iquest.net
> Richard Coleman wrote:
> > It seems /bin/sh is the real sticking point.
>
> There is a problem here: Unix systems have historically used
> /bin/sh for two somewhat contradictory purposes:
> * the system script interpreter
> * as a user shell
>
> The user shell must be dynamically linked in order
> to support centralized administration. I personally
> see no way around that. Given that many users do
> rely on /bin/sh, it seems that /bin/sh must be
> dynamically linked.
>
(efficiency issue WRT the sparse allocations and greater
COW overheads/etc) for the shell to programmatically link in
a module for optional feature sets. This can even be
placed under a libc call (which then wouldn't encumber
the shell unless the feature was active and increase the
footprint of generally all libc routines.)
John
Stijn Hoop
--tqI+Z3u+9OQ7kwn0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed, Nov 19, 2003 at 09:27:55PM -0800, Tim Kientzle wrote:
> Richard Coleman wrote:
> >It seems /bin/sh is the real sticking point.=20
>=20
> There is a problem here: Unix systems have historically used
> /bin/sh for two somewhat contradictory purposes:
> * the system script interpreter
> * as a user shell
>=20
> The user shell must be dynamically linked in order
> to support centralized administration. I personally
> see no way around that. Given that many users do
> rely on /bin/sh, it seems that /bin/sh must be
> dynamically linked.
>=20
> There are good reasons to want the system script
> interpreter statically linked.
>=20
> Maybe it's time to separate these two functions?
> I would be content to have a static /sbin/sh
> that is used as the system script interpreter for
> rc scripts, etc.
And /usr/bin/sh as a user shell?
--Stijn
--=20
"I'm not under the alkafluence of inkahol that some thinkle peep I am. It's
just the drunker I sit here the longer I get."
--tqI+Z3u+9OQ7kwn0
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (FreeBSD)
iD8DBQE/vGumY3r/tLQmfWcRAtvfAJ90nry5K1mAlaWz6Yvj/+Gw2HosyACfVVbA
ZG15mlSsgjLLwpjMlbOUn50=
=gyDY
-----END PGP SIGNATURE-----
--tqI+Z3u+9OQ7kwn0--
Peter Jeremy
>--On Wednesday, November 19, 2003 12:30 AM -0500 Garance A Drosihn
><dro...@rpi.edu> wrote:
>
>>have a: chflags ldcache /bin/sh
>
>Shouldn't that be 'chmod +t /bin/sh' ???
Definitely. Why waste a new bit when there's already a perfectly good
one that is (or was) defined for the purpose.
As for the implementation, I presume the desired behaviour would be to
snapshot the process image (make it copy-on-write) at the point where
ld-elf.so invokes main(). You'd probably want LD_BIND_NOW behaviour
to minimise the runtime overheads. I don't see any need for this to
need massive amounts of RAM - there's no reason why the "snapshot"
couldn't be paged normally.
I think this would be a big win compared to what we have now - the
full benefits of dynamic linking remain and most of the run-time
binding overheads are removed.
Of course it's not perfect. The snapshot image permanently occupies
virtual space (RAM/swap). And there's still the PIC overhead -
especially on register-starved architectures like the i386.
I wonder how difficult this would be to implement in userland only?
Peter
Steve Kargl
> From: "Dimitry Andric" <dim...@andric.com>
>
> % sudo ldd /sbin/init
> /sbin/init:
> libutil.so.3 => /lib/libutil.so.3 (0x28074000)
> libcrypt.so.2 => /lib/libcrypt.so.2 (0x2807f000)
> libc.so.5 => /lib/libc.so.5 (0x28097000)
>
> Yes, working fine here. What should the problem be?
>
> the day /lib gets smashed.
>
This is only marginally different than /sbin/init
getting smashed. If the root partition develops
problems, you need to restore for tape.
--
Steve
Richard Coleman
> From: "Dimitry Andric" <dim...@andric.com>
>
> % sudo ldd /sbin/init
> /sbin/init:
> libutil.so.3 => /lib/libutil.so.3 (0x28074000)
> libcrypt.so.2 => /lib/libcrypt.so.2 (0x2807f000)
> libc.so.5 => /lib/libc.so.5 (0x28097000)
>
> Yes, working fine here. What should the problem be?
>
> the day /lib gets smashed.
>
> you're building a house of cards. once, if /etc/init and
> /bin/sh and some other pieces where in place a smashed
> file-system could be easily fixed. now you have to have
> 3 shared libs and a viable /lib.
>
> do you want systems that work? or houses of cards?
I would prefer to solve this problem using a fixit floppy or cdrom
anyways. I don't think that creates a house of cards. My systems work
just fine.
But I've often wondered how frequently a production system has such
problems. I've been a sysadmin for many years and can't remember this
ever happening. It's much more common to blow a hard drive, or have
flaky memory, etc.
Richard Coleman
richard...@mindspring.com
Richard Coleman
> From: "Christopher Vance" <va...@aurema.com>
>
>>Personally, I think init should be static, and can't think of any way
>>it would benefit from shared libraries.
>
>
> plan 9 has everything static. the kernel compiles in about 20 seconds
> or less -- no compression -- and you can boot it off a floppy.
>
> if i can sit in /sys/src and type:
>
> mk install
>
> and have everything re-built (and i could do it for all the supported
> architectures) in minutes i have eliminated unnecessary complexity.
>
> if it's not there, it can't break.
>
>
> btw: say hi to maltby for me.
plan9 doesn't count. It's so minimalistic, it's useless. It has many
beautiful and brilliant ideas. But it's not useful to many people as a
production system. It's a shame, really.
boyd, rounin
well, try to think about non-minimalism when you're trying to track
down a kernel bug in a zillion SLOC ...
Garrett Wollman
> well, try to think about non-minimalism when you're trying to track
> down a kernel bug in a zillion SLOC ...
How about trying to think about FreeBSD when posting on the FreeBSD
mailing-lists.
-GAWollman
boyd, rounin
> During my time in an investment bank, installations were usually hosed
> in this way by human error (systems administrators removing a file by
> accident, etc) ...
yup, it's rare i've seen flakey h/w. but i do remember one sysadmin
(when i was a contract sysadmin) who on day 2 chown'd the whole
source tree to himself on a development m/c. ugly. there were
backups but 'that would be too costly [in time]' to do a clean restore.
Cy Schubert
write
s:
> From: "Bruce M Simpson" <b...@spc.org>
> > During my time in an investment bank, installations were usually hosed
> > in this way by human error (systems administrators removing a file by
> > accident, etc) ...
>
> yup, it's rare i've seen flakey h/w. but i do remember one sysadmin
> (when i was a contract sysadmin) who on day 2 chown'd the whole
> source tree to himself on a development m/c. ugly. there were
> backups but 'that would be too costly [in time]' to do a clean restore.
I've seen that too. An end user got permission from management to the root
account, e.g. management ordered us to give her the root pw on a Tru64 box.
She chowned every file on the system to herself. Very ugly indeed.
Cheers,
--
Cy Schubert <Cy.Sc...@komquats.com> http://www.komquats.com/
BC Government . FreeBSD UNIX
Cy.Sc...@osg.gov.bc.ca . c...@FreeBSD.org
http://www.gov.bc.ca/ . http://www.FreeBSD.org/
Peter Jeremy
>But I've often wondered how frequently a production system has such
>problems. I've been a sysadmin for many years and can't remember this
>ever happening. It's much more common to blow a hard drive, or have
>flaky memory, etc.
We've had a customer whose "system administrator" installed a root
cronjob including "rm -r / somedirectory/tmp" (note the space) - on
all four production machines as well as their model system. (The
person in question is a classic example of "a little knowledge is a
dangerous thing"). The console logs (which were on another unaffected
system) made interesting reading - he had spent a significant amount
of time attempting to reboot the systems before calling for help.
Peter
Peter Jeremy
>From: "Peter Jeremy" <peter...@optushome.com.au>
>> >Shouldn't that be 'chmod +t /bin/sh' ???
>>
>> Definitely. Why waste a new bit when there's already a perfectly good
>> one that is (or was) defined for the purpose.
>
>sharable (judged by a human) text segments in core. since then
>that bit has been overloaded to death.
The purpose for setting the 't' bit was to speed up exec'ing of
frequently used executables by avoiding the need to load the text
segment from the filesystem. This is exactly what the suggested
pre-linking achieves.
As for overloading the 't' bit, I don't believe it's ever been used
for anything else on executable files. I agree it grew a distinct
meaning for directories but the 'x' bits have always meant different
things on files and directories and the setgid bit means different
things on executable and non-executable files.
>shared libraries have always been a mistake.
That is a matter of opinion. Shared libraries have both advantages
and disadvantages. Statically linking /usr/X11R6/bin demonstrates
one advantage of shared libraries.
boyd, rounin
> As for overloading the 't' bit, I don't believe it's ever been used
> for anything else on executable files.
directories
Bruce M Simpson
> But I've often wondered how frequently a production system has such
> problems. I've been a sysadmin for many years and can't remember this
> ever happening. It's much more common to blow a hard drive, or have
> flaky memory, etc.
During my time in an investment bank, installations were usually hosed
in this way by human error (systems administrators removing a file by
accident, etc) or by third party package installation. The OS in
question was Solaris.
BMS
William Josephson
> On Thu, Nov 20, 2003 at 09:29:30PM -0500, Richard Coleman wrote:
> > But I've often wondered how frequently a production system has such
> > problems. I've been a sysadmin for many years and can't remember this
> > ever happening. It's much more common to blow a hard drive, or have
> > flaky memory, etc.
>
> During my time in an investment bank, installations were usually hosed
> in this way by human error (systems administrators removing a file by
> accident, etc) or by third party package installation. The OS in
> question was Solaris.
People at Berkeley (and elsewhere) have done user studies to try to
quantify this sort of thing. It is pretty clear that with modern
hardware, most failures are due to human error. That's not to say
that hardware and software faults aren't real problems, too, but it
is more common that someone, say, pulls the wrong drive from the
RAID-5 array, resulting in an unnecessary double disk fault.
Garance A Drosihn
>On Wed, Nov 19, 2003, Lyndon Nerenberg wrote:
> >--On Wed, Nov 19, 2003, Garance A Drosihn <dro...@rpi.edu> wrote:
> > >
>> > have a: chflags ldcache /bin/sh
> >
>
>Definitely. Why waste a new bit when there's already a
>perfectly good one that is (or was) defined for the purpose.
Two reasons I didn't suggest that:
a) Well, actually it never occurred to me...
b) I thought that you might want to have this an "admin-only"
command, so nefarious users couldn't abuse it on a shared
system.
I'm certainly fine with it using "+t". Now it's just a matter
of figuring out how to implement it... :-)
--
Garance Alistair Drosehn = g...@gilead.netel.rpi.edu
Senior Systems Programmer or g...@freebsd.org
Rensselaer Polytechnic Institute or dro...@rpi.edu
Tim Kientzle
> To boot a machine into single user mode you need a kernel, init,
> and /bin/sh (minimally). It would seem to me that alone is a good
> argument for those three things to be static.
You need a static shell, yes. That does not have to be /bin/sh.
init does prompt, and /rescue/sh is (and always will be) static.
As I pointed out earlier, some of the heat here comes
from the fact that /bin/sh is currently overloaded:
* It is the default system script interpreter, used
by the rc scripts and many other things. As such,
it must start quickly.
* It is the default user shell for many users. As such,
it must support NSS.
So far, I haven't seen anyone in this thread seriously
argue against either of these points.
Today, these two issues are somewhat contradictory. Right now,
NSS requires dlopen() which requires dynamic linking which
negatively impacts startup.
There have been a lot of proposed solutions:
* Rewrite NSS to not require dlopen().
A lot of people have suggested this, but Jacques
made an important point about long-term support.
In short, this would probably just gaurantee that our NSS
is perpetually out-of-date, since it would become a major
headache to port existing NSS modules.
* Rewrite dlopen() to not require dynamic linking.
There were some patches for this submitted at one point.
As I recall, the people who looked at them were not entirely
comfortable with them. (I'd be concerned about version
conflict problems with this approach: what happens when
a dynamically-loaded NSS module refers to a libc function?
Does that get resolved to the already statically-linked
version? Or does another copy of libc get dynamically linked
with potential version conflicts? Does anyone know?)
I personally think this is worth researching, though I
have my doubts.
* Don't support NSS in /bin/sh.
This essentially amounts to: Don't use /bin/sh as
an end-user shell. Given the number of commits to
improve command-line usage of /bin/sh, I think it's clear
that a lot of people do rely on /bin/sh as a user shell.
* Change the default script interpreter for rc and such.
(e.g., use a statically-linked, optimized /sbin/sh for
that purpose.) Again, not particularly attractive, and
it certainly breaks a lot of administrator's assumptions.
* Make dynamic linking faster.
As many people have pointed out, FreeBSD's dynamic linking is
slower than it should be. Advances here would benefit Mozilla,
OpenOffice, KDE, Gnome, Apache, Perl, bash, TCL, Python, PHP,
etc. (I believe these are all slower to start on FreeBSD
than on Linux, for example.) This is a major end-user
experience issue that I find much more compelling than
anything mentioned so far. I personally find OpenOffice only
barely usable on FreeBSD, in large part because it takes so
long to open.
I suspect that most of our users would gladly give up a few
seconds booting (or a few minutes from portupgrade or buildworld)
if they could have Mozilla and OpenOffice open a few seconds faster.
Benchmarks do not necessarily reflect user experience.
As many people have pointed out, there are other advantages and
disadvantages to dynamic linking: library upgrades are simplified, disk
requirements are reduced, memory usage is less efficient, system upgrade
and repair is more complicated, etc., but I think the above summarizes
the issues that people seem to really care about.
Tim
Tim Kientzle
> The more I think about init the more I don't like dynamic linking for
> it. init needs to have as few failure modes as possible. I do still
> think it's fine for all the other /bin and /sbin things.
Right now, /sbin/init is statically linked.
Tim Kientzle
Garrett Wollman
> There have been a lot of proposed solutions:
> * Rewrite NSS to not require dlopen().
> * Rewrite dlopen() to not require dynamic linking.
> * Don't support NSS in /bin/sh.
> * Change the default script interpreter for rc and such.
> * Make dynamic linking faster.
You forgot:
* Allow statically-linked programs to use dynamic NSS modules
by forking a (dynamically-linked) resolver process when
needed.
This leads to a related, but widely disparaged option:
* Have a persistent NSS caching daemon with an RPC interface
that all programs can access for NSS lookups. You might
call such a program `nscd'. (Might as well be honest about
it.)
Both of these options may incidentally help to resolve threading
issues in the C library (although that would not be the preferred way
of doing so).
-GAWollman
Tim Kientzle
> It obviously uses NSS. How else could it be so bloated? :
>
> $ ls -l /sbin/init
> -r-x------ 1 root wheel 453348 Nov 18 10:30 /sbin/init
I believe it's actually DNS, not NSS.
Pre-5.0, the resolver ballooned significantly.
A lot of the bloat in /bin and /sbin came
from the NIS functions which in turn pull in
the resolver.
Example: /bin/date on 5.1 is also over 450k
because of a single call to getservbyname().
Removing that one call shrinks a static /bin/date
to a quite reasonable size. (I seem to recall 80k when
I did this experiment.)
I note that /sbin/init calls getpwnam();
I expect that's where the bloat gets pulled in.
Tim Kientzle
Tim Kientzle
> <<On Fri, 21 Nov 2003 15:38:49 -0800, Tim Kientzle <kien...@acm.org> said:
>>There have been a lot of proposed solutions:
>> * Rewrite NSS to not require dlopen().
>> * Rewrite dlopen() to not require dynamic linking.
>> * Don't support NSS in /bin/sh.
>> * Change the default script interpreter for rc and such.
>> * Make dynamic linking faster.
>
> You forgot:
>
> * Allow statically-linked programs to use dynamic NSS modules
> by forking a (dynamically-linked) resolver process when
> needed.
>
> This leads to a related, but widely disparaged option:
>
> * Have a persistent NSS caching daemon with an RPC interface
> that all programs can access for NSS lookups. You might
> call such a program `nscd'. (Might as well be honest about
> it.)
Jacques seems to be of the opinion that the current
'nscd' is not up to the task. He would know better than I.
Tim
Bruce Evans
> Bruce Evans wrote:
> > It obviously uses NSS. How else could it be so bloated? :
> >
> > $ ls -l /sbin/init
> > -r-x------ 1 root wheel 453348 Nov 18 10:30 /sbin/init
>
> I believe it's actually DNS, not NSS.
>
> Pre-5.0, the resolver ballooned significantly.
> A lot of the bloat in /bin and /sbin came
> from the NIS functions which in turn pull in
> the resolver.
Perhaps both.
> Example: /bin/date on 5.1 is also over 450k
> because of a single call to getservbyname().
> Removing that one call shrinks a static /bin/date
> to a quite reasonable size. (I seem to recall 80k when
> I did this experiment.)
The 2 calls to logwtmp() must also be removed, at least now.
I get the following text sizes: for /bin/date:
RELENG_4: 137491
-current*: 93214 (* = getservbyname() and logwtmp() calls removed)
-current: 371226 (only 412492 total, not 450K yet)
> I note that /sbin/init calls getpwnam();
> I expect that's where the bloat gets pulled in.
Yes, except it's only the latest 200+K of bloat (from 413558 bytes text
to 633390). Before that there was 100+K of miscellaneous bloat
relative to RELENG_4 (text size 305289 there). Before that there was
another 200+K of bloat from implementing history. Compiling with
-DNO_HISTORY removes history support and reduces the text size to
162538 (this is without getpwnam()). Then there is another 30K of
mostly non-bloat for actual changes within /bin/sh, since compiling
the FreeBSD-1 /bin/sh with current libraries gives a text size of
132966. Finally, IIRC the text size of the FReeeBSD-1 /bin/sh is
70K (total size 90K), so there is another 60K of miscellaneous bloat
in current libraries to increase the text size from 70K to 130K.
Total text sizes for /bin/sh's internals:
FreeBSD-1 sh compiled with -current's compiler: 55350
current sh compiled with -current's compiler: 87779
87:55 is about right for the increased functionality.
Bruce
boyd, rounin
> People at Berkeley (and elsewhere) have done user studies to try to
> quantify this sort of thing. It is pretty clear that with modern
> hardware, most failures are due to human error. That's not to say
> that hardware and software faults aren't real problems, too, but it
> is more common that someone, say, pulls the wrong drive from the
> RAID-5 array, resulting in an unnecessary double disk fault.
that means your raid 5 is bust. i've seen raid 5 fail and it just picks
another disk in the 'free' pool like nothing has happened.
a study? it's bleeding obvious.
Peter Jeremy
>>>Shouldn't that be 'chmod +t /bin/sh' ???
>
> command, so nefarious users couldn't abuse it on a shared
> system.
I would make one change to your proposal: Instead of having the
pre-linked image wired in RAM, I'd allow it to be paged as for an
ordinary process image. This means the overhead of starting a
pre-linked process would be similar to starting a statically-linked
executable.
Once you remove the "store image in RAM only" requirement, I don't see
that allowing an ordinary user to set the sticky bit would allow any
new attacks. The user can only set the bit on their own files. The
address space overhead of holding a pre-linked image is roughly the
same as executing that image so allowing a user to create sticky
executables is roughly the same as allowing the user to create an
additional process running that executable.
>I'm certainly fine with it using "+t". Now it's just a matter
>of figuring out how to implement it... :-)
I thought it was obvious so I left it as an exercise for the reader :-).
My guess is that if a pre-linked image didn't exist, imgact_elf()
would run as normal, transferring control to ld-elf.so. ld-elf.so
would do symbol binding as per normal (probably as if with LD_BIND_NOW
was set) and would then invoke a new save_image() syscall just before
it transferred control to main(). This would trigger the kernel to
save the whole process image as COW. This image would be associated
back to the original executable using a mechanism similar to that used
for sharing the text segment. Next time that executable is execve()'d,
imgact_elf() would note that there was a pre-linked image available and
attach to it, rather than re-mapping the executable. It would pass a
flag to ld-elf.so to inform the latter that it should just start main().
A few rough edges still need to be resolved:
- ld-elf.so runs as the invoking user but the above model has it telling
the kernel what the process image is - which shouldn't be trusted by a
different uid. The easiest solution is to make save_image() a root-
only syscall so pre-binding only works if the executable has been run
by root. An alternative would be to have the process images tied into
the user's credentials - this would mean more significant changes
because there's nothing that currently maintains information about
an executable on a per-uid basis outside the process.
- ld-elf.so environment (eg LD_LIBRARY_PATH, LD_PRELOAD) impact. This
gets messy - it is undesirable to totally inhibit pre-linking if
either is set so maybe ld-elf.so needs to verify that the pre-linked
image matches the result if the environment is taken into account.
- ld-elf.so needs to make sure that the libraries haven't changed since
the original image was mapped - to handle (eg) libc.so being replaced.
- There needs to be some way for the VM system to throw away the
pre-linked images if the system runs low on RAM+swap.
An alternative approach might be to have a dummy 'placeholder' process
which solely exists to hold the pre-linked image. This process would
be started by invoking the executable with a special "LD_PRELINK_ONLY"
flag. When the executable was later exec'd, the kernel would fork() the
dummy process giving it the credentials, pid and ppid of the process
being exec'd.
I'm sure deeper thought will reveal problems...
Peter
Dimitry Andric
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
On 2003-11-22 at 00:39:45 Tim Kientzle wrote:
> Right now, /sbin/init is statically linked.
Not here... I've built everything with WITH_DYNAMICROOT since the time
the option was introduced, and as such:
# file /sbin/init
/sbin/init: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), fo=
r FreeBSD 5.0.1, dynamically linked (uses shared libs), stripped
# ldd /sbin/init
/sbin/init:
libutil.so.3 =3D> /lib/libutil.so.3 (0x28074000)
libcrypt.so.2 =3D> /lib/libcrypt.so.2 (0x2807f000)
libc.so.5 =3D> /lib/libc.so.5 (0x28097000)
In fact, the only statically linked executable I can currently find in
my base system (=3D -CURRENT as of 2003-11-11) is /sbin/devd...
------------10B1427C296342E8
Content-Type: application/pgp-signature
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.2.3 (MingW32)
iD8DBQE/v4jCsF6jCi4glqMRAgvxAJ9hqJLYDkaubTZzSPTRslVrLdadbgCfY9Fi
+8Q/PVwq/ZnNH/jt8ew+9U8=
=hi6H
-----END PGP MESSAGE-----
------------10B1427C296342E8--
Michael Edenfield
--Xm/fll+QQv+hsKip
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
* Tim Kientzle <kien...@acm.org> [031121 18:40]:
> Leo Bicknell wrote:
> >To boot a machine into single user mode you need a kernel, init,
> >and /bin/sh (minimally). It would seem to me that alone is a good
> >argument for those three things to be static.
> * Rewrite dlopen() to not require dynamic linking.
>=20
> There were some patches for this submitted at one point.
> As I recall, the people who looked at them were not entirely
> comfortable with them. (I'd be concerned about version
> conflict problems with this approach: what happens when
> a dynamically-loaded NSS module refers to a libc function?
> Does that get resolved to the already statically-linked
> version? Or does another copy of libc get dynamically linked
> with potential version conflicts? Does anyone know?)
>=20
> I personally think this is worth researching, though I
> have my doubts.
I took a look at the glibc implementation of dlopen() breifly, since
that does function from within libc.a. It appears that you *do* get
more than one loaded copy of libc. The copy of dlopen() that is built
when #ifndef SHARED includes a flag: __libc_multiple_libcs that is set
to 1.
Additionally, I was reading comments from some of the glibc developers
who basically claim that dlopen() in a static binary *only* works if you
dlopen() a NSS module. It isn't guaranteed to work in the general case
because the static binary has no DYNAMIC elf section to resolve external
references etc. I suspect this means NSS modules are limited in what
they are allowed to reference and still work? I haven't looked in much
detail on their implementation but it certainly looks like a hack just
to make NSS work, which I don't think is a good long-term solution.
--Mike
--Xm/fll+QQv+hsKip
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (FreeBSD)
iD8DBQE/v58gCczNhKRsh48RAgLCAJ9C/TIJ6cRTldlNNLj3n3ml0aimUwCgqwzf
e0HG+k1+0IUq+SqurFA2QuQ=
=DbPO
-----END PGP SIGNATURE-----
--Xm/fll+QQv+hsKip--
Robert Watson
On Sat, 22 Nov 2003, Dimitry Andric wrote:
> On 2003-11-22 at 00:39:45 Tim Kientzle wrote:
>
> > Right now, /sbin/init is statically linked.
>
> Not here... I've built everything with WITH_DYNAMICROOT since the time
> the option was introduced, and as such:
>
> # file /sbin/init
> /sbin/init: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 5.0.1, dynamically linked (uses shared libs), stripped
> # ldd /sbin/init
> /sbin/init:
> libutil.so.3 => /lib/libutil.so.3 (0x28074000)
> libcrypt.so.2 => /lib/libcrypt.so.2 (0x2807f000)
> libc.so.5 => /lib/libc.so.5 (0x28097000)
>
> In fact, the only statically linked executable I can currently find in
> my base system (= -CURRENT as of 2003-11-11) is /sbin/devd...
The commit to force init to be linked statically was made on 2003/11/19,
so your system is too old to see the change. Commit message attached
below.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Network Associates Laboratories
revision 1.28
date: 2003/11/19 19:57:20; author: gordon; state: Exp; lines: +2 -0
Make init statically linked by default. It's not worth the pain of having
a dynamically linked init as recently seen by ia64 woes.
Approved by: re (jhb)
Duncan Barclay
From: "David O'Brien" <obr...@freebsd.org>
> I'll seriously argue against the 2nd point above. I don't know of a
> SINGLE person that uses /bin/sh as their interactive shell when
> multi-user. Not ONE. Every Bourne shell'ish user I've ever met uses
> Bash, AT&T ksh, pdksh, zsh.
I don't know anyone that farms lama's, so there cannot be any lama farmers.
computer$ grep dmlb /etc/passwd
dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh
Duncan
Thomas David Rivers
> > argue against either of these points.
>
> SINGLE person that uses /bin/sh as their interactive shell when
> multi-user. Not ONE. Every Bourne shell'ish user I've ever met uses
> Bash, AT&T ksh, pdksh, zsh.
>
I'm one...
I have to operate on many disparate systems with the
same home directory.... some of the bourne-shell variants
(e.g. HP/UX's ksh) will lock-up on my simple bourne-shell .profile.
So - I stick with the "plain/old/boring" sh.
- Dave R. -
--
riv...@dignus.com Work: (919) 676-0847
Get your mainframe programming tools at http://www.dignus.com
David O'Brien
>
> From: "David O'Brien" <obr...@freebsd.org>
>
> > SINGLE person that uses /bin/sh as their interactive shell when
> > multi-user. Not ONE. Every Bourne shell'ish user I've ever met uses
> > Bash, AT&T ksh, pdksh, zsh.
>
One has to make a strong statement to get the people to come out of the
woodwork.
> computer$ grep dmlb /etc/passwd
> dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh
Good. Now do you need NSS support? Do the benefits of supporting NSS in
/bin/sh for you out-weigh the performance issue of building it
dynamically? Couldn't you just as easily use the pdksh port?
--
-- David (obr...@FreeBSD.org)
David O'Brien
> But it would be sorta odd to have statically linked versions of sh in
> both /bin and /rescue.
We'd remove it from /rescue if the /bin/sh one was static. :-)
Maxim M. Kazachek
>
>From: "David O'Brien" <obr...@freebsd.org>
>
>> I'll seriously argue against the 2nd point above. I don't know of a
>> SINGLE person that uses /bin/sh as their interactive shell when
>> multi-user. Not ONE. Every Bourne shell'ish user I've ever met uses
>> Bash, AT&T ksh, pdksh, zsh.
>
>I don't know anyone that farms lama's, so there cannot be any lama farmers.
>
>computer$ grep dmlb /etc/passwd
>dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh
>
>Duncan
So, imagine, i'm accidentally deleted /bin with your most wanted
static sh... And, of course, due to static nature of /bin/sh it was
removed from /rescue? Nothing will protect you from shooting in the leg,
neither static linking, nor assumption that /lib is OK.
MOST people uses /bin/sh only for rc scripts (to be correct, their system
uses it). David O'Brien just tried to told, that NOBODY he knows will be
REALLY impacted by performance loss, caused due dynamic /bin/sh linking.
You will... So, because Duncan Barclay is impacted by performance
loss due dynamic /bin/sh linking, ENTIRE FreeBSD community will have
troubles (at least with NSS) due to static linking...
Sincerely, Maxim M. Kazachek
mailto:stra...@sberbank.sibnet.ru
mailto:stra...@fpm.ami.nstu.ru
David Wolfskill
>From: "Maxim M. Kazachek" <stra...@sberbank.sibnet.ru>
> So, imagine, i'm accidentally deleted /bin with your most wanted
>static sh... And, of course, due to static nature of /bin/sh it was
>removed from /rescue? Nothing will protect you from shooting in the leg,
>neither static linking, nor assumption that /lib is OK.
So go ahead and make /bin/sh also have a (hard) link to /rescue/sh.
Then the referenced action merely decrements the link count, and the
executable itself doesn't go away.
Sure, you could be more imaginative with foot-shooting, but the stated
problem is really easy to avoid.
Peace,
david (reluctant to contribute to this thread....)
--
David H. Wolfskill da...@catwhisker.org
If you want true virus-protection for your PC, install a non-Microsoft OS
on it. Plausible candidates include FreeBSD, Linux, NetBSD, OpenBSD, and
Solaris (in alphabetical order).
Maxim M. Kazachek
>>Date: Mon, 24 Nov 2003 09:34:08 +0600 (NOVT)
>>From: "Maxim M. Kazachek" <stra...@sberbank.sibnet.ru>
>
>> So, imagine, i'm accidentally deleted /bin with your most wanted
>>static sh... And, of course, due to static nature of /bin/sh it was
>>removed from /rescue? Nothing will protect you from shooting in the leg,
>>neither static linking, nor assumption that /lib is OK.
>
>So go ahead and make /bin/sh also have a (hard) link to /rescue/sh.
>Then the referenced action merely decrements the link count, and the
>executable itself doesn't go away.
>
>Sure, you could be more imaginative with foot-shooting, but the stated
>problem is really easy to avoid.
>
>Peace,
>david (reluctant to contribute to this thread....)
>--
>David H. Wolfskill da...@catwhisker.org
>If you want true virus-protection for your PC, install a non-Microsoft OS
>on it. Plausible candidates include FreeBSD, Linux, NetBSD, OpenBSD, and
>Solaris (in alphabetical order).
I'm a real masochist, and removed /rescue/sh /sbin/sh whatsoever.
And after that will begin complain, that FreeBSD is ugly, non bullet-proof
operating system... :-)
But, in fact, as I can remember, all /rescue stuff is one
hadlinked executable. I don't think that hardlinking /bin/sh into /rescue
would be nice idea... IMHO it's not clear.
Sincerely, Maxim M. Kazachek
mailto:stra...@sberbank.sibnet.ru
mailto:stra...@fpm.ami.nstu.ru
_______________________________________________
Duncan Barclay
> >From: "David O'Brien" <obr...@freebsd.org>
> >
> >> I'll seriously argue against the 2nd point above. I don't know of a
> >> SINGLE person that uses /bin/sh as their interactive shell when
> >> multi-user. Not ONE. Every Bourne shell'ish user I've ever met uses
> >> Bash, AT&T ksh, pdksh, zsh.
> >
> >I don't know anyone that farms lama's, so there cannot be any lama
farmers.
> >
> >computer$ grep dmlb /etc/passwd
> >dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh
> >
> >Duncan
> static sh... And, of course, due to static nature of /bin/sh it was
> removed from /rescue? Nothing will protect you from shooting in the leg,
> neither static linking, nor assumption that /lib is OK.
>
>
> uses it). David O'Brien just tried to told, that NOBODY he knows will be
> REALLY impacted by performance loss, caused due dynamic /bin/sh linking.
> You will... So, because Duncan Barclay is impacted by performance
> loss due dynamic /bin/sh linking, ENTIRE FreeBSD community will have
> troubles (at least with NSS) due to static linking...
Maxim, I was merely rising to David's bait for some proof that people use
/bin/sh as an interactive shell. You're correct in saying that if /bin/sh on
my machines gets hosed it won't
matter too much - I'll use another shell to rebuild the machine, or boot
from an install CD
to get a shell. But as part of me fixing the machine, I'll put /bin/sh back
on.
I didn't say anything about NSS and I don't intend to as I don't need it.
The debate should be held between people that need the functionality but
want it implemented in different ways.
As to performance loss, I really don't think I'm going to notice it - the
cheapest machine I can find in the UK is an Athlon 1800XP, that has a lot
more grunt than I need. This may not be true for others.
Duncan
Duncan Barclay
> >
> >
> > > I'll seriously argue against the 2nd point above. I don't know of a
> > > SINGLE person that uses /bin/sh as their interactive shell when
> > > multi-user. Not ONE. Every Bourne shell'ish user I've ever met uses
> > > Bash, AT&T ksh, pdksh, zsh.
> >
> > I don't know anyone that farms lama's, so there cannot be any lama
farmers.
>
> woodwork.
Ack.
> > computer$ grep dmlb /etc/passwd
> > dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh
>
> Good. Now do you need NSS support? Do the benefits of supporting NSS in
> /bin/sh for you out-weigh the performance issue of building it
> dynamically? Couldn't you just as easily use the pdksh port?
The machine use I generate doesn't really require a lot of /bin/sh
invocations. Either I have file servers, shell boxes, or compute engines
running CPU bound jobs for half an hour upwards. Whether it takes a gnats
longer to start /bin/sh doesn't really matter to me. However, NSS is likely
to feature as needed sometime soon.
Duncan
> --
> -- David (obr...@FreeBSD.org)
Robert Watson
On Mon, 24 Nov 2003, Maxim M. Kazachek wrote:
> MOST people uses /bin/sh only for rc scripts (to be correct, their
> system uses it). David O'Brien just tried to told, that NOBODY he knows
> will be REALLY impacted by performance loss, caused due dynamic /bin/sh
> linking. You will... So, because Duncan Barclay is impacted by
> performance loss due dynamic /bin/sh linking, ENTIRE FreeBSD community
> will have troubles (at least with NSS) due to static linking...
Actually, you appear to be agreeing with him, not disagreeing with him.
Duncan was pointing out that he *does* use /bin/sh as his shell, in
response to David's suggestion that on one uses it and therefore that
making it statically linked wouldn't hurt.
It strikes me that this whole conversation has gotten a little
confrontational... The "middle ground" of adding a static /sbin/sh for
scripts soudds like a reasonable choice, and has precedent in other
systems (Solaris). We can set the boot and periodic scripts to use that,
and interactive users can keep using /bin/sh. Someone must be using
/bin/sh as a shell, because apparently someone spent a lot of time adding
things like character input editing, filename completion, etc. We even
use "sh" as the default in adduser(8).
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Network Associates Laboratories
Andrew Gallatin
Robert Watson writes:
>
> It strikes me that this whole conversation has gotten a little
> confrontational... The "middle ground" of adding a static /sbin/sh for
> scripts soudds like a reasonable choice, and has precedent in other
> systems (Solaris). We can set the boot and periodic scripts to use that,
> and interactive users can keep using /bin/sh. Someone must be using
> /bin/sh as a shell, because apparently someone spent a lot of time adding
> things like character input editing, filename completion, etc. We even
> use "sh" as the default in adduser(8).
Tru64 also does the /sbin/sh (static) and /bin/sh (dynamic) thing.
I've never liked it because standard, portable shell scipts expect to
use /bin/sh. Standard, portable shell scripts don't need tilde
expansion for ldap/nss usernames.
So I think the best solution (*) would be to keep /bin/sh statically
linked, and build a dynamic version in /usr/bin that people can use as
an interactive shell. Root's shell remains /bin/sh
1) All three (;-) interactive bourne shell users that use nss/ldap get
tilde expansion.
2) 3rd party scripts, which are written to be portable and don't use
tilde, don't have to pay for dynamic linking.
3) System startup scripts are faster because they use the static
/bin/sh
(*) The real best solution would be to backout the dynamic linking
changes and put the onus on those who want/need nss/ldap to prove that
the dynamic linking changes do not produce a measurable slowdown (as
they were asked to do 6 months ago, and never did). If it really
slows things down, then they need to fix the performance problems by
either speeding up dynamic linking, or getting the features they need
in another, less invasive way (as discussed previously in this
thread).
Drew
[reply-to set to freebsd-current, as I don't want to be CC'ed on this
thread forever.. ]
David O'Brien
> It strikes me that this whole conversation has gotten a little
> confrontational... The "middle ground" of adding a static /sbin/sh for
> scripts soudds like a reasonable choice, and has precedent in other
> systems (Solaris).
Time for a pdksh import! We can install it as the dynamic /bin/sh, and
there is precedent in other systems (Solaris) for /bin/ksh.
The issue with static /sbin/sh is that we have to fix all the base
scripts (doable), and then try to educate people that they need to do the
same to their scripts (very hard to do).
> Someone must be using /bin/sh as a shell, because apparently someone
> spent a lot of time adding things like character input editing,
> filename completion, etc.
Filename complete isn't in stock /bin/sh as its a private hack someone
did, it is a hack and isn't commitable.
Andrew Gallatin
Richard Coleman writes:
> Are you suggesting that (t)csh also move to /usr/bin to match
> /usr/bin/sh? The screams caused by such a change would be deafening.
Of course not. Nobody in their right mind uses csh for scripting.
Drew
Matthias Andree
> Are you suggesting that (t)csh also move to /usr/bin to match
> /usr/bin/sh? The screams caused by such a change would be deafening.
Would there be any screams at all?
chsh -s /bin/sh root # prevent lock-out
rm -f /bin/csh /bin/tcsh
echo NO_TCSH=true >>/etc/make.conf # avoid resurrection of evil
# next "make world"
Looks that a system without [t]csh is supported. And it works for me.
--
Matthias Andree
Encrypt your mail: my GnuPG key ID is 0x052E7D95
Tony Finch
>
>Not just the startup scripts, but ANY script. I dare say there's a long,
>long list of scripts that use ~-expansion, to say nothing of the
>homegrown ones we all have working quietly and forgotten for years.
It's required for POSIX compliance.
Tony.
--
f.a.n.finch <d...@dotat.at> http://dotat.at/
IRISH SEA: SOUTHWESTERLY 5 OR 6. SHOWERS. GOOD.
Tony Finch
>
>Someone must be using /bin/sh as a shell, because apparently someone
>spent a lot of time adding things like character input editing, filename
Command-line editing is required for POSIX compliance.
Tony.
--
f.a.n.finch <d...@dotat.at> http://dotat.at/
FAIR ISLE: SOUTHWESTERLY BACKING EASTERLY 5 TO 7, PERHAPS GALE 8 LATER.
SHOWERS THEN RAIN. MODERATE OR GOOD.