/dev/fd/62: No such file or directory

[Linda Walsh]

When a system is not fully up or limping, various
services like udevd may not be mounted yet -- proc may not be
mounted yet. But it appears there are features in bash that
try to use external paths to access process descriptors in
itself (rather than just using them directly).

If there is a need to pick up those fd's, I think it would
be prudent if bash, **internally**,
did not rely on os-mechanisms that may not be there ...

Note -- I'm not saying bash should attempt to do something
about a user's use of /dev/stdin or whatever in a script.

but specifically I've seen above error message /dev/fd/62 (not sure
if it was just 62, maybe 63 as well?) a fair amount after,
I think, 4.3.

I certainly wouldn't be against a builtin that would return
the same thing as /dev/fd/62 -- just one that doesn't try to
walk an external path to perform internal functions (have looked
several times at scripts that have shown such messages, and
have yet to see any mention of /dev/fd/xx in the script, so I'm
/presuming/ (sigh) that it is something internal to bash?

[Chet Ramey]

On 3/27/14, 8:42 PM, Linda Walsh wrote:

> I certainly wouldn't be against a builtin that would return
> the same thing as /dev/fd/62 -- just one that doesn't try to
> walk an external path to perform internal functions (have looked
> several times at scripts that have shown such messages, and
> have yet to see any mention of /dev/fd/xx in the script, so I'm
> /presuming/ (sigh) that it is something internal to bash?

Process substitution, whose entire reason for existence is to turn a
(pipe) file descriptor into a filename. If you want to use process
substitution in a context where /dev/fd might not be available, make
sure you don't tell bash to use it at build time or use another
scripting technique.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU ch...@case.edu http://cnswww.cns.cwru.edu/~chet/

[Linda Walsh]

Chet Ramey wrote:
> On 3/27/14, 8:42 PM, Linda Walsh wrote:
>
>> I certainly wouldn't be against a builtin that would return
>> the same thing as /dev/fd/62 -- just one that doesn't try to
>> walk an external path to perform internal functions (have looked
>> several times at scripts that have shown such messages, and
>> have yet to see any mention of /dev/fd/xx in the script, so I'm
>> /presuming/ (sigh) that it is something internal to bash?
>
> Process substitution, whose entire reason for existence is to turn a
> (pipe) file descriptor into a filename. If you want to use process
> substitution in a context where /dev/fd might not be available, make
> sure you don't tell bash to use it at build time or use another
> scripting technique.
>
>

----
So whether or not to use /def/fd is a build time option?

It's a minority of scripts but have noticed the message
in some startup scripts. It's not just my scripts but vendor scripts
as well. I'll have to find out why.. but it seems like it a
"gotcha"...

[Greg Wooledge]

On Thu, Mar 27, 2014 at 07:31:31PM -0700, Linda Walsh wrote:
> So whether or not to use /def/fd is a build time option?

On many commerical Unix systems (which don't have /dev/fd/), Bash
falls back to using named pipes.

> It's a minority of scripts but have noticed the message
> in some startup scripts. It's not just my scripts but vendor scripts
> as well. I'll have to find out why.. but it seems like it a
> "gotcha"...

If your OS vendor's scripts are generating error messages at boot time,
with the version of Bash that the OS vendor shipped, then this is a bug
in those scripts.

If you replaced the version of Bash that the vendor supplied, but did
not change the scripts that may be relying upon the vendor's Bash patches
or version-specific behavior, then the ball is in your own court.

[Linda Walsh]

Greg Wooledge wrote:
> On Thu, Mar 27, 2014 at 07:31:31PM -0700, Linda Walsh wrote:
>> So whether or not to use /def/fd is a build time option?
>
> On many commerical Unix systems (which don't have /dev/fd/), Bash
> falls back to using named pipes.

---
Right... but this is a case where "normally" /dev/fd is
there, but because it is a boot script /dev hasn't been mounted
yet. These are script before runlevel1 or single user and have
been in a "boot.d" directory.

What I'd prefer to see is that bash do what you say at
runtime, rather being limited to that choice at build time.

I just checked... it is partly my fault in that I
upgraded my bash from 4.2 -> 4.3 to get around a limitation in 4.2.
Am not looking forward to upgrading to 4.4, since both times I've tried,
autocompletion completely broke and performance was seriously dog'ed even
in non-debug versions.

Both of those were prerelease 4.3's, so I decided I'd hold off for
a bit, but from what I saw others ran into similar problems later
on and several other "minor syntax" changes went
in which means it is likely I'll hit some of them given
the large number of scripts...like this one -- an edge case,
where bash bash is relying on /dev/fd -- which is true for a running
system, but doesn't fall-back dynamically, which can cause rare
probs in boot scripts and I'm not seeing where process substitution
is used.

Isn't it only things that are like "read xxx < <(cmd)" ? or is there
something else that uses process substitution??

[Eduardo A. Bustamante López]

On Fri, Mar 28, 2014 at 08:21:40AM -0700, Linda Walsh wrote:
> I just checked... it is partly my fault in that I
> upgraded my bash from 4.2 -> 4.3 to get around a limitation in 4.2.
> Am not looking forward to upgrading to 4.4, since both times I've tried,
> autocompletion completely broke and performance was seriously dog'ed even
> in non-debug versions.

There's no bash 4.4.

> Isn't it only things that are like "read xxx < <(cmd)" ? or is there
> something else that uses process substitution??

<(cmd) and >(cmd) are process substitutions. If you want to work
around, use named pipes. You won't have issues with these.

--
Eduardo Alan Bustamante López

[Greg Wooledge]

On Fri, Mar 28, 2014 at 08:21:40AM -0700, Linda Walsh wrote:

> What I'd prefer to see is that bash do what you say at
> runtime, rather being limited to that choice at build time.

If you require process substitution features in a script that may be
executed when the OS is not fully booted, I'd say you should explicitly
used a named pipe instead. An alternative would be to use a
specially-built bash in the boot environment, which uses the named pipe
implementation internally.

> Isn't it only things that are like "read xxx < <(cmd)" ? or is there
> something else that uses process substitution??

My guess would be that < <(cmd) on a read or a while loop is the most
common use of process substitutions. I sometimes see diff <(cmd1) <(cmd2).
The >(cmd) form seems to be much less common, mostly used for logging
a whole script through a filtering command (or simply tee).

I don't think I've ever seen a boot script use process substitutions,
but I admit I haven't studied a lot of them. Mostly the ones I've seen
are written for /bin/sh = dash (or ksh or Bourne shell) anyway.

[Chet Ramey]

On 3/28/14, 11:21 AM, Linda Walsh wrote:
>
>
> Greg Wooledge wrote:
>> On Thu, Mar 27, 2014 at 07:31:31PM -0700, Linda Walsh wrote:
>>> So whether or not to use /def/fd is a build time option?
>>
>> On many commerical Unix systems (which don't have /dev/fd/), Bash
>> falls back to using named pipes.
> ---
> Right... but this is a case where "normally" /dev/fd is
> there, but because it is a boot script /dev hasn't been mounted
> yet. These are script before runlevel1 or single user and have
> been in a "boot.d" directory.
>

> What I'd prefer to see is that bash do what you say at
> runtime, rather being limited to that choice at build time.

The source code is available, and all of the necessary pieces exist. There
is nothing stopping you from implementing your preference and, if you would
like, sending any patches back upstream.

> I just checked... it is partly my fault in that I
> upgraded my bash from 4.2 -> 4.3 to get around a limitation in 4.2.
> Am not looking forward to upgrading to 4.4, since both times I've tried,
> autocompletion completely broke and performance was seriously dog'ed even
> in non-debug versions.

It's true that bash-completion did not keep up with changes made over
several years. Rather than wait for bash-completion to change, I
made some changes to the way bash-4.3 does programmable completion and
sent those out to the debian folks who maintain that package. Initial
reaction is favorable, so I will be releasing another patch with those
changes soon.

Your experiences are your own, of course, but it's my experience that
performance and memory consumption have improved between bash-4.2 and
bash-4.3. I periodically do performance testing to verify that.

> Isn't it only things that are like "read xxx < <(cmd)" ? or is there
> something else that uses process substitution??

Only you can answer questions about where your scripts, and your `system
scripts', use process substitution.

[Linda Walsh]

Eduardo A. Bustamante López wrote:
> On Fri, Mar 28, 2014 at 08:21:40AM -0700, Linda Walsh wrote:

>> Isn't it only things that are like "read xxx < <(cmd)" ? or is there
>> something else that uses process substitution??

> <(cmd) and >(cmd) are process substitutions. If you want to work
> around, use named pipes. You won't have issues with these.

Does read varname <<<$(...) use process substitution?

I'd think not.. but just to be safe... it's only the things
with '<(' in them?

[Greg Wooledge]

On Fri, Mar 28, 2014 at 06:14:27PM -0700, Linda Walsh wrote:
> Does read varname <<<$(...) use process substitution?

I wouldn't dare write it like that, because who knows how the parser
will treat it. I'd write it this way:

read varname <<< "$(...)"

This is a command substitution and a here-string. Here-strings are
implemented with a temporary file, I believe.

> I'd think not.. but just to be safe... it's only the things
> with '<(' in them?

Process substitutions are <(...) and >(...). These are the ones that
use either a named pipe, or a /dev/fd/* node.

[Linda Walsh]

Greg Wooledge wrote:
> On Fri, Mar 28, 2014 at 06:14:27PM -0700, Linda Walsh wrote:
>> Does read varname <<<$(...) use process substitution?
>
> I wouldn't dare write it like that, because who knows how the parser
> will treat it. I'd write it this way:
>
> read varname <<< "$(...)"
>
> This is a command substitution and a here-string. Here-strings are
> implemented with a temporary file, I believe.

----
Well don't know if it circumvents the /fd/62 prob
yet (got a few places more to check & convert),
but this seems to work for checking if a file
or dir is empty:

function empty {
[[ $# -lt 1 ]] && return -1
[[ -f $1 && ! -s $1 ]] && return 0
[[ -d $1 ]] && {
readarray entries<<<"$(cd "$1" && printf "%s\n" * 2>/dev/null)"
((${#entries[@]} < 3)) && return 0
}
return 1
}

Had one with find+wc, but this one doesn't rely on any
sub-utils.

[Pierre Gaston]

> why not simply: entries=("$1"/*) ?

[Greg Wooledge]

On Wed, Apr 02, 2014 at 07:54:58AM +0300, Pierre Gaston wrote:
> On Wed, Apr 2, 2014 at 6:04 AM, Linda Walsh <ba...@tlinx.org> wrote:
> > Well don't know if it circumvents the /fd/62 prob
> > yet (got a few places more to check & convert),
> > but this seems to work for checking if a file
> > or dir is empty:
> >
> > function empty {
> > [[ $# -lt 1 ]] && return -1
> > [[ -f $1 && ! -s $1 ]] && return 0
> > [[ -d $1 ]] && {
> > readarray entries<<<"$(cd "$1" && printf "%s\n" * 2>/dev/null)"
> > ((${#entries[@]} < 3)) && return 0
> > }
> > return 1
> > }

That's unnecessarily complex.

> why not simply: entries=("$1"/*) ?

You need to activate nullglob and dotglob first, but yes, that would be
the way I'd recommend. That's what we use in
http://mywiki.wooledge.org/BashFAQ/004 as well.

[Linda Walsh]

Greg Wooledge wrote:
> On Wed, Apr 02, 2014 at 07:54:58AM +0300, Pierre Gaston wrote:
>>> [[ -d $1 ]] && {
>>> readarray entries<<<"$(cd "$1" && printf "%s\n" * 2>/dev/null)"
>>> ((${#entries[@]} < 3)) && return 0
>

> That's unnecessarily complex.
>
>> why not simply: entries=("$1"/*) ?

----
Indeed -- I just got around to changing this (been an odd week)...
combining it with a hint from greg:

> You need to activate nullglob and dotglob first, but yes, that would be
> the way I'd recommend.

----
dotglob is already activated, nullglob is not. I thought about it
if nullblob is unset, then I'll always have 1 element in the array --
so I don't have to worry about coding for an 'unset' value.

So all I need do is test the first entry:

local -a entries=("$1"/*)
[[ ${entries[0]} == $1/* ]] && return 0

--- the $1 doesn't need quotes in [[]] and '*' won't expand or
am missing something? Thanks for the tip Pierre, I often
don't see forests because of all the trees...

[Chris Down]

Linda Walsh writes:
> So all I need do is test the first entry:
>
> local -a entries=("$1"/*)
> [[ ${entries[0]} == $1/* ]] && return 0
>
> --- the $1 doesn't need quotes in [[]] and '*' won't expand or
> am missing something? Thanks for the tip Pierre, I often
> don't see forests because of all the trees...

The RHS of [[ has pattern matching applied on unquoted parts, so yes,
you probably want quotes around $1.

[Linda Walsh]

----
Pattern matching? Why doesn't '*' match anything then?

Do you mean pathname expansion? or are you thinking of the =~
operator? I.e. where would it match a pattern? It can't match
in the current directory, since it just failed a pathname
expansion in the line before. But for a regex, I thought you
needed '=~' ??

[Chris Down]

Linda Walsh writes:
> Pattern matching? Why doesn't '*' match anything then?

I've no idea what you're doing, but:

$ var='*bar*'
$ [[ abarb = $var ]]
$ echo $?
0
$ [[ aquxb = $var ]]
$ echo $?
1

[Pierre Gaston]

On Sat, Apr 5, 2014 at 1:46 PM, Linda Walsh <ba...@tlinx.org> wrote:

>
>
> Chris Down wrote:
>
>> Linda Walsh writes:
>>
>>> So all I need do is test the first entry:
>>>
>>> local -a entries=("$1"/*)
>>> [[ ${entries[0]} == $1/* ]] && return 0
>>>
>>> --- the $1 doesn't need quotes in [[]] and '*' won't expand or
>>> am missing something? Thanks for the tip Pierre, I often
>>> don't see forests because of all the trees...
>>>
>>
>> The RHS of [[ has pattern matching applied on unquoted parts, so yes,
>> you probably want quotes around $1.
>>
> ----

> Pattern matching? Why doesn't '*' match anything then?
>

> Do you mean pathname expansion? or are you thinking of the =~
> operator? I.e. where would it match a pattern? It can't match
> in the current directory, since it just failed a pathname
> expansion in the line before. But for a regex, I thought you
> needed '=~' ??
>

> * matches everything and nothing, so you need to quote the whole thing

$ [[ foo == * ]] && echo true
true

your test will also fail if there is one file named "*" it' better to just
[[ -e ${entries[0]} ]]

[Linda Walsh]

Pierre Gaston wrote:
> your test will also fail if there is one file named "*" it' better to
> just [[ -e ${entries[0]} ]]

Sigh...so you are saying that:

[[ ${entries[0]} == "$1/*" ]] && return 0

would fail if someone has a file named '*'.

Hmmm...and whether the file exists or not, 'entries' will
still have 'arg1/*' in it so I can test it without worrying
about an 'unset' error. It's not that I'm running with
-u set normally, but these scripts get called from
various contexts, so want to make sure they don't
suffer from things that I know are possible.

This script would normally only be called at login or
by some system-start scripts. Either way, the env
is fairly predictable at those points (e.g. - not likely
'nullglob'). But protecting against refs to "unset vars"
is necessary, as I've seen some scripts that start
with "bash -u" at the top. Though, in my particular
use case, if there was only a file named '*' in
the directory -- having it return 0 would be ok, as
this is used to decide whether or not to add a directory
to your PATH when bash is starting -- so even if '*'
was an executable, and it's the only thing in a 'dir',
I don't think that "not adding" that dir to the PATH
would be a problem, and it might be a positive (who
names files '*' and expects to invoke them via
the PATH?).

I have never used the [[ xxx == ?xx*xx ]] construct
to do matching. I would have defaulted to =~ and
used an RE... but I can see there might be times
the other might be useful...but I see it as bit
arcane and would likely still tend to use an RE,
instead, in my own code.

So thanks Chris for the fast reminder lesson (I vague
remember having known that in the past, but it wasn't
something I actively remembered)..

[Linda Walsh]

Chet Ramey wrote:
> Only you can answer questions about where your scripts, and your `system
> scripts', use process substitution.

----
I tried to switch to in-line HERE documents instead of
/dev/fd/62... but those don't work either.

I thought HERE docs were pretty basic?

Are HERE documents also an optional feature that
are not included in POSIX?

Thanks...

[Chet Ramey]

Maybe you should post your script so readers can take a look. It's
unlikely that there is a bug in here-documents, but it's possible.

[Linda Walsh]

Chet Ramey wrote:
> Maybe you should post your script so readers can take a look. It's
> unlikely that there is a bug in here-documents, but it's possible.

----
If you ask for it... I "inlined" the needed library functions,
so should run standalone.

FWIW, this script DOES work interactively in normal operation.

Just when the system is in pre-single-user state and not
much in the way of resources is available that it blows.

Right now only have 'start' implemented, as there is little call
for deconstructing network configs when the system is going down.

'ifmap' by itself should show current eth-dev mappings.

Note, buried in this code are absolute numbers (MAC addrs),
so to test on another machine they would need to be changed.

Have intended to put "site-local" info in a config file, but
as it doesn't work....haven't spent alot of time on expanding it.



--- ifmap -------------------------------------------------------------

#!/bin/bash

### BEGIN INIT INFO
# Provides: net-devices
# Required-Start: boot.udev boot.device-mapper boot.localfs
# Required-Stop: $null
# Default-Start: B
# Default-Stop:
# Short-Description: order net devices
# Description: order net devs if needed
### END INIT INFO
#
# assign network names as rc-script
# L A Walsh, (free to use/modify/distribute to nice people) (c) 2013-2014
#
#include standard template:
# gvim=:SetNumberAndWidth
echo "assign_netif_names=$0 $@"
_prgpth=${0:?}; _prgpth=${_prgpth#-} _prg=${_prgpth##*/}; _prgdr=${_prgpth%/$_prg}
[[ -z $_prgdr || $_prg == $_prgdr ]] && _prgdr="$PWD"
#if ! typeset -f include >&/dev/null ;then
# source ${_LOCAL_DIR:=/etc/local}/bash_env.sh;
#fi
export PATH="/etc/local/bin:/etc/local/lib:$PATH"
export
PS4='>>${BASH_SOURCE:+${BASH_SOURCE[0]}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}> '

#include stdalias (needed entries included below)
shopt -s extglob expand_aliases

alias dcl=declare sub=function
alias int=dcl\ -i map=dcl\ -A hash=dcl\ -A array=dcl\ -a
alias lower=dcl\ -l upper=dcl\ -u string=dcl my=dcl
alias map2int=dcl\ -Ai intArray=dcl\ -ia

#include rc.status -- essential funcs included below:
int rc_status=0
sub rc_reset { rc_status=0; }
sub rc_status {
rc_status=$?;
if ((rc_status)) && { (($#)) && [[ $1 = -v ]] ; }; then
echo "Abnormal rc_status was $rc_status)."
elif (($#)) && [[ $1 = -v ]] ; then
echo "rc_status: ok"
fi
}
sub rc_exit {
rc_status=$?;
rc_status
exit $rc_status
}

# need to list commands here:
# modprobe

sub warn () { local msg="Warning: ${1:-"general"}"
echo "$msg" >&2
}

sub die () { int stat=$?; local msg="Error. ${1:-"unknown"}"
echo "$msg (errno=$stat)" >&2
(exit $stat);
rc_status -v
rc_exit
exit $stat
}

if [[ -z $(type -P modprobe) ]]; then
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH
fi

if [[ -n $(type -P modprobe) ]]; then
alias modprobe="$(type -P modprobe)"
else
#delay failure until use
alias modprobe="die 'cannot load required modules'"
fi

if [[ -z $(type -P ip ) ]]; then
die "Cannot find 'ip' util -- needed for network setup"
fi

alias ip="$(type -P ip)"


sub varflags() {
my var="${1:-""}"
read out <<<$(declare -p "$var" )
[[ $out =~ /^declare.*=.*$/ ]] || die "no such variable"
out="${out%% +([^-])=*}"
out="${out#declare }"
[[ ${out:0:1} == - ]] || { echo ""; return 0 ; }
echo "${out#-}"
}

sub isarray() {
my name="${1:-""}"
flags=$(varflags $name)
[[ $flags =~ a ]] && return 0
return 1
}

sub ipcmd () {
my ipcmd="${1:?}"; shift; array tmpbuff
my outbuff="${2:-tmpbuff}"
}

sysfs=/sys
sysnet=$sysfs/class/net
sys_modules=$sysfs/module

sub rev () {
(($#==0)) && { echo ""; return 0 ;}
my element=${1:?}; shift;
(($#==0)) && { echo "$element"; return 0;}
echo "$(rev "$@") $element"
}

sub rename_if () {
my old_name=${1:?} new_name=${2:?}
echo ip link set name "$new_name" dev "$old_name"
}


sub down_if () {
my if_name=${1:?}
echo ip link set down dev "$if_name"
}


sub set_links_down() { # can't operate on up links
down_if eth2 down
down_if eth3 down
down_if eth4 down
down_if eth5 down
}


map act_hw2if=() #actual values (to be read in)
map act_if2hw=()
map XIF=() #tmp array to hold exchanged IF's



##inline data (should be in external file)
map hw2if=( [00:15:17:bf:be:b2]=eth0 [00:15:17:bf:be:b3]=eth1
[00:26:b9:48:71:e2]=eth2 [00:26:b9:48:71:e4]=eth3
[a0:36:9f:15:c9:c0]=eth4 [a0:36:9f:15:c9:c2]=eth5 )

map if2hw=( [eth0]=00:15:17:bf:be:b2 [eth1]=00:15:17:bf:be:b3
[eth2]=00:26:b9:48:71:e2 [eth3]=00:26:b9:48:71:e4
[eth4]=a0:36:9f:15:c9:c0 [eth5]=a0:36:9f:15:c9:c2 )
#


#needed_drivers=(e1000e bnx2 ixgbe bonding)
needed_drivers=(e1000e bnx2 ixgbe)

sub vrfy_drivers () {
int errors=0;
for i in ${needed_drivers[@]} ; do
if [[ ! -d $sys_modules/$i ]]; then
modprobe "$i" || {
warn "Module $i is not in kernel and can't be loaded"
errors+=1
}
fi
done
return $errors
}



sub get_net_IFnames_hwaddrs () {
vrfy_drivers
array pseudo_devs=(br bond ifb team)
string pseudo_RE='^(?:'"$(echo "${pseudo_devs[@]}"|tr " " "|")"')\d+$'
string netdev_pat="+([_0-9a-z])+([0-9])"
( cd "$sysnet" &&
for nm in $(eval "echo $netdev_pat" | tr ' ' "\n" |
sort | grep -Pv "$pseudo_RE"); do
echo "$nm $(<$nm/address)"
done )
}


sub read_actuals () {
my ifname hwaddr

while read ifname hwaddr; do
printf "ifname=%s, hwaddr=%s\n" "$ifname" "$hwaddr"
act_hw2if["$hwaddr"]="$ifname"
act_if2hw["$ifname"]="$hwaddr"
done <<<"$(get_net_IFnames_hwaddrs)"
}


sub ifaddr_cmd () {
if ((${#act_hw2if[@]:-0}==0)) ;then read_actuals; fi
my hwaddr
for ifname in $(printf "%s\n" "${act_hw2if[@]}"|sort|tr "\n" " ") ; do
my first_ifn="$ifname"
if [[ $ifname =~ \+ ]] ; then
first_ifn="${ifname%%+*}"
fi
printf "%s\t%s\n" "$ifname" "${act_if2hw["$first_ifn"]}"
done
}

sub ifmap_cmd () {
ifaddr_cmd "$@"
}


sub remap_cmd () {
if ((${#act_hw2if[@]}==0)); then read_actuals; fi
my key ifname
int count=0
map XIF act_hw2if act_if2hw

array ifnames=$(printf "%s\n" "${!if2hw[@]}"|sort|
grep -P '^[^~+]*$' |tr "\n" " ")

array rev_ifns=($(rev "${ifnames[@]}" ))

for key in "${rev_ifns[@]}"; do
int is_regex=0;
ifname="$key"
if [[ ${key:0:1} == ~ ]];then ifname=${key:1}; is_regex=1; fi

my hwaddr="${if2hw["$key"]:-""}"

my actual_hw="${act_if2hw["$ifname"]:-""}"

my actual_if="${act_hw2if["$actual_hw"]:-""}" ##line 233

if [[ ${actual_hw:-""} && ! $actual_hw =~ \+ ]]; then ##
if ((is_regex)); then [[ $actual_hw =~ $hwaddr ]] && continue
else [[ $actual_hw == $hwaddr ]] && continue; fi

if [[ ! ${act_if2hw["$ifname"]:-} ]]; then
#Nobody has the name, use it
down_if "$actual_hw"
rename_if "$actual_hw" "$ifname" ; count+=1
else
rename_if "$actual_if" "X$ifname"; #don't count temp renames 2x
XIF["X$ifname"]="$hwaddr"
fi
fi
done
if ((${#XIF[@]}==0)); then
echo "HW interfaces appear to be in order."; return 0; fi
int count=0
for ifname in "${!XIF[@]}"; do
hwaddr=${XIF[$ifname]};
ifname=${ifname#X}
my destname=${hw2if[$hwaddr]}
rename_if "$ifname" "$destname" ; count+=1
done
printf "%d interface%s renamed\n" $count "$( (($count!=1)) && echo "s" )"
}

sub start_cmd () {
remap_cmd "$@"
}

hash switches=([ifmap]=1 [remap]=1 [start]=1)

sub help () {
echo "$_prg:"
echo "Options: ifmap - show current hw# -> IF map"
echo " remap - verify & remap ifnames if needed"
return 1
}


if (($#)) ; then

dcl op="${1#:-}"

if [[ ${switches[$op]:-} ]]; then
cmd="${op}_cmd"
shift
$cmd "$@"
else
echo "Unknown switch :-$op"
fi
else
help
fi

[Greg Wooledge]

On Fri, Jul 25, 2014 at 04:44:27PM -0700, Linda Walsh wrote:
> FWIW, this script DOES work interactively in normal operation.
>
> Just when the system is in pre-single-user state and not
> much in the way of resources is available that it blows.

Then you are attempting to use operating system features that haven't
been initialized yet. Stop doing that.

Here documents create temporary files, possibly in /tmp, or possibly in
whatever directory $TMPDIR points to. So, /tmp (or /var/tmp or whatever)
must be mounted and writable.

Process substitutions **ON LINUX** use /dev/fd/* entries, which are
in a special file system that must be mounted. Process substitutions
on other operating systems may use that, or named pipes, or even
temporary files.

In any case, the problem is clearly that your script is trying to use
features that aren't available yet, at that point in the boot process.
Rewrite the script to avoid using those features, or run it at a
different point in the boot process.

> #!/bin/bash
>
> ### BEGIN INIT INFO
> # Provides: net-devices

Too large. Trim it down to the smallest possible case that still
exhibits the problem. But even then, I'm sure you already know what
the problem is.

(P.S. I despise that "let me figure out the order for you" crap that
Debian has started doing. What a disaster. How in the hell is a
sysadmin supposed to add a boot script now, and make sure it runs at
the right point....)

[Linda Walsh]

Greg Wooledge wrote:
> On Fri, Jul 25, 2014 at 04:44:27PM -0700, Linda Walsh wrote:
>> FWIW, this script DOES work interactively in normal operation.
>>
>> Just when the system is in pre-single-user state and not
>> much in the way of resources is available that it blows.
>
> Then you are attempting to use operating system features that haven't
> been initialized yet. Stop doing that.

----
Except that in-line HERE docs don't need to be implemented
through a tmp file unless you want to slow things down.
They should be read out of memory and NOT transfered to
to non-existent, external storage.

Maybe POSIX says differently, but I doubt it differentiates
when you are allowed to use features based on when you run the shell.

>
> Here documents create temporary files, possibly in /tmp, or possibly in
> whatever directory $TMPDIR points to. So, /tmp (or /var/tmp or whatever)
> must be mounted and writable.

----
Is that part of a standard somewhere? The point is to similate
an input stream. Wouldn't it be more portable to do that from memory?

>
> Process substitutions **ON LINUX** use /dev/fd/* entries, which are
> in a special file system that must be mounted. Process substitutions
> on other operating systems may use that, or named pipes, or even
> temporary files.

----
I'll allow those as they are too new to really be relying on,
but HERE docs are basic, and shouldn't need external run-time support for
the type of stuff I'm doing.

> In any case, the problem is clearly that your script is trying to use
> features that aren't available yet, at that point in the boot process.

---
Like reading another process's output so I can alter
variables that are in the main thread.

The other methods for reading in data reverse the reader/writer
and put the reader into a child such that it can't modify variables
in the running script.

> Rewrite the script to avoid using those features, or run it at a
> different point in the boot process.

----
I have rewritten it to avoid the process substitutions and fell
back to HERE docs for that. There is no standard or documentation about
what parts of a POSIX shell are not available at boot time or how to
work around the deficiencies.

Also it isn't easy to run things elsewhere in the boot script, as I'll
likely have to run systemd at soem point and it will just take over and
I'll have no control of when things run, so I'm doing some early initialization
so I can then turn over the control thread to something else.

For example, I also have to mount "/usr" and "/usr/share" before
I can "boot", or systemd won't be happy.

> Too large. Trim it down to the smallest possible case that still
> exhibits the problem. But even then, I'm sure you already know what
> the problem is.

----
That's not easy, since by definition, it only works when there
is no environment to support trimming it.

>
> (P.S. I despise that "let me figure out the order for you" crap that
> Debian has started doing. What a disaster. How in the hell is a
> sysadmin supposed to add a boot script now, and make sure it runs at
> the right point....)

----
You aren't supposed to be adding boot scripts...

Wait till they get the walled garden up around PC's w/secure boot
and only allowing signed apps.

GRRRR.

[Andreas Schwab]

Linda Walsh <ba...@tlinx.org> writes:

> Except that in-line HERE docs don't need to be implemented
> through a tmp file unless you want to slow things down.
> They should be read out of memory and NOT transfered to
> to non-existent, external storage.

You need a file descriptor for your memory storage.

Andreas.

--
Andreas Schwab, SUSE Labs, sch...@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[Linda Walsh]

Andreas Schwab wrote:
> Linda Walsh <ba...@tlinx.org> writes:
>
>> Except that in-line HERE docs don't need to be implemented
>> through a tmp file unless you want to slow things down.
>> They should be read out of memory and NOT transfered to
>> to non-existent, external storage.
>
> You need a file descriptor for your memory storage.

---
Why?

If it is the case that you can't rely on OS services, you are
saying you can't access memory or create a virtual descriptor
that reads out of a large malloc'ed buffer?

Both C++ and Perl have routines to do I/O on strings directly
that don't go through system file descriptors to do it.

Last I checked, 'C' was a common denominator for both as well
as Bash. So why should it be the case that Bash can't do it when
it is the primary system shell available at boot.

I wrote my own line-splitting, word splitting and buffering code
to avoid libc buffering to talk to files in "proc", repeatedly
w/o closing & reopening. It's mostly in C and was relatively
simple compared to the work of writing a simple scheduler.

There's no reason for me to expect that a system shell wouldn't
have such available.

[Greg Wooledge]

On Tue, Jul 29, 2014 at 12:06:41PM -0700, Linda Walsh wrote:
> Andreas Schwab wrote:
> >You need a file descriptor for your memory storage.
> ---
> Why?

A here-document is a redirection. All it does it change where stdin
comes from. There has to be a place for file descriptor 0 to point to.
This can be a real file on disk, or one end of a pipe().

I'm not aware of any shells that use a multi-process pipe() implementation
for here-documents. Maybe this is just historical inertia, or maybe it's
because a temporary file can be lseek()ed, which gives the most flexibility
for the processes that are reading them. Or maybe it's because a temp
file is considered less of a resource waste than an extra process.

I really do advise you to simplify your boot scripts. Boot scripts run in
minimalist environments, where advanced features are not always available.
Boot scripts also typically are not run by bash; they're run by /bin/sh
(or even /sbin/sh), which is more likely to be some stripped-down POSIX
shell than it is to be bash.

If you've got some big static block of text that you need to feed to a
daemon at boot time, maybe it would be appropriate to stick this text in
a config file (/etc/default/yourdaemon or /etc/yourdaemon/startup.text or
something like that). Then you can redirect from that file instead of
using a here-document buried in a boot script. From a system administration
perspective, this separation of code from data would be a lot cleaner.

[Angel]

Linda Walsh wrote
>
> Andreas Schwab wrote:

> > Linda Walsh <ba...@tlinx.org> writes:
> >
> >> Except that in-line HERE docs don't need to be implemented
> >> through a tmp file unless you want to slow things down.
> >> They should be read out of memory and NOT transfered to
> >> to non-existent, external storage.
> >

> > You need a file descriptor for your memory storage.
> ---
> Why?

Because that's what the input command expects: your HERE-string *when it
reads its standard input* (fd 0)

You could implement it with pipes, but as you would need a thread to
write the partially-read string, just using a file is simpler. And as
you can write it into a deleted file, the effect on storage should be
minimal.

You should however be able to translate “foo <<<SOMETHING” into “echo
SOMETHING | foo”


What surprises me is that the only use of here-docs (actually
here-strings) in your script are process substitutions:

> read out <<<$(declare -p "$var" )

> while ... done <<<"$(get_net_IFnames_hwaddrs)"

When it looks simpler to write the non-here version:
> declare -p "$var" | read out
> get_net_IFnames_hwaddrs | while ...


Regards

[Greg Wooledge]

On Tue, Jul 29, 2014 at 10:21:36PM +0200, Angel wrote:
> What surprises me is that the only use of here-docs (actually
> here-strings) in your script are process substitutions:
> > read out <<<$(declare -p "$var" )
> > while ... done <<<"$(get_net_IFnames_hwaddrs)"
>
> When it looks simpler to write the non-here version:
> > declare -p "$var" | read out
> > get_net_IFnames_hwaddrs | while ...

The 'read' example will not work as you've written it. The change to
the shell variable 'out' will be lost when the pipeline terminates.
(But you can get a very recent bash release and set the "lastpipe"
shopt to work around this.)

If the while loop also tries to set shell variables, then it will have
the same problem (and the same possible workaround).

It looks like Linda's original script was doing something like this:

read out < <(declare -p "$var")

and she discovered that she couldn't use process substitutions in this
boot script (presumably because Linux's /dev/fd/ file system wasn't
mounted yet), so she tried to replace it with a here-document, and now has
learned that /tmp is not mounted (or is mounted read-only) at that point.

I won't even try to guess why she can't just do out=$var ... there is
probably some extremely silly reason that will just make me want to slam
my head into my desk....

[Linda Walsh]

Greg Wooledge wrote:
> read out < <(declare -p "$var")

----
That code actually isn't called. It's used by "isarray" to tell me whether
or not a var is an array.

The code that doesn't work is the line with the comment "line 233".
(which is now line 235 from the latest error message.

<notice -- Jul 22 19:02:49.388388000> service boot.clock start
assign_netif_names=/etc/init.d/boot.assign_netif_names start
ifname=eth0, hwaddr=00:15:17:bf:be:b2
ifname=eth1, hwaddr=00:15:17:bf:be:b3
ifname=eth2, hwaddr=00:26:b9:48:71:e2
ifname=eth3, hwaddr=00:26:b9:48:71:e4k
ifname=eth4, hwaddr=a0:36:9f:15:c9:c0
ifname=eth5, hwaddr=a0:36:9f:15:c9:c2
/etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array
subscript
/etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array
subscript
/etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array
subscript
/etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array
subscript
/etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array
subscript
/etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array
subscript

Note that according to the "printf in "read_actuals", the values are
correctly
being read in from my "here var".

Note, you can't use an assignment there.

It's reading names+hwaddrs of the net devs @ boot
in "get_net_IFnames_hwaddrs". That requires line and field
splitting.


Now that I'm writing about it -- I'm thinking that
it doesn't like me using the hwaddr's @ boot time as hash subscripts.

That's what I don't understand. It works "normally", but
why would "hashes" fail at boot? I couldn't think of how they'd
be implemented to use some non-existent service, so I don't
know which resource lack is causing the fail.

> I won't even try to guess why she can't just do out=$var ... there is
> probably some extremely silly reason that will just make me want to slam
> my head into my desk....
>

---
Yeah, like multiple lines and fields in "$var"...

[Angel]

Greg Wooledge wrote:
> The 'read' example will not work as you've written it. The change to
> the shell variable 'out' will be lost when the pipeline terminates.
> (But you can get a very recent bash release and set the "lastpipe"
> shopt to work around this.)
>
> If the while loop also tries to set shell variables, then it will have
> the same problem (and the same possible workaround).

You are right. Turns out I didn't reset the variable between tests.


Or you can put the inside-code into brackets :)

[Linda Walsh]

I got various suggestions for getting my script to work at boot time,

and any of them might have been applicable before I'd gotten to the current
problem...

In particular the <<<"$VAR" construct where VAR was holding
output from a program.. was something on my "iffy" list,
but it seemed to work and not be at fault upon further
exploration. Now it's one of 2 Associative arrays (often
called "'map's" in the code where they are used as such)
that is failing due to illegal subscript messages.

The fact that one of the maps works and the other does not
seems odd. They are both initialized the same way.

I've also tried resetting the environment...
(env -i bash --norc --noprofile;
bash-4.2$ ...command works...)

Is it really the case that now that the problem is better
defined, that it really is just some problem in bash
that can only reproduced on a booting system?

Note -- if you invoke the script with with no parms, it
display normal options "ifmap" (shows current mapping)
and "remap" which does a verify on the names and does
renaming if needed (based on it's internal table -- i.e.
don't use this unless you've adapted it to your system).

undocumented: if you invoke it with 'start' it does the "remap" function.

Other than a few left-over functions from earlier debug
attempts (tests removed, but some of the functions remain
so I can maybe move them to a library later (e.g. - isarray
& varflags), it is fairly lean for design of non-throwaway code.

[Chet Ramey]

> but it seemed to work and not be at fault upon further
> exploration. Now it's one of 2 Associative arrays (often
> called "'map's" in the code where they are used as such)
> that is failing due to illegal subscript messages.
>
> The fact that one of the maps works and the other does not
> seems odd. They are both initialized the same way.

Have you considered printing the value you're trying to use as a
subscript before the failing line is executed?

> Is it really the case that now that the problem is better
> defined, that it really is just some problem in bash
> that can only reproduced on a booting system?

Unlikely.

[Linda Walsh]

Chet Ramey wrote:
>> but it seemed to work and not be at fault upon further
>> exploration. Now it's one of 2 Associative arrays (often
>> called "'map's" in the code where they are used as such)
>> that is failing due to illegal subscript messages.
>>
>> The fact that one of the maps works and the other does not
>> seems odd. They are both initialized the same way.
>
> Have you considered printing the value you're trying to use as a
> subscript before the failing line is executed?

----
It's been a while, but I seem to remember doing that
and setting -x at the beginning of the script and picking through
that post-boot.

I try it again... might easily get a different result
with how things are going for me lately...