combine xargs with EOF

[giuseppe...@gmail.com]

Hi
i'm using often xargs to run process in parallel

ls *.txt | xargs -n 1 -P 10 bash myscript.sh

where myscript.sh is (for examples of course is longer):
echo $1

i would like to perform the same operation using the EOF syntax and import the arguments inside the eof

I tried

ls *.txt | xargs -n 1 -P 10 <<
echo $1
EOF

also with the option -E and -e but did not work

any ideas?
thanks Giuseppe

[DJ Mills]

xargs can't read both from a pipe and a here document, they're both on stdin.

Also, don't parse ls. http://mywiki.wooledge.org/ParsingLs

But I'm not entirely sure what you're trying to do here... Can you
explain the whole goal, in plain language, without using code?

Also, make sure you quote your expansions. "$1", never $1. See
http://mywiki.wooledge.org/Arguments.

The other issue is that since <<EOF is unquoted, $1 will be expanded
before the here document is passed to xargs. If there are no
arguments, it will simply get 'echo'.

[Greg Wooledge]

On Wed, Oct 17, 2012 at 09:49:10AM -0700, giuseppe...@gmail.com wrote:
> ls *.txt | xargs -n 1 -P 10 bash myscript.sh

This has some serious flaws already. ls may mangle the filenames that
it is feeding to xargs. Any filenames that have spaces or quotes in
them will be bungled by xargs itself. Any filenames that contain newlines
will most definitely be mishandled.

> where myscript.sh is (for examples of course is longer):
> echo $1

Use more quotes!

> i would like to perform the same operation using the EOF syntax and import the arguments inside the eof
>
> I tried
>
> ls *.txt | xargs -n 1 -P 10 <<
> echo $1
> EOF

You mean a here document. But it's not clear what you intended. A here
document is a redirection of stdin. But xargs's stdin is already coming
from the pipeline. (Also you forgot the EOF sentinel after <<.)

> also with the option -E and -e but did not work

Presumably you mean the -E option of xargs, which is "-E eof-str ...
Set the end of file string to eof-str. If the end of file string occurs
as a line of input, the rest of the input is ignored."

You seem to be conflating this with bash's here documents somehow.
xargs's -E is just an emergency "abort!" button, to stop processing input
at a certain point.

But... I am still having trouble guessing what result you want.
Was the purpose of the here document simply to avoid having a second
file containing a bash script? If so, then what you probably wanted
was something like:

find . -maxdepth 1 -name '*.txt' -print0 |
xargs -0 -n 1 -P 10 bash -c '
echo "$1"
'

Note that there are several GNU extensions in this example (on both find
and xargs). There is no POSIX equivalent to xargs -P, so this whole
approach is non-portable in the first place.

[giuseppe...@gmail.com]

Sorry i was not clear,

yes my purpose is

" simply to avoid having a second file containing a bash script "

but

find . -maxdepth 1 -name '*.txt' -print0 | xargs -n 1 -P 10 bash -c 'echo "$1" '

or

ls '*.txt' | xargs -n 1 -P 10 bash -c 'echo $1 '

do not print $1 so the argument (-n 1) is not passed inside.

any idea?

Thanks
Giuseppe
p.s. my files are all without space.

[giuseppe...@gmail.com]

Sorry i was not clear,

yes my purpose is

" simply to avoid having a second file containing a bash script "

[Greg Wooledge]

On Wed, Oct 17, 2012 at 10:52:37AM -0700, giuseppe...@gmail.com wrote:
> find . -maxdepth 1 -name '*.txt' -print0 | xargs -n 1 -P 10 bash -c 'echo "$1" '

> do not print $1 so the argument (-n 1) is not passed inside.

OK, first thing: you omitted the -0 on xargs.

Next, please realize that I'm not accustomed to this -n -P stuff at all,
so it's not clear what you're trying to accomplish beyond the basic
mechanics of passing the filenames to bash. You want to print filenames
10 at a time in parallel? It's a weird question.

If you want the filename to be given as an argument to bash, you actually
need a placeholder after the -c 'script' part. The first argument after
the script becomes $0, and then the arguments after that become $1, $2, etc.
So:

find . -maxdepth 1 -name '*.txt' -print0 |

xargs -0 -n 1 -P 10 bash -c 'echo "==start $$"; echo "$1"; echo "==end $$"' _

(I added echoes of the start/end of each invoked bash process so I could
see what it's doing.)

[DJ Mills]

"Simplified examples" do not help, they just confuse things even more.

Can you explain what the ACTUAL goal is? And just because you
currently have filenames without spaces doesn't mean that will always
be the case, or that it's OK to write unsafe code.

[giuseppe...@gmail.com]

so this is my actual gol:

change this loop ( which is working fine )

for file in /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_tiff/*.tif ; do
filename=`basename $file .tif`
tile=${filename:3:6}
echo processin $file
oft-stat $file /weldgfs/p51/gius_urban/LandCover/tree_cover/TREE_COVER_$tile.tif /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_tree/UA_tree$tile.txt
oft-stat $file /weldgfs/p51/gius_urban/LandCover/bare_ground/BARE_GROUND_$tile.tif /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_bare/UA_bare$tile.txt
done

in

ls /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_tiff/*.tif | xargs -n 1 -P 10 bash -c '
file="$1"
filename=`basename $file .tif`
tile=${filename:3:6}
echo processin $file
oft-stat $file /weldgfs/p51/gius_urban/LandCover/tree_cover/TREE_COVER_$tile.tif /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_tree/UA_tree$tile.txt
oft-stat $file /weldgfs/p51/gius_urban/LandCover/bare_ground/BARE_GROUND_$tile.tif /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_bare/UA_bare$tile.txt
' _


for multi process in parallel, without call an external script. . of course the echo $1 was a way to simplify the full task. Now the script works fine
Thanks to all you
Ciao
Giuseppe

[giuseppe...@gmail.com]

[DJ Mills]

You're still lacking quotes. "$file", never $file. And you have the
same issue with ls.

printf '%s\0' /weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_tiff/*.tif
| xargs -0 -n 1 -P 10 bash -c '
file="$1"
filename=${file##*/} filename=${filename%.tif}
tile=${filename:3:6}
echo "processing $file"

oft-stat "$file"
"/weldgfs/p51/gius_urban/LandCover/tree_cover/TREE_COVER_$tile.tif"
"/weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_tree/UA_tree$tile.txt"
oft-stat "$file"
"/weldgfs/p51/gius_urban/LandCover/bare_ground/BARE_GROUND_$tile.tif"
"/weldgfs/p51/gius_urban/pop_urban/tl_2010_us_uac10/UA_bare/UA_bare$tile.txt"
' _

would be one option...

or do the parallel processing in bash itself... see
http://mywiki.wooledge.org/ProcessManagement
and
https://github.com/e36freak/templates/blob/master/threads or
https://github.com/e36freak/templates/blob/master/parallel