Process Duplication Within for Loop

Java Jive

unread,

Jul 24, 2021, 9:40:11 AM7/24/21

to

I have the following piece of code creating an rsync process launched in
the early hours of every morning to sync a NAS from another upstream. I
have only included the business part of the file, the beginning of it is
to do with setting up the variable values used in these lines. As you
will note, there are two debugging statements trapping the output of ps ...

# Debug Line 1:
ps | grep -v grep | grep tutsync.sh > $HD/rsync.log
echo "Syncing the following shares with rsync: ${SHARES}" >> $HD/rsync.log
for SHARE in ${SHARES}
do
# Debug Line 2:
ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
echo "nice -n ${NICE} rsync ${OPTIONS}
${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
${BU}/${SHARE}/
done | grep -v '/$' >> $HD/rsync.log 2>&1

... because, when run, the process appears to duplicate itself within
the for loop - not only does the tutsync.sh process itself get
duplicated, but also the rsync process it launches as well, as can be
proven by launching with a trailing & and running a similar command from
the shell:

~ # cat $HD/rsync.log
32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
Syncing the following shares with rsync: A List Of Shares
32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
32458 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
receiving incremental file list
etc

~ # pf rsync
9680 root 1972 S /opt/bin/rsync --ipv4 --daemon
--config=/etc/rsyncd.conf
32467 root 2908 R N rsync -Drltugpv --exclude=*.flv
--exclude=*.partial.* --exclude=*.part --exclude=**/.streams
32468 root 49772 S N rsync -Drltugpv --exclude=*.flv
--exclude=*.partial.* --exclude=*.part --exclude=**/.streams

What is going on, why are the processes duplicating themselves?

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

Martin Gregorie

unread,

Jul 24, 2021, 11:28:41 AM7/24/21

to

FWIW my weekly backup process is run from the host with the backup disk
mounted on it and uses rsync to back up itself as well as other hosts on
my LAN. Backups are run from a menu with each backup being triggered
manually. I've watched this run using 'top' and seen duplicate copies of
rsync get spawned from time to time: the duplicated rsync processes are
not present in all backup runs and and can come and go during a backup.

I have no idea why the duplicate processes are created, but the backups
always complete without errors reported and seem to not duplicate,
corrupt or omit any files, so I just accept that this is what rsync does.

For an explanation, look at this (which I've just found):
https://serverfault.com/questions/547165/why-does-rsync-spawn-multiple-
processes-for-me#547194

--
Martin | martin at
Gregorie | gregorie dot org

Richard Kettlewell

unread,

Jul 24, 2021, 12:35:47 PM7/24/21

to

The first half of the main pipeline (for ... done) runs in a separate
process.

rsync uses multiple processes as part of its normal operation.

--
https://www.greenend.org.uk/rjk/

Java Jive

unread,

Jul 24, 2021, 1:15:52 PM7/24/21

to

I have the following piece of code creating an rsync process launched in
the early hours of every morning to sync a NAS from another upstream. I
have only included the business part of the file, the beginning of it is
to do with setting up the variable values used in these lines. As you
will note, there are two debugging statements trapping the output of ps ...

# Debug Line 1:
ps | grep -v grep | grep tutsync.sh > $HD/rsync.log
echo "Syncing the following shares with rsync: ${SHARES}" >> $HD/rsync.log
for SHARE in ${SHARES}
do
# Debug Line 2:
ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
echo "nice -n ${NICE} rsync ${OPTIONS}
${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
${BU}/${SHARE}/
done | grep -v '/$' >> $HD/rsync.log 2>&1

.... because, when run, the process appears to duplicate itself within

the for loop - not only does the tutsync.sh process itself get
duplicated, but also the rsync process it launches as well, as can be
proven by launching with a trailing & and running a similar command from
the shell:

~ # cat $HD/rsync.log
32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
Syncing the following shares with rsync: A List Of Shares
32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
32458 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
receiving incremental file list
etc

~ # pf rsync
9680 root 1972 S /opt/bin/rsync --ipv4 --daemon
--config=/etc/rsyncd.conf
32467 root 2908 R N rsync -Drltugpv --exclude=*.flv
--exclude=*.partial.* --exclude=*.part --exclude=**/.streams
32468 root 49772 S N rsync -Drltugpv --exclude=*.flv
--exclude=*.partial.* --exclude=*.part --exclude=**/.streams

What is going on, why are the processes duplicating themselves?

Martin Gregorie

unread,

Jul 24, 2021, 1:15:53 PM7/24/21

to

On Sat, 24 Jul 2021 14:40:08 +0100, Java Jive wrote:

Richard Kettlewell

unread,

Jul 24, 2021, 1:15:53 PM7/24/21

to

Java Jive

unread,

Jul 24, 2021, 5:21:47 PM7/24/21

to

Thanks for both the replies. It seems there's nothing to worry about,
and the script seems to have been working aright anyhow.

Java Jive

unread,

Jul 24, 2021, 7:18:58 PM7/24/21

to

J.O. Aho

unread,

Jul 25, 2021, 5:08:10 AM7/25/21

to

Running ps with the f option will give you a clearer picture of process
relations

5467 ? Ss 0:02 sshd: /usr/sbin/sshd -f /etc/ssh/sshd_config
13958 ? Ss 0:00 \_ sshd: user1 [priv]
13960 ? S 0:01 \_ sshd: user1@pts/4
13962 pts/4 Ss 0:00 \_ -bash
18569 pts/4 S 0:00 \_ su -
18571 pts/4 S 0:00 \_ -su
20700 pts/4 R+ 0:00 \_ ps axf

--

//Aho

Java Jive

unread,

Jul 25, 2021, 6:39:34 AM7/25/21

to

On 25/07/2021 10:08, J.O. Aho wrote:
>
> Running ps with the f option will give you a clearer picture of process
> relations
>
> 5467 ?        Ss     0:02 sshd: /usr/sbin/sshd -f /etc/ssh/sshd_config
> 13958 ?        Ss     0:00 \_ sshd: user1 [priv]
> 13960 ?        S      0:01      \_ sshd: user1@pts/4
> 13962 pts/4    Ss     0:00          \_ -bash
> 18569 pts/4    S      0:00              \_ su -
> 18571 pts/4    S      0:00                  \_ -su
> 20700 pts/4    R+     0:00                      \_ ps axf

That's a useful general tip which I didn't know about before, thanks.
However, in this particular instance, this is an embedded device, and
the busybox version of ps doesn't support that option, more's the pity.

J.O. Aho

unread,

Jul 25, 2021, 7:16:03 AM7/25/21

to

Running ps with the f option will give you a clearer picture of process
relations

5467 ? Ss 0:02 sshd: /usr/sbin/sshd -f /etc/ssh/sshd_config
13958 ? Ss 0:00 \_ sshd: user1 [priv]
13960 ? S 0:01 \_ sshd: user1@pts/4
13962 pts/4 Ss 0:00 \_ -bash
18569 pts/4 S 0:00 \_ su -
18571 pts/4 S 0:00 \_ -su
20700 pts/4 R+ 0:00 \_ ps axf

--

//Aho

Java Jive

unread,

Jul 25, 2021, 7:16:04 AM7/25/21

to

On 25/07/2021 10:08, J.O. Aho wrote:
>

> Running ps with the f option will give you a clearer picture of process
> relations
>
> 5467 ?        Ss     0:02 sshd: /usr/sbin/sshd -f /etc/ssh/sshd_config
> 13958 ?        Ss     0:00 \_ sshd: user1 [priv]
> 13960 ?        S      0:01      \_ sshd: user1@pts/4
> 13962 pts/4    Ss     0:00          \_ -bash
> 18569 pts/4    S      0:00              \_ su -
> 18571 pts/4    S      0:00                  \_ -su
> 20700 pts/4    R+     0:00                      \_ ps axf