bash question

[OBBARD Darren]

Hi!

Does the code monkeys list still work?


My objective is to read through a list of file stem names, test if that file does not exist on a remote computer, and wait for it to appear (separate to this loop) before continuing to read through the list.



This "while" approach does not work, if the remote file does exist, the outer loop exits after a single step

while read stem
do
echo "looking for $stem"
while ssh -q dob...@computer.bio.ed.ac.uk [[ ! -f /path/$stem/$stem.log ]] ; do echo "waiting for $stem" ; done
echo "found $stem"
done < ../sstems



This "for" approach DOES work.

for stem in $(cat ../sstems)
do
echo "looking for $stem"
while ssh -q dob...@computer.bio.ed.ac.uk [[ ! -f /path/$stem/$stem.log ]] ; do echo "waiting for $stem" ; done
echo "found $stem"
done

Why? This has just wasted an hour of my life!

Thanks!

D
--

Darren Obbard
darren...@ed.ac.uk

Institute of Evolutionary Biology
University of Edinburgh
Ashworth Laboratories, Charlotte Auerbach Road
EdinburghEH9 3FL

Office 0131 651 7781
Mobile: 07968 838 635

http://obbard.bio.ed.ac.uk/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

[CHALLIS Richard]

Hi Darren

The list still works:

Ssh reads from STDIN so there is nothing left to read after the first iteration of the while loop.

Rich

> --
> The wiki is at:
> https://www.wiki.ed.ac.uk/display/AshCodes/Ashworth+Codemonkeys
> The mailing list archive is at:
> https://groups.google.com/forum/?fromgroups#!forum/ashworth-code-monkeys
> If you have trouble editing the wiki or emailing the group, let me know: sujai...@ed.ac.uk
> ---
> You received this message because you are subscribed to the Google Groups "Ashworth Codemonkeys" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ashworth-code-mo...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/ashworth-code-monkeys/VI1PR0502MB2894FBDF17FDB6CD212B33F1DFC60%40VI1PR0502MB2894.eurprd05.prod.outlook.com.

[Dominik R. Laetsch]

Hmm... no idea.

But there are tools that do these things and maybe you want to do it that way:

https://superuser.com/a/1114538

https://stackoverflow.com/questions/12460279/how-to-keep-two-folders-automatically-synchronized 

There is also https://github.com/axkibe/lsyncd#readme (see https://stackoverflow.com/a/15283061)

Cheers, 

dom

On 2 Apr 2020, at 09:32, OBBARD Darren <darren...@ed.ac.uk> wrote:

[CHALLIS Richard]

And I just discovered that you can add -n to your ssh to prevent reading from stdin so 

while read stem
do
echo "looking for $stem"

while ssh -n -q dob...@computer.bio.ed.ac.uk [[ ! -f /path/$stem/$stem.log ]] ; do echo "waiting for $stem" ; done

echo "found $stem"
done < ../sstems

should work

To view this discussion on the web, visit https://groups.google.com/d/msgid/ashworth-code-monkeys/D2FD8A85-3264-446C-B9BD-E92671BA373F%40ed.ac.uk.

[OBBARD Darren]

Hi all!

> But there are tools that do these things and maybe you want to do it that way:

I don't wish to sync two folders! (broadly: test if a remote file is created yet, if so download some other files and analyze them, then upload the result, and delete everything temporary. Waiting time is likely to be minutes, but not longer)

> Ssh reads from STDIN so there is nothing left to read after the first iteration of the while loop.

Fantastic! Thank you

D.

--

Darren Obbard
darren...@ed.ac.uk

Institute of Evolutionary Biology
University of Edinburgh
Ashworth Laboratories, Charlotte Auerbach Road

Edinburgh EH9 3FL

Office 0131 651 7781
Mobile: 07968 838 635

http://obbard.bio.ed.ac.uk/

> https://groups.google.com/d/msgid/ashworth-code-monkeys/D2FD8A85-
> 3264-446C-B9BD-E92671BA373F%40ed.ac.uk.

[Tim Booth]

Hi Darren,

Good to see the old code monkeys list back in action.

My only comment here would be to consider if you can use shfs or gvfs to mount the remote location directly into the local filesystem for this task. This would simplify your code a lot - unfortunately there are several other gotchas to be discovered when making direct calls to SSH like this.

Cheers,

TIM

-- 
====================================================
Edinburgh Genomics' Privacy Notice can be viewed at:
http://genomics.ed.ac.uk/about-us/privacy-notice

This email and any attachments are confidential and
intended solely for the use of the recipient(s) to
whom they are addressed. If you have received it in
error, please destroy all copies and inform the
sender.
====================================================

[Hywel Dunn-Davies]

Hi Darren,

Just to add to Tim's answer, I've been sshfs (https://wiki.archlinux.org/index.php/SSHFS), which makes it pretty straightforward to mount a remote filesystem over ssh.

All the best,

Hywel

To view this discussion on the web, visit https://groups.google.com/d/msgid/ashworth-code-monkeys/0b794c33954c1f6328114c9e052341fe9aef1c80.camel%40ed.ac.uk.