Persistent ssh connection?

[Christopher Alemany]

Hi all,

I've been having a lot of trouble lately (actually for many months) with wview not being able to upload via SSH reliably. It seems the 1min upload period is too much now combined with my web host and provider possibly.... I don't know. I have to either restart wview or the computer as a whole to reestablish the updates.

My thought is whether it would be feasible for wview to setup a persistent ssh connection to avoid constant connection/disconnection traffic. This could perhaps support updates of 60sec or less more robustly?

In the meantime I'm going to see if I can setup a script that could run and make the connection itself and avoid using wviewssh to see if that helps.

Thanks for any thoughts or ideas.

Sent from my iPad
Chris Alemany
3854 6th Ave Port Alberni, BC
V9Y 4M2

[Stan]

I don't have a solution, but I'd be interested in anything  that might help.  Failures for me might be after a couple of days or sometimes a month or more.  Restart or reboot is the only way I know to get it back up.

 

Thanks,

Stan

 

[Christopher Alemany]

I've been working on a script with the 'expect' system. I haven't fully automated the startup and shutdown of the script yet and need to simplify things a bit, but I've got the basics for now. It's at least allowed me to turn off wviewssh. If all I need to do is rerun the script after 84+ hours then I'm already much further ahead.

I'm running it on MacOS 10.6. You need to have 'expect' installed for this to all work. Easy to do if you've installed wview with ports..... 'ports install expect' Other systems do what you need to do :)

Here's the script I've made and am running manually for now.

#!/usr/bin/expect
set user "username"
set pass "password"
set host "host address"
set count 0

spawn /usr/bin/sftp -oBatchMode=no $user@$host
expect "password"
send "$pass\r"
expect "sftp>"
send "cd img\r"
expect "sftp>"
send "lcd /opt/local/var/wview/img\r"
while {$count<10000} {
expect "sftp>"
send "put 1minArc.htm\r"
expect "sftp>"
send "put phpparameterlist.htm\r"
sleep 30
incr count
}
send "exit"

> --
> You received this message because you are subscribed to the Google Groups "wview" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/wview/-/IUYKYeIX6FoJ.
> To post to this group, send email to wv...@googlegroups.com.
> To unsubscribe from this group, send email to wview+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/wview?hl=en.

Chris Alemany
3854 6th Ave
Port Alberni, BC
V9Y4M2

h:250-723-0889
w: 250-753-3245 x2106
e: chri...@gmail.com
w: http://www.norailnocoal.ca

[Warren Gill]

Why wouldn't you just use rsync? You just pass everything the command
line, and rsync figures out automagically which files are newer and
sends only those.
\\'9

[Christopher Alemany]

Hi Warren,

rsync is what wviewssh uses and I think (but don't have any proof) that while doing its automagic figuring, rsync is actually causing a backlog on my webhost so that it doesn't finish before the next connection/update in 60sec.... eventually either crashing wviewssh or causing my webhost to become unresponsive (both of which I've observed).

I only have 5 files to update, and only 2 that need to be updated every 60sec so checking the 200 files in the img directory every 60sec doesn't make much sense to me.

So my hope is that by setting up a semi-permanent connection, I take away those constant connection attempts and also only update the files that need to be updated.

I've been dealing with wview crashing on me for months now and thought I had it mostly licked with a script I had that tried to restart wview when it was not uploading properly, but it's gotten really bad of late, and has led to some people complaining who rely on the 60sec feed for research... I needed to look for a new approach! :)

Chris

[Graham Eddy]

hello chris,

the researchers who need the data every 60 secs, is it just the data they need, or is it the files generated by htmlgend?

if the former (i.e. the data), just use the "client interface" to pass it along - write a simple program that reads from a particular port (defined by wview), and wview will automatically push the binary data into that port for you every 60 secs until you close the port.

(internally, it is wvalarmd that writes to the port, so alarms need to be enabled even if you don't have any)

OR

could generate the files you want to transfer in a subdirectory and only transfer that subdirectory

(i haven't looked to see if wviewssh supports wildcards; if not, use a whole subdir).

if necessary, generate those files twice, one in main dir and second ion subdir

---

Graham Eddy

[Michael]

Which version are you running - there's a fix in the current version
for 60 sec rsync updates

[Christopher Alemany]

5.19.0

What does the fix entail?

Chris

> --
> You received this message because you are subscribed to the Google Groups "wview" group.
> To post to this group, send email to wv...@googlegroups.com.
> To unsubscribe from this group, send email to wview+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/wview?hl=en.
>

Chris Alemany

3854 6th Ave
Port Alberni, BC

[Christopher Alemany]

Hi Graham.

Thanks. Ya unfortunately the researchers (at http://www.islandweather.ca), as well as my web software that I created, depend on single files in the root of the website. I also don't trust my home ISP and their router enough right now to use the data ports. I'd love to use them for an iOS app.

Not only that, I've also realized that I could consolidate some transfers that I'm doing right now for webcams and things using this script down to a 15sec interval, which would be ideal.. at least if I could get it running at boot time.

Chris

On 2011-07-12, at 11:44 PM, Graham Eddy wrote:

> hello chris,
> the researchers who need the data every 60 secs, is it just the data they need, or is it the files generated by htmlgend?
> if the former (i.e. the data), just use the "client interface" to pass it along - write a simple program that reads from a particular port (defined by wview), and wview will automatically push the binary data into that port for you every 60 secs until you close the port.
> (internally, it is wvalarmd that writes to the port, so alarms need to be enabled even if you don't have any)
>
> OR
>
> could generate the files you want to transfer in a subdirectory and only transfer that subdirectory
> (i haven't looked to see if wviewssh supports wildcards; if not, use a whole subdir).
> if necessary, generate those files twice, one in main dir and second ion subdir

[Michael]

I tried to find it in mantis but its not there anymore.

The issue I found was if the rsync update took longer than 60 seconds
then wview would reenter the code already running and start another
update

I found once that happened things would hang and all you could do is
restart wview

I think you need to set a timeout to abort the update after about 10
seconds. I used 10 seconds because I figured the network is either up
and working so it should happen quickly. If not then its probably a
network problem and its going to take a while and maybe not update at
all.

I realise you are using rsync however I think you may find the issue
is applicable

On Jul 14, 12:55 am, Christopher Alemany <chris...@gmail.com> wrote:
> 5.19.0
>
> What does the fix entail?
>
> Chris
> On 2011-07-13, at 3:39 AM, Michael wrote:
>
>
>
>
>
>
>
>
>
> > Which version are you running - there's a fix in the current version
> > for 60 sec rsync updates
>
> > On Jul 12, 4:38 am, Christopher Alemany <chris...@gmail.com> wrote:
> >> Hi all,
>
> >> I've been having a lot of trouble lately (actually for many months) with wview not being able to upload via SSH reliably.   It seems the 1min upload period is too much now combined with my web host and provider possibly.... I don't know.  I have to either restart wview or the computer as a whole to reestablish the updates.
>
> >> My thought is whether it would be feasible for  wview to setup a persistent ssh connection to avoid constant connection/disconnection traffic.   This could perhaps support updates of 60sec or less more robustly?  
>
> >> In the meantime I'm going to see if I can setup a script that could run and make the connection itself and avoid using wviewssh to see if that helps.
>
> >> Thanks for any thoughts or ideas.
>
> >> Sent from my iPad
> >> Chris Alemany
> >> 3854 6th Ave Port Alberni, BC
> >> V9Y 4M2
>
> > --
> > You received this message because you are subscribed to the Google Groups "wview" group.
> > To post to this group, send email to wv...@googlegroups.com.
> > To unsubscribe from this group, send email to wview+un...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/wview?hl=en.

>
> Chris Alemany
> 3854 6th Ave
> Port Alberni, BC
> V9Y4M2
> h:250-723-0889
> w: 250-753-3245 x2106

> e: chris...@gmail.com
> w:http://www.norailnocoal.ca

[Christopher Alemany]

Thanks Michael I'll give a try!

> For more options, visit this group at http://groups.google.com/group/wview?hl=en.
>

Chris Alemany
3854 6th Ave
Port Alberni, BC
V9Y4M2
h:250-723-0889
w: 250-753-3245 x2106

e: chri...@gmail.com
w: http://www.norailnocoal.ca

[Christopher Alemany]

Hi everyone.

Well it appears that I have found the problem that was causing the headaches. It appears to have been a resource limitation on my webhosting service providers server that specifically throttles SSH connections.

I've pasted in the explanation from my webhost provider (who are THE BEST when it comes to customer service...). I don't know if there are any strategies on use of rsync/ssh in wview that could make things more gentle on webhosts so that these sorts of issues. Something to think about perhaps... or at least just something to be aware of if you have similar problems with your webhost. My host has very generously removed the protection on my account on their end so that I have unimpeded access and it seems much happier now.

Chris

[Sir Geek]

Thing is using rsync/ssh shouldn't be THAT bad.  That's the whole purpose of it.  And to hit it every 5 minutes isn't so horrid either.  I mean, how large is the directory that is being rsync'd ?

Matt

[Christopher Alemany]

Not very large at all. I don't think it's the amount of connections so much as the accumulation of concurrent connections that trips their systems if/when they start to timeout or otherwise stall.

I forgot to actually paste in their assessment in my last message, here it is:

"You were just hitting your resource limits for shell use. What was happening is that your account was spawning a lot of SSH activity over jailshell(a rescricted SSH suitable for shared servers). All the processes were using a larger number of combined resources then what we allow in our shell fork protection settings for each user. Fork Bomb Protection prevents users with terminal access (ssh/telnet) from using up all the resources on the server. Unchecked resource allocation can potentially lead to a server crash."