Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

wget and ftp download problems

2,005 views
Skip to first unread message

Dave Selby

unread,
Feb 9, 2003, 3:50:06 AM2/9/03
to
Having a bit of trouble with wget loading from ftp sites, have set it up to
load urls from a text file.

In a shell script I have ...

#!/bin/sh
# Script to read list of URLs form /mnt/archive/wget/sources.list
# download the files and leave them in /usr/archive/wget/downloads
cd /mnt/archive/wget/downloads
wget -t inf -T 600 -S -c --progress=bar -i '/mnt/archive/wget/url.list'

and in /mnt/archive/wget/url.list I have ...

http://www.mikebonnell.com/pub_html/0252/pub0252.html
ftp://debian.org/debian/dists/potato/main/binary-i386/base/kernel-image-2.0.38_2.0.38-3.deb

The first part works great, I get my file from the http, happy bunney .....
However the ftp load appears to log on then just hangs for a couple of mins
then issues an error message, re-tries,re-tries etc etc etc. I have enabled
debug to get as much info as possible.


test@debian:~$ wget.sh
--08:16:20-- http://www.mikebonnell.com/pub_html/0252/pub0252.html
=> `pub0252.html'
Resolving www.mikebonnell.com... done.
Connecting to www.mikebonnell.com[216.105.35.35]:80... connected.
HTTP request sent, awaiting response...
1 HTTP/1.0 200 OK
2 Date: Sun, 09 Feb 2003 08:12:23 GMT
3 Server: Apache/1.3.27 (Unix) mod_perl/1.27 mod_auth_pam/1.1.1
mod_ssl/2.8.12 OpenSSL/0.9.6h
4 Last-Modified: Sun, 15 Dec 2002 05:29:21 GMT
5 ETag: "198819-2653-3dfc1331"
6 Accept-Ranges: bytes
7 Content-Length: 9811
8 Content-Type: text/html
9 Age: 578
10 Connection: keep-alive
11 Via: HTTP/1.1 ntl-site (Traffic-Server/5.1.3-50227-50963-51463 [cHs f ])

100%[==================================================================>]
9,811 11.87K/s ETA 00:00
08:16:22 (11.87 KB/s) - `pub0252.html' saved [9811/9811]

--08:16:22--
ftp://debian.org/debian/dists/potato/main/binary-i386/base/kernel-image-2.0.38_2.0.38-3.deb
=> `kernel-image-2.0.38_2.0.38-3.deb'
Resolving debian.org... done.
Connecting to debian.org[192.25.206.10]:21... connected.
Logging in as anonymous ...
220 gluck.debian.org FTP server (vsftpd)
--> USER anonymous

331 Please specify the password.
--> PASS Turtle Power!
230 Login successful. Have fun.
--> SYST

215 UNIX Type: L8
--> PWD

257 "/"
--> TYPE I

200 Binary it is, then.
--> CWD /debian/dists/potato/main/binary-i386/base

250 Directory successfully changed.
--> PORT 62,253,48,102,5,26

200 PORT command successful. Consider using PASV.
--> RETR kernel-image-2.0.38_2.0.38-3.deb

425 Failed to establish connection.

Error in server response, closing control connection.
Retrying.

This is a bit of a problem since I wanted to use wget to download open office
etc all from ftp sites .....

Any ideas ?

Dave, learning fast but a way to go !


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Rus Foster

unread,
Feb 9, 2003, 4:10:05 AM2/9/03
to
On Sun, 9 Feb 2003, Dave Selby wrote:

> Having a bit of trouble with wget loading from ftp sites, have set it up to
> load urls from a text file.
>
> In a shell script I have ...

<snip>

Are you sitting behind a firewall at all? Have you allowed both port 20
and 21 for FTP? Also does it work on other FTP sites as it could just be
that debian.org is overloaded

Rgds

Rus
--
http://www.65535.net | MSN: sup...@65535.net
Lifetime Linux or FreeBSD account: $100 || Lifetime Hosting: $150
Offsite Backups - Remote System Monitoring - Email Hosting

Jeffrey Taylor

unread,
Feb 9, 2003, 4:10:09 AM2/9/03
to
Quoting Dave Selby <dave_...@yahoo.co.uk>:
[snip]

> 200 PORT command successful. Consider using PASV.
>
> 425 Failed to establish connection.
>
> Error in server response, closing control connection.
> Retrying.
>
> This is a bit of a problem since I wanted to use wget to download open office
> etc all from ftp sites .....
>
> Any ideas ?

There may be a firewall between you and the FTP server that is
blocking active FTP. Try adding --passive-ftp to the wget command
line.

HTH,
Jeffrey

Dave Selby

unread,
Feb 9, 2003, 4:20:08 AM2/9/03
to
On Sunday 09 February 2003 8:45 am, you wrote:
> On Sun, 9 Feb 2003, Dave Selby wrote:
> > Having a bit of trouble with wget loading from ftp sites, have set it up
> > to load urls from a text file.
> >
> > In a shell script I have ...
>
> <snip>
>
> Are you sitting behind a firewall at all? Have you allowed both port 20
> and 21 for FTP? Also does it work on other FTP sites as it could just be
> that debian.org is overloaded
>
> Rgds
>
> Rus

Sure am sitting behind a firewall, Its a pretty crude affair, blocks all NEW
packets from outside, lets in ESTABLISHED & RELATED.

So should allow a transmit from my machine via any port, and accept any
ESTABLISHED, RELATED packet back in.

I have unfortunately got the same problem on all ftp sites I have tried.

Dave

Dave Selby

unread,
Feb 9, 2003, 4:50:06 AM2/9/03
to
On Sunday 09 February 2003 8:50 am, Jeffrey Taylor wrote:
> Quoting Dave Selby <dave_...@yahoo.co.uk>:
> [snip]
>
> > 200 PORT command successful. Consider using PASV.
> >
> > 425 Failed to establish connection.
> >
> > Error in server response, closing control connection.
> > Retrying.
> >
> > This is a bit of a problem since I wanted to use wget to download open
> > office etc all from ftp sites .....
> >
> > Any ideas ?
>
> There may be a firewall between you and the FTP server that is
> blocking active FTP. Try adding --passive-ftp to the wget command
> line.
>
> HTH,
> Jeffrey

wget -t inf -T 600 -c -v --passive-ftp --progress=bar -i
'/mnt/archive/wget/url.list'

AND IT WORKS !!!!

I never realised ftp would initiate a NEW connection from ftp site, bizzare,
I assumed it was like a web page.

Anyhow many thanks, I learn a little every day !

Sean Burlington

unread,
Feb 9, 2003, 6:40:10 AM2/9/03
to
Dave Selby wrote:
> On Sunday 09 February 2003 8:45 am, you wrote:
>
>>On Sun, 9 Feb 2003, Dave Selby wrote:
>>
>>>Having a bit of trouble with wget loading from ftp sites, have set it up
>>>to load urls from a text file.

>>


>>Are you sitting behind a firewall at all? Have you allowed both port 20

>

> Sure am sitting behind a firewall, Its a pretty crude affair, blocks all NEW
> packets from outside, lets in ESTABLISHED & RELATED.
>
> So should allow a transmit from my machine via any port, and accept any
> ESTABLISHED, RELATED packet back in.
>
> I have unfortunately got the same problem on all ftp sites I have tried.
>

ESTABLISHED can match the incoming part of ftp downloads if you

modprobe ip_tables

(some people seem to have trouble with passive downloads)

--

Sean

Rob Weir

unread,
Feb 10, 2003, 8:30:31 AM2/10/03
to
On Sun, Feb 09, 2003 at 11:20:32AM +0000, Sean Burlington wrote:
> Dave Selby wrote:
> >On Sunday 09 February 2003 8:45 am, you wrote:
> >
> >>On Sun, 9 Feb 2003, Dave Selby wrote:
> >>
> >>>Having a bit of trouble with wget loading from ftp sites, have set it up
> >>>to load urls from a text file.
>
> >>
> >>Are you sitting behind a firewall at all? Have you allowed both port 20
>
> >
> >Sure am sitting behind a firewall, Its a pretty crude affair, blocks all
> >NEW packets from outside, lets in ESTABLISHED & RELATED.
> >
> >So should allow a transmit from my machine via any port, and accept any
> >ESTABLISHED, RELATED packet back in.
> >
> >I have unfortunately got the same problem on all ftp sites I have tried.
> >
>
> ESTABLISHED can match the incoming part of ftp downloads if you
>
> modprobe ip_tables

I think you mean the 'ip_conntrack_ftp' module. Possibly also
'ip_nat_ftp', too.

--
Rob Weir <rw...@ertius.org> http://ertius.org/

Sean Burlington

unread,
Feb 10, 2003, 9:10:12 AM2/10/03
to

oops yes - cut n pasted the wrong bit

though I haven't used ip_nat_ftp

0 new messages