Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Batch commands at a remote http site - Possible?

0 views
Skip to first unread message

Friar Broccoli

unread,
Dec 13, 2009, 9:09:00 AM12/13/09
to
I would like to download a lot of images from a remote site.
I can right click and press "save image as" etc for each image
but I would like to be able to do this in batch mode (or just group
select) as I can do with an FTP server tool (I use "Core FTP
Light Edition").

Is there some way to accomplish with free tools?

Scott Bryce

unread,
Dec 13, 2009, 10:19:08 AM12/13/09
to

Assuming what you want to do is legal, Perl is free. Depending on how
many "a lot" is, it may take longer to write the script than it takes to
grab each image manually.

Jake Jarvis

unread,
Dec 13, 2009, 10:31:14 AM12/13/09
to

firefox and the DownThemAll! extension.

--
Jake Jarvis

Manuel Collado

unread,
Dec 13, 2009, 4:30:53 PM12/13/09
to
Friar Broccoli escribi�:

You can look at the 'wget' utility.

--
Manuel Collado - http://lml.ls.fi.upm.es/~mcollado

Friar Broccoli

unread,
Dec 13, 2009, 6:20:06 PM12/13/09
to

This is a very nice addon.
Thanks for recommending it.

Friar Broccoli

unread,
Dec 13, 2009, 6:22:57 PM12/13/09
to
On Dec 13, 4:30 pm, Manuel Collado <m.coll...@invalid.domain> wrote:
> Friar Broccoli escribió:

>
> > I would like to download a lot of images from a remote site.
> > I can right click and press "save image as" etc for each image
> > but I would like to be able to do this in batch mode (or just group
> > select) as I can do with an FTP server tool (I use "Core FTP
> > Light Edition").
>
> > Is there some way to accomplish with free tools?
>
> You can look at the 'wget' utility.

Thanks for the recommendation, but this looks more
like work than is needed for my purposes.

Greg Russell

unread,
Dec 13, 2009, 11:53:47 PM12/13/09
to
In news:8c388936-77f3-43cc...@f16g2000yqm.googlegroups.com,
Friar Broccoli <eli...@gmail.com> typed:

>>> I would like to download a lot of images from a remote site.

...


>>> Is there some way to accomplish with free tools?
>>
>> You can look at the 'wget' utility.
>
> Thanks for the recommendation, but this looks more
> like work than is needed for my purposes.

Is that a joke?

wget -A jpg,jpeg,gif,png http://some.site.of.interest

is "more like work than is needed"? Geez, you must be a lazy fucker!

Of course just scanning "wget --help" probably defeats you.


Swifty

unread,
Dec 14, 2009, 3:17:59 AM12/14/09
to
Greg Russell wrote:
> Of course just scanning "wget --help" probably defeats you.

It doesn't defeat me (I've just described my interests as "anything with
a reference manual") but I have to factor in a day of reading the man
page whenever I have a new task for wget.

I was going to suggest wget, It's what I would have used, after a day or
two.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

Andy Dingley

unread,
Dec 14, 2009, 5:36:48 AM12/14/09
to
On 14 Dec, 04:53, "Greg Russell" <gruss...@invalid.com> wrote:

> Is that a joke?

No, it's a concern.

>     wget -A jpg,jpeg,gif,png  http://some.site.of.interest
>
> is "more like work than is needed"? Geez, you must be a lazy fucker!

That depends on whether wget * will work for what you need, which
depends on remote server config. If the paths to the images are simple
and the server supports directory listing, then wget is fine. If the
image URLs aren't obvious, then it won't. In that case DownThemAll is
a better option, as it uses the starting page as the source of the
URLs, so doesn't care about directory listings. It also has quite good
abilities to traverse beyond that, also (and very usefully) good
ignore facilities so that you can download the "content" assets
without also downloading all of the layout furniture.

Friar Broccoli

unread,
Dec 14, 2009, 12:38:11 PM12/14/09
to
On Dec 13, 11:53 pm, "Greg Russell" <gruss...@invalid.com> wrote:
> Innews:8c388936-77f3-43cc...@f16g2000yqm.googlegroups.com,
> Friar Broccoli <elia...@gmail.com> typed:

>
>
>
> >>> I would like to download a lot of images from a remote site.
> ...
> >>> Is there some way to accomplish with free tools?
>
> >> You can look at the 'wget' utility.
>
> > Thanks for the recommendation, but this looks more
> > like work than is needed for my purposes.
>
> Is that a joke?
>
>     wget -A jpg,jpeg,gif,png  http://some.site.of.interest
>
> is "more like work than is needed"?

.

> Geez, you must be a lazy fucker!

Also slow and stupid. I've got it ALL!

> Of course just scanning "wget --help" probably defeats you.

I would have had to download and install it first.
(MORE work!)

Thanks for the sample command line.
Now that I have a shortcut into its use, I will definitely
give it a try.

Swifty

unread,
Dec 30, 2009, 5:46:35 AM12/30/09
to
Greg Russell wrote:
> wget -A jpg,jpeg,gif,png http://some.site.of.interest

Well, I find myself needing this function, but

wget -A jpg http://swiftys.org.uk

Downloaded index.html and nothing else.

Admittedly there are no *.jpg files visible on that site (via index.html
and linked documents), but I don't understand why it downloaded an
"unacceptable" file extension.

Presumably it would also need some sort of recursion specification.

David Postill

unread,
Dec 30, 2009, 8:37:07 AM12/30/09
to

In article <ouSdnSPhnOQOsqbW...@brightview.com>, on Wed, 30

Dec 2009 10:46:35 +0000, Swifty wrote:

| Greg Russell wrote:
| > wget -A jpg,jpeg,gif,png http://some.site.of.interest
|
| Well, I find myself needing this function, but
|
| wget -A jpg http://swiftys.org.uk
|
| Downloaded index.html and nothing else.
|
| Admittedly there are no *.jpg files visible on that site (via index.html
| and linked documents), but I don't understand why it downloaded an
| "unacceptable" file extension.
|
| Presumably it would also need some sort of recursion specification.

That would be "wget -r" then I suppose :)

<http://linuxreviews.org/man/wget/>

Try the following:

wget -q -r --accept=*.jpg 'url'
--
David Postill
Dance your Life - Biodanza in Holland - <http://www.danceyourlife.eu>

Swifty

unread,
Dec 30, 2009, 9:36:38 AM12/30/09
to
David Postill wrote:
> Try the following:
>
> wget -q -r --accept=*.jpg 'url'

That created a directory corresponding to the url (of my website) that I
specified and downloaded just robots.txt

I suspect that wget is honouring my robots.txt - I'll find the switch to
make it ignore that.

David Postill

unread,
Dec 31, 2009, 3:26:19 AM12/31/09
to

In article <2pKdnUiTRv3j-KbW...@brightview.com>, on Wed, 30

Dec 2009 14:36:38 +0000, Swifty wrote:

| David Postill wrote:
| > Try the following:
| >
| > wget -q -r --accept=*.jpg 'url'
|
| That created a directory corresponding to the url (of my website) that I
| specified and downloaded just robots.txt
|
| I suspect that wget is honouring my robots.txt - I'll find the switch to
| make it ignore that.

Try adding the option:

--user-agent="Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

Cheers,

Swifty

unread,
Dec 31, 2009, 9:21:35 AM12/31/09
to
David Postill wrote:
> Try adding the option:
>
> --user-agent="Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

Thanks. I'll try that if the "native" mechanism of "-e robots=off --wait
1" fails.

This is just an intellectual exercise now, so it may be a day or two
before I do anything (just in case anyone is holding their breath).
For those of you not holding your breath� Happy New Year!

0 new messages