Save pages from command line

[nyath...@hotmail.com]

Hi,
I was wondering, is there a way to save web pages from a command line
utility (basically from outside of firefox)?

So for example if at the command line i type "MyArchiver.exe
'C:\Archives'" It should be the equivalent of choosing File -> Save As
-> (Navigate to C:\Archives) -> Save Complete in the firefox browser.

Would writing a custom extension/plugin to firefox be the way to go or
is there an another alternative? What would be the best suited language
for the command line util? Java, Perl, C# or does it really not matter?

Thanks in advance for the help

[Tony Mechelynck]

You could install the Vim editor, which is available on many platforms
including, but not limited to, Linux, Unix, Windows and MacOsX, and which has
support to edit files online. If the network options are correctly configured,
you could save a file from the command-line using a command similar to

vim http://www.example.net/index.htm -c "saveas ~/example.net.html |q"

or you could start gvim and then do interactively (on any platform: Vim for
Windows accepts the Unix-like ~ and / and translates them to something that
Windows understands):

:edit http://www.example.net/index.htm
:saveas ~/example.net.html
[ ... repeat for any number of additional files ... ]
:quit

Vim is available (mostly in source form) from http://ftp.vim.org/pub/vim/ ;
for Windows, bugfixed precompiled distributions are available as
self-installing .exe archives from
https://sourceforge.net/project/showfiles.php?group_id=43866&package_id=39721
. If you want to compile it yourself (e.g. on Unix/Linux), see the two HowTo
pages linked to by the bottom two links at top left of my Vim page,
http://users.skynet.be/antoine.mechelynck/vim/

Best regards,
Tony.

[CBFalconer]

Look up wget and curl.

--
Merry Christmas, Happy Hanukah, Happy New Year
Joyeux Noel, Bonne Annee.
Chuck F (cbfalconer at maineline dot net)
<http://cbfalconer.home.att.net>

[nyath...@hotmail.com]

Thanks to you and Tony. Its closer to the answer i was looking for, but
not quite it. Both Wget and vim solution seem to require me to specify
the http address of the page. While the http address is available in
the address bar of the browser, i don't want to cut and paste it into
the command line util (If i can avoid it).

For example, if the command line util is setup as a shortcut or
assigned to a hotkey, I want to be able to just double click or press
the appropriate key and i want the executable to save the current page
that is being displayed in the web browser (without me explicitly
providing the http address). Note that the command line util does some
other stuff as well, but those tasks are relatively simple and most
importantly independent of other applications.

This would need some sort of inter-process communication, but i am not
sure if firefox provides the appropriate hooks to do that or how I
would go about using those hooks.

And for the sake of completion, how would you try and implement this
with IE? If it is actually possible in IE at all !

Ernie wrote:

> Google for "wget win32" for a windows version of a unix utility that
> has been around for ages. Should do the job you want.

[Tony Mechelynck]

nyath...@hotmail.com wrote:
> Thanks to you and Tony. Its closer to the answer i was looking for, but
> not quite it. Both Wget and vim solution seem to require me to specify
> the http address of the page. While the http address is available in
> the address bar of the browser, i don't want to cut and paste it into
> the command line util (If i can avoid it).

If the URI is in the address bar of the browser (and the page is displayed),
just use "File => Save Page As...". No need from _any_ auxiliary program,
command-line or otherwise.

>
> For example, if the command line util is setup as a shortcut or
> assigned to a hotkey, I want to be able to just double click or press
> the appropriate key and i want the executable to save the current page
> that is being displayed in the web browser (without me explicitly
> providing the http address). Note that the command line util does some
> other stuff as well, but those tasks are relatively simple and most
> importantly independent of other applications.
>
> This would need some sort of inter-process communication, but i am not
> sure if firefox provides the appropriate hooks to do that or how I
> would go about using those hooks.
>
> And for the sake of completion, how would you try and implement this
> with IE? If it is actually possible in IE at all !
>

In IE too, IIRC, there's a "File => Save As" menu.

Best regards,
Tony.

[Dennis McCunney]

nyath...@hotmail.com wrote:

> Thanks to you and Tony. Its closer to the answer i was looking for, but
> not quite it. Both Wget and vim solution seem to require me to specify
> the http address of the page. While the http address is available in
> the address bar of the browser, i don't want to cut and paste it into
> the command line util (If i can avoid it).

It sound like you would like FF to be able to launch another program as
an asynchronous process, and pass it the value currently in the Address
Bar as an argument. You might want to look at Henrik Gemal's "Launchy"
extension. It looks like it might be able to do what you wish.

http://gemal.dk/mozilla/launchy.html?project=launchy
______
Dennis

--
As a general rule the most successful man in life is the man who has the
best information.
Benjamin Disraeli (1804 - 1881)

[nyath...@hotmail.com]

Thanks Dennis,
Actually what I am looking for is the other way around. Rather than
Firefox launch any applications/processes, I want an independently
running application/process to query Firefox for the current address in
the address bar (or initiate the File -> Save As mechanism with a
directory name as an argument)

I think iMacros might be what i am looking for
(https://addons.mozilla.org/firefox/3863/). This needs further
investigation. Has anyone used this plugin or have any comments
regarding this?

[Dennis McCunney]

nyath...@hotmail.com wrote:
> Thanks Dennis,

> Dennis McCunney wrote:
>> nyath...@hotmail.com wrote:
>>
>>> Thanks to you and Tony. Its closer to the answer i was looking for, but
>>> not quite it. Both Wget and vim solution seem to require me to specify
>>> the http address of the page. While the http address is available in
>>> the address bar of the browser, i don't want to cut and paste it into
>>> the command line util (If i can avoid it).
>>

>> It sounds like you would like FF to be able to launch another program as

>> an asynchronous process, and pass it the value currently in the Address
>> Bar as an argument. You might want to look at Henrik Gemal's "Launchy"
>> extension. It looks like it might be able to do what you wish.
>>
>> http://gemal.dk/mozilla/launchy.html?project=launchy
>

> Actually what I am looking for is the other way around. Rather than
> Firefox launch any applications/processes, I want an independently
> running application/process to query Firefox for the current address in
> the address bar (or initiate the File -> Save As mechanism with a
> directory name as an argument)

Er, what triggers the independently running process to do this? Polling
on a specified basis? Something like a named pipe where FF writes the
value of whatever is in the URL bar whenever that changes? Is this
process supposed to grab *every* page FF looks at? What happens if the
new page is a link off the top level on the same site? What else would
this process do besides grab the page?

I have a clearer idea now of what you want to do, but I don't understand
*why* you want to.
______
Dennis

--
As a rule we disbelieve all the facts and theories for which we have no use.

[nyath...@hotmail.com]

Thanks for the reply Dennis,

The independent process is triggerred by the user. From a command
line/short cut etc. etc.

Basically, what I am trying to do is automate my job application
lodging/archiving process. Currently whenever I search an online job
site (seek.com for example), if i find a job that I like and want to
apply for, I create a directory on my local drive with a date ( "GAP
Consulting - 2 Jan 2007" for example) then I copy my standard, template
resume and cover letter into that directory. Then I change the date to
match the current date, I might also change some other bits and pieces
in the resume. Then I save the web page in that directory as well, for
future reference/archiving. Then I go back to the web page and submit
my resume.

Now most of the other stuff is quite easy to automate. Its just
creating directories with time stamps, manipulating text based on
parameters passed etc. etc. I can use any language (and I intend to
play around with a few) and have a relatively straight forward approach
here. The only thing thats stumping me at the moment is getting this
program to talk to the web browser to get the currently displayed web
page and save it to disk.

The way I envision it, once its built, I will have a window with
command line program open and i will be surfing the job site. If i come
across a job that I like, i switch to the command line and type
something like "C:\MyJobApplications\ResumeGen.exe -GAP Consulting".
Then I switch back to the browser and when I go to apply for the job,
hey presto there is a directory called GAP Consulting with a copy of my
CV that contains the current date at the top and text such as "I would
like to apply for a position at GAP Consulting that was...", "... A
valuable asset to GAP Consulting..." etc. etc.. And the directory also
contains the web page with the ad so that even after the job listing is
taken down, I still have access to it for future
reference/comparisions.

[nyath...@hotmail.com]

Hello? Anyone? Any bright ideas from the gurus out there? Requires some
unconventional thinking, i know....

[Irwin Greenwald]

Please bottom post in newsgroups on this server. Thanks!

Sounds like you need a script writing program that gives you access to
(and some control over) windows that are open on your screen. I have
dabbled with a freebie called autoit <http://www.hiddensoft.com/autoit3/>
which may serve your needs.

--
Irwin

Please do not use my email address to make requests for help.

Knowledge Base: http://kb.mozillazine.org/Main_Page

[nyath...@hotmail.com]

Cool. Thanks heaps Irwin, that just might be what I need. I will have a
play around and see how it goes.

Cheers.