DL web pages w/o browser?
Dave
Also have a look at the TWebBrowser, just to make you application
cool, you can watch the web pages your application will go to while
it is running. It's good to check if everything is going ok (you will
rapidly see if any 404 errors come up, this means you got to
check the Urls or the parameters... just some friendly advice. I like
to check out visually that the application goes to the right pages b4
I start inserting in a db or whatever.)
Build what I put below, you will quickly understand. What you want to
retrieve
and parse is put into the memo (I put it in the memo here so you can
visualize,
you don't actualy need it if you put it straight into a variable).
(After this short tutorial :-) you will need to check the TNMHTTP post
method ...)
procedure TForm1.Button1Click(Sender: TObject);
begin
NMHTTP1.Get('http://www.realtor.com/default.asp?');
WebBrowser1.Navigate('http://www.realtor.com/default.asp?');
end;
procedure TForm1.NMHTTP1Success(Cmd: CmdType);
begin
memo1.Lines.Text := NMHTTP1.Body;
end;
Have a good MMIII (2003)!
"Mike Austin" <mau...@hevanet.com> wrote in message
news:3e10...@newsgroups.borland.com...
> Thanks a lot, Dave!
>
> My (potential) client conducts real estate seminars around the US. They
want
> to get contact info from the national assoc of "realtors". There is a
> "static" search page that has 15 search fields. I only need 2. All 15
fields
> resolve to variable names and are not included in any URLs. So, I need to
> have something where I can set these variables before I submit it to the
> server.
>
> I was told on the winsock news group to the Indy
TIdMultipartFormDataStream
> in Indy to do this.
>
> TIdMultipartFormDataStream.AddFormField(sName, sValue);
>
> Unfortunately, this is not well documented, so I am working my way through
> things.
>
> Once I have the results page for each city/state (which is a list of
> realtors), I intend to parse through it to retrieve the individual
> information for each realtor. Figuring out how to "spoof" the server by
> setting the variables on the web page programmatically is the biggest
hurdle
> (so far).
>
> Thanks for your comments.
>
> Mike
>
> "Dave" <dave...@hotmail.com> wrote in message
> news:3e0e2a00$1...@newsgroups.borland.com...
> > Hi mike,
> >
> [snip]
> > So you will need to:
> >
> > 1- Find out how the search engine works (list page Urls)
> > 2- Build the array(s) of elements you will need to build the list page
> Urls
> > and retrieve them
> > 3- Loop through the array(s)
> > for each country
> > for each state
> > get the list page
> > (www.webpage.com?country=element[i]&state=otherelement[j])
> > For each element of the list page
> > get the element "title" information and the detail page
Url
> > (usually <a herf=www.something.com>)
> > get the detail page and parse the information you want
> > do whatever you need to do with the information (DB
> insertion
> > or whatever)
> >
> >
> > Once you have a program that does this, you just click and it gets all
you
> > need but BEWARE:
> > Web sites change!
> > You will need to check that the Web site you are visiting does not
change
> to
> > use that one program.
> > Any change in a Web page can screw everything up (text, image, or
> whatever).
> > If it does, your program may be inserting unusable information into a
> > database, for example (or
> > whatever you will be doing with it)
> > And please do not over use it, you do not want to use all of that Web
> site's
> > server band width. That
> > usually get some unneeded attention. If your program is to demanding on
> the
> > other Web server
> > you may want to put some small delay in a loop (a half a second or so).
> >
> > Enjoy!
> >
> >
> >
> > "Mike Austin" <mau...@hevanet.com> wrote in message
> > news:3e00...@newsgroups.borland.com...
> > > All,
> > >
> > > I have a potential client who wants me to write an application that
can
> > > "scrape" data off of web pages. Essentially, the client wants to go to
a
> > > very specific web site and search for particular items by city and
> state.
> > At
> > > present, the client goes to the home page and then navigates down to
the
> > > page(s) of interest and cuts and pastes the information into a
> > spreadsheet.
> > > He wants me to automate this process.
> > >
> > > The process really starts with the "Search" page, which requires a
City
> > and
> > > State.
> > >
> > > My idea is to write an application that performs the following
process:
> > >
> > > 1. Using the URL request on the "Search" page, fill in the City and
> State
> > > parameters and then submit the request to the server. The returned
page
> > > consists of a list of items, each having a link to the details.
> > > 2. Accept the returned page, discarding all non-text, and place it
into
> a
> > > string list (or ?).
> > > 3. Parse through the list for the individual links.
> > > 4. Request each individual link from the server.
> > > 5. Accept the returned page, discarding all non-text.
> > > 6. "Scrape" (parse) the information from the returned text.
> > >
> > > Is this possible? My thinking is that I should be able to do this
> outside
> > of
> > > a browser. My web experience is very rudimentary, but it seems like
this
> > > should be possible to do in a "desktop" Delphi application.
> > >
> > > Any comments or suggestions very welcome!
> > >
> > > TIA,
> > >
> > > Mike
> > >
> > >
> >
> >
>
>
Chad Z. Hower aka Kudzu
news:3e11...@newsgroups.borland.com:
Ugh. Not unless you want a bugfest. I hope you didnt want to count him as a
friend. :)
Try ICS or Indy, they are both free and work.
--
Chad Z. Hower (a.k.a. Kudzu) - http://www.hower.org/Kudzu/
"Programming is an art form that fights back"
Instant Web Applications with IntraWeb
Try a fully functional evaluation at:
http://www.atozedsoftware.com/
ELKNews - Get your free copy at http://www.atozedsoftware.com