Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Newbie - How do you request a web page from within Java?

1 view
Skip to first unread message

Jacky

unread,
Apr 24, 2005, 1:26:46 PM4/24/05
to
Hi,

How do you request a web page from within Java?

Regards,
Jacky

Ryan

unread,
Apr 24, 2005, 1:38:31 PM4/24/05
to

jacky.n...@gmail.com

unread,
Apr 24, 2005, 1:47:58 PM4/24/05
to
thanks! :)

jacky.n...@gmail.com

unread,
Apr 24, 2005, 1:54:54 PM4/24/05
to
Hi,

It seemed a bit complicated. Is there a source code sample, that
requests "http://www.google.com" for example?

Thanks,
Jacky

Roland

unread,
Apr 24, 2005, 2:05:27 PM4/24/05
to

Wendy S

unread,
Apr 24, 2005, 4:16:44 PM4/24/05
to
<jacky.n...@gmail.com> wrote:
>
> It seemed a bit complicated. Is there a source code sample, that
> requests "http://www.google.com" for example?
>

There's also Jakarta Commons HTTPClient, which hides some of the complexity.
http://jakarta.apache.org/commons/httpclient/

--
Wendy


Dag Sunde

unread,
Apr 24, 2005, 4:44:32 PM4/24/05
to
<jacky.n...@gmail.com> wrote in message
news:1114365294.7...@l41g2000cwc.googlegroups.com...

> Hi,
>
> It seemed a bit complicated. Is there a source code sample, that
> requests "http://www.google.com" for example?
>

try {
URL url = new URL(http://www.google.com);
BufferedReader in = new BufferedReader(new
InputStreamReader(url.openStream()));
String str;
StringBuffer page = new StringBuffer();
while ((str = in.readLine()) != null) {
page.append(str + "\n");
}
in.close();
}
catch (MalformedURLException e) {
//...
}
catch (IOException e) {
//...
}

// And now 'page' contains the webpage...

--
Dag.


Ross Bamford

unread,
Apr 24, 2005, 4:53:25 PM4/24/05
to

If you know you're going to get textual data back, something like the
following might work:

StringBuffer bigBuf = new StringBuffer();
try {
InputStream strm = new URL("http://www.google.com/").openStream();
BufferedReader rdr = new BufferedReader(new InputStreamReader(strm));

String thisLine = rdr.readLine();
while (thisLine != null) {
bigBuf = bigBuf.append(thisLine);
thisLine = rdr.readLine();
}
rdr.close();
} catch (Exception e) {
/* You might catch URLException, network exception, or IOException
}
String wholeThing = bigBuf.toString();

After which the 'wholeThing' string would contain all the data, unless
an exception occured.

You could easily modify this to check the URL's Content type - Read up
on the classes others have suggested. This is just a starter :)

Hope this helps,
Ross

--
[Ross A. Bamford] [ro...@synergix-systems.co.uk]
Synergix Systems +++ Linux Admin + Support + Consultancy
www.synergix-systems.co.uk + in...@synergix-systems.co.uk

Jacky

unread,
Apr 25, 2005, 6:15:21 AM4/25/05
to
Thanks guys!

Jacky

unread,
Apr 25, 2005, 10:08:00 AM4/25/05
to
Hi Guys,

I have the following code.
It displays alright when URL = http://google.com

It gives me an error when URL =
http://www.google.com/search?hl=en&safe=off&q=Media+studies+and+gender+studies+with+content+on+queer+theory%2C+gender&meta=

===============================================================
Error Message:
===============================================================
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 403 for URL: htt
p://www.google.com/search?hl=en&safe=off&q=Media+studies+and+gender+studies+with+content+on+queer+th
eory%2C+gender&meta=
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:789)
at java.net.URL.openStream(URL.java:913)
at Project.main(Project.java:19)
Press any key to continue...

===============================================================
Code:
===============================================================
import java.io.*;
import java.net.*;
import java.lang.*;


public class Project {
public static void main(String[] args) throws Exception {
URL search_term = new URL("http://www.yahoo.com/");
BufferedReader in = new BufferedReader(
new InputStreamReader(
search_term.openStream()));

String inputLine;


StringBuffer bigBuf = new StringBuffer();


search_term = new URL("http://www.google.com/search?hl=en&safe=off&q=Media+studies+and+gender+studies+with+content+on+queer+theory%2C+gender&meta=");
in = new BufferedReader(new
InputStreamReader(search_term.openStream()));


while ((inputLine = in.readLine()) != null) {
bigBuf = bigBuf.append(inputLine);
bigBuf = bigBuf.append("\n\n\n");
}


String wholeThing = bigBuf.toString();


System.out.println(wholeThing);

in.close();
}
}

jacky.n...@gmail.com

unread,
Apr 25, 2005, 10:14:37 AM4/25/05
to
okay, I've checked out the Error code, google.com is forbidding me to
use Java program to use it's search function.

Any ways to get around it?

Real Gagnon

unread,
Apr 26, 2005, 8:23:42 PM4/26/05
to
> okay, I've checked out the Error code, google.com is forbidding me to
> use Java program to use it's search function.
>
> Any ways to get around it?

You need to pretend to be a "real" browser by setting a request property.

See http://www.rgagnon.com/javadetails/java-0399.html
for an example.

bye.
--
Real Gagnon from Quebec, Canada
* Looking for Java or PB code examples ? Visit Real's How-to
* http://www.rgagnon.com/howto.html

jacky.n...@gmail.com

unread,
Apr 27, 2005, 3:30:49 AM4/27/05
to
thanks! :)

btw, the code at the page has some "bugs":
webData = new BufferedReader(new
InputStreamReader(connection.getInputStream()));

connection should be replaced with con, because:
URLConnection con = urlObject.openConnection();
was declared eariler...


Here's the code:


=====================================================
import java.io.*;
import java.net.*;
import java.lang.*;


public class Project {
public static void main(String[] args) throws Exception {

String search= "dogs";
String google="http://www.google.com/search?q=" + search +
"&hl=en&ie=UTF-8&oe=UTF8";

URL urlObject = new URL(google);
URLConnection con = urlObject.openConnection();
con.setRequestProperty
( "User-Agent", "Mozilla/4.0 (compatible;MSIE 5.5; Windows NT 5.0;
H010818)" );

BufferedReader in = new BufferedReader(new

InputStreamReader(con.getInputStream()));


String inputLine;
StringBuffer bigBuf = new StringBuffer();

while ((inputLine = in.readLine()) != null) {


bigBuf = bigBuf.append(inputLine);
bigBuf = bigBuf.append("\n\n\n");
}


String wholeThing = bigBuf.toString();
System.out.println(wholeThing);

in.close();
}
}
=====================================================

0 new messages