Remote HTTP fetch from Server not working?

[vegbenz]

Hi, when i run the code below as standalone pojo, it works fine, but
in my "GreetingServiceImpl" on server it returns 500 error;

** please note i DO want to do this from the server. i saw the
tutorial about adding script tags to accomplish this from the client.
I need to do the fetch from the server side for other reasons.
thanks!

import java.io.IOException;

import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HttpMethodParams;

public class TestRemoteCall {

public static void main(String args[]) {

HttpClient client = new HttpClient();

GetMethod method = new GetMethod("http://waterdata.usgs.gov/tx/nwis/
current/?
type=flow&group_key=basin_cd&search_site_no_station_nm=barton");

// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new
DefaultHttpMethodRetryHandler(3, false));

try {
// Execute the method.
int statusCode = client.executeMethod(method);

if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}

// Read the response body.
byte[] responseBody = method.getResponseBody();

// Deal with the response.
// Use caution: ensure correct character encoding and is not binary
data
// System.out.println(new String(responseBody));
System.out.println(new String(responseBody));

} catch (HttpException he) {
System.err.println("Fatal protocol violation: " + he.getMessage());
he.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}
}

}

HERE is the error message

import java.io.IOException;

import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HttpMethodParams;

public class TestRemoteCall {

public static void main(String args[]) {

HttpClient client = new HttpClient();

GetMethod method = new GetMethod("http://waterdata.usgs.gov/tx/nwis/
current/?
type=flow&group_key=basin_cd&search_site_no_station_nm=barton");

// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new
DefaultHttpMethodRetryHandler(3, false));

try {
// Execute the method.
int statusCode = client.executeMethod(method);

if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}

// Read the response body.
byte[] responseBody = method.getResponseBody();

// Deal with the response.
// Use caution: ensure correct character encoding and is not binary
data
// System.out.println(new String(responseBody));
System.out.println(new String(responseBody));

} catch (HttpException he) {
System.err.println("Fatal protocol violation: " + he.getMessage());
he.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}
}

}

[vegbenz]

So I discovered that I need to add any extra libraries into /war/WEB-
INF/lib -- added apache httpclient, apache logging, and apache codec
jars.

Now I get this error ---

java.net.Socket is a restricted class. Please see the Google App
Engine developer's guide

But the gwt docs seem to suggest i can use anything i want on the
server side:

"Tip: Although GWT translates Java into JavaScript for client-side
code, GWT does not meddle with your ability to run Java bytecode on
your server whatsoever. Server-side code doesn't need to be
translatable, so you're free to use any Java library you find useful."

Well this one would be useful... any ideas why it is a restricted
class? and what that means?

Anecdotally i saw that I should be able to use URLConnection? I guess
I'll try that.

Thanks!

[vegbenz]

figured it out ! so for the benefit of anybody searching for this
problem in the future, here you go:

try {
java.net.URL url = new URL(input);
java.net.URLConnection urlConn = url.openConnection();

urlConn.getContent();

int c;
StringBuilder sb = new StringBuilder();
InputStream stream = urlConn.getInputStream();
int i = urlConn.getContentLength();
while (((c=stream.read())!=-1)){// && (--i > 0){
System.out.print((char)c);
sb.append((char)c);
}
stream.close();
return sb.toString();
}
catch (java.io.IOException ioe)
{
ioe.printStackTrace();
}
return null;

[jhulford]

If you're not actually deploying to the App Engine environment you can
remove the GAE class restrictions in the GWT plugin options and use
whatever classes you want.

[Jason C]

Hello,

I'm having a similar issue where my servlet can not reach a URL within
my company's intranet. My company has a proxy and firewall, but I've
written other servlets that don't have any issue reaching URLs on my
intranet. It's possible that gwt requires me to manually specify a
proxy, but I've noticed trying to use the java.net.Proxy class is
unsupported by appengine, and was wondering if anyone had other
methods of connecting. I will paste the code below, with the
exception. I've tried this in development mode and I've tried creating
a war and posting it to my company's sunone server. No luck either
way.

System.setProperty("http.proxyHost", "my-company-proxy");
System.setProperty("http.proxyPort", "80");
String urlStr = "http://my-company-intranet.com";

try
{
StringWriter content = new StringWriter();
URL url = new URL(urlStr);

// Proxy proxy = new Proxy(Proxy.Type.HTTP, new
InetSocketAddress("my-company-proxy.com", 80));

URLConnection spoof = url.openConnection();
InputStream i = spoof.getInputStream();
spoof.setRequestProperty("User-Agent", "Mozilla/4.0
(compatible; MSIE 5.5; Windows NT 5.0; H010818)" );
InputStreamReader isr = new InputStreamReader(i);
BufferedReader in = new BufferedReader(isr);

and the exception is...

Initializing AppEngine server
The server is running at http://localhost:8888/
java.io.IOException: Could not fetch URL: http://my-company-intranet.com
at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.handleApplicationException(URLFetchServiceImpl.java:
75)
at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:
45)
at
com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler
$Connection.fetchResponse(URLFetchServiceStreamHandler.java:409)
at
com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler
$Connection.getInputStream(URLFetchServiceStreamHandler.java:290)
at
myproject.server.search.GreetingServiceImpl.getURLTextContent(GreetingServiceImpl.java:
118)
at
myproject.server.search.GreetingServiceImpl.greetServer(GreetingServiceImpl.java:
50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.google.appengine.tools.development.agent.runtime.Runtime.invoke(Runtime.java:
100)
at
com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:
562)
at
com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:
188)
at
com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:
224)
at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:
62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
487)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1093)
at
com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:
51)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1084)
at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:
43)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1084)
at
com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:
121)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1084)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
360)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
405)
at
com.google.apphosting.utils.jetty.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:
70)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
139)
at com.google.appengine.tools.development.JettyContainerService
$ApiProxyHandler.handle(JettyContainerService.java:352)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
139)
at org.mortbay.jetty.Server.handle(Server.java:313)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
506)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:844)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:644)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
396)
at org.mortbay.thread.BoundedThreadPool
$PoolThread.run(BoundedThreadPool.java:442)

Any ideas?

- Jason