Downloading HTML files from Server..
Patrick
England
They put up a list of the players and the points they have
I am running a fantasy football for just a few friends
I want to download the scores from this website
Then parse them
Then calculate the points for all the teams in our little Fantasy
Football league
Now to my problems...
If I navigate to
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
And click on a link which calls the follow javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
A new page pops up, the browser say its url is
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
Now with my code
public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}
}
}
I can download the page at
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
But I cannot download the page at
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
That the javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
In the page
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
calls.
When I try to download
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
With my code, all i get is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I noticed that when I access
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
The server sends a cookie. And then when I access
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
I get the table of players and their respective points.
But when I try to access
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
without accessing
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
first, I just get redirected to
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
I used the following code
public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167");
URLConnection uc = url.openConnection();
System.out.println(uc.getHeaderField("Set-Cookie"));
}
To get the cookie, which was
CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
22:25:46 GMT; Path=/
I think
CF_HA, is just a unique identifier, a variable which in incremented
by the server for each new client
Domain, is just the domain
expires, is just the expiry date
Path, hmm dunno
Now I hardcoded the cookie into the code, with a valid expiry date
public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167");
URLConnection uc = url.openConnection();
String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
uc.setRequestProperty("cookie",cookie);
int i = 0;
while ((i = uc.getInputStream().read()) != -1)
{ System.out.print((char) i);
}
}
Now, when I run this code I get the following error
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1133)
at Test.main(Test.java:46)
Am I sending the cookie correctly?
Is there something else I must do?
Any help/advice appreciated,
regards
pat