Q) Why is URLConnection not downloading all the renderable content for me?

[sande...@yahoo.com]

Hi,

I am using URLConnection to download the contents of the following URL in my Java program:

http://www.pollutionequipmentdirectory.com/category/air?Page=2


My code snippet is as follows:

// ******** Code Snippet Begin ********

try
{
outputStream = new PrintWriter(new FileOutputStream(outputFileName));

// try opening the URL
URLConnection urlConnection = url.openConnection();

urlStream = url.openStream();

byte b[] = new byte[1000];
int numRead = urlStream.read(b);
String content = new String(b, 0, numRead);
while ((numRead != -1))
{
outputStream.write(content);
numRead = urlStream.read(b);
if (numRead != -1)
{
content = new String(b, 0, numRead);
outputStream.write(content);
}
}
urlStream.close();
}
catch (IOException e)
{
System.err.println("ERROR: couldn't open URL " + url);
}
finally
{
if(outputStream != null) outputStream.close();
}

// ******** Code Snippet End ********


However, I am finding that URLConnection does not download the content that I can see if I save the contents of the browser (Chrome in my case).
For example, if I save the browser contents, I see the following content, which I don’t see when I use URLConnection:

// ********Expected content not seen BEGIN ********/
MIL-RAM TECHNOLOGY, Inc. is a leading innovator in industrial gas detection specializing in fixed systems utilizing our unique no false alarms sensor technology. We are a technical, engineering based company with strong emphasis on product design and customization to meet specific application...


Lufft - our philosophy, our services and the reasons why our products are used throughout the world in the fields of industrial climate measurement technology, Meteorology, Renewable Energy, and Intelligent Transportation Systems.


Schutte & Koerting is a designer and manufacturer of equipment for the process and power industries serving the chemical, petroleum, petrochemical, food, pulp and paper, power, metal, refining, water and waste treatment industries among others. Schutte & Koerting products are custom...

// ********Expected content not seen END ********/


Any suggestions would be welcome?

Regards,
Sandeep