Problem downloading large files with Apache fronting WAS

[Ma...@softlanding.com]

WebSphere 4.03 AE
OS/400 V4R5M0
Apache - latest Group PTF's (including recent CERT fix).

We have a WebSphere app we have developed where we have a servlet that
downloads files, such as zip files or PDF's to the user. This works great
when using the built-in WAS HTTP transport.

In our production site, the HTTP server is Apache. The servlet works for
small files, but fails for anything over 10-16 MB, do not know the exact
amount. After pulling our hair out on this all week, tweaking our code to
read/write the file different ways etc., we finally got the idea of
hitting the WAS transport directly, which is how we discovered that it
works fine in that mode.

Could there be any settings in the Apache/WebSphere configuration that
could effect this? Is it a known bug? We were about to go live with this
application and this has turned into a major setback. While we do see
stack traces in stdout, I do not think they are necessarily meaningful. I
am not sure how to look for errors in Apache.

The site is also using SSL, but we tried SSL on the WAS transport and it
worked fine.

The problems happen in both IE 6 and Mozilla 1.0.

Thanks for any help.

Mark Phippard

[Ma...@softlanding.com]

Just as an update, I tried using the Original HTTP Server instead of
Apache and have the same problem.

What happens on the downloads that fail is that almost immediately upon
clicking the link that invokes the servlet I get a "Server Error", which I
believe is error code 500.

So, based on what I have tested, the problem has to lie in the plugin.

Mark

[Ma...@softlanding.com]

OK, this problem just gets worse as I look for a solution.

Now, if I take my servlet out of the equation and just put the file I want
to download in a path that ican be straight-served by my web app, I still
get the same problem. In other words, even the built in WAS file-serving
servlet cannot download the big files when communicating over the plugin.
Nothing shows up in stdout or stderr in this scenario. The browser just
gives an immediate server error. Same test with a smaller file works
fine.

All works fine when talking directly to WAS HTTP transport.

By the way, I cannot use the WAS transport because it has SSL problems
with IE that have not been resolved.

Thanks

Mark

[Art Smet]

Yup, I finally got around to trying this out myself, I get the same failure as
you describe when trying to download a large .zip file. I suggest reporting
this problem to IBM support. In the meantime, you might try circumventing the
problem by having your servlet stat'ing files and when they are >16MB read the
file and write it back to the browser in pieces <16MB each (issue a
response.flushBuffer() between writes).

[Ma...@softlanding.com]

Art,

Thanks. Since you reproduced it, can you report it? If not, I guess I
can post something via PartnerWorld.

Given that this fails instantly, do you really think rewriting the servlet
would make a difference? I do not even think the servlet gets a chance to
send any data. If you do think it would make a difference could you give
me more information about what you think I should do? Are you saying to
just flush the output stream before I get to 16MB? I already know the
file size.

The way that I have worked around the problem, which while not ideal at
least works is to do the following:

When a file over a certain size is to be download I create a unique
session folder under my HTTP server, copy the file to that folder and then
send a redirect to the browser. I then add an object to the session that
implements HttpSessionBindingListener to delete the file and folder when
the session expires. This works, but there is a slight sever-side delay
while the file is created. One advantage is that I can make the download
happens over a non-SSL connection.

By the way, the same problem appears to happen when posting/uploading
large files. I am writing a small document management system so the user
can upload files that I store in a database. I then make the files
available for download.

Thanks

Mark

[Art Smet]

The failure was near immediate just as you say, so I can understand why you'd
think it never reached the app server. But there are indications in the
plugin trace, right before the failue, that it knows the size of the large
file being returned, which means it must have reached the app server. Yes,
doing the flush is what I had in mind. If using the file-serving servlet to
serve the large file, you wont have control to issue the flush somewhere
before the 16M point. Sounds like you have your own servlet doing the
download so it maybe it will be possible in you situation. I'll see how far I
can get contacting IBM.

[Corey]

I am encountering the same problem. We have written a java app which
is registered within WebSphere. The application deals with uploading
files. When uploading a small .pdf, everything is fine. It's when I
use a file bigger than 16Mb that I encounter the problems. I get the
same symptoms both of you have discussed in this thread. Did we find
any answers to this problem? Please let me know.

Corey Cartwright

Art Smet <sm...@us.ibm.com> wrote in message news:<3D3EC3BE...@us.ibm.com>...

[Bob Kittelson]

Has there been any response from IBM as to a fix for this problem?

"Art Smet" <sm...@us.ibm.com> wrote in message
news:3D3EC3BE...@us.ibm.com...

[Ma...@softlanding.com]

Bob,

No, I never got any respose. Since I had developed a workaround I have
not pressed the issue further.

If you have a WebSphere support contract I would recommend reporting the
problem to IBM via official channels. It is very simply to duplicate so
it shouldn't be difficult to get a problem opened.

Mark

[Bob Kittelson]

I called the support line yesterday and opened up a PMR about this. I am
currently waiting to see what they tell me, but the person I am dealing with
has noted that he has queried Art S. and is awaiting a response.

<Ma...@softlanding.com> wrote in message
news:al5d3q$3e5m$1...@news.boulder.ibm.com...

[Byron Bailey]

Mark,

The file download problem that you are seeing is fixed in WAS 4.0.4, so please
get the latest ptf to fix this problem. In addition, the file upload problem
has also been fixed, but due to java object size restrictions at V4R5, such
large uploads will only work if you move to V5R1. In order to make them work
you will also have to tweak your plugin-cfg.xml file. The tweak is present to
prevent potential DOS attacks on the HTTP server.

Open your plugin-cfg.xml file in an editor, and add the attribute
PostSizeLimit to all applicable server groups. Set the attribute to the
mzximum value in bytes that you would like to be able to upload. For example:

<ServerGroup Name="iSeriesBox/Default Server" PostSizeLimit="30000000">
<Server CloneID="u0dea013" Name="Default Server">
<Transport Hostname="iSeriesBox" Port="10542" Protocol="http"/>
</Server>
</ServerGroup>

Unfortunately, any time the plugin file is regenerated you will need to add
this property manually.
-Byron

[Frances Stewart]

Note: V4.0.4 for iSeries is not yet available but should be available over this
weekend or early next week.

[Bob Kittelson]

Here is the latest reply I have gotten from the support line
.
Bob,
Here is the word from the developer about WAS 3.5
Download does work at v3.56 across all servers & OS releases. Upload
only works for apache http server at v5r1. The reasons the other
uploads fails is due to non-WebSphere reasons (original http server cant
handle requests 16MB+), nor can the JDK objects be greater than 16MB.
.
So, there is nothing to fix with 3.5. The problems are software
restrictions.

This statement proves to be wrong as we are on V5R1 running WAS 3.5.3 on the
Apache http server and we still have the upload barrier of 16 megs.

<Ma...@softlanding.com> wrote in message
news:ah78f2$etk$1...@news.boulder.ibm.com...

[Art Smet]

Bob, the upload was meant to apply to WAS v3.56 as well as the download
statement. I'm not sure why it is failing at WAS v3.53, but that level is over
a year old and it's possible the limitation got fixed somewhere along the way to
WAS v3.56 or perhaps in the latest level of the apache http server itself, which
WAS plugs into. Are you able to move up to WAS v3.56, if the upload still fails
you should contact IBM support.