Load Balancer causes PR_CONNECT_RESET_ERROR when grading Samigo attachments

[Austin]

Hello Sakai Devs,

We've run into an issue with our Load Balancer where if a Samigo test has a "file upload" question and you try to view a submission's uploaded file while grading, the browser (firefox) fails with a PR_CONNECT_RESET_ERROR (other browsers fail too, but they don't give a reason).    Our load balancer shows an error

"http_handle_pipelined_data - Invalid action:0x108000 Server sends too much data. serverside..."

an article, https://my.f5.com/manage/s/article/K42151600 ,mentions that

• The virtual server is configured with an HTTP profile, and the BIG-IP system receives a response from the pool member that either contains more data than indicated in the Content-Length header or has additional data following the ending chunk of a chunked response.
  

Has anyone else run into this?  Is Sakai sending an incorrect content-length or additional data following the ending chunk?  for a URL like eg. 

/samigo-app/servlet/ShowMedia?mediaId=1234567&sakai.tool.placement.id=abcd-12334

note - we prefer not to apply the workaround mentioned in the article if we don't have to since our load balancer handles traffic for other apps besides Sakai.

Thanks,

Austin

[Sam Ottenhoff]

Here is how I would try to diagnose:

1) Connect directly to Tomcat and record the HTTP headers including content-length

2) Connect via loadbalancer and record the HTTP headers including content-length

3) Measure the file size on local desktop

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/CAKL84%3Dmc%2Bjb_%3DGnOpa1xn3gkqNGPwxSZ8NNRDqTP4Z0gshhpRQ%40mail.gmail.com.

[Austin]

the filesizes are all equal in all of those cases, 15621 bytes.  I also debugged line by line and the filesize in ShowMediaServlet.java is the same when setting

res.setContentLength(fileSize)

Also, that servlet uses a BufferedOutputStream and res.flushBuffer() to display the image.  Could that cause the data to somehow not match or to "send additional data following the ending chunk"?

[Sam Ottenhoff]

Please show the headers from the direct Tomcat vs the loadbalancer output

[Austin]

please see attached headers from chrome developer

[Sam Ottenhoff]

The only oddities I see are a double HTTP status code of 200 (strange)

Your direct Tomcat "Server" is Apache

Our servers do not send this HTTP header:

  

    Transfer-Encoding: chunked

[Austin]

we run each node with apache -> mod_jk -> tomcat.

if the transfer isn't chunked, could the size be incorrect?  stepping through the following block from ShowMediaServlet.java, when the highlighted "while section" gets to the last block/batch of bytes to write, the buffer size "looks" like it's still 1024 even though the "i" and bytesLeft are e.g. 261.. could it be writing the full 1024 bytes (instead of 261) to the outputstream... thus making the stream bigger than the actual filesize?

      //** note that res.setContentType() must be called before res.getOutputStream(). see javadoc on this
      try (BufferedInputStream buf_inputStream = (StringUtils.isBlank(mediaLocation)) ?
              new BufferedInputStream(new ByteArrayInputStream(mediaData.getMedia()))
              :
              new BufferedInputStream(getFileStream(mediaLocation));
              BufferedOutputStream buf_outputStream = new BufferedOutputStream(res.getOutputStream());){

        int i=0;
        if (buf_inputStream != null)  {
            // skip to the start of the possible range request
            buf_inputStream.skip(start);
           
            int bytesLeft = rangeContentLength;
            byte[] buffer = new byte[1024];

            while ( (i = buf_inputStream.read(buffer)) != -1 && bytesLeft > 0){
                buf_outputStream.write(buffer);
                bytesLeft -= i;
            }
        }
        log.debug("**** mediaLocation="+mediaLocation);
       
        res.flushBuffer();
      }
      catch(IOException ioe) {
          log.warn("Error handling with IO in doPost", ioe);
      }
      catch(Exception e){
        log.warn(e.getMessage());
      }

[Sam Ottenhoff]

> we run each node with apache -> mod_jk -> tomcat.

So who is adding the Transfer-Encoding: chunked header? That seems key to your problem.

> if the transfer isn't chunked, could the size be incorrect?  stepping through the following block

> from ShowMediaServlet.java, when the highlighted "while section" gets to the last block/batch 

> of bytes to write, the buffer size "looks" like it's still 1024 even though the "i" and bytesLeft are

>  e.g. 261.. could it be writing the full 1024 bytes (instead of 261) to the outputstream... thus making

> the stream bigger than the actual filesize?

The behavior looks correct to me. Sakai is sending the correct number of bytes and the correct Content-Length.

[Austin]

I don't see Transfer-Encoding: chunked in our headers?  Are you saying it should be there? Or did you think we had it set and who/what is setting it?  I mentioned "chunked" earlier because I was guessing that could have happened based on the F5 load balancer's error documentation.

If you compare the while snippet in my previous email to ShowAttachentMedia.java's snippet below.  it doesn't read/write to a 1024 buffer.  Is there a performance advantage to read/writing 1024 bytes at a time vs one byte at a time?  Also, viewing an attachment that calls ShowAttachentMedia.java doesn't error with our LB.  So, I'm thinking of replacing the while loop in my previous email to the one below and see if that gets around the error.

    try{
        buf_outputStream = new BufferedOutputStream(outputStream);

        int i=0;
        while ((i=buf_inputStream.read()) != -1){
            buf_outputStream.write(i);
        }

        //res.setContentLength(count);
        res.flushBuffer();
    }

[Austin]

fyi, using the simplified while loop (from ShowAttachentMedia.java) worked on our test server.

[Sam Ottenhoff]

I don't see Transfer-Encoding: chunked in our headers? 

Are you just relying on your Chrome inspector, or are you using tools like curl to show the full raw headers? 

If you want help interpreting your headers, please don't obfuscate data like URLs. There is nothing private about Hawaii's Sakai URL and obfuscating it just makes it more difficult for people to help you out.

--Sam