Jetty does not return full image to browser when use ResponseBytes

[Anton Moiseev]

Hello, I am trying to experiment with basic web application structure on Unfiltered, have some success but met a strange problem.

Here is full application code (maven project):

Scala code:
https://github.com/1i7/snippets/blob/master/scala-web/ScalaUnfilteredBasicDemo/src/main/scala/edu/nntu/scalaunfiltereddemo/UnfilteredBasicDemo.scala

I load images from resources inside application jar in the following way:

case Path(Seg("lasto4ka.png" :: Nil)) =>    val in = getClass.getResourceAsStream("/public/lasto4ka.png")    val bytes = new Array[Byte](in.available)    in.read(bytes)
 
 
    object JpegImageContent extends ContentType("image/jpeg")    object PngImageContent extends ContentType("image/png")
    Ok ~> PngImageContent ~> ResponseBytes(bytes)
 
case Path(Seg("led6_on.png" :: Nil)) =>
 val in = getClass.getResourceAsStream("/public/led6_on.png")
 val bytes = new Array[Byte](in.available)
 in.read(bytes)
 
 object JpegImageContent extends ContentType("image/jpeg")
 object PngImageContent extends ContentType("image/png")
 
 
 Ok ~> PngImageContent ~> ResponseBytes(bytes)

and then show it on html page like this:
https://github.com/1i7/snippets/blob/master/scala-web/ScalaUnfilteredBasicDemo/src/main/resources/html/static_html.html

 <img src="lasto4ka.png" alt="lasto4ka"/>
 <img src="led6_on.png" alt="led6_on"/>

So, when I run this code locally and open address http://localhost:8080/static_html in browser, I can see both images on the page, all is fine.

But then when I load jar to Amazon virtual server instance, run web application remotely and open the same application page in Firefox, the 1st image on the page
https://github.com/1i7/snippets/blob/master/scala-web/ScalaUnfilteredBasicDemo/src/main/resources/public/lasto4ka.png

loads and shows fine, but the 2nd image
https://github.com/1i7/snippets/blob/master/scala-web/ScalaUnfilteredBasicDemo/src/main/resources/public/led6_on.png

fails to load.  Actually, it shows the upper part of the image for a second and then hides it with error message in the browser console:

> Image corrupt or truncated: http://robotc.lasto4ka.su:8080/led6_on.png

In Google Chrome the browser shows the upper part of the image.

[Chris Lewis]

Hello Anton,

I've only glanced at this but I have a few thoughts. If you want to serve static resources through unfiltered I recommend using the resource handling provided by server builders (see https://github.com/unfiltered/unfiltered/blob/0.8.2/jetty/src/main/scala/Server.scala#L59). I know you mentioned this is an experimental app, but applications dealing with moderate to heavy load would do better to use a dedicated asset server/cluster/CDN rather than burdening a java process with streaming bytes.

As to why you are seeing this particular behavior, I suspect it has to do with the behavior of the available method. Per the api docs this method does not necessarily give you the total number of bytes that make up the underlying resource, but rather an "estimate." If that less than the size of the file then you'll allocate an array too small, and the call to read will only read up to bytes.length, so it won't overflow the array. If this is the case then you will be streaming only a portion of the bytes for the image.

If you want to confirm this you can log the allocated array size, or you can probably just curl the image served from amazon directly to a local blob and compare its byte size with the image locally (ie curl http://whatever/img.png > blob && ls -l blob).

--
You received this message because you are subscribed to the Google Groups "Unfiltered" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unfiltered-sca...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-chris

[Anton Moiseev]

Chris, thank's for input, I have solved this.

The problem was in this line:

      val bytes = new Array[Byte](in.available)
      in.read(bytes)

in.available returns correct value, but in.read in some cases does not read the whole strem at a time, but only part of it (and returns number of read bytes).

so I have replaced these two line with a bigger section which always reads all bytes from input stream:

      val buffer = new java.io.ByteArrayOutputStream()
      val data = new Array[Byte](16384)
      var nRead = in.read(data, 0, data.length)
      while (nRead != -1) {
        buffer.write(data, 0, nRead)
        nRead = in.read(data, 0, data.length)
      }
      buffer.flush()
      val bytes = buffer.toByteArray();

and in this way all works now.



For the reason why this could happen. On the local machine I run application from netbeans, and as I have found out, it does not run it from jar, but from output directory on filesystem - for example, this:

println(getClass.getResource("/public"))

on local machine (with netbeans) will print:

> file:/C:/xxx/snippets/scala-web/ScalaUnfilteredBasicDemo/target/classes/public

and on the remote machine I run application with java -jar and same call will print:

> jar:file:/home/ubuntu/scala/scala-unfiltered-basic-demo-1.0-SNAPSHOT.jar!/public


So, as I can see, when in.read(bytes) is called for local file, it will try to read the whole file at once (at least in my case with bigger image), and when called for resource file inside jar, it will read it in multiple steps (I have downloaded corrupted image from initial setup with curl and it gave me file with correct size, but it was filled with zeros at the end).

I also run Oracle JDK on local machine and OpenJDK on remote machine, which also might affect this in some way.

---

Anyway, by following your advice, I have added 'public' dir in resource to jetty startup line in this way:

unfiltered.jetty.Http.apply(8080).resources(getClass.getResource("/public")).plan(handlePath).run()

and now all images from '/public' are loaded automatically without any of the above code. The only trap here may be - the last section with any path handler should be removed from custom filter:

//    case Path(Decode.utf8(Seg(path :: Nil))) =>
//      Ok ~> PlainTextContent ~> ResponseString("Your segment: " + path)

because it will capture all requests and resources will never be loaded.


This also works for resources inside jar file and I really love this because in this way creating and running web application as normal jar file with all resources inside is much simpler, than deploying war file to tomcat (no need to install and confugure tomcat). For high loaded installations with many static content I would use nginx frontend with virtual filesystem directory for images and other static content.

[Chris Lewis]

Glad you sorted it out! Yeah the behavior of read can be different according to the JVM implementation, server hardward, OS .. a plethora of things really. Yay resource handling!