More questions about dynamic image generation

[Strom]

My goals are to be able to fetch and display user uploaded images from
the DB in a way that won't bog down a high-traffic production server.
Something like what eBay or Amazon does in their search result pages.

I found an old thread on dynamic image generation that might be
relevant to what I'm doing. Couldn't reply to the old thread, so I
thought I'd make a new one. Here's the original:
http://groups.google.com/group/liftweb/browse_thread/thread/a4a04b145128e6f8/cb5e7026cb361fa6?lnk=gst&q=image#cb5e7026cb361fa6

I have a boat load of questions about David's code, and I can't find
any documentation that has a clear explanation of what's going on.

Here's the code:
object ImageLogic {
object TestImage {
def unapply(in: String): Option[Image] =
Image.find(By(Image.lookup, in.trim))
}

def matcher: LiftRules.DispatchPF = {
case r @ Req("image_logic" :: TestImage(img) ::
Nil, _, GetRequest) => () => servImage(img, r)
}

def servImage(img: Image, r: Req): Box[LiftResponse] = {
if (r.testIfModifiedSince(img.saveTime))
Full(InMemoryResponse(new Array[Byte](0),
List("Last-Modified" ->
toInternetDate(img.saveTime.is)), Nil,
304))
else Full(InMemoryResponse(img.image.is,
List("Last-Modified" ->
toInternetDate(img.saveTime.is),
"Content-Type" -> img.mimeType.is,
"Content-Length" ->
img.image.is.length.toString),
Nil,
200))
}

}

Questions
1. I see "unapply" it in the Req object's methods list in the API, but
I have NO idea what it's doing. Lift book has 1 sentence about it, but
I'm still lost.

2. What's the difference between DispatchPF and dispatch.append? I see
no documentation on what DispatchPF is, but from reading the API it
sounds like some sort of member object or something to be associated
with custom dispatches.

3. What is the "r @ Req" syntax? I'm having a hard time searching for
the syntax on a web search engine. Is it specific to Lift, or is it
something in Scala?

4. For the "servImage" method, what is the effect of the HTTP 304 code
if the image hasn't been modified? I looked it up and it's "Error Not
Modified". Will this image generation fail?

Other, more academic curiosities:
1. What's the advantage of using the lookup for the image class (GUID
vs primary key)? The image class in the original thread has lookup
extending MappedUniqueId, which is a randomly generated unique id. Is
there some security issue for using the Long PK of the image if I'm
going to be serving images (e.g. "site.com/image/4", where 4 is the Id
of the image instance in the DB), or is having a indexed lookup field
inherently better?

2. What are the performance/compatibility issues with using
StreamingResponse vs InMemoryResponse for the LiftResponse? If I'm
going to be displaying these images I have on a search results page
with 30 or more images, should I be using one or the other? Is
StreamingResponse even compatible with generating images in this
scenario?

3. When the site I'm building moves to production, I'd like to
incorporate EHCache or some other (forum recommended ;) ) image
caching tools to ease the load on the production server. I'd also like
to use subdomains for image servers...something like img1.site.com,
img2.site.com. I'm not sure how to go about it all really, have to do
more research on multiple image servers and such. Is the dynamic image
generation the best method to move forward, or is there another
alternative method to fetch and display user uploaded images from the
database?

Thanks,
Strom

[Ross Mellgren]

Some answers inline:

On Nov 10, 2009, at 6:05 PM, Strom wrote:

Questions
1. I see "unapply" it in the Req object's methods list in the API, but
I have NO idea what it's doing. Lift book has 1 sentence about it, but
I'm still lost.

unapply is part of scala proper, not lift particularly. It is the converse of apply, and is used in pattern matching:

class MyMatchableObject(val value: Int)

object MyMatchableObject {

    def unapply(o: MyMatchableObject): Option[Int] = Some(o.value)

}

val foo = new MyMatchableObject(1)

foo match { case MyMatchableObject(value) => ... } // calls MyMatchableObject.unapply to yield value

    

2. What's the difference between DispatchPF and dispatch.append? I see
no documentation on what DispatchPF is, but from reading the API it
sounds like some sort of member object or something to be associated
with custom dispatches.

DispatchPF is a type alias. vscaladoc is pretty bad about showing inner types, so here's its definition from LiftRules.scala:

1.   type DispatchPF = PartialFunction[Req, () => Box[LiftResponse]];  

That is, a dispatching partial function is a partial function taking a Req and yielding a function that generates a response. Partial functions, in case you aren't already aware are extensions of simple functions (Function1) except that in addition to application, you can test whether or not the function is defined for a particular argument.

3. What is the "r @ Req" syntax? I'm having a hard time searching for
the syntax on a web search engine. Is it specific to Lift, or is it
something in Scala?

It's scala -- this captures the req itself as the variable "r" -- that is, you can refer to "r" in the case statement and it'll be the whole Req, not some of Req's arguments.

4. For the "servImage" method, what is the effect of the HTTP 304 code
if the image hasn't been modified? I looked it up and it's "Error Not
Modified". Will this image generation fail?

This indicates to the browser that the image has not changed since the last time it asked, as an optimization. The browser will sometimes send a header "If-Modified-Since" which says basically "give me this resource if it has changed since this date when I last got it, else give me 304 and I'll use what I already have". I assume the test on If-Modified-Since is wrapped up in Req.testIfModifiedSince.

-Ross

[Strom]

Thanks for the reply. I implemented the code and it's working, but
sporadically.

I'm getting these issues on my Win XP browsers:
1. Internet Explorer 6 and Safari - the image is broken altogether.
2. FireFox - The initial attempt to display or view the image via URL
(e.g. http:localhost/image/{lookup}) results in an XML error (I can't
seem to find why, nor can I reproduce it all the time), but subsequent
requests to the url succeed.

I used David's code, except the Dispatch URL is "image/{lookup}" vs
"image_logic{lookup}", and when I'm trying to bind to a page I'm
calling:
bind("item", xhtml, "image" -> <img src={image.lookup} />)

Any ideas on what's happening? Am I doing something wrong?

[Ross Mellgren]

Any chance you could post your code or a reproducible example so we can poke at it?

If you're getting an XML error looking at the image URL directly, I suspect that something is wrong with your Content-Type, therefore with your img.mimeType. Perhaps use Firebug or the like to see what Content-Type you're getting back? It should always be image/png or image/jpeg or something along those lines -- if it's something else then the browser may try to interpret the result in whatever what it can.

If everything is going to the right place, then you shouldn't have anything that even vaguely looks like XML though, which is why it would be very helpful to see the code.

-Ross

> --
>
> You received this message because you are subscribed to the Google Groups "Lift" group.
> To post to this group, send email to lif...@googlegroups.com.
> To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/liftweb?hl=.
>
>

[Strom]

Thanks Ross,
I'll try to get some code up in a bit. In the meantime, can you
explain how to use Firebug to check the content type? I looked under
the HTML and DOM inspectors and didn't find how to look for it.

Also, the error I'm getting is an XML parsing error:
XML Parsing Error: no element found
Location: http://localhost:8080//image/2WAW0HOHBIG03VYR
Line Number 1, Column 1

Probably doesn't help too much, but thought I'd share before getting
some code up.

On Nov 16, 5:19 pm, Ross Mellgren <dri...@gmail.com> wrote:
> Any chance you could post your code or a reproducible example so we can poke at it?
>

> If you're getting an XML error looking at theimageURL directly, I suspect that something is wrong with your Content-Type, therefore with your img.mimeType. Perhaps use Firebug or the like to see what Content-Type you're getting back? It should always beimage/png orimage/jpeg or something along those lines -- if it's something else then the browser may try to interpret the result in whatever what it can.

>
> If everything is going to the right place, then you shouldn't have anything that even vaguely looks like XML though, which is why it would be very helpful to see the code.
>
> -Ross
>
> On Nov 16, 2009, at 8:14 PM, Strom wrote:
>
> > Thanks for the reply. I implemented the code and it's working, but
> > sporadically.
>
> > I'm getting these issues on my Win XP browsers:

> > 1. Internet Explorer 6 and Safari - theimageis broken altogether.
> > 2. FireFox - The initial attempt to display or view theimagevia URL

> >>> if theimagehasn't been modified? I looked it up and it's "Error Not
> >>> Modified". Will thisimagegenerationfail?
>
> >> This indicates to the browser that theimagehas not changed since the  

[Strom]

Thanks Ross,
I'll try to get some code up in a bit. In the meantime, can you
explain how to use Firebug to check the content type? I looked under
the HTML and DOM inspectors and didn't find how to look for it.

Also, the error I'm getting is an XML parsing error:
XML Parsing Error: no element found
Location: http://localhost:8080//image/2WAW0HOHBIG03VYR
Line Number 1, Column 1

Probably doesn't help too much, but thought I'd share before getting
some code up.

On Nov 16, 5:19 pm, Ross Mellgren <dri...@gmail.com> wrote:
> Any chance you could post your code or a reproducible example so we can poke at it?
>

> If you're getting an XML error looking at theimageURL directly, I suspect that something is wrong with your Content-Type, therefore with your img.mimeType. Perhaps use Firebug or the like to see what Content-Type you're getting back? It should always beimage/png orimage/jpeg or something along those lines -- if it's something else then the browser may try to interpret the result in whatever what it can.

>
> If everything is going to the right place, then you shouldn't have anything that even vaguely looks like XML though, which is why it would be very helpful to see the code.
>
> -Ross
>
> On Nov 16, 2009, at 8:14 PM, Strom wrote:
>
> > Thanks for the reply. I implemented the code and it's working, but
> > sporadically.
>
> > I'm getting these issues on my Win XP browsers:

> > 1. Internet Explorer 6 and Safari - theimageis broken altogether.
> > 2. FireFox - The initial attempt to display or view theimagevia URL

> >>> if theimagehasn't been modified? I looked it up and it's "Error Not
> >>> Modified". Will thisimagegenerationfail?
>
> >> This indicates to the browser that theimagehas not changed since the  

[Ross Mellgren]

It's in the "net" tab of firebug, if I recall. I haven't used firebug in a little bit though so my memory may be foggy.

-Ross

[Strom]

/** Image.scala **/
object Image extends Image with LongKeyedMetaMapper[Image] {
override def dbTableName = "images"
}

class Image extends LongKeyedMapper[Image] with IdPK {
def getSingleton = Image

object image extends MappedBinary(this)
object listing extends MappedLongForeignKey(this, Listing)
object lookup extends MappedUniqueId(this, 16) {
override def dbIndexed_? = true
}
object saveTime extends MappedLong(this) {
override def defaultValue = Helpers.millis //TimeHelpers method
}
object mimeType extends MappedString(this, 256)
def imageUrl : NodeSeq = {
<img src={"/image/"+lookup} />
}
}

/** ImageLogic.scala **/

object ImageLogic {
object TestImage {
def unapply(in: String): Option[Image] =
Image.find(By(Image.lookup, in.trim))
}

def matcher: LiftRules.DispatchPF = {

case req @ Req("image" :: TestImage(img) :: Nil, _, GetRequest) =>
() => serveImage(img, req)
}

def serveImage(img: Image, req: Req) : Box[LiftResponse] = {
if (req.testIfModifiedSince(img.saveTime)) {
//if not modified, optimized to tell browser to use last good
version
Full(InMemoryResponse(

new Array[Byte](0),
List("Last-Modified" -> toInternetDate(img.saveTime.is)),
Nil,

HttpCode.NotModified))
} else {
//serve the image
Full(InMemoryResponse(

img.image.is,
List("Last-Modified" -> toInternetDate(img.saveTime.is),
"Content-Type" -> img.mimeType.is,
"Content-Length" -> img.image.is.length.toString),

Nil /*cookies*/,
HttpCode.Ok)
)
}
}//end serveImage

}//end object ImageLogic

/** Template HTML **/
<lift:surround with="default" at="content">
<h2>Details</h2>
<br/>
<lift:ListingDetailsPage.summary>
<h2><listing:title /></h2>
<listing:desc />
<br/>
<listing:image />
<br/>
</lift:ListingDetailsPage.summary>
</lift:surround>

/** ListingDetailsPage snippet **/
class ListingDetailsPage {
def summary(xhtml : NodeSeq) : NodeSeq = S.param("listingId") match
{
case Full(listingId) => {
Listing.find(By(Listing.id, listingId.toLong)) match {
case Full(l) => {
bind("listing", xhtml,
"title" -> l.title.asHtml,
"desc" -> l.description.asHtml,
"image" -> bindImages(l.images))
}
case _ => Text("Could not find any listing with id:
"+listingId)
}
}
case _ => {
Text("No listing id provided")
}
} //end listingId match, summary method

private def bindImages(images: List[Image]) : NodeSeq = {
if(images.isEmpty) {
Text("No image or default")
} else {
images(0).imageUrl
}
}

} //end class

[Ross Mellgren]

I tried to hack your code into a lift-archetype-basic and eventually had success but could not reproduce the problem (the image downloaded as I intended). I did hardcode the image data and mime type because I didn't want to try and wedge an image file into the H2 database. I did have one issue where the savetime on the image was 0, and that caused it to generate 304 Not Modified. You might want to make your savetime in the database is not 0.

-Ross

[Strom]

Thanks Ross,
I looked up the mime type and save time in the DB (using postgres),
and mime type is "image/jpeg" while save time is non-zero. The save
time override is the current time in millis, so I don't think there
should be a 0 save time.

Anyways, thanks for your help in this. I'll poke around a bit more,
and reply to this if I find out anything more.

On Nov 16, 9:23 pm, Ross Mellgren <dri...@gmail.com> wrote:
> I tried to hack your code into a lift-archetype-basic and eventually had success but could not reproduce the problem (theimagedownloaded as I intended). I did hardcode theimagedata and mime type because I didn't want to try and wedge animagefile into the H2 database. I did have one issue where the savetime on theimagewas 0, and that caused it to generate 304 Not Modified. You might want to make your savetime in the database is not 0.

>
> -Ross
>
> On Nov 16, 2009, at 9:03 PM, Strom wrote:
>
> > /**Image.scala **/

> > objectImageextendsImagewith LongKeyedMetaMapper[Image] {

> >  override def dbTableName = "images"
> > }
>

> > classImageextends LongKeyedMapper[Image] with IdPK {
> >  def getSingleton =Image
>
> >  objectimageextends MappedBinary(this)

> >      Text("Noimageor default")

[Strom]

Actually, can I see the code you used? It's weird yours works and mine
doesn't. I'd just like to compare.

> ...
>
> read more »

[Ross Mellgren]

Sure, I'll send it along when I get home, as it's not on the computer
I'm at now.

-Ross

[Ross Mellgren]

http://github.com/Dridus/test-image

-Ross

On Nov 17, 2009, at 1:52 PM, Strom wrote:

[Hannes]

Hi Ross,

I've took the code from GitHub and modified it a little bit to fit my needs - works great!

The only thing is that its not 100% compatible with Jetty, because the app root changed.

The method you used to declare this was:

def matcher: LiftRules.DispatchPF = {
    case req @ Req("image" :: TestImage(img) :: Nil, _, GetRequest) =>
      () => serveImage(img, req)
  }

Now, when I have an image tag like <img src="/image/12344567889" /> and I use Jetty, it puts the name of the applications war file in front of it so it looks like <img src="/myapp/image/12344567889" />

What do I've to change to get it working, must be something really small I guess...

thanks!