Problem when run in Jetty or Tomcat

[Tom Vaughan]

Hi,

Does anyone have an idea why the `update-in` call here:

    http://stackoverflow.com/questions/7729628/serve-index-html-at-by-default-in-compojure/7730478#7730478

would work under ring but not under Jetty or Tomcat?

I can run `lein ring server` without problem. But when I load a .war produced by `lein ring uberwar` in Jetty or Tomcat I can see this this is being called and `:uri` is being updated within this wrapper, but any subsequent wrappers get the origin, unmodified value of just "/". Of course the problem may not be `update-in` but the problem appears to be that the result of update-in doesn't "stick".

Thanks.

-Tom

[Sean Bowman]

Why not use this instead (right below your answer):

http://stackoverflow.com/a/7730904

 

Or extend the "not-found" route functionality, have it check for a static index.html on the given path, and return that or a 404.html.  With this solution, you're closer to the Apache and Nginx standards of returning index.html by default in any path on the server, e.g. /users -> /users/index.html.

If you're serving a lot of static content, in particular your home page (/index.html) why not put Nginx or Apache in front of Compojure and have it serve the static HTML content instead?  Free performance boost right there, and you don't have to worry about the /index.html default at all.

[James Reeves]

The :uri key is always the URI passed to the web server, even if your application is deployed via a war in some subpath. If you want the context-specific path, try the :path-info key.

Alternatively, just add a route:

    (GET "/" [] (io/resource "public/index.html"))

There's no need to resort to middleware in this case.

- James

-Tom

--
You received this message because you are subscribed to the Google Groups "Compojure" group.
To view this discussion on the web visit https://groups.google.com/d/msg/compojure/-/_dCjY0WX17AJ.
To post to this group, send email to comp...@googlegroups.com.
To unsubscribe from this group, send email to compojure+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/compojure?hl=en.

[Tom Vaughan]

Thanks Sean.

Extending the 404 Not Found route sounds reasonable. But I'm stuck on
what that would look like in implementation. To me a handler that
"fixes-up" a request for another handler is middleware, right? What I
like about the middleware approach is that I don't have to re-invent
an existing handler, and I don't need to "keep up with it" if I did.
Sorry, I don't think I fully understand everything that's going on
here just yet.

Curious that Ring's `file-response` takes an `index-files?` option,
but `resource-response` doesn't:

https://github.com/ring-clojure/ring/blob/1.1.6/ring-core/src/ring/util/response.clj#L72
https://github.com/ring-clojure/ring/blob/1.1.6/ring-core/src/ring/util/response.clj#L104

Also, I'm just curious for my own edification why `update-in` works in
one environment and not the other.

We would like to put a webserver in front of Compojure to serve the
static content but we plan to continuously deploy to Amazon Beanstalk.
The only way we know how to do that is with the static resources and
the Compojure app bundled together in a .war. The static resources and
the Compojure app are in the same Git repo too (although they need not
be). Are there any tools we could leverage for this sort of continuous
deployment setup, or would we have to roll our own?

Thanks.

-Tom

> --
> You received this message because you are subscribed to the Google Groups
> "Compojure" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/compojure/-/mQ_sttM-GTYJ.

> To post to this group, send email to comp...@googlegroups.com.
> To unsubscribe from this group, send email to
> compojure+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/compojure?hl=en.

--
http://flavors.me/tvaughan

[Tom Vaughan]

Thanks James. That does indeed work. And this works for all routes,
not just the root route. I still wonder why the update to `:uri` was
lost (or was never really made), but I appreciate that I have a
solution that works. Best.

-Tom

--
http://flavors.me/tvaughan

[Sean Bowman]

On Dec 23, 2012, at 9:57 AM, Tom Vaughan <thomas.dav...@gmail.com> wrote:

> Thanks Sean.
>
> Extending the 404 Not Found route sounds reasonable. But I'm stuck on
> what that would look like in implementation. To me a handler that
> "fixes-up" a request for another handler is middleware, right? What I
> like about the middleware approach is that I don't have to re-invent
> an existing handler, and I don't need to "keep up with it" if I did.
> Sorry, I don't think I fully understand everything that's going on
> here just yet.

The "trick" with middleware is that it's just a series of functions that are called that modify the request map along the way down to the final function, your handler. Take a look at the source code for the not-found function, and you'll see it's pretty easy to override based on the code you already wrote around looking for an index.html file. I'd say write a function to take the :path-info, tack index.html, prepend your base directory for your static files, and see if there's an index.html. Make sure you filter for /../ in that path, though, or you could wind up serving unintended files!

> Curious that Ring's `file-response` takes an `index-files?` option,
> but `resource-response` doesn't:
>
> https://github.com/ring-clojure/ring/blob/1.1.6/ring-core/src/ring/util/response.clj#L72
> https://github.com/ring-clojure/ring/blob/1.1.6/ring-core/src/ring/util/response.clj#L104

It's because "resource" is looking on the classpath. It's incredibly tedious to discover files on the classpath. I know there are some libraries that do that, in particular Eclipse and Intellij dynamically scan classpaths for valid classes, but I don't think the resource-response function is that adventurous (I've written these functions before, and they're hard on your application, so I wouldn't recommend them for a live web site trying to do that 1000 times per second). For Java, looking for the files in a directory is a simple prospect.

> Also, I'm just curious for my own edification why `update-in` works in
> one environment and not the other.

I'm like you: I'd like to know the answer to that as well, and if I were in your shoes, it would be driving me nuts as well. I'm not sure why you're seeing something different inside a J2EE container than from without. Unfortunately I stopped deploying into containers with our Compojure apps and just run them with Jetty or Aleph (Netty) embedded directly in the app (I found it makes deployments easier), so I haven't fought this battle you're fighting right now.

> We would like to put a webserver in front of Compojure to serve the
> static content but we plan to continuously deploy to Amazon Beanstalk.
> The only way we know how to do that is with the static resources and
> the Compojure app bundled together in a .war. The static resources and
> the Compojure app are in the same Git repo too (although they need not
> be). Are there any tools we could leverage for this sort of continuous
> deployment setup, or would we have to roll our own?

I haven't worked with Beanstalk; we deploy our own EC2 instances and either deploy from S3 (I upload packages to S3), Git, or we use tools like Capistrano (Rails apps) or Puppet. That said, you do want to isolate your static files and deploy to and serve them from S3 rather than Beanstalk. Then you can use CloudFront to distribute the files on Amazon's CDN. That will allow you to scale up your requests and cut down on network load times as the files will be served from servers closer to the user.

You might also want to throw Jenkins into the mix. You could push your app through Jenkins, and if it passes all its tests, have Jenkins deploy it to Beanstalk. The Leiningen wiki has details on deploying to Jenkins, and I would imagine someone has a Jenkins-Beanstalk module by now.

[James Reeves]

On 24 December 2012 11:30, Sean Bowman <pic...@gmail.com> wrote:

> Also, I'm just curious for my own edification why `update-in` works in
> one environment and not the other.

I'm like you:  I'd like to know the answer to that as well, and if I were in your shoes, it would be driving me nuts as well.  I'm not sure why you're seeing something different inside a J2EE container than from without.  Unfortunately I stopped deploying into containers with our Compojure apps and just run them with Jetty or Aleph (Netty) embedded directly in the app (I found it makes deployments easier), so I haven't fought this battle you're fighting right now.

It's likely because when he deploys it to Tomcat or Jetty, the path of the application changes to a subdirectory.

For instance, in development you might access your application at http://localhost:3000, but when you deploy it to a servlet container as a war file, the path might be http://localhost:8080/testapp.

It's important to realise that the :uri key contains the URI of the request, not the relative path. So if :uri is "/" in development, when you deploy it to a servlet container, that same :uri key might be "/testapp" instead.

So what do you use instead? Well, this (or (:path-info request) (:uri request)) should cover all bases. When deployed under a context macro, or as a war file, the :path-info key is set, so if you check that first for the relative path, and then fall back on the war if that's not found, that should work.

You should also avoid changing the :uri key - update the :path-info key instead if you want to change the relative path in some way.

- James

[Tom Vaughan]

On Mon, Dec 24, 2012 at 9:27 AM, James Reeves <ja...@booleanknot.com> wrote:
> On 24 December 2012 11:30, Sean Bowman <pic...@gmail.com> wrote:
>>
>> > Also, I'm just curious for my own edification why `update-in` works in
>> > one environment and not the other.
>>
>> I'm like you: I'd like to know the answer to that as well, and if I were
>> in your shoes, it would be driving me nuts as well. I'm not sure why you're
>> seeing something different inside a J2EE container than from without.
>> Unfortunately I stopped deploying into containers with our Compojure apps
>> and just run them with Jetty or Aleph (Netty) embedded directly in the app
>> (I found it makes deployments easier), so I haven't fought this battle
>> you're fighting right now.
>
>
> It's likely because when he deploys it to Tomcat or Jetty, the path of the
> application changes to a subdirectory.

No, that's not it. The path is always "/".

I was too quick with my "it works" proclamation. Sort of.

This works in Ring but not Jetty or Tomcat.

(defn- wrap-dir-index [handler]
(fn [req]
(handler
(update-in req (filter #(contains? req %) [:path-info :uri])
#(if (re-find #"/$" %) (format "%sindex.html" %) %)))))

When I remove `:uri` in the sequence above it does work in Jetty or
Tomcat. I can reduce this further such that just this:

(defn- wrap-dir-index [handler]
(fn [req]
(do
(update-in req [:path-info :uri] str)
(handler req))))

doesn't work in Jetty or Tomcat. The error is a little bit more
constructive in this case. For example:

java.lang.ClassCastException: java.lang.String cannot be cast to
clojure.lang.Associative
at clojure.lang.RT.assoc(RT.java:691)
at clojure.core$assoc.invoke(core.clj:187)
at clojure.core$update_in.doInvoke(core.clj:5472)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:607)
at clojure.core$update_in.doInvoke(core.clj:5471)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at mametipsum.middleware$wrap_dir_index$fn__74.invoke(middleware.clj:10)

If I look at the type of (:uri req) it is java.lang.String.

The plot thickens...

Thanks.

-Tom

>
> For instance, in development you might access your application at
> http://localhost:3000, but when you deploy it to a servlet container as a
> war file, the path might be http://localhost:8080/testapp.
>
> It's important to realise that the :uri key contains the URI of the request,
> not the relative path. So if :uri is "/" in development, when you deploy it
> to a servlet container, that same :uri key might be "/testapp" instead.
>
> So what do you use instead? Well, this (or (:path-info request) (:uri
> request)) should cover all bases. When deployed under a context macro, or as
> a war file, the :path-info key is set, so if you check that first for the
> relative path, and then fall back on the war if that's not found, that
> should work.
>
> You should also avoid changing the :uri key - update the :path-info key
> instead if you want to change the relative path in some way.
>
> - James
>

> --
> You received this message because you are subscribed to the Google Groups
> "Compojure" group.

[Tom Vaughan]

On Mon, Dec 24, 2012 at 8:30 AM, Sean Bowman <pic...@gmail.com> wrote:
>
> On Dec 23, 2012, at 9:57 AM, Tom Vaughan <thomas.dav...@gmail.com> wrote:
>
>> Thanks Sean.
>>
>> Extending the 404 Not Found route sounds reasonable. But I'm stuck on
>> what that would look like in implementation. To me a handler that
>> "fixes-up" a request for another handler is middleware, right? What I
>> like about the middleware approach is that I don't have to re-invent
>> an existing handler, and I don't need to "keep up with it" if I did.
>> Sorry, I don't think I fully understand everything that's going on
>> here just yet.
>
> The "trick" with middleware is that it's just a series of functions that are called that modify the request map along the way down to the final function, your handler. Take a look at the source code for the not-found function, and you'll see it's pretty easy to override based on the code you already wrote around looking for an index.html file. I'd say write a function to take the :path-info, tack index.html, prepend your base directory for your static files, and see if there's an index.html. Make sure you filter for /../ in that path, though, or you could wind up serving unintended files!

Good point about tainted inputs. Thanks. So it sounds like middleware
is the correct approach, right?

>
>> Curious that Ring's `file-response` takes an `index-files?` option,
>> but `resource-response` doesn't:
>>
>> https://github.com/ring-clojure/ring/blob/1.1.6/ring-core/src/ring/util/response.clj#L72
>> https://github.com/ring-clojure/ring/blob/1.1.6/ring-core/src/ring/util/response.clj#L104
>
> It's because "resource" is looking on the classpath. It's incredibly tedious to discover files on the classpath. I know there are some libraries that do that, in particular Eclipse and Intellij dynamically scan classpaths for valid classes, but I don't think the resource-response function is that adventurous (I've written these functions before, and they're hard on your application, so I wouldn't recommend them for a live web site trying to do that 1000 times per second). For Java, looking for the files in a directory is a simple prospect.

Ok. That makes sense. Thanks.

>
>> Also, I'm just curious for my own edification why `update-in` works in
>> one environment and not the other.
>
> I'm like you: I'd like to know the answer to that as well, and if I were in your shoes, it would be driving me nuts as well. I'm not sure why you're seeing something different inside a J2EE container than from without. Unfortunately I stopped deploying into containers with our Compojure apps and just run them with Jetty or Aleph (Netty) embedded directly in the app (I found it makes deployments easier), so I haven't fought this battle you're fighting right now.

I just posted some additional details about this. That's a deployment
strategy I hadn't considered. Thanks for that.

>
>> We would like to put a webserver in front of Compojure to serve the
>> static content but we plan to continuously deploy to Amazon Beanstalk.
>> The only way we know how to do that is with the static resources and
>> the Compojure app bundled together in a .war. The static resources and
>> the Compojure app are in the same Git repo too (although they need not
>> be). Are there any tools we could leverage for this sort of continuous
>> deployment setup, or would we have to roll our own?
>
> I haven't worked with Beanstalk; we deploy our own EC2 instances and either deploy from S3 (I upload packages to S3), Git, or we use tools like Capistrano (Rails apps) or Puppet. That said, you do want to isolate your static files and deploy to and serve them from S3 rather than Beanstalk. Then you can use CloudFront to distribute the files on Amazon's CDN. That will allow you to scale up your requests and cut down on network load times as the files will be served from servers closer to the user.

I didn't know about CloudFront. Thanks for that too.

>
> You might also want to throw Jenkins into the mix. You could push your app through Jenkins, and if it passes all its tests, have Jenkins deploy it to Beanstalk. The Leiningen wiki has details on deploying to Jenkins, and I would imagine someone has a Jenkins-Beanstalk module by now.

We do use Jenkins. I think that's one of the reasons why a .war with
the static resources included is preferable at this point.

Best.

-Tom


>
>
> --
> You received this message because you are subscribed to the Google Groups "Compojure" group.

[Tom Vaughan]

OK. This now looks to be my fault and not Jetty's or Tomcat's. I can't
reproduce my initial problem anymore. That was where it seemed like
the result of `update-in` didn't stick when I updated `:uri`. I now
can see that `:uri` *is* being updated. But Jetty and Tomcat ignore
this more or less and require instead that `:path-info` be updated. So
that's helpful.

I assume now I've just made some sort of rookie mistake with my
`update-in` statement above in particular the sequence with more than
one value, right?

Thanks a lot.

-Tom

--
http://flavors.me/tvaughan

[James Reeves]

On 24 December 2012 14:35, Tom Vaughan <thomas.dav...@gmail.com> wrote:

I assume now I've just made some sort of rookie mistake with my
`update-in` statement above in particular the sequence with more than
one value, right?

Pretty much. Take the middleware you had in your previous post:

(defn- wrap-dir-index [handler]
  (fn [req]
    (do
      (update-in req [:path-info :uri] str)
      (handler req))))

The update-in function returns a new, modified map, but it doesn't change the original map. Remember, in Clojure types are immutable.

The other problem is you've written your update-in function to expect a data structure like:

{:path-info {:uri "foo"}

You're saying update the :uri key in a map referenced by the :path-info key.

Instead, you want something like:

(defn wrap-dir-index [handler]

  (fn [request]

    (let [path-info (or (:path-info request) (:uri request))]

      (if (= path-info "/")

        (handler (assoc request :path-info "/index.html"))

        (handler request)))))

But honestly, it's easier just to add a route in:

(GET "/" []

  (io/resource "public/index.html"))

- James

[Tom Vaughan]

On Mon, Dec 24, 2012 at 11:48 AM, James Reeves <ja...@booleanknot.com> wrote:
> On 24 December 2012 14:35, Tom Vaughan <thomas.dav...@gmail.com>
> wrote:
>>
>> I assume now I've just made some sort of rookie mistake with my
>> `update-in` statement above in particular the sequence with more than
>> one value, right?
>
>
> Pretty much. Take the middleware you had in your previous post:
>
> (defn- wrap-dir-index [handler]
> (fn [req]
> (do
> (update-in req [:path-info :uri] str)
> (handler req))))
>
> The update-in function returns a new, modified map, but it doesn't change
> the original map. Remember, in Clojure types are immutable.
>
> The other problem is you've written your update-in function to expect a data
> structure like:
>
> {:path-info {:uri "foo"}
>
> You're saying update the :uri key in a map referenced by the :path-info key.

Ah! Right. Great. Thanks for that.

>
> Instead, you want something like:
>
> (defn wrap-dir-index [handler]
> (fn [request]
> (let [path-info (or (:path-info request) (:uri request))]
> (if (= path-info "/")
> (handler (assoc request :path-info "/index.html"))

This is awesome. Thanks. But does this only update the value
associated with the :path-info key? I need to update :path-info or
:uri, right?

> (handler request)))))
>
> But honestly, it's easier just to add a route in:
>
> (GET "/" []
> (io/resource "public/index.html"))

Cool. Thanks.

-Tom

[Tom Vaughan]

This is what I came up with:

(defn- wrap-dir-index [handler]
(fn [request]
(handler
(let [k (if (contains? request :path-info) :path-info :uri) v
(get request k)]
(if (re-find #"/$" v)
(assoc request k (format "%sindex.html" v))
request)))))

Thanks a lot Sean and James. I really appreciate it.

-Tom

>
>> (handler request)))))
>>
>> But honestly, it's easier just to add a route in:
>>
>> (GET "/" []
>> (io/resource "public/index.html"))
>
> Cool. Thanks.
>
> -Tom
>
>
>>
>> - James
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Compojure" group.
>> To post to this group, send email to comp...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> compojure+...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/compojure?hl=en.
>
>
>
> --
> http://flavors.me/tvaughan

--
http://flavors.me/tvaughan

[James Reeves]

On 24 December 2012 15:20, Tom Vaughan <thomas.dav...@gmail.com> wrote:

This is awesome. Thanks. But does this only update the value
associated with the :path-info key? I need to update :path-info or
:uri, right?

Nope. You should always update the :path-info key, because that's the key that defines an application-specific path. The :uri key should (ideally) remain the same, because it's independent of the application.

- James

[Tom Vaughan]

OK. Then at this point a middleware approach isn't workable at all
because `:path-info` will only exist in `request` in a .war context.
This won't work with ring.

Thanks again.

-Tom

[James Reeves]

It will, because Compojure routes match against the :path-info key first (if it exists), and then the :uri key.

- James