Advanced Configuration for es-jetty

388 views
Skip to first unread message

Pulkit Singhal

unread,
Apr 26, 2012, 8:56:37 AM4/26/12
to elasticse...@googlegroups.com
@imotov - BCC'ed you since you aren't on the list yet :)

Question:
It may just be my ignorance of Jetty (though I have tried to fill the gap) but based on the REAME on github, it seems that the JettyHttpServerTransportModule is supposed to go hand in hand with JettyHttpServerTransportHandler.

But in the files provided for the advanced configuration:
config/elasticsearch.yml
config/jetty.xml
... it is the FilterHttpTransportModule (yml file)which is being configured with the JettyHttpServerTransportHandler (xml file).

Is this intended? I suppose I simply don't understand the similarities and differences between JettyHttpServerTransport and FilterHttpTransportModule ... could you elaborate? Which of these two is a subset of the other? And which one is the superset in terms of functionality and features? Is there any thing that only one of them does and the other doesn't?

Thanks!

imotov

unread,
Apr 26, 2012, 12:58:31 PM4/26/12
to elasticse...@googlegroups.com
The http.type parameter in elasticsearch.yml defines which http transport should be used to process http requests. The default transport that comes with elasticsearch is netty. The elasticsearch-jetty plugin adds two additional options: JettyHttpServerTransportModule and FilterHttpTransportModule. 

JettyHttpServerTransportModule wraps jetty server and allows elasticsearch to use it as HttpServerTransport. As you know, the jetty server by itself doesn't know how to process any requests - it needs to be configured. By default, JettyHttpServerTransportModule configures jetty using the jetty.xml file. The supplied jetty.xml file contains the following stack of handlers:

GzipHandler (from jetty - handles gzip compression)
  RestConstraintSecurityHandler (from elasticsearch-jetty - handles authorization)
    JettyHttpServerTransportHandler (from elasticsearch-jetty - dispatches http requests to elasticsearch)

Jetty handles http requests and passes them to the top handler in the stack. Each handler processes requests and then passes them to the underlying handler. When requests reach JettyHttpServerTransportHandler, it dispatches them to elasticsearch. 

FilterHttpTransportModule wraps another HttpServerTransport. It can wrap JettyHttpServerTransportModule (that's used by default) or NettyHttpServerTransportModule. Wrapping underlying HttpServerTransport allows us to intercept http requests after they are parsed but before they are dispatched to elaticsearch, and pass them through a series of filters. FilterHttpTransportModule accepts a stack of filters defined in the sonian.elasticsearch.http.filter parameter in elasticsearch.yml file. We are using several filters in our deployment including logging filter (LoggingFilterHttpServerAdapter) that logs all http requests sent to elasticsearch.

Pulkit Singhal

unread,
Apr 27, 2012, 8:34:31 AM4/27/12
to elasticse...@googlegroups.com
Thank you for the awesome explanation. Its a lot clearer to me now.

I suppose that sicne FilterHttpTransportModule wraps JettyHttpServerTransportModule by default, I didn't see a configuration setting for it anywhere that would setup this relationship. If I had seen a commented out version of it, I might have caught on sooner ... though I doubt it :p

--
You received this message because you are subscribed to the Google Groups "elasticsearch-jetty" group.
To view this discussion on the web visit https://groups.google.com/d/msg/elasticsearch-jetty/-/ZRd5ezxHYPAJ.
To post to this group, send email to elasticse...@googlegroups.com.
To unsubscribe from this group, send email to elasticsearch-j...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/elasticsearch-jetty?hl=en.

Pulkit Singhal

unread,
Apr 27, 2012, 10:04:21 AM4/27/12
to elasticse...@googlegroups.com
Found it! Silly me:
====
371     # Filter should use netty instead of jetty (default)
372     #transport_type": "netty"
====

Another follow-up Q that is based in my ignorance of Jetty + the plugin alike:

Q1: Does the following line have to be uncommented for jetty-ssl.xml to be picked up?
====
461 #    config: jetty.xml,jetty-ssl.xml
====

Q2: Or is it simply enough for jetty-ssl.xml to be present in the config directory?

Pulkit Singhal

unread,
Apr 27, 2012, 10:07:17 AM4/27/12
to elasticse...@googlegroups.com
Re-pasting the chunk for Q1:
====
447 # Jetty Settings
448 sonian.elasticsearch.http.jetty:
449     port: XXXX
450     ssl_port: XXXX
451     keystore_password: "blah"
452     config: jetty.xml,jetty-ssl.xml
====

Pulkit Singhal

unread,
Apr 27, 2012, 10:37:05 AM4/27/12
to elasticse...@googlegroups.com
With the realm.properties, keystore.jks, jetty.xml, jetty-ssl.xml in place ... and the http.type, sonian.elasticsearch.http.jetty.config, and other settings in elasticsearch.yml  ... the server just spits out:
====
WARNING: ElasticSearch may have failed to start.
====

The logs don't have much more than:
====
[2012-04-27 14:31:15,276][INFO ][node                     ] [Patriot I] {0.19.0.RC4-SNAPSHOT}[19402]: initializing ...
[2012-04-27 14:31:15,289][INFO ][plugins                  ] [Patriot I] loaded [jetty, river-couchdb], sites [head]
====

Can anyone offer some suggestion on how to get more debug info and proceed forward?

Pulkit Singhal

unread,
Apr 27, 2012, 10:46:44 AM4/27/12
to elasticse...@googlegroups.com

imotov

unread,
Apr 27, 2012, 8:34:04 PM4/27/12
to elasticse...@googlegroups.com
Do you still have an issue, or the issue was caused by MD5 instead of OBF?
To post to this group, send email to elasticsearch-jetty@googlegroups.com.
To unsubscribe from this group, send email to elasticsearch-jetty+unsub...@googlegroups.com.

Pulkit Singhal

unread,
Apr 28, 2012, 1:45:01 PM4/28/12
to elasticse...@googlegroups.com
@imotov - the issue that was caused by MD5 was resolved by using OBF instead ... but that only allows me to start up a serve that has JettyHttpServerTransportHandler being used.

When trying to use FilterHttpTransportModule, nothing works for me right now, the server simply doesn't start up and I don't know why.

- Pulkit

To view this discussion on the web visit https://groups.google.com/d/msg/elasticsearch-jetty/-/QzKas6rkjfMJ.

To post to this group, send email to elasticse...@googlegroups.com.
To unsubscribe from this group, send email to elasticsearch-j...@googlegroups.com.

Igor Motov

unread,
Apr 28, 2012, 1:46:38 PM4/28/12
to elasticse...@googlegroups.com
Could you send me your configuration files?

imotov

unread,
Apr 28, 2012, 2:18:31 PM4/28/12
to elasticse...@googlegroups.com
When you remove a filter definition form http_filter, you also should remove this filter from http_filter_chain:

# Filter setup
sonian.elasticsearch.http.filter:
    # Filter should use netty instead of jetty (default)
    #transport_type": "netty"

    http_filter_chain: ["logging", "jsonlog"]
    http_filter:
        # Simulated timeout filter - remove in production
        #timeout.type: com.sonian.elasticsearch.http.filter.timeout.TimeoutFilterHttpServerAdapter
        # Request logging filter
        logging:
To unsubscribe from this group, send email to elasticsearch-jetty+unsubscribe...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/elasticsearch-jetty?hl=en.

--
You received this message because you are subscribed to the Google Groups "elasticsearch-jetty" group.
To view this discussion on the web visit https://groups.google.com/d/msg/elasticsearch-jetty/-/QzKas6rkjfMJ.

To post to this group, send email to elasticsearch-jetty@googlegroups.com.
To unsubscribe from this group, send email to elasticsearch-jetty+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/elasticsearch-jetty?hl=en.

pulkit...@gmail.com

unread,
Apr 28, 2012, 2:28:35 PM4/28/12
to elasticse...@googlegroups.com, elasticse...@googlegroups.com
Aha! Would you mind getting an exception to be thrown if they don't match up?

Sent from my iPhone
To view this discussion on the web visit https://groups.google.com/d/msg/elasticsearch-jetty/-/7e859HvU-ykJ.
To post to this group, send email to elasticse...@googlegroups.com.
To unsubscribe from this group, send email to elasticsearch-j...@googlegroups.com.

Igor Motov

unread,
Apr 28, 2012, 2:53:39 PM4/28/12
to elasticse...@googlegroups.com
It's actually getting thrown. It's just elasticsearch logs it to stderr instead of normal log for some reason. 

Pulkit Singhal

unread,
Apr 28, 2012, 7:41:45 PM4/28/12
to elasticse...@googlegroups.com
Thanks Igor, it works now.
Reply all
Reply to author
Forward
0 new messages