My website(niusb.com) deploy question!

[刘久武]

Hi All!

  My website(http://www.niusb.com,use java7、liftweb 2.6-M2) deploy tomcat7,website is worked,but over time,the tomcat don't accept user's requests,tomcat process still exists.

  I need restart tomcat website is work.

  My tomcat7 main config:
 
  <Connector port="9001" protocol="org.apache.coyote.http11.Http11NioProtocol"  URIEncoding="UTF-8" compression="on" compressionMinSize="2048" enableLookups="false"
               connectionTimeout="30000"
               redirectPort="8443" acceptCount="500" maxThreads="400"/>

 
 and catalina.sh

  JAVA_OPTS="$JAVA_OPTS -Drun.mode=production -server -XX:+CMSClassUnloadingEnabled -Xms256m -Xmx256m -XX:PermSize=128m -XX:MaxPermSize=250m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/data/log/tomcat/gc.log -XX:HeapDumpPath=/data/log/tomcat/dumpfile.hprof -XX:+HeapDumpOnOutOfMemoryError"


The server top info:

  Cpu(s):  0.7%us,  0.0%sy,  0.0%ni, 98.7%id,  0.3%wa,  0.0%hi,  0.0%si,  0.3%st
  Mem:   1534064k total,  1120424k used,   413640k free,    33436k buffers
  Swap:  2097148k total,   357388k used,  1739760k free,   180012k cached


BTW,I'm try deploy to jetty9,start server report error info
 
 java.lang.VerifyError: Bad type on operand stack|Exception Details:|  Location:|    net/liftweb/util/SecurityHelpers$class.base64Encode(Lnet/liftweb/util/SecurityHelpers;[B)Ljava/lang/String; @13: invokevirtual|  Reason:|    Type 'org/apache/commons/codec/binary/Base64' (current frame, stack[3]) is not assignable to 'org/apache/commons/codec/binary/BaseNCodec'|  Current Frame:|    bci: @13|    flags: { }|    locals: { 'net/liftweb/util/SecurityHelpers', '[B' }|    stack: { uninitialized 0, uninitialized 0, 'net/liftweb/util/SecurityHelpers', 'org/apache/commons/codec/binary/Base64', '[B' }|  Bytecode:|    0000000: bb00 6459 2abb 0066 59b7 0069 2bb6 006f|    0000010: b800 71b7 0074 b0                      |
    at net.liftweb.util.Helpers$.<init>(Helpers.scala:34)
    at net.liftweb.util.Helpers$.<clinit>(Helpers.scala)
    at net.liftweb.http.LiftRules.<init>(LiftRules.scala:165)
    at net.liftweb.http.LiftRules$.prodInstance$lzycompute(LiftRules.scala:77)
    at net.liftweb.http.LiftRules$.prodInstance(LiftRules.scala:77)
    at net.liftweb.http.LiftRulesMocker$$anonfun$9$$anonfun$apply$2.apply(LiftRules.scala:64)
    at net.liftweb.http.LiftRulesMocker$$anonfun$9$$anonfun$apply$2.apply(LiftRules.scala:64)
    at net.liftweb.common.EmptyBox.openOr(Box.scala:644)
    at net.liftweb.http.LiftRulesMocker$$anonfun$9.apply(LiftRules.scala:64)
    at net.liftweb.http.LiftRulesMocker$$anonfun$9.apply(LiftRules.scala:64)
    at net.liftweb.http.LiftRules$.realInstance(LiftRules.scala:85)
    at net.liftweb.http.LiftRulesMocker$.toLiftRules(LiftRules.scala:49)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$class.init(ServletFilterProvider.scala:38)
    at net.liftweb.http.LiftFilter.init(LiftServlet.scala:928)
    at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
    at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:810)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:300)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1347)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:745)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:492)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:117)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:281)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:213)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:763)
    at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:144)
    at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
    at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
    at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
    at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
    at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
    at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
    at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
    at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
    at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
    at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
    at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
    at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
    at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:560)
    at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:235)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:117)
    at org.eclipse.jetty.server.Server.start(Server.java:355)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:99)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
    at org.eclipse.jetty.server.Server.doStart(Server.java:324)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
    at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.eclipse.jetty.start.Main.invokeMain(Main.java:297)
    at org.eclipse.jetty.start.Main.start(Main.java:724)
    at org.eclipse.jetty.start.Main.main(Main.java:103)


so,I want to know how to solve two problems at the.

Thanks All!

[Peter Petersson]

Hi 刘久武

Problem 1
To find out what is going on I would try to check if I am getting
connection starvation? by checking the number of db connections in you
database to your lift application, are you using a connection pool ?
what persistence layer module module are you using in your lift
application ?
If you are getting connection starvation it may be due to a burst in
connection requests (by a crawler) or that connections dose not get closed.

Problem 2 (java.lang.VerifyError ...org/apache/commons/codec/...)
I think you may have encountered a similar problem I had some time back
using java7 with the Apache Geronimo JEE2 server that uses tomcat7.

For me the solution was to filter out the containers codec library as it
clashes with the one used by lift (not the same versions).

In Apache Geronimo [1] you can do this in the geronimo-web.xml by
filtering see for example [2]

< import-package > !org.apache.commons.codec* < / import-package > (here "!" negates the container import package statement)

I do not know how to do the same in tomcat7 but there is probably a way
to do the same, another solution (but probably not as good) would be to
reverse the class loading process. Another thing you could try to do to
resolve this would be to run with java6.

[1] https://geronimo.apache.org/
[2]
https://github.com/karma4u101/FoBo-Demo/blob/master/fobo-angular-sandbox/geronimo-web.xml

I hope that help you a bit on the way to solve the problems, hopefully
someone else here knows more, let us know what you find.

Best regards Peter Peterssson

> --
> --
> Lift, the simply functional web framework: http://liftweb.net
> Code: http://github.com/lift
> Discussion: http://groups.google.com/group/liftweb
> Stuck? Help us help you:
> https://www.assembla.com/wiki/show/liftweb/Posting_example_code
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Lift" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to liftweb+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[刘久武]

Hi Peter Petersson:

Problem 1 :
I use Mapper db connection pool.

package code.model

import net.liftweb.mapper.{ ConnectionIdentifier, ConnectionManager, Schemifier }
import java.sql.{ Connection, DriverManager }
import net.liftweb.common.{ Box, Empty, Full }
import net.liftweb.util.Props

object MyDBVendor extends ConnectionManager {
  private var pool: List[Connection] = Nil
  private var poolSize = 0
  private val maxPoolSize = 4

  private lazy val chooseDriver = Props.mode match {
    case Props.RunModes.Production => "com.mysql.jdbc.Driver"
    case _ => "com.mysql.jdbc.Driver"
  }

  private lazy val chooseURL = Props.mode match {
    case Props.RunModes.Production => "jdbc:mysql://localhost:3306/niusb"
    case _ => "jdbc:mysql://localhost:3306/niusb"
  }

  private def createOne: Box[Connection] = try {
    val driverName: String = Props.get("mysql.db.driver") openOr chooseDriver
    val dbUrl: String = Props.get("mysql.db.url") openOr chooseURL
    Class.forName(driverName)
    Full((Props.get("mysql.db.user"), Props.get("mysql.db.password")) match {
      case (Full(user), Full(pwd)) => DriverManager.getConnection(dbUrl, user, pwd)
      case _ => DriverManager.getConnection(dbUrl)
    })
  } catch {
    case e: Exception => e.printStackTrace; Empty
  }

  def newConnection(name: ConnectionIdentifier): Box[Connection] =
    synchronized {
      pool match {
        case Nil if poolSize < maxPoolSize => {
          val ret = createOne
          poolSize = poolSize + 1
          ret.foreach(c => pool = c :: pool)
          ret
        }

        case Nil =>
          wait(1000L)
          newConnection(name)

        case x :: xs => try {
          x.setAutoCommit(false)
          Full(x)
        } catch {
          case e: Exception => try {
            pool = xs
            poolSize = poolSize - 1
            x.close
            newConnection(name)
          } catch {
            case e: Exception => newConnection(name)
          }
        }
      }
    }

  def releaseConnection(conn: Connection): Unit = synchronized {
    pool = conn :: pool
    notify
  }

}

Boot.scala

if (!DB.jndiJdbcConnAvailable_?) {
      DB.defineConnectionManager(DefaultConnectionIdentifier, MyDBVendor)
}

Best regards

在 2014年2月27日星期四UTC+8上午3时31分56秒，Peter Petersson写道：

[刘久武]

I add code in boot.scala

S.addAround(DB.buildLoanWrapper)

then db connection is release, but 50 concurrent users test, the server's cpu has been at 100%.

Is this the reason for the difference with something server configuration?

Thanks All!

在 2014年2月26日星期三UTC+8下午8时02分18秒，刘久武写道：

[Diego Medina]

On Wed, Feb 26, 2014 at 10:10 PM, 刘久武 <liuj...@gmail.com> wrote:

I add code in boot.scala

S.addAround(DB.buildLoanWrapper)

then db connection is release, but 50 concurrent users test, the server's cpu has been at 100%.

so now you need to see where that CPU time is spent on. And this may or may not be an issue with your app, but giving only 256m of memory may be too little, again, maybe your app doesn't need much, but I figured to point it out.

 

--

--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code
 
---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Diego Medina
Lift/Scala consultant
di...@fmpwizard.com
http://fmpwizard.telegr.am

[刘久武]

Thanks!


在 2014年2月27日星期四UTC+8上午11时53分38秒，fmpwizard写道：

[Robert Freytag]

If you use S.addAround() with a DB loan wrapper, that means a database-connection from your database pool is taken for the whole duration of a request.

That means, if you do a lot of heavy lifting, your application may run into pool starvation. There are multiple ways of dealing with pool starvation: 

 - increase your pool-size 

 - optimize request to become shorter

 - do not use S.addAround (this is very effective but also dangerous, it can have unwanted side effects depending on how your code interacts with the database, and how much you need transactional behaviour)

The amount of RAM of your machine CAN be an issue, but that is heavily dependent on your use-case. I would think more often the database creates the bottleneck.

For your current setup you have a connection pool of 4. With S.addAround(...) we can do a simple calculation.

Say your requests which also use the database take 40 milliseconds then you can serve 1000/40 * 4 = 100 requests/second

If you do more intense stuff, say a request takes 80 millis then 1000/80 * 40 = 50 requests/second.

So S.addAround can be a dangerous bottleneck in contention scenarios

Regards

Robert

[刘久武]

Robert Freytag， I test it, thank you for your reply.

在 2014年3月13日星期四UTC+8上午2时12分46秒，Robert Freytag写道：