Tomcat is a Fantastic Server

[Mark van Wyk]

We're running a JSP web site on our server for a MXit portal, and we're right now pushing through 220 000 requests per minute, and Tomcat is smiling. Unfortunately, MySQL is struggling. Does anyone know if it's hard to implement something like memcache over MySQL as a cheat way to increase MySQL performance until I find the time to implement application layer caching?

--

Mark van Wyk

Email: ma...@foxbomb.com

Mobile: 082 831 9227

Landline: 021 789 1427

"Before printing this email, please consider the animals and eat less of them"

[Albert de Jongh]

Are you using some kind of persistence layer?

--
You received this message because you are subscribed to the Google Groups "CTJUG Forum" group.
To post to this group, send email to CTJUG...@googlegroups.com
To unsubscribe from this group, send email to CTJUG-Forum...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CTJUG-Forum
For the ctjug home page see http://www.ctjug.org.za
For jobs see http://jobs.gamatam.com/

[Moandji Ezana]

Terracotta, maybe?

Moandji

--
www.moandjiezana.com

Sent from my phone

[Len Weincier]

Terracotta is great but possibly overkill for a single app. I would certainly take a look at http://ehcache.org/

Len 

[James Mc Millan]

Hello

There is no "drop-in" memcache over MySQL that I know of, but MySQL already has the query cache, which is exactly what I imagine memcached over MySQL would do. I'd look at optimising the MySQL configuration first (make sure it's given as much memory as you can for it's various caches, by default it doesn't use a lot of memory). MySQL master-slave read replication can also work for distributing database load, but you may want to look at "application layer" caching first to see if you can reduce your number of queries.

There are lots of options for "application layer" caching, and depending on your app there could be some easy big wins with caching data that doesn't change very often or doesn't need to be entirely up to date but is requested often. Ehcache or memcached are both great options, memcached is great for distributing your cache when you want to scale out.

James

[David Tinker]

Are you running Apache in front of Tomcat or Tomcat on its own? If Apache in front, how is it communicating with Tomcat? What version of Tomcat are you using? Just interested as its not often you get first hand info from someone running such a busy site.

Probably you quickest option for MySQL is to just boost the machine its running on as much as possible and make sure it is using all the memory. Assuming you don't have any broken queries that is.

On Wed, Oct 12, 2011 at 7:32 PM, Mark van Wyk <ma...@foxbomb.com> wrote:

--
You received this message because you are subscribed to the Google Groups "CTJUG Forum" group.
To post to this group, send email to CTJUG...@googlegroups.com
To unsubscribe from this group, send email to CTJUG-Forum...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CTJUG-Forum
For the ctjug home page see http://www.ctjug.org.za
For jobs see http://jobs.gamatam.com/

--
Engage - Web based goal setting and performance management
http://engage.calibreapps.com/

[renier...@gmail.com]

We've ditched the Apache in front of Tomcat idea.

Compile APR... Then Apache ( well the portable runtime ) is natively embedded in Tomcat... Soooo much less schlep :)

Sent from my BlackBerry® wireless device

---

From: David Tinker <david....@gmail.com>

Sender: ctjug...@googlegroups.com

Date: Thu, 13 Oct 2011 07:52:08 +0200

To: <ctjug...@googlegroups.com>

ReplyTo: ctjug...@googlegroups.com

Subject: Re: [CTJUG Forum] Tomcat is a Fantastic Server

[Clinton Bosch]

How many connections are you using in your pool

[Clinton Bosch]

What are you using to manage your connection pool

[Moandji Ezana]

Indeed. I was thinking EHcache and said Terracotta.

Moandji

--
www.moandjiezana.com

Sent from my phone

[Mark van Wyk]

Hi David,

We're running Apache Tomcat 7, straight, no Apache Front, No APR.

Funnily enough, I tried out APR on another server, and found it used more memory, took 15 times longer to startup / shutdown, and offered no noticeable performance benefit. I'm starting to think that the JVM is some places with HotSpot, etc - is starting to outperform the natives. I gave Tomcat 256 MB RAM, but it's only using 40% of it.

*** cringe ***

I'm not using connection pools. I'm just using straight JDBC - on demand connections. The problem is that I'm inserting "best times" and "best score" into a table, then then running a top 50 leaderboard query ordering by "score, time" from a table of close to 500 000 entries. This is not really conducive to caching - I don't think. Obviously. So I think my options are (in order of importance):

1. Move the application to Google App Engine and forget about it.

2. Implement a custom server-side leaderboard cache.

3. Keep a 90 day history in the table and archive the rest.

4. Make sure that all my sort columns are indexed.

5. Allocate more memory to MySQL.

6. Implement Connection Pools

7. Use a Caching Solution like EHCache

8. Get a Dedicated Server

9. Set up a cloud.

Comments appreciated...

[Brian Silberbauer]

Hey Mark

I would first start by figuring out what is slowing your database access down. Your comment "I'm just using straight JDBC - on demand connections." immediately made my back spasm :) Have a look at your connection on mysql, maybe you are maxing out on connections or your connections are taking a long time to open. Or it might not be an issue at all, not sure about your architecture - like I said, try and find out what the issue is, might be a simple change like pooling will help.

Brian

Brian Silberbauer Software Consulting

 +27 (0)83 566 2705
 http://www.gamatam.com/

jobs site: http://jobs.gamatam.com/

[Mike Morris]

On 13 October 2011 09:48, Mark van Wyk <ma...@foxbomb.com> wrote:

6. Implement Connection Pools

Your simplest (litte/no programming) solution. You just have to do the configuration for a DataSource in server.xml and make sure that your code uses DataSource.getConnection() rather than DriverManager. Everything else is handled by MySQL drivers.

--
This email is [X]bloggable [ ]ask-first [ ]private

Mike Morris
http://mikro2nd.net/
EarthStuff: http://mikro2nd.net/blog/planb/
TechStuff: http://mikro2nd.net/blog/mike/

----- A day without chillies is a day wasted ------

[Clinton Bosch]

Without a doubt you are going to suffer without a connection pool, have a look at bonecp

Clinton Bosch
Cell: +27 (0)82 805 9479

[Mark van Wyk]

Hi Brian,

Yeah, I made the mistake of delivering some prototype/demo code for the client/partner - and they said "Perfect - Put it live". Now the prototype/demo code is running live in a heavy load production environment. Oops!

Think connection pools will really make that much difference? This is actually going to be a great test. I'll implement pooling, and then let you know. What do you guys prefer.

Commons DBCP or C3PO?

Thanks,

Mark

[Mark van Wyk]

UPDATE

On 13 October 2011 10:18, Mark van Wyk <ma...@foxbomb.com> wrote:

Hi Brian,

Yeah, I made the mistake of delivering some prototype/demo code for the client/partner - and they said "Perfect - Put it live". Now the prototype/demo code is running live in a heavy load production environment. Oops!

Think connection pools will really make that much difference? This is actually going to be a great test. I'll implement pooling, and then let you know. What do you guys prefer.

Commons DBCP or C3PO or BoneCP?

[Clinton Bosch]

Try BoneCP, it is the fastest. I surprised MySQL every handled 220 000 requests a minute with no connection pool

[Mark van Wyk]

Thank you so much everyone. Wow, what a blessing to have the CTJUG as a community caring for poor little me ;)

Will implement Bone Database Connection Pooling now (and that only). We're expecting the same amount of traffic tonight, and let you know what percentage performance improvement I got.

Is Database Connection Pooling really a performance benefit or is it an Old Wives Tale. - Tonight on MythBusters at 9:00pm SAST - Java Edition

;)

[Clinton Bosch]

Hi Mark

Connection pooling will make a HUGE difference, the overhead to establish a JDBC connection is very big (as I said I am surprised your app worked with those amounts of traffic establishing a new connection each time). Have a look at the performance graphs on the BoneCp site for an idea on what kind of improvements you can expect (but it be impressive)

C

[James Mc Millan]

Hello

The new tomcat 7 connection pooling is really great much better than dbcp:
http://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html

Otherwise I've found c3p0 to work brilliantly when configured correctly.

Also make sure you're using prepared statements and prepared statement caching with your connection pooling.

Otherwise - there are so many other optimisations possible! The trick is that your leaderboards don't need to be *really* live, they just need to look live. And you also definitely don't need every single score ever recorded - only the top however-many-thousand.

At a lower level, jdbc batching can be used for inserting records if that's where the issue is (gather up a few scores in memory and batch to the database every now and again).

James

[Mark van Wyk]

Does anyone have any idea if the folk at MySQL have fixed that "?autoconnect = true" bug that causes "Could not establish a database connection. Last successful connection was 59458954 milliseconds ago". That was a real headache. If the connection was closed (expired), the first attempt to reconnect would fail. It is confirmed in the source code with a comment in the driver source that the code to use ?autoconnect = true was never implemented. C3PO corrected this by either first sending a SELECT 1 before anything else, or reattempting a query to a specified number of times. The latter being ugly, but working spectacularly for my last issue.

[Eugene Robert De Villiers Adams]

Mark,

I've used this with great success : http://ehcache.org/documentation/user-guide/write-through-caching

So,
1. you can write your scores to the cache,
2. the cache(you can configure it), will queue and batch(configurable) your requests to the database
3. the cache has the latest data, so this is where you can sort your entries...

Before you roll it out into production, just check how big your VM Heap size get(JConsole is good enough), after you've loaded 500k entries.  I also suggest you use Terracotta as an NAM bank(it is super easy to configure).  I only suggest this, since I assume that Tomcat is using alot of memory already and if/when you throw your 500k objects on the same heap, you might start getting GC pauses...o, and you are going to be sorting data, so you will keep the GC on it's toes...

For reference : http://vimeo.com/21193026

Cheers,
Eugene

[Kirk Daries]

Agree! Tomcat JDBC pool is awesome....   ^_^

I've also used Proxool for quite a while in production as well if you need an alternative.

My newer project is using the Tomcat JDBC pool though. Smooth sailing so far.

[Marius Botha]

We had that exact same issue through JBoss/C3P0 after we upgraded to MySQL 5.5 from 5.0. I believe the rules in MySQL 5.5 may be slightly different with regards to closing of connections. In the end all we had to do to fix it (after COUNTLESS configuration tests) was to just install the latest mysql-connector driver (think it was 5.1.16) and it never happened again ...

 

We also checked or put the following properties in the config file (as an example) to ensure it was doing what it should and the pool size the app expected was the same as what MySQL made available, etc. etc.

    <min-pool-size>10</min-pool-size>
    <max-pool-size>80</max-pool-size>
    <idle-timeout-minutes>200</idle-timeout-minutes>
    <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>
    <blocking-timeout-millis>40000</blocking-timeout-millis>

Maybe it helps!

 

Marius

[Mark van Wyk]

They Fixed It? Yaaaaaay!!!

[Dirk le Roux]

Hi Mark, how did it go? Can you share the performance numbers?

[Mark van Wyk]

I was yaying for "they fixed the mysql driver error". Just launched a new product last week. Will tackle it again this week and let you know the results...

;)