Running out of Heap Space

[Chris Maness]

Hello all, I have a very simple topology that I'm running that takes in a tweet from the 1% sample and stores it in a mysql database. My problem comes when I allow the application to run overnight it runs out of heap space. No matter how much heap I allocate using the -Xmx argument it fills up after running for about 24 hours or longer (I've tried up to 16GB allocated to it). I use the code for the twitter sample spout that's included in the storm sample code and for the time being I'm running the topology from Eclipse until I get the bugs worked out of it. For the MySQL bolt I have the following code:

public void execute(Tuple input, BasicOutputCollector collector) {

try {

con = connector.getConnection(sqlDBUrl, sqlUser, sqlPassword);

} catch (ClassNotFoundException e) {

e.printStackTrace();

} catch (SQLException e) {

e.printStackTrace();

} finally {

//System.out.print("Connection created.");

}

PreparedStatement pst = null;

//parse out the Status object from the first tuple

Status s = (Status) input.getValue(0);

try {

pst = con.prepareStatement("INSERT INTO " + db + " (tweet)" +

"VALUES (?);");

pst.setString(1, s.toString());

//execute the SQL

pst.executeUpdate();

} catch (SQLException e) {

// TODO Auto-generated catch block

e.printStackTrace();

} finally {

    if(pst != null) {try {pst.close();} catch (SQLException e) {e.printStackTrace();}}

    if(con != null) {try {con.close();} catch (SQLException e) {e.printStackTrace();}}

}

} 

As you can see there doesn't seem to be anywhere for a memory leak. Can anyone point me in the right direction for fixing this problem? I'm using Storm 0.7.1 for the time being. 

[Michael Rose]

Have you added -XX:+HeapDumpOnOutOfMemoryError to your JVM options?

Once it OOMs with that you'll have a heap dump to work with and run jhat over to explore your object graph.

Alternately, you can attach a profiler like YourKit to your running topology and watch your object allocations.

-- 

Michael Rose (@Xorlev)

Senior Platform Engineer, FullContact
mic...@fullcontact.com

[Chris Maness]

I didn't even realize that was an option for the JVM. Thanks. I'll do that and be back if I have any problems or need any help.

[vinh]

Add to storm.yaml something like:

worker.childopts: "-Xmx768m -XX:+HeapDumpOnOutOfMemoryError"

For more info on what you can customize in storm.yaml:

https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml

On Jan 10, 2013, at 11:48 AM, Chris Maness <chrispray...@gmail.com> wrote:

I didn't even realize that was an option for the JVM. Thanks. I'll do that and be back if I have any problems or need any help.

On Thursday, January 10, 2013 1:58:04 PM UTC-5, Michael Rose wrote:

Have you added -XX:+HeapDumpOnOutOfMemoryError to your JVM options?

Once it OOMs with that you'll have a heap dump to work with and run jhat over to explore your object graph.

Alternately, you can attach a profiler like YourKit to your running topology and watch your object allocations.

-- 

Michael Rose (@Xorlev)

Senior Platform Engineer, FullContact
mic...@fullcontact.com

On Thursday, January 10, 2013 at 11:55 AM, Chris Maness wrote:

Hello all, I have a very simple topology that I'm running that takes in a tweet from the 1% sample and stores it in a mysql database. My problem comes when I allow the application to run overnight it runs out of heap space. No matter how much heap I allocate using the -Xmx argument it fills up after running for about 24 hours or longer (I've tried up to 16GB allocated to it). I use the code for the twitter sample spout that's included in the storm sample code and for the time being I'm running the topology from Eclipse until I get the bugs worked out of it. For the MySQL bolt I have the following code:

public void execute(Tuple input, BasicOutputCollector collector) {try {con = connector.getConnection(sqlDBUrl, sqlUser, sqlPassword);} catch (ClassNotFoundException e) {e.printStackTrace();} catch (SQLException e) {e.printStackTrace();} finally {

//System.out.print("Connection created.");

}PreparedStatement pst = null;

//parse out the Status object from the first tuple

Status s = (Status) input.getValue(0);try {pst = con.prepareStatement("INSERT INTO " + db + " (tweet)" +"VALUES (?);");

pst.setString(1, s.toString());

//execute the SQL

pst.executeUpdate();} catch (SQLException e) {// TODO Auto-generated catch blocke.printStackTrace();} finally {    if(pst != null) {try {pst.close();} catch (SQLException e) {e.printStackTrace();}}    if(con != null) {try {con.close();} catch (SQLException e) {e.printStackTrace();}}}} 

[Chris Maness]

So after poking around with the Eclipse Memory Analyzer I see that java.util.concurrent.LinkedBlockingQueue is the root of my problem. Why is this taking up so much room in memory? I'm guessing this is the queue that serves the tuples up to the next bolt.

[Chris Maness]

I stopped running it through Eclipse. There's something about running a topology through eclipse that messes up pieces of the configuration it seems. Make sure you run it through Storm or on a test cluster. 

On Wed, May 8, 2013 at 5:03 AM, William Sell <sell...@gmail.com> wrote:

Same problem here-- how did you resolve this?

On Monday, January 14, 2013 7:41:39 AM UTC-8, Chris Maness wrote:

So after poking around with the Eclipse Memory Analyzer I see that java.util.concurrent.LinkedBlockingQueue is the root of my problem. Why is this taking up so much room in memory? I'm guessing this is the queue that serves the tuples up to the next bolt.

--
You received this message because you are subscribed to a topic in the Google Groups "storm-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/storm-user/4CPHtPfSdVY/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

-Chris Maness

Software Developer and Graphic Designer

M#: 4235025115