Hypertable 0.9.5.6 RangeServer Memory 99% fragment or leak

[gcc.lua]

Hi,all
I have use c++ stream pipe to implement map/reduce logic handle
big data(hypertable/hadoop)
I had search the old issues like my problem, it's like CellCache
memmory fragment,
and fixed use CellCachePool import in 0.9.1.0/0.9.1.1, It be removed
in latest version 0.9.5.6(or sth. old version),
where can I commit the Hypertable.RangeServer.log, thanks

server:(16GB Memory)

20674 line:
1340084923 INFO Hypertable.RangeServer : (/home/hadoop/temp/
hypertable-0615/src/cc/Hypertable/RangeServe r/
MaintenanceScheduler.cc:255) Memory Allocation: BlockCache=0.00%
BlockIndex=0.00% BloomFilter=0.00% CellCache=99.51% ShadowCache=0.00%
QueryCache=0.49%


221688 line:
1340093490 INFO Hypertable.RangeServer : (/home/hadoop/temp/
hypertable-0615/src/cc/Hypertable/RangeServe r/RangeServer.cc:
3710) Application queue PAUSED due to low memory condition

[gcc.lua]

change to tcmalloc ,but no use

[Christoph Rupp]

Hi,

"Application queue PAUSED due to low memory condition" is actually quite common and usually not a problem, unless you have MANY of them. These are usually caused either by low memory (because load is too high) or because hypertable fails to run compactions. We had a bug in 0.9.5.4 which did not reset some internal flags correctly, thus stopping compactions, and the cellcaches were not flushed to disk. but this was fixed in 0.9.5.6 (or 0.9.5.5, if i remember correctly).

There are a couple of options you can use to control memory behavior:


  --Hypertable.RangeServer.MemoryLimit arg
                                        RangeServer memory limit
  --Hypertable.RangeServer.MemoryLimit.Percentage arg (=60)
                                        RangeServer memory limit specified as
                                        percentage of physical RAM
  --Hypertable.RangeServer.LowMemoryLimit.Percentage arg (=10)
                                        Amount of memory to free in low memory
                                        condition as percentage of RangeServer
                                        memory limit
  --Hypertable.RangeServer.MemoryLimit.EnsureUnused arg
                                        Amount of unused physical memory
  --Hypertable.RangeServer.MemoryLimit.EnsureUnused.Percentage arg
                                        Amount of unused physical memory
                                        specified as percentage of physical RAM

bye
Christoph

2012/6/19 gcc.lua <gcc...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

[BigQiao]

We have set Hypertable.RangeServer.MemoryLimit=4GB, but rangeserver still run out of memory(almost cost 10GB), and 99% cost by CellCache,

I wait it for almost 1 hour, but nothing changed.  I belive the compactions stopped. 

2012/6/19 gcc.lua <gcc...@gmail.com>

To post to this group, send email to hypertable-dev@googlegroups.com.
To unsubscribe from this group, send email to hypertable-dev+unsubscribe@googlegroups.com.

[BigQiao]

I think I figure it out.

The memory is cost by Ranges which load by replay_update() in RangeServer.cc

those Ranges will not be compact because their needs_compaction=false;

So I add one line to RangeServer.cc:3616, RangeServer::replay_load_range

3612     schema = table_info->get_schema();

3613 

3614     range = new Range(m_master_client, schema, range_entity, live_table_info.get());

3615 //##

3616     range->set_needs_compaction(true); //## Qiao Fix Memory Bug 2012-06-20

3617 //##

3618     range->recovery_initialize();

3619 

3620     table_info->add_range(range);

After replay finish, it will start to compact cellcaches.

[Christoph Rupp]

Thanks for sharing this....

Can you please switch off the cluster, then only start dfsbrokers and hyperspace, and run this:

/opt/hypertable/current/bin/ht metalog_dump /hypertable/servers/rs1/log/rsml

and grep the output for "load_acknowledged=false". If that flag is set then the ranges are not compacted, because the master is waiting for the rangeserver to confirm that it has loaded the range.

Thanks
Christoph

2012/6/20 BigQiao <zhen...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/oNBjwrL8XIYJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

[Doug Judd]

Hi,

This is a known problem and we're working on a fix for it now.  Your fix should work for now, but is not an ideal solution and could potentially cause worse problems.  You should be ok for now, but as soon as we come out with the fixed version, I advise that you upgrade to it immediately.

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/oNBjwrL8XIYJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

--

Doug Judd

CEO, Hypertable Inc.

[BigQiao]

Thanks.  If you have any news about this problem, please let me know.