Drop table issue

[lambgong]

Hi

  The version is 0.9.7.5.

  Here is my operation:

  At 10:17 am, i stop my writing application, and then drop some tables manually, two minutes later i restart my  writing application which will recreate tables droped before.

  After these operations as mentioned above, the cluster is not running properly. 

  First the commitlog purge seems don't work correctly, i find all purge log are as follow:

  10:18:07 INFO Hypertable.RangeServer : (/data/ht-pkg/hypertable-0.9.7.5-cdh/src/cc/Hypertable/Lib/CommitLog.cc:356) purge('/hypertab

le/servers/rs6/log/user') breaking on fragment 0 (rev=1368574108449055042 gen=2296); rev=1368576198913445891 gen=2705; ref_req=yes r

m_ok=no

......

  In my scene,   ref_req will be always yes, so if rm_ok=no, that meens the commitlog can't be pruge forever.

  

  Second, when i restart  my writing application, the writing speed are slower than before obviously.

  Thanks!

[Doug Judd]

Thanks for reporting this, I'll look into it today.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Doug Judd

CEO, Hypertable Inc.

[Doug Judd]

You can get around the problem for now by adding the following property to your hypertable.cfg file, pushing it out to all servers, and then restarting:

Hypertable.RangeServer.CommitLog.FragmentRemoval.RangeReferenceRequired=false

- Doug

[lambgong]

Thanks for your suggestion, but there are too many commitlogs now, about 1500(each 100M) every rangeserver， exceed the memory ( 64G) of the machine, is that ok for restart hypertable?

在 2013年5月15日星期三UTC+8下午9时55分04秒，lambgong写道：

[David]

I also encounter the  many commitlogs on the 0.9.7.5 version. I set the Hypertable.RangeServer.CommitLog.PruneThreshold.Max to 12G, originally, it looks ok, but after a week, the commit log of all RangeServer reach to  about 55G+.

[Doug Judd]

One thing you could do, which will take a little time, is to compact all of the ranges and then manually remove the files from the filesystem.  Here's how to do that:

for each RangeServer:

  1. Record the largest numerically named file in the user log directory (/hypertable/servers/rs?/log/user), call it $max_name

  2. Schedule a compaction for all user ranges:

      echo "compact ranges user;" | /opt/hypertable/current/bin/ht rsclient --batch

  3. Once the compactions complete (you can tell by the monitoring system or tailing the RangeServer logs), remove all user log fragment files that have name less than $max_name

Then you should be able to restart the system.  Add the following two config properties to your hypertable.cfg file before restarting:

Hypertable.Failover.Quorum.Percentage=100

Hypertable.RangeServer.CommitLog.FragmentRemoval.RangeReferenceRequired=false

If you run into the excessive memory usage at startup bug, let me know and I'll build you a pre-release of 0.9.7.6 that you can install to get around the problem.

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Doug Judd]

Ok, I'm fixing this now.  It will be in the 0.9.7.6 release.

- Doug

On Wed, May 15, 2013 at 7:55 PM, David <cwh...@gmail.com> wrote:

I also encounter the  many commitlogs on the 0.9.7.5 version. I set the Hypertable.RangeServer.CommitLog.PruneThreshold.Max to 12G, originally, it looks ok, but after a week, the commit log of all RangeServer reach to  about 55G+.

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David]

Hi,Doug,

I also find that when commit log piles up, even if the cluster is free state(under no any write workload circumstance),  it looks like the maintenace thread never purge the commit log.

[Doug Judd]

Thanks for the feedback.  I'm working on a fix for this now.

- Doug

On Wed, May 15, 2013 at 11:38 PM, David <cwh...@gmail.com> wrote:

Hi,Doug,

I also find that when commit log piles up, even if the cluster is free state(under no any write workload circumstance),  it looks like the maintenace thread never purge the commit log.

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[lambgong]

Thank  you for suggestion to restart the cluster, but it really take a long time.  

In our environment, 1 master, 19 rangrservers(24 CPU,64G RAM), 18T data total, cost 12 hours to complete  compaction, 

Is there any way to speed up, cause the machine resources isn't been full use of.

在 2013年5月15日星期三UTC+8下午9时55分04秒，lambgong写道：

[Doug Judd]

So did the compactions complete?  If so, then follow the instructions that I posted before.  Remove the user log files < $max_name, add the following config properties to your hypertable.cfg:

Hypertable.Failover.Quorum.Percentage=100

Hypertable.RangeServer.CommitLog.FragmentRemoval.RangeReferenceRequired=false

and restart Hypertable.  This will resolve the commit log purging problems.  If the compactions did not complete, do you have any idea when they'll complete?

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[lambgong]

The compactions have completed， and we restart the cluster successfully, but the compactions  cost 12 hours.

The environment : 1 master, 19 rangrservers(24 CPU,64G RAM), 18T data total. 

Is there any way to speed up the compactions , cause the machine resources isn't been full use of.

在 2013年5月15日星期三UTC+8下午9时55分04秒，lambgong写道：

Hi

[Doug Judd]

You could try increasing the number of maintenance threads.  You can do that by setting the Hypertable.RangeServer.MaintenanceThreads configuration property.

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Alex Kashirin]

regards compaction, currently not an issue, have restarted the cluster,

the log of RangeServer was:

1368784311 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'COMPACTION 6/62[TWFwIG9mIHRoZSBJbnZpc2libGUgV29ybGQgLSBUYXNoIEF3IC0gR2VuZXJhbCAmIExpdGVyYXJ5IEZpY3Rpb24=..TWVzc2lhaC1FeHRyYSBQYXJ0IC0gTXVzaWMgQm9vaw==]' failed, dropping task ...
1368784312 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'DEFERRED INITIALIZATION 6/62[U1RIIDIzMCAtIEFjY2Vzc29yeSBmb3IganVuY3Rpb24gYm94IC0gU1RIIDIzMA==..U29uaXgyIEJhc2ljIE1vYmlsZSBGbGlwLVRvcCBSZWN0YW5ndWxhciBUYWJsZSBXMTYwMHhEODAweEg3MjBtbSBNYXBsZQ==]' failed, dropping task ...
1368784312 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'COMPACTION 6/62[U29uaXgyIEJhc2ljIE1vYmlsZSBGbGlwLVRvcCBSZWN0YW5ndWxhciBUYWJsZSBXMTYwMHhEODAweEg3MjBtbSBNYXBsZQ==..U2V0IEFkaGVzaWZzIC1FTEVNRU5UIEJJTExBUkQtIE5vaXIgZXQgQmxhbmMgLSBDYXIgRGVjbw==]' failed, dropping task ...
1368784312 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'COMPACTION 6/64[..??]' failed, dropping task ...
1368784312 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'COMPACTION 6/45[4045-4046-4055-4060-..??]' failed, dropping task ...
1368784312 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'COMPACTION 6/62[MTRSOTI0OQ==..MjAxMC0xMSBNYW4gVXRkIE5pa2UgSG9tZSBTaGlydCAoQmVja2hhbSA3KQ==]' failed, dropping task ...
1368784312 WARN Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:169) Maintenance Task 'DEFERRED INITIALIZATION 6/62[Q2hhcmxlcyBGb3ggLSBIYXBweSBEYXlzIC0gU2hlZXQgTXVzaWMgKERpZ2l0YWwgRG93bmxvYWQp..QSBiYWxsZXJpbmEgdGllcyB1cCBoZXIgc2hvZWxhY2VzLCBwaG90byBIb3VzdG9uIFJvZ2Vycw==]' failed, dropping task ...

Just to let you know of.

Thanks,

Alex

[Doug Judd]

These are warnings.  Was there a problem with the database that required you to restart it?  Or did you just restart it due to these messages?

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.