0.9.3.0 minor release available
Doug Judd
alok mishra
cell_agt = Cell(timestamp = ts, row_key = client + request['adr'] + request['sid'], column_family = 'agt', value = request['agt'])
TypeError: __init__() got an unexpected keyword argument 'column_family'
alok
On Mar 24, 6:18 am, Doug Judd <d...@hypertable.com> wrote:
> Hypertable minor release 0.9.3.0 has been released and is now available for
> download athttp://www.hypertable.org/download.html(you may have to hit
alok mishra
I am using a mutator that flushes every 2 seconds.
The error seems like its in ThriftBroker
From the ThriftBroker log:
1269527167 ERROR ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:52) caught std::exception:
1269527167 ERROR ThriftBroker : run (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutatorFlushHandler.cc:35): Hypertable::Exception: interval flush - HYPERTABLE request timeout
at void Hypertable::TableMutatorShared::interval_flush() (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutatorShared.cc:66)
at void Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:406):
at bool Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71):
Also, there seems to be some sort of memoey leak in Thriftbroker. Should it take this much memory?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29392 xxxx 20 0 332m 140m 4584 S 0 6.8 0:16.53 ThriftBroker
Any help would be appreciated.
Alok
--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-us...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-user?hl=en.
Doug Judd
alok mishra
What OS and architecture (uname -a output) ==> Linux spidercrunch 2.6.24-24-xen #1 SMP Tue Aug 18 18:15:39 UTC 2009 x86_64 GNU/Linux
What does your hypertable.cfg file look like? default
How much data gets inserted before the system times out? Times out every 20-30 minutes. After the first timeout, if I start the processes immediately, it times out again but if I wait for few minutes and start then it starts working for the next 20-30 minutes.
Doug Judd
Doug Judd
Hi Doug,
I isolated the problems that I had mentioned. It was because of a mutator set to flush every 2 seconds. I made it non-timed flush and I flush using code after every 100 records or so. That fixed the timeout problem.
I also found that the ThriftBroker was getting bloated with time because of the python client which was not closing a scanner after using it. So the number of scanners multiplied with time.
If you still want to replicate this error
1269527167 ERROR ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:52) caught std::exception:
1269527167 ERROR ThriftBroker : run (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutatorFlushHandler.cc:35): Hypertable::Exception: interval flush - HYPERTABLE request timeout
at void Hypertable::TableMutatorShared::interval_flush() (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutatorShared.cc:66)
at void Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:406):
at bool Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71):
then I am going to setup a new Amazon instance for you with all the software.
Alok
Doug Judd wrote:
Is there anyway I could log into your machine to diagnose? Or, if it's easier, I can setup an Amazon machine instance and give you access to recreate the problem. Feel free to e-mail me directly at doug at hypertable dot com
- Doug
On Thu, Mar 25, 2010 at 8:50 AM, alok mishra <mish...@gmail.com <mailto:mish...@gmail.com>> wrote:
What DFS are your running on (local, hadoop, etc.)? ==> local
What OS and architecture (uname -a output) ==> Linux spidercrunch
2.6.24-24-xen #1 SMP Tue Aug 18 18:15:39 UTC 2009 x86_64 GNU/Linux
How many nodes in your setup and what processes are running on
each node. ==> 1, rabbitmq, apache
What does your hypertable.cfg file look like? default
How much data are you trying to load? Its a memory only table with
TTL=5 minutes for all data. Inserting very moderate amount of
data, 400KB per minute with 3000 calls per minute.
How much data gets inserted before the system times out? Times
out every 20-30 minutes. After the first timeout, if I start the
processes immediately, it times out again but if I wait for few
minutes and start then it starts working for the next 20-30 minutes.
On Thu, Mar 25, 2010 at 9:08 PM, Doug Judd <nugge...@gmail.com
<mailto:nugge...@gmail.com>> wrote:
Can you give us some more details of your system and your
test? Here are some things that would be helpful to know:
What DFS are your running on (local, hadoop, etc.)?
What OS and architecture (uname -a output)
How many nodes in your setup and what processes are running on
each node.
What does your hypertable.cfg file look like?
How much data are you trying to load?
How much data gets inserted before the system times out?
- Doug
On Thu, Mar 25, 2010 at 8:16 AM, alok mishra
<mailto:mish...@gmail.com>> wrote:
0.9.3.0 seems to have changed the Thrift interface. I
am getting this
error:
cell_agt = Cell(timestamp = ts, row_key = client +
request['adr'] +
request['sid'], column_family = 'agt', value =
request['agt'])
TypeError: __init__() got an unexpected keyword argument
'column_family'
On Mar 24, 6:18 am, Doug Judd <d...@hypertable.com
<mailto:d...@hypertable.com>> wrote:
> Hypertable minor release 0.9.3.0 has been released
and is now available for
> download
athttp://www.hypertable.org/download.html(you
have to hit
> refresh in your browser to get the correct download).
>
> NOTE: This release is incompatible with previous
releases. To preserve your
> data, you will need to first backup your tables,
upgrade Hypertable, and
> then restore your tables from backup.
>
> Version 0.9.3.0 <http://0.9.3.0>:
<mailto:hyperta...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/hypertable-user?hl=en.
-- You received this message because you are subscribed to
the Google Groups "Hypertable User" group.
To post to this group, send email to
hyperta...@googlegroups.com
<mailto:hyperta...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/hypertable-user?hl=en.
-- You received this message because you are subscribed to the
Google Groups "Hypertable User" group.
To post to this group, send email to
hyperta...@googlegroups.com
<mailto:hyperta...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/hypertable-user?hl=en.
-- You received this message because you are subscribed to the Google
Groups "Hypertable User" group.
To post to this group, send email to
hyperta...@googlegroups.com
<mailto:hyperta...@googlegroups.com>.
Sanjit Jhala
Are you still seeing this issue? If you can send me your script I'd like to look further into trying to reproduce the issue. I was unable to reproduce it with the following code:
import sys-Sanjit
import time
from hypertable.thriftclient import *
from hyperthrift.gen.ttypes import *
try:
client = ThriftClient("localhost", 38080)
print "HQL examples"
res = client.hql_query("drop table if exists test")
res = client.hql_query("create table test('col')")
print res
print "shared mutator examples";
mutate_spec = MutateSpec("issue_415_py", 2000, 0);
max_rows=8000000
#max_rows = 100
reporting_period=max_rows/50;
for ii in range(1,max_rows):
ii_str = str(ii);
client.put_cell("test", mutate_spec, Cell(Key("py-put-k" + ii_str, "col", None),
"py-put-v"+ii_str))
if ((ii-1) % reporting_period==0):
timestamp = str(time.localtime())
print timestamp + " On cell " + ii_str
except:
print sys.exc_info()
raise
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
alok mishra
Sorry for the late reply. As shown below, my code opens a mutator that
should auto-flush every 1 second. Then it registers the process_callback
with rabbitmq to receive events and then dump it to hypertable. I do not
explicitly flush. This setup works for 15 minutes or so and then fails
with the error that I had originally sent.
I fixed my problem by flushing manually after every 1000 events. But I
would love the elegance of auto-flush if it is fixed.
Please email me for any other questions,
Alok
class SessionDumper:
"""Dumps the session info into hypertable"""
def __init__(self):
self.hyperclient = ThriftClient("localhost", 38080)
self.mutator = self.hyperclient.open_mutator('session',0, 1000)
# auto dump after every 1 second
self.count=0
def process_callback(self, msg):
# insert it into the hypertable
cell_agt = Cell(Key(timestamp = ts, row = client +
request['adr'] + request['sid'], column_family = 'agt'), value =
request['agt'])
cell_uri = Cell(Key(timestamp = ts, row = client +
request['adr'] + request['sid'], column_family = 'uri'), value =
request['uri']+ request['qry'])
cell_ref = Cell(Key(timestamp = ts, row = client +
request['adr'] + request['sid'], column_family = 'ref'), value =
request['ref'])
celllist = [cell_agt, cell_uri, cell_ref]
self.hyperclient.set_cells(self.mutator, celllist)
> <mailto:mish...@gmail.com <mailto:mish...@gmail.com>>>
> <mailto:nugge...@gmail.com
> <mailto:nugge...@gmail.com>>> wrote:
>
> Can you give us some more details of your system
> and your
> test? Here are some things that would be helpful
> to know:
>
> What DFS are your running on (local, hadoop, etc.)?
> What OS and architecture (uname -a output)
> How many nodes in your setup and what processes are
> running on
> each node.
> What does your hypertable.cfg file look like?
> How much data are you trying to load?
> How much data gets inserted before the system times
> out?
>
> - Doug
>
> On Thu, Mar 25, 2010 at 8:16 AM, alok mishra
> <mish...@gmail.com <mailto:mish...@gmail.com>
> <mailto:mish...@gmail.com <mailto:mish...@gmail.com>>>
> <mailto:mish...@gmail.com
> <mailto:mish...@gmail.com>>> wrote:
>
> 0.9.3.0 seems to have changed the Thrift
> interface. I
> am getting this
> error:
>
> cell_agt = Cell(timestamp = ts, row_key =
> client +
> request['adr'] +
> request['sid'], column_family = 'agt', value =
> request['agt'])
>
> TypeError: __init__() got an unexpected
> keyword argument
> 'column_family'
>
> On Mar 24, 6:18 am, Doug Judd
> <d...@hypertable.com <mailto:d...@hypertable.com>
> <mailto:d...@hypertable.com
> <mailto:hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>>.
>
> To unsubscribe from this group, send email to
>
> hypertable-us...@googlegroups.com
> <mailto:hypertable-user%2Bunsu...@googlegroups.com>
>
> <mailto:hypertable-user%2Bunsu...@googlegroups.com
> <mailto:hypertable-user%252Buns...@googlegroups.com>>.
>
> For more options, visit this group at
>
> http://groups.google.com/group/hypertable-user?hl=en.
>
>
> -- You received this message
> because you are subscribed to
> the Google Groups "Hypertable User" group.
> To post to this group, send email to
> hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>
> <mailto:hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>>.
>
> To unsubscribe from this group, send email to
> hypertable-us...@googlegroups.com
> <mailto:hypertable-user%2Bunsu...@googlegroups.com>
>
> <mailto:hypertable-user%2Bunsu...@googlegroups.com
> <mailto:hypertable-user%252Buns...@googlegroups.com>>.
>
> For more options, visit this group at
>
> http://groups.google.com/group/hypertable-user?hl=en.
>
>
> -- You received this message because you
> are subscribed to the
> Google Groups "Hypertable User" group.
> To post to this group, send email to
> hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>
> <mailto:hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>>.
>
> To unsubscribe from this group, send email to
> hypertable-us...@googlegroups.com
> <mailto:hypertable-user%2Bunsu...@googlegroups.com>
>
> <mailto:hypertable-user%2Bunsu...@googlegroups.com
> <mailto:hypertable-user%252Buns...@googlegroups.com>>.
>
> For more options, visit this group at
> http://groups.google.com/group/hypertable-user?hl=en.
>
>
> -- You received this message because you are
> subscribed to the Google
> Groups "Hypertable User" group.
> To post to this group, send email to
> hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>
> <mailto:hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>>.
>
> To unsubscribe from this group, send email to
> hypertable-us...@googlegroups.com
> <mailto:hypertable-user%2Bunsu...@googlegroups.com>
> <mailto:hypertable-user%2Bunsu...@googlegroups.com
> <mailto:hypertable-user%252Buns...@googlegroups.com>>.
>
> For more options, visit this group at
> http://groups.google.com/group/hypertable-user?hl=en.
>
>
> --
> You received this message because you are subscribed to
> the Google Groups "Hypertable User" group.
> To post to this group, send email to
> hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>.
> To unsubscribe from this group, send email to
> hypertable-us...@googlegroups.com
> <mailto:hypertable-user%2Bunsu...@googlegroups.com>.
> For more options, visit this group at
> http://groups.google.com/group/hypertable-user?hl=en.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Hypertable Development" group.
> To post to this group, send email to
> hyperta...@googlegroups.com
> <mailto:hyperta...@googlegroups.com>.
> To unsubscribe from this group, send email to
> hypertable-de...@googlegroups.com
> <mailto:hypertable-dev%2Bunsu...@googlegroups.com>.
> For more options, visit this group at
Doug Judd
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-us...@googlegroups.com.