list_spaces question

[李俊良]

hyperdex start:

hyperdex coordinator -f -l 127.0.0.1 -p 1982
hyperdex daemon -f --listen=127.0.0.1 --listen-port=2016 --coordinator=127.0.0.1 --coordinator-port=1982 --data=./data

python debug:

[cnangel@localhost ~]$python
Python 2.7.10 (default, Sep  8 2015, 17:20:17)
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyperdex.admin
>>> a = hyperdex.admin.Admin('127.0.0.1', 1982)
>>> a.add_space('''
... space phonebook
... key username
... attributes first, last, int phone
... subspace first, last, phone
... create 8 partitions
... tolerate 2 failures
... ''')
True
>>> a.list_spaces()
['0\xbc\xf3\xe6\x1dV', '0\xbc\xf3\xe6\x1dV', '0\xe5\xf1\xe6\x1dV']
>>>

coordinator log:

I0129 18:01:48.825009  4350 object.cc:856] hyperdex.__tick__ @ slot=1018: establishing checkpoint 17
I0129 18:01:48.859936  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1020: checkpoint 17 done
I0129 18:01:48.860064  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1020: garbage collect <= checkpoint 16
I0129 18:02:18.772552  4350 object.cc:856] hyperdex.__tick__ @ slot=1080: establishing checkpoint 18
I0129 18:02:18.822522  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1083: checkpoint 18 done
I0129 18:02:18.822763  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1083: garbage collect <= checkpoint 17
I0129 18:02:34.714263  4346 daemon.cc:935] introducing 127.0.0.1:42498 to the cluster
I0129 18:02:34.749367  4350 object.cc:856] hyperdex.space_add @ slot=1114: successfully added space "�'�4XU" with space(139)
I0129 18:02:34.749449  4350 object.cc:856] hyperdex.space_add @ slot=1114: issuing new configuration version 7
I0129 18:02:34.799439  4350 object.cc:856] hyperdex.config_stable @ slot=1117: stable through version 7
I0129 18:02:34.799792  4350 object.cc:856] hyperdex.config_ack @ slot=1119: acked through version 7
I0129 18:02:48.791918  4350 object.cc:856] hyperdex.__tick__ @ slot=1147: establishing checkpoint 19
I0129 18:02:48.827745  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1149: checkpoint 19 done
I0129 18:02:48.827952  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1149: garbage collect <= checkpoint 18
I0129 18:03:18.796950  4350 object.cc:856] hyperdex.__tick__ @ slot=1209: establishing checkpoint 20
I0129 18:03:18.829062  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1212: checkpoint 20 done
I0129 18:03:18.829207  4350 object.cc:856] hyperdex.checkpoint_stable @ slot=1212: garbage collect <= checkpoint 19

my notebook : fedora23 x86_64
why?

[Robert Escriva]

This problem has popped up on systems with a newer glibc. How have you
installed HyperDex? Is this problem regularly reproduceable?

-Robert

> --
> You received this message because you are subscribed to the Google Groups
> "hyperdex-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to hyperdex-discu...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[cnangel]

i install rpm package from fedora20:http://fedora.hyperdex.org/src/x86_64/20/

and rpmbuild --rebuild *.src.rpm

this problem is regularly reproduceable!

You received this message because you are subscribed to a topic in the Google Groups "hyperdex-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hyperdex-discuss/v8X1hzo9nOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hyperdex-discu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

I love perl love me

[Robert Escriva]

The packages from fedora 20 are not intended for use with fedora 24.

Sent from my phone. Please excuse my brevity.

[李俊良]

common/hyperspace.cc:

then operate >>

 s.sc.attrs = &attrs.front();
 s.sc.attrs_sz = attrs.size();

attrs is stack space, so the pointer s.sc.attrs is problematic

在 2016年2月2日星期二 UTC+8下午1:09:26，Robert Escriva写道：

[cnangel]

only patch for c++

python strill is question

[Robert Escriva]

The Python code calls directly into the c++ code. The code in
reestablish_backing() adjusts and updates the pointers accordingly:
https://github.com/rescrv/HyperDex/blob/master/common/hyperspace.cc#L170-L186

After the call to reestablish_backing, the s.sc.attrs will equal
m_attrs, not the attrs vector on the stack.

If your patch fixes the issue, it points to a problem in
reestablish_backing, which makes sure all pointers point to internal
storage by copying/rewriting/etc. The problem should be fixed in that
function; otherwise, it could occur elsewhere and the patch is only a
fix for this particular crash and it might introduce a memory leak.

From your debugging, did you see the pointer to s.sc.attrs point to
the stack even at the return point from the function?

-Robert

> From 31d22cc7f6f7d09a18dfe41395308fbe6248fa43 Mon Sep 17 00:00:00 2001
> From: cnangel <junli...@alibaba-inc.com>
> Date: Wed, 3 Feb 2016 17:23:42 +0800
> Subject: [PATCH] patch space name
> MIME-Version: 1.0
> Content-Type: text/plain; charset=utf-8
> Content-Transfer-Encoding: 8bit
>
> ---
> common/hyperspace.cc | 17 +++++++++--------
> 1 file changed, 9 insertions(+), 8 deletions(-)
>
> diff --git a/common/hyperspace.cc b/common/hyperspace.cc
> index 0c2e1a8..2146393 100644
> --- a/common/hyperspace.cc
> +++ b/common/hyperspace.cc
> @@ -226,23 +226,25 @@ e::unpacker
> hyperdex :: operator >> (e::unpacker up, space& s)
> {
> e::slice name;
> - std::vector<std::string> strs;
> - std::vector<attribute> attrs;
> + std::auto_ptr<std::vector<std::string> > strs(new std::vector<std::string>);
> + e::array_ptr<attribute> attrs;
> uint16_t num_subspaces;
> uint16_t num_indices;
> up = up >> s.id >> name >> s.fault_tolerance >> s.sc.attrs_sz
> >> num_subspaces >> num_indices;
> - strs.push_back(std::string(name.cdata(), name.size()));
> - s.name = strs.back().c_str();
> + strs->push_back(std::string(name.cdata(), name.size()));
> + s.name = strs->back().c_str();
>
> // Unpack all attributes
> + attrs = new attribute[s.sc.attrs_sz];
> for (size_t i = 0; !up.error() && i < s.sc.attrs_sz; ++i)
> {
> e::slice attr;
> hyperdatatype type;
> up = up >> attr >> type;
> - strs.push_back(std::string(attr.cdata(), attr.size()));
> - attrs.push_back(attribute(strs.back().c_str(), type));
> + strs->push_back(std::string(attr.cdata(), attr.size()));
> + attrs[i].name = strs->back().c_str();
> + attrs[i].type = type;
>
> if (type == HYPERDATATYPE_MACAROON_SECRET)
> {
> @@ -250,8 +252,7 @@ hyperdex :: operator >> (e::unpacker up, space& s)
> }
> }
>
> - s.sc.attrs = &attrs.front();
> - s.sc.attrs_sz = attrs.size();
> + s.sc.attrs = attrs.get();
>
> // Unpack subspaces
> s.subspaces.resize(num_subspaces);
> --
> 2.5.0
>

[cnangel]

if used std::vector<std::string> strs and std::vector<attribute> attrs;

need strs.reserve();  otherwise the memory of attrs on the stack could be changed, it maybe g++ or glibc version question , but i think it is not a good way. :)

 now fixed :

       strs.reserve(s.sc.attrs_sz + 1);
       attrs.reserve(s.sc.attrs_sz);

before
      strs.push_back(std::string(name.cdata(), name.size()));

[Robert Escriva]

Thank you. I've committed a fix.

-Robert