Listing of the keys uploaded into elliptics

[Evgeniy Polyakov]

Hi

I often get requests on how to get a list of keys written into elliptics. Do not really understand
why is this really needed especially considering storage setups where billions of keys were uploaded,
but yet, this is one of the most frequently asked question.

Elliptics has secondary indexes for that purpose: http://doc.reverbrain.com/elliptics:secondary-indexes
Indexes are automatically sharded and evenly distributed across the nodes in the group.

One can tag own uploaded keys with special indexes and then intersect those indexes on servers or
read the whole index key-by-key. That's essentially what RIFT - http elliptics frontend does
when you upload file through its HTTP interface: http://doc.reverbrain.com/rift:rift

And I've added listing support into RIFT proxy via /list URI - it reads an index from the server,
iterates over the keys and creates a nice output json. It also prints a timestamp of the key update in the index,
both in seconds and current timezone.

URI accepts a namespace - bucket name to get indexes from and name - a placeholder for future indexes names
(if we will support multiple indexes).

$ curl "http://example.com/list?namespace=testns&name="

{
"indexes": [
{
"id": "4e040aa8a798d04d56548d4917460f5759434fdf3ed948fd1cf35fd314cad3290e69b80deb0fc9b87a6bfbcbd08583919eb5b966658b3ed65e127236e1632525",
"key": "test1",
"timestamp": "1970-01-01 03:00:00.0",
"time_seconds": "0"
},
{
"id": "e5b7143155f46c9e9023cbf5e04be7276ae2e9a7583fee655c32aaff39755fa213468217291f0e08428a787bf282b416be1d26a5211f244fc66d1ce8ce545382",
"key": "test7",
"timestamp": "2014-02-18 03:29:44.835283",
"time_seconds": "1392679784"
}
]
}

Zero timestamp is for older indexes when timestamps were not yet supported. key is an object name given at upload time,
id is numeric elliptics ID (one can read those objects directly from elliptics without namespace name),
time_seconds is a coarse grained timeout in seconds since the Epoch. timestamp is a real parsed timestamp with microsecond resolution.

There is also an example python script which does basically the same - reads an index, unpacks it and print to console:
https://github.com/reverbrain/rift/blob/elliptics-2.25/example/listing.py

[Konstantin Shcherban]

Just want to say thank you Evgeniy.

Is there any changelog available for elliptics minor/major versions? If it is can you please point to where it can be found?

вторник, 18 февраля 2014 г., 13:52:36 UTC+4 пользователь Evgeniy Polyakov написал:

[Evgeniy Polyakov]

Hi

27.02.2014, 17:12, "Konstantin Shcherban" <k.sch...@gmail.com>:
> Just want to say thank you Evgeniy.Is there any changelog available for elliptics minor/major versions? If it is can you please point to where it can be found?

Thank you!

One can check debian/changelog or rpm spec file, we put the most interesting bits there

Elliptics (master, recommended): https://github.com/reverbrain/elliptics/blob/master/debian/changelog
Elliptics (lts, no features): https://github.com/reverbrain/elliptics/blob/lts/debian/changelog

Rift (2.25 elliptics build): https://github.com/reverbrain/rift/blob/elliptics-2.25/debian/changelog

Eblob: https://github.com/reverbrain/eblob/blob/master/debian/changelog

Or 'git whatchanged' for more details

[Рустам Нарманов]

The actual script? I have it throws an error:

Python argument types in
    Session.set_namespace(Session, bool)
did not match C++ signature:
    set_namespace(ioremap::elliptics::python::elliptics_session {lvalue}, std::string namespace)

вторник, 18 февраля 2014 г., 13:52:36 UTC+4 пользователь Evgeniy Polyakov написал:

Hi

[Рустам Нарманов]

Это из-за того, что -b параметр считается bool, прописав нужное в default получаю

2019-01-11 19:49:10.446538 0000000000000000/31690/31687 ERROR: 11:200000000000...c4993b78b9a3: INDEXES_FIND: handled reply from: 10.100.10.215:1021, trans: 6, cflags: 0x200 [reply], status: -2, size: 0, client: 0, last: 1, attrs: []

2019-01-11 19:49:10.447523 0000000000000000/31690/31687 ERROR: 11:300000000000...c4993b78b9a3: INDEXES_FIND: handled reply from: 10.100.10.215:1021, trans: 7, cflags: 0x200 [reply], status: -2, size: 0, client: 0, last: 1, attrs: []

2019-01-11 19:49:10.448313 0000000000000000/31690/31687 ERROR: 11:400000000000...c4993b78b9a3: INDEXES_FIND: handled reply from: 10.100.10.215:1021, trans: 8, cflags: 0x200 [reply], status: -2, size: 0, client: 0, last: 1, attrs: []

2019-01-11 19:49:10.449159 0000000000000000/31690/31687 ERROR: 11:e00000000000...c4993b78b9a3: INDEXES_FIND: handled reply from: 10.100.10.215:1021, trans: 9, cflags: 0x200 [reply], status: -2, size: 0, client: 0, last: 1, attrs: []

2019-01-11 19:49:10.452559 0000000000000000/31690/31687 ERROR: 11:000000000000...c4993b78b9a3: INDEXES_FIND: handled reply from: 10.100.10.215:1021, trans: 5, cflags: 0x200 [reply], status: -2, size: 0, client: 0, last: 1, attrs: []

А на деле там много данных.

пятница, 11 января 2019 г., 19:29:10 UTC+3 пользователь Рустам Нарманов написал: