Upgrading old Elephantdb shards for 0.4.4?

40 views
Skip to first unread message

Jeroen van Dijk

unread,
May 17, 2013, 9:58:06 AM5/17/13
to elephan...@googlegroups.com
Hi, I've tried to upgrade my Elephantdb shards from 0.2.0 to 0.4.4. I figured the fastest way to do this is to use the old shards from 0.2.0 and create something that is compatible with the latest versions. Reading from Cascalog with 0.4.4 seems to work for old shards and my "upgraded" shards, but loading them into the server isn't working. I get nil values for known keys. I am starting to doubt what I want is possible, but someone might be able to see what is missing in my approach:


Another option would be to write data from the old code to intermediate seqfiles, that will be read by new code, that writes them to upgraded shards. However I think it would be more convenient if the old elephantdb-cascalog was able to write to the new format.

Any ideas?

Thanks,
Jeroen

Soren Macbeth

unread,
May 17, 2013, 11:23:12 AM5/17/13
to elephan...@googlegroups.com
Jeroen,

The code in the gist looks ok to me. 

How are you querying for your keys? Remember that keys and values are stored as byte arrays now, so you need to serialize the key before sending the get/multiGet request to EDB. 

All my domains were already using byte array keys and values and I was able to read and load shards into 0.4.4 without issue. 


--
You received this message because you are subscribed to the Google Groups "elephantdb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elephantdb-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
http://about.me/soren

Jeroen van Dijk

unread,
May 17, 2013, 11:37:06 AM5/17/13
to elephan...@googlegroups.com
I'm using this code to query https://github.com/jeroenvandijk/elephantdb-playground/blob/master/client/src/clj/client/core.clj It works perfectly for shards generated with 0.4.4. So I am assuming that part is ok...

I remember seeing somewhere you were using leveldb as backend, could that make a difference maybe?

I also tried a more straightforward way of converting keys (https://gist.github.com/jeroenvandijk/5599138#file-upgrade_shards-clj-L70) and this gives me keys (when inspecting them with Cascalog) that look like this "[Bnormal-key-here" when the byte-array key came from "normal-key-here". I have the feeling that with the Elephantdb 0.2.0-wip4 something is still being added to the keys even when using raw byte arrays.

Jeroen van Dijk

unread,
May 17, 2013, 12:01:00 PM5/17/13
to elephan...@googlegroups.com

Soren Macbeth

unread,
May 17, 2013, 12:06:32 PM5/17/13
to elephan...@googlegroups.com
The other option that might work: since I re-namespaced cascalog.elephantdb to elephantdb.cascalog for 0.4.4, you can probably read your domain in with cascalog.elephantdb.keyval/keyval-tap, then write it back out with elephantdb.cascalog.keyval/keyval-tap

Jeroen van Dijk

unread,
May 17, 2013, 12:11:36 PM5/17/13
to elephan...@googlegroups.com
Ah that could work, thanks! 

I'll try it asap, probably tomorrow again.

Jeroen van Dijk

unread,
May 22, 2013, 10:13:48 AM5/22/13
to elephan...@googlegroups.com
Hi Soren,

I have been able to successfully upgrade shards in my playground project. Unfortunately it wasn't as easy as your last suggestion. Because of version conflicts of classes, I went down the path of importing all the relevant Java classes through copy-pasting and giving them a "New" prefix. Painful, but it seems to work now.. If you are interested here is the code https://github.com/jeroenvandijk/elephantdb-playground/tree/master/data_gen_0_4_4

There were two main issues between the two versions:
- different serialization of keys
- new sharding because of the new serialization of keys

I'll try to use this code now to migrate my real data.. fingers crossed.

Reply all
Reply to author
Forward
0 new messages