News: custom serializer for index keys (for the advanced user)

[Luca Garulli]

Hi all,

I've just committed a quite internal feature someone of you could use: the support for custom serializer for index keys.

In order to let to the developer to optimize at the best the usage of memory, disk space and cpu, it's now possible to have custom binary serializer.

Example: index documents per SHA-256 key. Before now you had to use a String to represent it. A SHA-256 string occupies at least 64bytes in memory. Using your own byte[] you can save half space because SHA-256 is 256bits long = 32 bytes.

Unfortunately you can't use a raw byte[] because Index needs object that implements the Java Comparable interface. For this purpose let's create own binary type:

public static class ComparableBinary implements Comparable<ComparableBinary> {


  private byte[]        value;



  public ComparableBinary(byte[] buffer) {


    value = buffer;
  }



  public int compareTo(ComparableBinary o) {


    final int size = value.length;


    for (int i = 0; i < size; ++i) {


      if (value[i] > o.value[i])


        return 1;


      else if (value[i] < o.value[i])


        return -1;


    }
    return 0;


  }

  public byte[] toByteArray() {


    return value;
  }


}

Now create the serializer for this type using 32bytes long:

public static class OHash256Serializer implements OBinarySerializer<ComparableBinary> {



  public static final OBinaryTypeSerializer INSTANCE = new OBinaryTypeSerializer();


  public static final byte ID = 100;


  public static final int LENGTH = 32;



  public int getObjectSize(final int length) {


    return length;
  }



  public int getObjectSize(final ComparableBinary object) {


    return object.toByteArray().length;


  }

  public void serialize(final ComparableBinary object, final byte[] stream, final int startPosition) {


    final byte[] buffer = object.toByteArray();


    System.arraycopy(buffer, 0, stream, startPosition, buffer.length);


  }

  public ComparableBinary deserialize(final byte[] stream, final int startPosition) {


    final byte[] buffer = Arrays.copyOfRange(stream, startPosition, startPosition + LENGTH);


    return new ComparableBinary(buffer);


  }

  public int getObjectSize(byte[] stream, int startPosition) {


    return LENGTH;
  }



  public byte getId() {


    return ID;
  }


}

Now to use it register to the OrientDB kernel and create an index with it:

OBinarySerializerFactory.INSTANCE.registerSerializer(new OHash256Serializer(), null);


index = database.getMetadata().getIndexManager().createIndex("custom-hash", "UNIQUE", new ORuntimeKeyIndexDefinition(OHash256Serializer.ID), null, null);

How to use it? Like a normal value, OrientDB will do all the rest:

ComparableBinary key1 = new ComparableBinary(new byte[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1 });


ODocument doc1 = new ODocument().field("k", "key1");


index.put(key1, doc1);

For more information: http://code.google.com/p/orient/wiki/Indexes#Custom_keys

Lvc@

[Russ White]

Sorry if this is a noob question.

I am trying to use a custom index key type this against a remote server. How do I register the serializer?

I have put the jar containing the serializer in the remote server's lib directory - but when I try this:

OBinarySerializerFactory.getInstance().registerSerializer(new CFISerializer(), null);
val index = db.getMetadata.getIndexManager.createIndex("cfi","UNIQUE",new ORuntimeKeyIndexDefinition(CFISerializer.ID),null,null,null)

Enter code here...

I get the following exception:

com.orientechnologies.orient.core.exception.OConfigurationException: Runtime index definition cannot find binary serializer with id=100. Assure to plug custom serializer into the server.
    at com.orientechnologies.orient.core.index.ORuntimeKeyIndexDefinition.<init>(ORuntimeKeyIndexDefinition.java:48)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateIndex.execute(OCommandExecutorSQLCreateIndex.java:255)

The problem is I can't seem to find any mention on "how" to plug in the custom serializer...

Any help appreciated

Also - as long as my serialized key implements Comparable can I use it in a ranged query?

What about using custom keys in Vertex key indexes? Will the key index use the custom serializer? Can I make SQL queries use the serialized objects compare method for ordering and range queries?

Thanks!
Russ

[Colin]

Hi Russ.

Would you be able to send me a complete stack trace of the exception and the source to your CFISerializer code?  (You can send it privately or post it here, whichever you prefer.)

Thanks for your patience.

-Colin

Colin Leister

Orient Technologies

The Company behind OrientDB

[Russ White]

Hi Colin,

I am glad to provide that.

I have attached the Serializer (don't laugh too hard if it is rubish)

here is the entire stack trace:

Feb 25, 2015 3:35:08 PM com.orientechnologies.common.log.OLogManager log
INFO: OrientDB auto-config DISKCACHE=10,695MB (heap=3,641MB os=16,384MB disk=120,600MB)

com.orientechnologies.orient.core.exception.OConfigurationException: Runtime index definition cannot find binary serializer with id=100. Assure to plug custom serializer into the server.
    at com.orientechnologies.orient.core.index.ORuntimeKeyIndexDefinition.<init>(ORuntimeKeyIndexDefinition.java:48)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateIndex.execute(OCommandExecutorSQLCreateIndex.java:255)

    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:64)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:1184)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:1173)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:63)
    at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1178)
    at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:385)
    at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:220)
    at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)

[Colin]

Thanks Russ.

Which version of OrientDB are you working with?

[Russ White]

Version 2.0.3

I am running the db remote and the index creation code is using a remote url from a client app (API).

I feel like I am missing something simple - do I need to create some sort of plugin? Or is dropping the jar in the lib dir sufficient?

Thanks for the help!

[Russ White]

It just feel to me like this code:

OBinarySerializerFactory.getInstance().registerSerializer(new CFISerializer(), null);

should be running on the remote server - not on the client - but I am talking way out of school.

[Colin]

Hey Russ.

You are correct with your feeling.

I'm creating a quick howto demo to illustrate this and will be updating the documentation as well.

-Colin

Colin Leister

Orient Technologies

The Company behind OrientDB

[Russ White]

That's excellent news Colin! This will be a very important feature for us to adopt OrientDB vs Neo4J. I trying to code a proof of concept right now - so the sooner the better. :)

Can you comment about using the custom serializer in an index created this way:

g.createKeyIndex("spec",classOf[Vertex],new Parameter("class", "CFI"), new Parameter("type", "UNIQUE"))

If such an index is created on a custom serialized Comparable type - can I use that index within a SELECT WHERE clause to limit results to a range?

Thanks!

[Colin]

Hi Russ.

I'll have the mock-up posted in just a bit.

However, in regards to your "SELECT WHERE" question, it depends on the type of index you use.

If you stick with the default SB-Tree, you should be fine.

http://www.orientechnologies.com/docs/last/orientdb.wiki/Indexes.html

-Colin

Colin Leister

Orient Technologies

The Company behind OrientDB

[Russ White]

Excellent.

The reason I ask is I have been using the default SB-Tree *without* the custom serializer and it treats my CFI like a string - sorting/comparing it like a string. What I am hoping is that f I use the custom serializer the Index will treat the key as the CFI type it really is - and use the CFI implementation of Comparable (Comparable<CFI>) rather than the String (Comparable<String>)

[Colin]

That's my understanding too.

So, let me know what you find out as soon as you implement the plug-in to register it (I'm almost done).

-Colin

Colin Leister

Orient Technologies

The Company behind OrientDB

On Thursday, February 26, 2015 at 11:52:02 AM UTC-6, Russ White wrote:

Excellent.

The reason I ask is I have been using the default SB-Tree *without* the custom serializer and it treats my CFI like a string - sorting/comparing it like a string. What I am hoping is that f I use the custom serializer the Index will treat the key as the CFI type it really is - and use the CFI implemation of Comparable (Comparable<CFI>) rather than the String (Comparable<String>)

[Russ White]

Here is an example of creating the Vertex I want to query against:

 execute(_.addVertex("class:CFI","spec",CFIParser.parse("epubcfi(/6/100)")))

That works for me right now - but the "spec" (which is a CFI instance) seems to be treated as a normal String. Queries come back in string order and not CFI order.

On Thursday, February 26, 2015 at 11:52:02 AM UTC-6, Russ White wrote:

Excellent.

The reason I ask is I have been using the default SB-Tree *without* the custom serializer and it treats my CFI like a string - sorting/comparing it like a string. What I am hoping is that f I use the custom serializer the Index will treat the key as the CFI type it really is - and use the CFI implementation of Comparable (Comparable<CFI>) rather than the String (Comparable<String>)

On Thursday, February 26, 2015 at 11:45:40 AM UTC-6, Colin wrote:

[Russ White]

Colin,

Will you be posting here when you are finsihed? I just want to be sure I don't miss it.

Thanks!
Russ

[Colin]

Hi Russ.

I will be posting an update here.  If you want to reply to me privately and send me your email address, I'll be happy to shoot you an email too.

I've run into a similar snag as you even though I'm registering the serializer on the server in a plugin.  So, I'm looking into that.

I'll let you know the moment I have a proven answer.

Thanks for your patience!

-Colin

Colin Leister

Orient Technologies

The Company behind OrientDB

[Colin]

This issue has now been added to the 2.1 milestone.

You can find more information here:
https://github.com/orientechnologies/orientdb/issues/3661

[Russ White]

Yikes - I hope that does not mean a long time before it is working :(

When will the 2.1milestone be available?

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/QwAxzvWOyy4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Colin]

The 2.1 milestone is an important release and has a high priority.

I doubt it will be a long time, I just can't define what that it is right now.  :-)

I'll update this thread once a snapshot is available for testing.

-Colin

Colin Leister

Orient Technologies

The Company behind OrientDB