Unable to store byte[] when using ManyAssociation

[Jaydatt Desai]

Hello Gentlemen,

I am not able to store byte[] when using ManyAssociation.

I am trying to store a byte[] in an entity which is of ManyAssociation of another entity. For example 

EmailAttachementEntity {

   Property<byte[]> data();

}

EmailEntity {

     ManyAssociation<EmailAttachementEntity> attachments();

}

So, when associating multiple EmailAttachementEntity to EmailEntity, I am getting an exception such as "org.qi4j.api.value.ValueSerializationException: Could not deserialize value"

Exception: http://pastebin.com/xNn2CXDS

So, it seems to be problem when I am trying to store the byte[]

I read on a topic of this group that byte[] is not supported and if it need to support than there needs to fix the following bugs:

- fix the equality bug
- (de)serialization support

I read it in one of the reply of :https://groups.google.com/forum/#!msg/qi4j-dev/kp9dgsc3O1A/-ePeS3TbttoJ

So, Am I getting the exception due to same or I am missing something to achieve this?

How ever on the above pointed thread I found that we can achieve this by using List<byte> instead of byte[], but this seems to be a kind of patch.

Thanks in advance,

Jaydatt

[Niclas Hedhman]

No, this was never resolved. And I guess won't be fixed in the near future. I don't think that your use of ManyAssociation is at all related, it is inherent to the ValueSerialization system itself.

I think Paul knows a lot more about this than I do.

Cheers

Niclas

--
You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/d/optout.

--
Niclas Hedhman, Software Developer
河南南路555弄15号1901室。
http://www.qi4j.org - New Energy for Java

I live here; http://tinyurl.com/3xugrbk
I work here; http://tinyurl.com/6a2pl4j
I relax here; http://tinyurl.com/2cgsug

[Jaydatt Desai]

hi Niclas,

Thanks for the reply, however the reason I thought that it might be due to the existing bug is, in my ManyAssociation when I don't use a byte[] it works perfectly and all the associated entities stored perfectly, but once I start adding byte array in this associated entity it throws me the said exception. however if I only associate only one entity with byte[] it works fine. but at the time I associate the second entity with byte array it throws the decoding exception from Base64Encoder. But as you wrote may be Paul can help me figuring out the problem cause. And thanks a lot that you figured out some time for the issue I am facing.

Thanking you,
Jaydatt

[Jaydatt Desai]

Hi Niclas,

I dig into some more detail of this issue and I figured out that this is not due to ManyAssociation but this is because when we create an entity with binary data and after that when we try to read this entity than this Deserialize exception occurred.

Sample code:

@Test
public void failStoringBinaryData() throws Exception{
UnitOfWork uow = module.newUnitOfWork();
TestEntity2 testEntity2 = uow.newEntity(TestEntity2.class);
String entityId = testEntity2.identity().get();
testEntity2.binaryProperty().set("test".getBytes()); // we are setting binary data here
uow.complete();

uow = module.newUnitOfWork();
TestEntity2 testEntity2Other = uow.get(TestEntity2.class, entityId);
testEntity2Other.binaryProperty().get(); // This is where the exception is generated.
uow.complete();

}

public interface TestEntity2
extends EntityComposite
{
@Optional
Property<String> property();

@Optional
Property<byte[]> binaryProperty();

}

[Niclas Hedhman]

I guess it also happens if you wrap it in a ValueComposite?

 public interface TestEntity2
            extends EntityComposite
    {
        @Optional
        Property<String> property();

        @Optional

        Property<Attachment> attachment();

public interface Attachment extends ValueComposite 

{

    Property<String> name();

    Property<String> mimetype();

    Property<byte[]> data();

}

    }

--
You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/d/optout.

[Jaydatt Desai]

Hi Niclas,

I have tried with List<Byte> instead of byte[] as per you mentioned in one of your previous email chain, and it seems to be working with List<Byte>, so should I use this List<Byte> instead of byte[]?

Thank you,
Jaydatt

[Niclas Hedhman]

I don't think List<Byte> is very effective. You might end up with 32byte overhead per byte...

--
You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/d/optout.

[Jaydatt Desai]

Hi Niclas,

I am sorry but I didn't get that what do you mean by "32 byte overhead", can you please give me some understanding.
Also what do you think should be a proper solution of this case, as I want to store the binary data?

Niclas, thanks a lot again for your time for answering me.

Thank you,
Jaydatt

[Paul Merlin]

Jaydatt,

A List<Byte> property will be stored as a JSON array, one value per entry in the List. Hence the "32 bytes overhead per byte". So, that's not a good idea for binary data storage that exceed a few bytes.

Entities storage and Values serialization/deserialization is done using JSON. JSON do not support binary data. One way to work around this is to store the data in a String, say Base64 encoded. It works but is not size-effective at all.

Il you need to store relatively large binary data, don't store it in entities but in a dedicated system (filesystem, blob-service etc...).

HTH

/Paul


Jaydatt Desai a écrit :

[Paul Merlin]

Paul Merlin a écrit :

A List<Byte> property will be stored as a JSON array, one value per entry in the List. Hence the "32 bytes overhead per byte". So, that's not a good idea for binary data storage that exceed a few bytes.

Under the hood, that's why support for Property<Whatever[]> was postponed after Qi4j 2.0.

Ideally one would want a String[] serialized as a JSON array of strings.
But one would also want a byte[] serialized as ... NOT a JSON array of bytes.

We'll want to fix this in future releases.
Feel free to push the discussion forward.

Cheers

/Paul

[Jaydatt Desai]

Hi Paul,

Thanks a lot for your time and the provided solution about "Storing it to Base64 String". It also make sense to store the bigger file on the file system and give their references to an entity of entity store. And for small binary chunks we can store it using Base64 encoded string in entities as well. I can see only one negative point when storing on to file system is it will increase manageability and maintainability , where we need to take care while moving data to for example some another machine, also we need to keep it strict rule that file should not be deleted anyhow by a garbage collector or so on. But for bigger file system we should not store it to database is also true

Thanks again for your time,
Jaydatt

[Paul Merlin]

Jaydatt Desai a écrit :

You're welcome Jaydatt.

About manageability, you still can use any system to store the big binaries, filesystem is one of them. In fact any store thought for big binaries would do. A dedicated SQL table, a MondoDB collection using GridFS etc... Could be implemented as a Service used by your Entities.
 
Cheers

/Paul

[Jaydatt Desai]

Hi Paul,

Thanks for the suggestions here for storing the big binaries, it really make sense to store large binaries on dedicated sql table or a mongodb collection. We are also thinking to create a kind of java plug-in to deal with this big binaries and where ever we need to store the binary data we can simply use this plugin for storing this big binaries. So, with plug in as you suggested we can use the "Filesystem" or a sql table or any other option to store the big binaries.

Thank you,
Jaydatt