How to handle binary data

[John Doe]

Hello
Is there any example available showing how to transfer large binary data with YAMI4?

For example frames caputred from the camera(http://inspirel.com/yami4/book/1-2-4.html)

Thanks

[Maciej Sobczak]

Hello,

On 23 Kwi, 17:24, John Doe <johndoeilikeche...@gmail.com> wrote:

> Is there any example available showing how to transfer large binary data
> with YAMI4?

The YAMI4 distribution contains not only examples, but also a set of
unit tests - please check the Agent test for your target language, it
should contain a unit for large binary data transfer.
If you fail to find it, don't hesitate to ask for more details, but
please let us know which language do you intend to use, as the actual
API is a bit different.
Note also that in addition to binary data YAMI4 allows to send raw,
unformatted messages:

http://inspirel.com/yami4/book/3-13.html

This can be particularly useful if the message is either already
formatted or has a trivial structure - just like with binary opaque
blocks.

Regards,

--
Maciej Sobczak * http://www.inspirel.com/

[John Doe]

Hey, thanks for reply.
Based on aget-test code I tried to setup simplest possible binary transfer: client is sending some data, server will receive and print size.

Code here:
http://pastebin.com/20tG5YkT

Everything compiles fine, I'm starting server:
./server tcp://localhost:12345

But when call client:
./client tcp://localhost:12345 somefile

it throws yami_runtime_error what: The value that was given or received is incorrect.

Any clue what's wrong here?

TIA

[John Doe]

I was just looking at the code of raw_buffer_data_source, and it looks like the data has to be aligned to multiple of 4 by client code.

Am I right? If so is it mentioned somewhere in the book? If not, what does "check_buffer_sizes" do?

Thanks

[Maciej Sobczak]

On 24 Kwi, 14:38, John Doe <johndoeilikeche...@gmail.com> wrote:

> I was just looking at the code of raw_buffer_data_source, and it looks like
> the data has to be aligned to multiple of 4 by client code.
> Am I right?

There is no such requirement, although later in code the data transfer
is implemented in terms of words, not individual bytes. It is safer
(and most likely faster) if the buffer is aligned.

Please note that normally there is no need to worry about it, as any
binary buffer that you will ever allocate in your program will be
already aligned anyway.

> If not, what does
> "check_buffer_sizes" do?

It checks whether the sizes of buffers are multiple of 4-byte words
(that is, you can use buffer of 100 bytes but not 101 bytes, even if
it is aligned).

I will try to check your example as well.

[Maciej Sobczak]

On Apr 24, 2:19 pm, John Doe <johndoeilikeche...@gmail.com> wrote:

> Code here:http://pastebin.com/20tG5YkT
>
> Everything compiles fine

Uhm... I don't think so. The construct:

const::std::vector<char>

is not likely to compile at all, and:

yami::client client(options);

there is no such class in the yami namespace.
You seem to be missing some #includes, too, and finally, we don't see
the "load a bunch of data into vector" part - this is actually the
most probable source of problems, as the buffer needs to have a size
that is multiple of 4.

Please verify that all these issues are cleared.

[John Doe]

On Tuesday, April 24, 2012 8:50:46 PM UTC, Maciej Sobczak wrote:

Please verify that all these issues are cleared.

Hi.

Sample project attached.

I hope it's OK to post binaries to this group :)

I have two problems with that code.

1) If the size of the file being send is not multiple of 4, then client throws yami_runtime_error: The value that was given or received is incorrect.

2) If the size is multiple of 4, server is receiving the file, but client never exits.

Looks like it's not leaving "wait_for_completion()" - are there any steps that server should perform to fix this?

TIA

best regards

[Maciej Sobczak]

On 25 Kwi, 12:37, John Doe <johndoeilikeche...@gmail.com> wrote:

> I have two problems with that code.
>
> 1) If the size of the file being send is not multiple of 4, then client
> throws yami_runtime_error: The value that was given or received is
> incorrect.

Yes. This is correct.

> 2) If the size is multiple of 4, server is receiving the file, but client
> never exits.
> Looks like it's not leaving "wait_for_completion()" - are there any steps
> that server should perform to fix this?

Yes. The message becomes "completed" when there is some feedback from
the server - this can be either a reply message or a rejection
message. The client can then inspect this feedback and continue
processing accordingly. Please check the calculator example to see how
such interaction can look like.
The client can also send the message as a one-way message to express
that it is not interested in any feedback. Or, the client can send the
regular message but instead of waiting for completion do something
else. Please check the print example.

Regards,

[John Doe]

On Thursday, April 26, 2012 8:23:54 AM UTC, Maciej Sobczak wrote:

> 1) If the size of the file being send is not multiple of 4, then client
> throws yami_runtime_error: The value that was given or received is
> incorrect.

Yes. This is correct.

Hello.

I'm a bit lost, is it or is it not required, to align data to 4 bytes in client code?(by client I mean consumer of a library, not "client" program from my example)

You said earlier that "There is no such requirement" to align data to 4, but now you're saying that YAMI4 will throw an exception if data is not multiple of 4.

TIA for enlightenment

Brgds.

[Maciej Sobczak]

On 26 Kwi, 12:22, John Doe <johndoeilikeche...@gmail.com> wrote:

> Hello.
> I'm a bit lost, is it or is it not required, to align data to 4 bytes in
> client code?

Hello.
You seem to be confusing two things: data alignment and buffer size.
These are different things and you have asked about them as if they
are the same.

You should not worry about alignment (that is, at what address your
data starts), because in normal code you will always have that right.
In particular, if your buffer was dynamically allocated (this includes
vector<char> and similar), then it is aligned.

You should still worry about buffer size (that is, how big is your
data) - it has to be a multiple of 4.

[John Doe]

Thanks for clarification.

I think I got it now.

From the usage point of view, it would be beneficial not to worry about the size of the data.

What is the reason that library is not taking care of it?

Brgds. 

[Maciej Sobczak]

On 26 Kwi, 15:29, John Doe <johndoeilikeche...@gmail.com> wrote:

> From the usage point of view, it would be beneficial not to worry about the
> size of the data.
> What is the reason that library is not taking care of it?

The library is taking care of it if you use typed data - that is, if
you put your data in the parameters object.
But if you use raw binary data transfer, then some simplifying
assumptions are made and one of them is that the buffer has complete
words.

[John Doe]

Thanks, fair enough.