Building a custom assembler (Complying with BM&FBovespa's UMDF)

[indiosmo]

Hello, I'm trying to get QuickFAST to work with BM&FBovespa's new feed
(UMDF).

Seeing as there is currently no support for messages spanning multiple
packets (which the exchange sends), I'm trying to figure out the best
way to extend QuickFAST in order to support it.

I've been reading the source code for a couple of days and I'm still
not sure what's the best way to go about it, so I was wondering if you
guys might have any suggestions.

UMDF is multicast and each packet has an unencoded header (described
below), and a maximum size of 1400 bytes (header included).
Messages larger than 1400 bytes may be split across packets in
multiple chunks.

MsgSeqNum - 4 bytes - Repeated in case of multiple chunks
NoChunks - 2 bytes - Total chunks in this message
CurrentChunk - 2 bytes
MsgLength - 2 bytes - Length of the payload
Payload - [MsgLength] bytes - FAST encoded message (may be partial)

I was thinking about implementing two classes, the Assembler itself
and a HeaderAnalyzer. So far what I've got is:

1)UMDFAssembler::serviceQueue(...) would call
UMDFHeaderAnalyzer.analyzeHeader(...), which in turn reads the chunk
information.

2)If the CurrentChunk is smaller than NoChunks, skip is set to true,
serviceQueue(..) returns and we wait for more packets, which I
understand would be read into a new buffer (i.e., for a message with 3
chunks, I would have 3 filled buffers)
It appears that blockSize would be unused here, so maybe we could
use it to store NoChunks.
If NoChunks is greater than the number of buffers, then call
receiver_->addBuffers(...) to allocate as needed.

3)If we are at the last chunk, skip is false, serviceQueue(...) calls
decoder_.decodeMessage(...) and we are ready for the next message.


Would something like that work? I'm not sure if I get the underlying
handling of the buffers right. (In step 2 for example)

I understand that ultimately the call to decodeMessage will result in
a call to DataSource::getByte(...).
Is getByte(...) able to traverse the multiple buffers, and if so, will
it be done sequentially? (In the same order the buffers were filled).

I also feel like I'm missing a cleanup step here, don't I have to
release the buffers somewhere?


Thanks,
Luiz Siqueira

[Leonardo Arnt]

Hi, Luiz

We've been able to do what you're trying to achieve. Feel free to get in touch if you want to chat about it.

Regards,

Leonardo B. Arnt
Mobile: +55 21 9847-0030
Inoa Sistemas
Phone: +55 21 3795-8197

--
You received this message because you are subscribed to the Google Groups "quickfast_users" group.
To post to this group, send email to quickfa...@googlegroups.com.
To unsubscribe from this group, send email to quickfast_use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/quickfast_users?hl=en.

[Renato Laiz]

Hi Luiz,

I don't know if QuickFast yourself can do this job, I didn't find anyway to do this. In my case I handle all the multicast packet by myself, order this and use QucikFast just to decode the complete Fast message, I think that this is a good idea, because in BVMF UMDF we can receive the same packet from backup and from primary streams, if you handle this packets by yourself you can throw out the second one, without lot of processing just analyzing the header.

If you want some help, contact me, I already have a solution running using QuickFast.

Thanks,
Renato Pereira Laiz
EzMarket - Soluções inovadoras
Email: ren...@ezmarket.com.br

Sent from my iPhone