DNDecoderConnection best practices
Олег
Thank you for this amazing and free project!
I have several questions about DNDecoderConnection
Should I run "run" method for every packet or it's better to
"agregate" several packets and run "run" method only once per pack?
Does it make sens to use "run" method which uses "extraThreadCount"?
How to do that?
How can I know how to tune DNDecoderConnection to better fit my needs?
For example I just found in this newsgroup that I can play with
"BufferCount" parameter but I don't know how to find "ideal" value.
And I'm sure there are other parameters that can be tuned but there
are no information about that...
Thanks,
Oleg
Олег
I want to make sure that my question is not lost and you just have no
time to answer... I will appreciate If you can answer it in nearest
2-3 weeks or so.
Thanks,
Oleg
Malcolm Spence
I will follow up with Dale when he gets in the office.
regards Malcolm
Malcolm D. Spence
Director of Business Development
OCI "Use our reach to exceed your grasp."
(Phone 1- 314-579-0066 ext. 206 or FAX -0065)
www.theaceorb.com www.ociweb.com
> --
> You received this message because you are subscribed to the Google Groups
> "quickfast_users" group.
> To post to this group, send email to quickfa...@googlegroups.com.
> To unsubscribe from this group, send email to
> quickfast_use...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/quickfast_users?hl=en.
Dale Wilson
Hi Oleg,
On 3/21/2012 2:56 AM, Олег wrote:
> Hi Dale
> Thank you for this amazing and free project!
You are welcome.
> I have several questions about DNDecoderConnection
>
> Should I run "run" method for every packet or it's better to
> "agregate" several packets and run "run" method only once per pack?
The design is that you only need to call run once to start listening for
incoming messages on all incoming feeds.
As each message is received it will be decoded and the fields will be
delivered to the message builder. This process should continue
automatically until you call the stop() method (and possibly the
joinThreads() method) to shut the system down.
If you have multiple DNDecoderConnections in your system -- meaning you
are accepting data from more than one feed simultaneously -- you only
need to call run() on one of them (however, it doesn't hurt to call
run() on each of them -- the number of extra threads will be cumulative.)
> Does it make sens to use "run" method which uses "extraThreadCount"?
> How to do that?
QuickFAST can use a varying number of threads to get some parallel
processing benefits although the nature of FAST data means that it must
be decoded sequentially so there is a limit to the number of threads
that can be kept busy. For each DNDecoderConnection in your system
(i.e. for each incoming data feed) I recommend two threads. These
threads all go into a common thread pool which is why you can call run
on any DNDecoderConnection to start all of them.
The extraThreadCount and useThisThread parameters together determine how
many threads will be available for QuickFAST. Examples:
run(builder, 1, false);
This would allocate a thread for use by QuickFAST and return
immediately so the main thread could be used for other purposes.
run(builder, 0, true);
This would use the main thread to run QuickFAST. This method
would not return until QuickFAST is shut down (presumably stop() would
be called from the
message builder at some p
run(builder, 1, true);
This would allocate a new thread and also use the main thread to
run QuickFAST. This is a good option to use if you have nothing
worthwhile for the main thread to do.
run(builder, 2, false);
This would allocate two threads for use by QuickFAST. The main
thread would return immediately. This is a good option if the main
thread has other work to do like accepting input from a user.
>
> How can I know how to tune DNDecoderConnection to better fit my needs?
> For example I just found in this newsgroup that I can play with
> "BufferCount" parameter but I don't know how to find "ideal" value.
> And I'm sure there are other parameters that can be tuned but there
> are no information about that...
There are very few parameters to QuickFAST that tune its behavior. Look
at the methods in the DNDecoderConnection class to see these parameters.
Most of the parameters configure QuickFAST to match the characteristics
of a particular data feed so they can't really be adjusted unless the
data feed changes. The buffer count is the most significant tunable
parameter.
The C++ Receiver class keeps some statistics about the behavior of the
system which might help with tuning. Interpreting some of these
statistics requires a detailed understanding about how the communication
layer of QuickFAST works. Right now these statistics are not exposed
through the .NET wrapper, however that could be changed.
The statistics are:
/////////////
// Statistics
/// No buffers avaliable when we could have started a read
size_t noBufferAvailable_;
/// Packets accepted (includes error & empty)
size_t packetsReceived_;
/// Bytes in those packets
size_t bytesReceived_;
/// Packets received with errors (usually disconnect or EOF)
size_t errorPackets_;
/// Packets received in error due to a linux bug.
size_t pausedPackets_;
/// Packets containing no data (usually during shutdown)
size_t emptyPackets_;
/// Packets containing valid data: queued to be processed
size_t packetsQueued_;
/// Batches of packets collected by queue
size_t batchesProcessed_;
/// Individual packets in the batches
size_t packetsProcessed_;
/// Bytes in the processed packets.
size_t bytesProcessed_;
/// Largest single packet received
size_t largestPacket_;
Hope this helps,
Dale
> Thanks,
> Oleg
--
Dale Wilson
Principal Software Engineer
Object Computing, Inc.
Олег
You wrote that I should call "run" just once to start listening.
But that's how I do things:
1. Myself I create Socket and listen for UDP Multicast
2. Each time I receive new datagram I call "run" method like that:
public void NewMessage(object sender, NewMessageEventArgs
args)
{
int currentNumber = BitConverter.ToInt32(args.byteArray,
0);
Log.PushDebug("received message " + currentNumber + "
count=" + args.count);
if (!checkbook.AddSeqNum(currentNumber))
{
return;
}
try
{
lock (Decoder)
{
Decoder.run(Builder, args.byteArray, 4, args.count
- 4, true);
}
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
Am I doing something wrong? Note I call "run" method several thousands
times per second and this works!
Note that MICEX adds "seqNum" so I can ignore packet without decoding
it in case I have already received it.
Олег
"May be called repeatedly. Should be called instead of the run()
method with thread counts. i.e. do not call both run() methods."
So it seems I correctly use API. But which of these two "run" methods
is faster? It seems In my situation I should always use "run" method
with "reset" flag.
How to use "reset" parameter? Should I set it to true or false?
Thank you Dale for your help, I think some parts of your messages may
be copypasted to QuickFAST documentation. Currently documentation is
very pure :(
Олег
> very pure :(