Sending blobs through CDS
79 views
Skip to first unread message
Mark Weston
May 17, 2012, 10:42:44 AM5/17/12
to IADS
We are currently using the custom data interface to send data to IADS
through the CDS. We have only been using the non-blob types up to this
point. We are planning to start using blobs to send message structures
to IADS through the CDS.
At our external data source driving the CDS interface, different
message structures will be used to communicate different properties of
the multiple systems. However, each system will use the same message
to send its data for a given property.
This gives us two choices (at least) for sending the messages for a
specific property:
1. Define a unique blob parameter for each source system like
sys1_prop1, sys2_prop1, sys3_prop1,etc.
2. Use a single blob parameter like prop1 that is used to send
Property1 data for all systems.
These blob parameters would be handled by a custom ActiveX control to
display the message structure data in expandable tree-type displays.
We plan to use a single control per property that will be able to
display the fields of the property for each system in different
branches of the tree. A field in each message would identify which
system the message applied to.
We would prefer to use option 2 since we would only need to associate
a single blob parameter with the control so as more systems are added
the interface does not change. Option 1 would require us to associate
all of the blob parameters from each system for a given property with
the control.
However this does mean that we could have two copies of the message
that need to be sent with the same timetag since the different systems
could try to communicate their property values at the same time. Was
wondering if the CDS interface would pass more than one instance of
the same blob parameter if they both had the same time associated with
them. Would also need to be able to get them from the CDS to our
control. We would need for every sample of the parameter sent to the
CDS to get to our control which will use the internal identifier data
in the message to figure out which system sent the data.
We would use 0 for the data rate in the CDS Parameter Definiton File
since the data is expected to be asynchronous.
Is this possible? Do you envision any issues?
through the CDS. We have only been using the non-blob types up to this
point. We are planning to start using blobs to send message structures
to IADS through the CDS.
At our external data source driving the CDS interface, different
message structures will be used to communicate different properties of
the multiple systems. However, each system will use the same message
to send its data for a given property.
This gives us two choices (at least) for sending the messages for a
specific property:
1. Define a unique blob parameter for each source system like
sys1_prop1, sys2_prop1, sys3_prop1,etc.
2. Use a single blob parameter like prop1 that is used to send
Property1 data for all systems.
These blob parameters would be handled by a custom ActiveX control to
display the message structure data in expandable tree-type displays.
We plan to use a single control per property that will be able to
display the fields of the property for each system in different
branches of the tree. A field in each message would identify which
system the message applied to.
We would prefer to use option 2 since we would only need to associate
a single blob parameter with the control so as more systems are added
the interface does not change. Option 1 would require us to associate
all of the blob parameters from each system for a given property with
the control.
However this does mean that we could have two copies of the message
that need to be sent with the same timetag since the different systems
could try to communicate their property values at the same time. Was
wondering if the CDS interface would pass more than one instance of
the same blob parameter if they both had the same time associated with
them. Would also need to be able to get them from the CDS to our
control. We would need for every sample of the parameter sent to the
CDS to get to our control which will use the internal identifier data
in the message to figure out which system sent the data.
We would use 0 for the data rate in the CDS Parameter Definiton File
since the data is expected to be asynchronous.
Is this possible? Do you envision any issues?
Mike Burt
May 17, 2012, 2:23:35 PM5/17/12
to IADS
> We would prefer to use option 2 since we would only need to associate
> a single blob parameter with the control so as more systems are added
> the interface does not change. Option 1 would require us to associate
> all of the blob parameters from each system for a given property with
> the control.
No problem...you can setup the blob content any way that would make
> a single blob parameter with the control so as more systems are added
> the interface does not change. Option 1 would require us to associate
> all of the blob parameters from each system for a given property with
> the control.
things easier downstream...just keep in mind that the CDS expects a
static data sample size per blob parameter.
> Was wondering if the CDS interface would pass more than one instance of
> the same blob parameter if they both had the same time associated with
> them.
> We would use 0 for the data rate in the CDS Parameter Definiton File
> since the data is expected to be asynchronous.
No problem...in this case the CDS will handle duplicate times across
> since the data is expected to be asynchronous.
samples by internally incrementing the time stamp(s) by 1 nanosecond
in order to be able to uniquely identify the samples.
> Would also need to be able to get them from the CDS to our
> control. We would need for every sample of the parameter sent to the
> CDS to get to our control which will use the internal identifier data
> in the message to figure out which system sent the data.
recommend obtaining the code examples contained on the Programming
Examples page of the IADS web site (http://iads.symvionics.com/
programs.html). See the section named 'Custom Display Tutorials' and
download the appropriate example for your development environment.
If you run into problems or have further questions please let us know.
Mark Weston
May 24, 2012, 11:33:20 AM5/24/12
to IADS
We have been able to get the blob parameter into IADS working.
However, we are having issues getting all samples of the blob
parameter sent to our ActiveX control. It does not appear all of the
samples are being sent to the Blob viewer either. When we look at the
IADS recording file for the parameter, we see all of the samples in
the data file.
We are sending the blob parameter as an async parameter (rate of 0 in
parameter def file). In our current testing we are sending three
samples of the blob parameter about twice a second. Initally they all
had the same timetag and we saw in the IADS recording that IADS was
adding 1 nanosecond to the samples with duplicate timetags. We have
also tried were we increment the timetag at the source so that all
samples have unique timetags (each sample in the group of three have
timetags 100 microseconds apart).
Testing has found that for both real-time and local-mode playback that
it does not appear that all of the samples are being sent to our
control and the blob viewer. The last sample of the three received by
IADS seems to always be seen by our control and the blob viewer, but
the other two samples in each group of three are only sporadically
sent to our control and the blob viewer.
We can provide a copy of the output directory that can be used to do
local mode playback. Our control would not be disaplyed without our
dll file, but we do not think the data is being sent to the blob
viewer either.
Note that are time parameters are also defined as async parameters in
the CDS parameter def file as well (smaple rate of 0). We only put
timetags in the messages sent to IADS for those timetag values that
actually have an parameter present. All of the parameters we are
sending are defined as async.
Any suggestions for how to proceed?
However, we are having issues getting all samples of the blob
parameter sent to our ActiveX control. It does not appear all of the
samples are being sent to the Blob viewer either. When we look at the
IADS recording file for the parameter, we see all of the samples in
the data file.
We are sending the blob parameter as an async parameter (rate of 0 in
parameter def file). In our current testing we are sending three
samples of the blob parameter about twice a second. Initally they all
had the same timetag and we saw in the IADS recording that IADS was
adding 1 nanosecond to the samples with duplicate timetags. We have
also tried were we increment the timetag at the source so that all
samples have unique timetags (each sample in the group of three have
timetags 100 microseconds apart).
Testing has found that for both real-time and local-mode playback that
it does not appear that all of the samples are being sent to our
control and the blob viewer. The last sample of the three received by
IADS seems to always be seen by our control and the blob viewer, but
the other two samples in each group of three are only sporadically
sent to our control and the blob viewer.
We can provide a copy of the output directory that can be used to do
local mode playback. Our control would not be disaplyed without our
dll file, but we do not think the data is being sent to the blob
viewer either.
Note that are time parameters are also defined as async parameters in
the CDS parameter def file as well (smaple rate of 0). We only put
timetags in the messages sent to IADS for those timetag values that
actually have an parameter present. All of the parameters we are
sending are defined as async.
Any suggestions for how to proceed?
James Bretz
May 24, 2012, 12:32:04 PM5/24/12
to ia...@googlegroups.com
Hi Mark,
Interesting that you mention that subject. It seems to the hot topic for
the last two week. Another convergence I suppose ;)
You'll need to 'beef up' your display to "every data sample" status. True,
the blob viewer (and most of the ActiveX controls) simply display the
current value only and don't bother with the extra processing it takes to
ensure every data point. However, most of the displays on the first tab of
the display builder (what we call Iads 'core' displays) do actually process
every data point (stripchart, crossplot, freqPlots, etc).
If this wasn't an aperiodic blob, I would direct you to the example project
"Creating an ActiveX display using C++ VS2005"
(http://iads.symvionics.com/programs.html), specifically the
SampleEveryDataPoint class. In fact, you've probably already implemented the
example yet you're still missing data. What makes your job so difficult
right now is that aperiodic blobs aren't supported in the example's
StartFrameRead/GetNextFrameRead functions that make this operation
relatively easy. We have actually fixed this issue in the latest release.
It's a long shot, but if you can upgrade to the version 7.3.3 (beta version
available), this problem will go away and the example project's code will
work fine. I'll continue along assuming you can't upgrade.
Essentially, what you need to do is follow the SampleEveryDataPoint class,
except you'll need to program the StartFrameRead and GetNextFrameRead
functions by hand. What they accomplish is rather simple, but the devil is
in the details. It might take a while to get things working 100%.
Let me give you a sense of the overall work that needs to be done, and then
we can translate that into the example code. A display is 'rendered' at a
frequency of the update rate of the AnalysisWindow (depends on processing
load, but it's approximately 50 times per second). What you need to do is
keep track of the last time you've processed data, and when the next display
cycle comes around, you simply get all of the data from the 'lastTime' to
the 'currentTime', one point (blob) at a time from the Parameter
(IParameter) using the Value2 function. Does this make sense?
The StartFrameRead call in the SampleEveryDataPoint class marks the current
time, so at that point, you would need to copy that time into an IIadsTime
'lastTime' member variable of your class. When the GetNextFrameFread is
called, you would then have to set the parameter's current time to your
'lastTime', then call the Value2 function with the argument
'iadsTimeAdvanceFoward'. This will essentially return the value (blob) at
the specific time, and then walk to the next sequential data point in the
parameter as you call it multiple times. The Value2 function has a argument
to return the time of the data point, and you would use that time to compare
to your 'currentTime'. When the time is >= currentTime you're done. After
that step you'll mark the new 'lastTime' as your current time and then wait
for the next drawing cycle.
There are a number of issues here that makes this complex. The worse part is
that the data is aperiodic, so you can't actually predict how many point to
read in a given time frame... and it's difficult to keep track of the time
values for the data ranges read, so you'll most likely be processing
duplicate points. Also, in order to read aperiodic data from a parameter,
you'll need to set the InterpolatedSampleRate value to zero. The
StartFrameRead/GetNextFrame read takes care of all these issues and more.
Too bad we can't use it ;)
I'll let you absorb this for a bit, then I'll try to build up an
SampleEveryDataPoint example without using StartFrameRead/GetNextFrameRead
functions. It might take a while because I'm pretty busy, sorry.
Jim
--------------------------------------------------
From: "Mark Weston" <mark.w...@lmco.com>
Sent: Thursday, May 24, 2012 8:33 AM
To: "IADS" <ia...@googlegroups.com>
Subject: [IADS] Re: Sending blobs through CDS
> --
> You received this message because you are subscribed to the Google Groups
> "IADS" group.
> To post to this group, send email to ia...@googlegroups.com.
> To unsubscribe from this group, send email to
> iads+uns...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/iads?hl=en.
>
Interesting that you mention that subject. It seems to the hot topic for
the last two week. Another convergence I suppose ;)
You'll need to 'beef up' your display to "every data sample" status. True,
the blob viewer (and most of the ActiveX controls) simply display the
current value only and don't bother with the extra processing it takes to
ensure every data point. However, most of the displays on the first tab of
the display builder (what we call Iads 'core' displays) do actually process
every data point (stripchart, crossplot, freqPlots, etc).
If this wasn't an aperiodic blob, I would direct you to the example project
"Creating an ActiveX display using C++ VS2005"
(http://iads.symvionics.com/programs.html), specifically the
SampleEveryDataPoint class. In fact, you've probably already implemented the
example yet you're still missing data. What makes your job so difficult
right now is that aperiodic blobs aren't supported in the example's
StartFrameRead/GetNextFrameRead functions that make this operation
relatively easy. We have actually fixed this issue in the latest release.
It's a long shot, but if you can upgrade to the version 7.3.3 (beta version
available), this problem will go away and the example project's code will
work fine. I'll continue along assuming you can't upgrade.
Essentially, what you need to do is follow the SampleEveryDataPoint class,
except you'll need to program the StartFrameRead and GetNextFrameRead
functions by hand. What they accomplish is rather simple, but the devil is
in the details. It might take a while to get things working 100%.
Let me give you a sense of the overall work that needs to be done, and then
we can translate that into the example code. A display is 'rendered' at a
frequency of the update rate of the AnalysisWindow (depends on processing
load, but it's approximately 50 times per second). What you need to do is
keep track of the last time you've processed data, and when the next display
cycle comes around, you simply get all of the data from the 'lastTime' to
the 'currentTime', one point (blob) at a time from the Parameter
(IParameter) using the Value2 function. Does this make sense?
The StartFrameRead call in the SampleEveryDataPoint class marks the current
time, so at that point, you would need to copy that time into an IIadsTime
'lastTime' member variable of your class. When the GetNextFrameFread is
called, you would then have to set the parameter's current time to your
'lastTime', then call the Value2 function with the argument
'iadsTimeAdvanceFoward'. This will essentially return the value (blob) at
the specific time, and then walk to the next sequential data point in the
parameter as you call it multiple times. The Value2 function has a argument
to return the time of the data point, and you would use that time to compare
to your 'currentTime'. When the time is >= currentTime you're done. After
that step you'll mark the new 'lastTime' as your current time and then wait
for the next drawing cycle.
There are a number of issues here that makes this complex. The worse part is
that the data is aperiodic, so you can't actually predict how many point to
read in a given time frame... and it's difficult to keep track of the time
values for the data ranges read, so you'll most likely be processing
duplicate points. Also, in order to read aperiodic data from a parameter,
you'll need to set the InterpolatedSampleRate value to zero. The
StartFrameRead/GetNextFrame read takes care of all these issues and more.
Too bad we can't use it ;)
I'll let you absorb this for a bit, then I'll try to build up an
SampleEveryDataPoint example without using StartFrameRead/GetNextFrameRead
functions. It might take a while because I'm pretty busy, sorry.
Jim
--------------------------------------------------
From: "Mark Weston" <mark.w...@lmco.com>
Sent: Thursday, May 24, 2012 8:33 AM
To: "IADS" <ia...@googlegroups.com>
Subject: [IADS] Re: Sending blobs through CDS
> You received this message because you are subscribed to the Google Groups
> "IADS" group.
> To post to this group, send email to ia...@googlegroups.com.
> To unsubscribe from this group, send email to
> iads+uns...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/iads?hl=en.
>
Mark Weston
May 24, 2012, 12:56:58 PM5/24/12
to IADS
Thanks for the response. We will look at trying to use the workaround
steps since we are not able to upgrade right now.
Another option would be to investigate making the parameters
"periodic". We have defined all of our parameters as aperiodic because
several are aperiodic and the source for all of them is a bit less
deterministic than a telemetry front-end. Without preprocessing I
cannot guarantee a sample would always be sent every time it needs to
or that the timetags would be perfectly spaced. I do not understand
the constraints IADS puts on us for sample rate accuracy and timetag
accuracy if we define our parameters as periodic parameters for both
the data and the timetags. Is there any info that clearly defines
these constraints so we can investigate switching to periodic
parameters?
Thanks again.
Mark
steps since we are not able to upgrade right now.
Another option would be to investigate making the parameters
"periodic". We have defined all of our parameters as aperiodic because
several are aperiodic and the source for all of them is a bit less
deterministic than a telemetry front-end. Without preprocessing I
cannot guarantee a sample would always be sent every time it needs to
or that the timetags would be perfectly spaced. I do not understand
the constraints IADS puts on us for sample rate accuracy and timetag
accuracy if we define our parameters as periodic parameters for both
the data and the timetags. Is there any info that clearly defines
these constraints so we can investigate switching to periodic
parameters?
Thanks again.
Mark
Kathy Rodittis
May 24, 2012, 1:07:50 PM5/24/12
to ia...@googlegroups.com
Hi Mark-
Before you change this over to periodic, can you tell me a little bit about
how the data is created? You mentioned that duplicate time tags are are
being sent. Is there any way to stop this? Do you have control over this
process?
I know that when I sent aperiodic data to the CDS I make sure that time has
changed. If it hasn't, I don't send it again. This may automatically clear
up some of the problems you're having.
-Kathy
-----Original Message-----
From: ia...@googlegroups.com [mailto:ia...@googlegroups.com] On Behalf Of Mark
Weston
Sent: Thursday, May 24, 2012 9:57 AM
To: IADS
Subject: [IADS] Re: Sending blobs through CDS
Before you change this over to periodic, can you tell me a little bit about
how the data is created? You mentioned that duplicate time tags are are
being sent. Is there any way to stop this? Do you have control over this
process?
I know that when I sent aperiodic data to the CDS I make sure that time has
changed. If it hasn't, I don't send it again. This may automatically clear
up some of the problems you're having.
-Kathy
-----Original Message-----
From: ia...@googlegroups.com [mailto:ia...@googlegroups.com] On Behalf Of Mark
Weston
Sent: Thursday, May 24, 2012 9:57 AM
To: IADS
Subject: [IADS] Re: Sending blobs through CDS
James Bretz
May 24, 2012, 1:10:32 PM5/24/12
to IADS
Mark,
If you define a sample rate, the server (CDS) will simply force the data
into this rate (either drop data if it's over the specified rate, or pad
data with last good value if it's less than the specified rate).
Essentially, you need to weigh the possible result of this behavior with the
quality of the data. In other words, if the data to be processed is to
simply obtain the current value, then it can be forced to a periodic rate.
However, if you need to ensure that you have every single point without
potential duplicates or drops, then leave it aperiodic. Since you're
discussing how to process every point at the display level, and the data
source is a bit less deterministic, I think you made the right decision to
make it aperiodic.
Also, one more bit of information. I misspoke in the last post. The
StartFrameRead/NextFrameRead functions don't work with blobs in any case
(either periodic or aperiodic). The only choice is to either program by
hand, or upgrade.
Sent: Thursday, May 24, 2012 9:56 AM
If you define a sample rate, the server (CDS) will simply force the data
into this rate (either drop data if it's over the specified rate, or pad
data with last good value if it's less than the specified rate).
Essentially, you need to weigh the possible result of this behavior with the
quality of the data. In other words, if the data to be processed is to
simply obtain the current value, then it can be forced to a periodic rate.
However, if you need to ensure that you have every single point without
potential duplicates or drops, then leave it aperiodic. Since you're
discussing how to process every point at the display level, and the data
source is a bit less deterministic, I think you made the right decision to
make it aperiodic.
Also, one more bit of information. I misspoke in the last post. The
StartFrameRead/NextFrameRead functions don't work with blobs in any case
(either periodic or aperiodic). The only choice is to either program by
hand, or upgrade.
Sent: Thursday, May 24, 2012 9:56 AM
Kathy Rodittis
May 24, 2012, 1:16:14 PM5/24/12
to ia...@googlegroups.com
Apologies -- I just re-read your question and noticed that you did try
something similar. Ignore this!
something similar. Ignore this!
Mark Weston
May 24, 2012, 2:27:17 PM5/24/12
to IADS
Actually I do not think using periodic will work without us changing
how we handle the data. I think our main issue is due to the multi
samples with the same timetag. We have about 140 "systems" reporting
the same info to our source app using the same structure. Unique info
is in each sample so our control knows which message came from what
system and it then places the info in the control separately for each
source. We chose to use a single blob parameter rather than having to
use ~140 parameters that would then require ~140 parameter properties
for our control. This is what can cause us to get the multi samples
and is what makes the data non-periodic.
The simple test case I have right now is only simulating 3 of the
source systems so the problem is actually going to be worse when we
have all of the system reporting. Only one is reliably getting to the
control. We are working with info Jim posted to see if we can get it
to work right with our current design.
Thanks.
how we handle the data. I think our main issue is due to the multi
samples with the same timetag. We have about 140 "systems" reporting
the same info to our source app using the same structure. Unique info
is in each sample so our control knows which message came from what
system and it then places the info in the control separately for each
source. We chose to use a single blob parameter rather than having to
use ~140 parameters that would then require ~140 parameter properties
for our control. This is what can cause us to get the multi samples
and is what makes the data non-periodic.
The simple test case I have right now is only simulating 3 of the
source systems so the problem is actually going to be worse when we
have all of the system reporting. Only one is reliably getting to the
control. We are working with info Jim posted to see if we can get it
to work right with our current design.
Thanks.
James Bretz
May 24, 2012, 5:11:56 PM5/24/12
to ia...@googlegroups.com
Mark,
I'm pretty close right now. I'm just testing and fine tuning.
The funny part is that the example looks a bit simpler than the original ;)
If I don't have it tonight, it'll definitely be ready tomorrow,
Sent: Thursday, May 24, 2012 11:27 AM
To: "IADS" <ia...@googlegroups.com>
I'm pretty close right now. I'm just testing and fine tuning.
The funny part is that the example looks a bit simpler than the original ;)
If I don't have it tonight, it'll definitely be ready tomorrow,
Sent: Thursday, May 24, 2012 11:27 AM
To: "IADS" <ia...@googlegroups.com>
Subject: [IADS] Re: Sending blobs through CDS
James Bretz
May 25, 2012, 12:30:29 PM5/25/12
to ia...@googlegroups.com
Mark,
Ok I take that back. This example isn't simpler than the original. During
testing I found a couple of missing pieces.
The good news is that it all appears to work fine. I ran through test data
and in each instance it succeeded to get every blob in a sequence. There may
be some more fine tuning to do, but not much.
Hope this helps (see attached),
Jim
--------------------------------------------------
From: "James Bretz" <j...@iads-soft.com>
Sent: Thursday, May 24, 2012 2:11 PM
To: <ia...@googlegroups.com>
Subject: Re: [IADS] Re: Sending blobs through CDS
Ok I take that back. This example isn't simpler than the original. During
testing I found a couple of missing pieces.
The good news is that it all appears to work fine. I ran through test data
and in each instance it succeeded to get every blob in a sequence. There may
be some more fine tuning to do, but not much.
Hope this helps (see attached),
Jim
--------------------------------------------------
From: "James Bretz" <j...@iads-soft.com>
Sent: Thursday, May 24, 2012 2:11 PM
To: <ia...@googlegroups.com>
Subject: Re: [IADS] Re: Sending blobs through CDS
Reply all
Reply to author
Forward
0 new messages