Fine-grained synchronization on modern networks
D Yuniskis
Modern network infrastructure uses *lots* of buffering;
memory is (now) cheap enough to embed throughout the network
fabric.
With that, fine-grained synchronization over (wired) networks
becomes problematic -- there's no deterministic way for a
processor in a particular node to have any idea of its
relative packet time wrt any other node in the network
(though it is pretty obvious that a packet arrives at
its destination some time *after* leaving its source! :> )
Sure, things like NTP *try* to quantify this skew. But,
its goals are much more long-term... if it is wrong on
the short term, there is no significant consequence.
(I also suspect the apparent precision and accuracy that
NTP provides is largely delusional :-/ )
So, how *do* you achieve fine-grained synchronization
nowadays? What is *practical*? And theoretically
*achievable* (without an a priori characterization
of the network infrastructure and topology)?
Spehro Pefhany
Hey, Don:-
PTP.. IEEE-1588.
Vladimir Vassilevsky
IEEE 1588.
Basically, it is about getting/setting send/receive time for packets at
the physical layer, as opposed to going through a whole networking
protocol stack.
Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com
Spehro Pefhany
<nos...@nowhere.com> wrote:
>
>IEEE 1588.
>
>Basically, it is about getting/setting send/receive time for packets at
>the physical layer, as opposed to going through a whole networking
>protocol stack.
And figuring out between the nodes which node has the best clock, and
synchronizing the other nodes to the best clock with an algorithm that
rejects network communication jitter but compensates for long term
drift between clocks.
Paul E. Bennett
> Hi,
[%X ---- Stuff about NTP ---- X%]
> So, how *do* you achieve fine-grained synchronization
> nowadays? What is *practical*? And theoretically
> *achievable* (without an a priori characterization
> of the network infrastructure and topology)?
As well at the IEEE 1588 standard that has already been mentioned, you
should look at LXI also. This builds on IEEE 1588 and is used for
instrumentation purposes across networks.
See <http://www.lxistandard.org/>
--
********************************************************************
Paul E. Bennett...............<email://Paul_E....@topmail.co.uk>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************
D Yuniskis
> On Wed, 23 Dec 2009 10:08:53 -0700, D Yuniskis
>
>> becomes problematic -- there's no deterministic way for a
>> processor in a particular node to have any idea of its
>> relative packet time wrt any other node in the network
>> (though it is pretty obvious that a packet arrives at
>> its destination some time *after* leaving its source! :> )
>>
>> Sure, things like NTP *try* to quantify this skew. But,
>> its goals are much more long-term... if it is wrong on
>> the short term, there is no significant consequence.
>> (I also suspect the apparent precision and accuracy that
>> NTP provides is largely delusional :-/ )
>>
>> So, how *do* you achieve fine-grained synchronization
>> nowadays? What is *practical*? And theoretically
>> *achievable* (without an a priori characterization
>> of the network infrastructure and topology)?
>
> Hey, Don:-
>
> PTP.. IEEE-1588.
Hi Spehro,
Yes, but this makes lots of assumptions (constraints)
about deployment.
Let me rephrase my question in a more general way:
What level of synchronization can you expect to achieve
in an unconstrained deployment environment? E.g.,
imagine you are trying to coexist within an *existing*
network structure -- you can't tell the customer
to replace his switches; or flatten his network;
or stop running applications that generate lots of
network traffic, etc.
I.e., you *can't* count on the user having special hardware
to generate good timestamps; special switches to propagate
this behaviour across the switch; "benign" applications that
let the network quiesce during times when synchronization
activities are happening; hosts that are NOT evenly loaded
wrt network traffic; NICs vary from machine to machine; etc.
I.e., you have to coexist with what is rather than
redesign what you would *like*...
Note that in days past, this was a lot easier to
achieve as broadcasts tended to *truly* be broadcasts,
propagation delays were simply K * distance * speed of
light, etc.
Vladimir Vassilevsky
D Yuniskis wrote:
> What level of synchronization can you expect to achieve
> in an unconstrained deployment environment? E.g.,
> imagine you are trying to coexist within an *existing*
> network structure -- you can't tell the customer
> to replace his switches; or flatten his network;
> or stop running applications that generate lots of
> network traffic, etc.
J = packet delivery jitter (sec)
V = number of sync packets per sec
D = drift of the local clock (Hz/sec)
D J^4 1
Time error (sec) = [-------] ^ ---
2 V^2 5
Ed Prochak
> Spehro Pefhany wrote:
[]
>
> > Hey, Don:-
>
> > PTP.. IEEE-1588.
>
> Hi Spehro,
>
> Yes, but this makes lots of assumptions (constraints)
> about deployment.
>
> Let me rephrase my question in a more general way:
>
> What level of synchronization can you expect to achieve
> in an unconstrained deployment environment? E.g.,
> imagine you are trying to coexist within an *existing*
> network structure -- you can't tell the customer
> to replace his switches; or flatten his network;
> or stop running applications that generate lots of
> network traffic, etc.
>
> I.e., you *can't* count on the user having special hardware
> to generate good timestamps; special switches to propagate
> this behaviour across the switch; "benign" applications that
> let the network quiesce during times when synchronization
> activities are happening; hosts that are NOT evenly loaded
> wrt network traffic; NICs vary from machine to machine; etc.
>
> I.e., you have to coexist with what is rather than
> redesign what you would *like*...
Question is: what are your trying to achieve? What really are your
real-time constraints?
Is it soft real time? Is the timestamped data essentially historical?
(i.e. late delivery updates a database of some sort. It is not used
immediate calculations for process control) You need all the data, but
late or out of order delivery does not create serious error
conditions.
IMO, This case could essentially be solved at the application layer.
(Timestamps generated by the end point devices, correlated to the
destination device's clock.)
Is it really hard real time? IOW, you are doing some kind of process
control?
Sorry, but ethernet being what it is does not provide any way to
assure delivery times. That's why there tends to be distributed
process control. Put as much intelligence as needed at the point of
control.
>
> Note that in days past, this was a lot easier to
> achieve as broadcasts tended to *truly* be broadcasts,
> propagation delays were simply K * distance * speed of
> light, etc.
Only if you happened to be on the same wire. Since hubs and bridges,
it has never been that simple.
Ed
D Yuniskis
I want to know what is *possible* before even considering various
communication media. E.g., if the best I can achieve with
"ethernet" is O(100us) then I will see how that reflects to
what the *product* can realistically claim. If that ends up
looking like crap, then I rule out "ethernet" and move on to other
technologies.
> Is it soft real time? Is the timestamped data essentially historical?
> (i.e. late delivery updates a database of some sort. It is not used
> immediate calculations for process control) You need all the data, but
> late or out of order delivery does not create serious error
> conditions.
> IMO, This case could essentially be solved at the application layer.
> (Timestamps generated by the end point devices, correlated to the
> destination device's clock.)
>
> Is it really hard real time? IOW, you are doing some kind of process
> control?
> Sorry, but ethernet being what it is does not provide any way to
> assure delivery times. That's why there tends to be distributed
> process control. Put as much intelligence as needed at the point of
> control.
If I can get verifiable synchronization points *reasonably*
often, I can calibrate the local oscillator to track time
*between* those synchronization points. Then I just have to
characterize my local oscillator and controls wrt the
implied accuracy and precision of the synchronization means.
>> Note that in days past, this was a lot easier to
>> achieve as broadcasts tended to *truly* be broadcasts,
>> propagation delays were simply K * distance * speed of
>> light, etc.
>
> Only if you happened to be on the same wire. Since hubs and bridges,
> it has never been that simple.
Hubs, for the most part, were "fixed delays". And, all ports
saw the same data at the same time. (contrast with switches
that store and forward with elastic stores)
Dombo
> Is it really hard real time? IOW, you are doing some kind of process
> control?
> Sorry, but ethernet being what it is does not provide any way to
> assure delivery times. That's why there tends to be distributed
> process control. Put as much intelligence as needed at the point of
> control.
For real-time applications one could consider ethercat.
larwe
> So, how *do* you achieve fine-grained synchronization
> nowadays? What is *practical*? And theoretically
> *achievable* (without an a priori characterization
> of the network infrastructure and topology)?
To summarize your question: If nothing is known, or can be assumed,
about the network, what is the best performance I can achieve? The
solution space to such a question is unbounded. Tomorrow your customer
may upgrade some haul of the network from 10bT to fiber optic. He may
perform a firmware upgrade on a router somewhere. Someone in the
office may accidentally install a virus while downloading pornography,
it may propagate through the network, and packet losses may jump up
through the roof while your office is firehosing Viagra spam into the
outside world. Someone on a low-bandwidth branch of the network may
copy a big file into the server and degrade everything on that branch
for half an hour.
To provide /guarantees/ you must constrain the network.
To provide statements of /typical performance/ you must benchmark in
typical scenarios.
Without constraints one may hypothesize a scenario to break any
implementation.
Vladimir Vassilevsky
D Yuniskis wrote:
> I want to know what is *possible* before even considering various
> communication media.
Anything is possible. Providing everything is done right, snake networks
achieve picosecond level of accuracy over 100M Ethernet. However, this
doesn't work with WiFi because of unpredictable delays.
BTW, I've done project where I synchronized a network with microsecond
accuracy over voiceband wireless link.
larwe
> D Yuniskis wrote:
> > I want to know what is *possible* before even considering various
> > communication media.
>
> Anything is possible. Providing everything is done right, snake networks
I would not have bothered to reply if I had noticed who the OP was.
All his posts seem to follow the general pattern "Let us hypothesize a
universe constructed entirely of cheese. What is the possibility in
such a universe of being eaten by a purple dragon between the hours of
three and four on a Wednesday afternoon?"
D Yuniskis
> D Yuniskis wrote:
>
>> I want to know what is *possible* before even considering various
>> communication media.
>
> Anything is possible. Providing everything is done right, snake networks
> achieve picosecond level of accuracy over 100M Ethernet. However, this
> doesn't work with WiFi because of unpredictable delays.
>
> BTW, I've done project where I synchronized a network with microsecond
> accuracy over voiceband wireless link.
Was the existing network tweeked in any way? I.e., did you
layer an application on top of an existing COTS network
*without* any special constraints placed on it? If not,
what did you have to "retrofit" in terms of equipment
and/or topology? I.e., how painful/expensive was it for
the customer to make this adaptation?
If the network architecture/implementation was designed
intentionally for this application, what penalties/constraints
were imposed? What were their costs (i.e., what was "traded"
for those constraints)?
I've looked at a proprietary wireless approach using a timing
beacon and hardware designed specifically for the application
and should be able to get O(50ns) by calibrating the receivers
in manufacturing and just running things open loop. But, I
haven't look at temperature effects on that accuracy and
atmospheric effects (the wireless approach tends to suggest
deployment over considerably larger areas -- square miles -- and
I don't want to trade one set of problems for yet another :<
"Gee, why can't we use this to replace our XYZ product, too?")
The wired approach silently and implicitly limits expectations
(deploying miles of wire has a significant cost associated with
it :> )
Vladimir Vassilevsky
Don,
I am consultant. If you have particular project, I can work on it.
Contact is at the web site.
Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com
D Yuniskis
Vladimir Vassilevsky wrote:
>
> I am consultant. If you have particular project, I can work on it.
> Contact is at the web site.
I'll run this by client (after New Year's). Your location (KC)
may be a problem for them (I'm sure you've dealt with that
"issue" in thepast... seems like most consultants have! :< ).
(BTW, is "Stevenson's" still in business? I can't recall if
they were in KC, MO or KC, KS)
I may try some approaches on a personal project just to see how
well they work "at no risk" (cf my "Bottom feeding..." posts
elsewhere in this newsgroup). Timing constraints there aren't
quite as important (since most folks have "tin ears" :> )
Thanks!
--don
Ed Prochak
Well said. Thanks.
Dan Henry
wrote:
I remember enjoying Mr. Yuniskis' posts and the occasional interaction
with him in the mid '90s, probably more on comp.realtime when it had a
pulse. There are few folks I admire (you know who you are) and he's
one of those. I don't remember when, but he went silent. Maybe I
wasn't looking in the right places, but to me, he'd fallen off the
map, or died, or something, based on the sudden silence. It's nice to
see that he's back, if it's really him.
FWIW, I hold Don in high regard now from my perception of him then
(again, the mid '90s). I understand what you're saying, but in my
opinion, he deserves more respect than you are giving him, but your
respect is yours to give and I respect that. Hey, you're a winner
too!
--
Dan Henry
Leon
XMOS chips and the XLinks that they use for inter-core and inter-chip
comms are completely deterministic:
http:www.xmos.com
Leon
Paul Carpenter
0cacf4...@d20g2000yqh.googlegroups.com>, leo...@btinternet.com
says...
Which has nothing to do with
"embed throughout the network fabric"
unless the network fabric is VERY small.
Obviously, you use full TCP/IP between these chips so that you can run
"things like NTP *try* to quantify this skew"
--
Paul Carpenter | pa...@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/> PC Services
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.gnuh8.org.uk/> GNU H8 - compiler & Renesas H8/H8S/H8 Tiny
<http://www.badweb.org.uk/> For those web sites you hate
Leon
wrote:
> In article <ec8c6001-f408-43d9-97fc-
> 0cacf4201...@d20g2000yqh.googlegroups.com>, leon...@btinternet.com
They have Ethernet connectivity as well, with TCP/IP stacks available.
They are fast enough to implement Ethernet in software.
Leon
Paul Carpenter
8ec7d5...@o28g2000yqh.googlegroups.com>, leo...@btinternet.com
says...
Whoosh...
Leon <---------------------------------------> point
E. Douglas Jensen
wrote:
>On Dec 23, 12:08�pm, D Yuniskis <not.going.to...@seen.com> wrote:
Don is right. It is possible to (and people do) build network-
based systems that have highly predictable timeliness, and
in which it is possible to do fine grained synchronization.
You have to make and adhere to an appropriate set of
strong presumptions about everything in the system and its
applications. Kopetz writes a lot about his approach, and
the AN/BSY-1 submarine signal processing system is one
that I worked a lot on. But most network-based systems
are too dynamic for fine grained synchronization; there
is a vast body of literature about synchronization in
dynamic distrobuted systems.
A bit more abstractly, network-based systems may be dynamic
in various ways -- e.g., task arrivals and durations, resource
contention and overloads, network latencies and bandwidths,
changing topology and end-node reachability, partial failures, etc.
You gave some good examples; mine are all in the military combat
context. Of course the Internet has some of these dynamics
network-centric warfare has them all.
Regardless, some of these systems include activities having
mission-critical and safety-critical (in the sense that people may
die, not in the usual DO-178B certification sense) time constraints,
and the need to resolve contending activities such that these
time constraints are collectively satisfied as best as circumstances
allow in the context of a specific application. Guarantees are
impossible in dynamic systems; but formal assurances of performance
(e.g., timeliness, predictability) are not.
Non-deterministic time-critical systems require non-deterministic
resource management concepts and techniques for specifying,
designing, implementing, and evaluating them. Obviously this cannot
be accomplished with traditional real-time concepts and techniques.
But outside of the real-time computing domain, there is a great
deal of theory and practice for non-deterministic systems -- c.f.
scheduling theory (Pineto wrote the best textbook), and decision
making under uncertainty -- some of which has been successfully
applied to to achieve system timeliness that is acceptably optimal
with acceptable predictability, as defined by the application.
Fortunately these dynamic network-based time-critical systems
normally operate in time frames that are a few orders of magnitude
longer than the ones that traditional real-time systems operate in.
This allows time to perform the necessarily more complex
resource management algorithms.
Doug Jensen
www.real-time.org