Improving multi-user consistency

[nonlinear5]

The long time JBTers are aware of this problem: when multiple people
run the same strategy in the same time frame, their results may be
different. For the benefit of everyone, I'd like to explain why this
happens, and perhaps someone would come up with an idea to improve the
consistency.

When JBT runs in the "trade" and "forward test" mode, it's driven by a
timer which fires 1 time per second. Every time the timer fires, JBT
calculates the depth balance as the midpoint between the lowest and
the highest depth balance observed during that second. This midpoint
balance is what gets fed to the indicators, which ultimately generate
trading signals. This approach works fairly well. We've run multiple
multi-user tests in the past, and I am also regularly checking my
results with another JBTer who runs the same strategies. Most of the
time, the results match well, with occasional exceptions. Here is why
these exceptions occur:

The timer uses the computer clock, and obviously, everyone's clock is
different. When user A samples 1-second interval, it may be, say,
10:00:05.200 to 10:00:06.200 real time period. By "real time", I mean
the most precise, atomic time, which is the same for everyone. At the
same time, user B sample could be 10:00:05.900 to 10:00:06.900,
measured by the same atomic clock. So, users A and B are "out of
phase" by about 700 milliseconds. As a consequence, their min/max
depth balance observations would differ by small amount. Now, imagine
that some strategy triggers a long trade when the indicator value
exceeds 100. Because of the small differences, the indicator for user
A may reach 100.01 and trigger a trade, while the same indicator for
user B may reach 99.99 and drop off, never triggering a trade. An
additional source of the differences is the PC clock drift, which may
cause JBT to sample overlapping periods, or to have gaps in the
samples.

Here are a couple of ideas that I have to improve the multi-user
consistency:

1. Instead of the midpoint depth balance, use a 1-second average (or
an exponential average) of the depth balance.

2. Instead of using the PC clock, obtain the atomic time from a time
server (such as the US Naval Observatory time server), and trigger
depth balance sampling based on the changes in that atomic clock. This
would involve continuous polling of the time server, multiple times
per second. This is easy to do in Java, but it may not be reliable, as
the server may go down. The server may also deny service, if the
requests are deemed to be too frequent.

If you have any other ideas, feel free to share with the group.

[Paul]

From your description, it sounds like the inconsistencies are caused
by timers starting off out of phase (or drifting out of phase),
resulting in different sampled depth balances.

One solution might be to calculate all possible depth samplings, and
take the average for each second. So for each market update that you
get (100's per second), you could calculate a 'rolling' depth balance
based on all the updates that arrived in the previous second. Then
when the timer fires, this mean rolling depth should be closely
matched even if timers are out of phase between computers.

Obviously, the 'chunkier' the order sizes are in the time interval of
interest, the more sensitive the depth sample will be to the precise
sampling time (even when the sample is 'rolling'). Then the only
solution is to estimate that variability (i.e., what is the
distribution of depths when sampled at a random phase), and adjust
your strategy so it only trade when the signals are above this known
noise.

[ecthx]

I don't believe that any time server would allow you to check the time
that often, it would probably consider it as a DoS attack or some
sort of abuse.
We could try to use NTP to synchronize our clocks and then record
snapshots based not on regular interval but on PC clock.
This may not be perfect solution but I use NTP and it keeps my
computers
clocks synchronized to few hundred microseconds. It can supposedly
maintain
time to within 10 milliseconds over the internet and it takes care of
the clock drift too.
We might do a multi-user test since all it requires is NTP client and
small change to the code.
NTP client for windows:
http://www.meinberg.de/english/sw/ntp.htm#ntp_nt_stable

[phili...@gmail.com]

Same problem occurs with JSystemTrader, if you compare live records of
the market with historical download on the same periode, some
differences happen.
I try to synchronise my PC with an atomic time server every 15 min. I
still have differences.
Would it be better to use IB time server Instead of using an atomic
time server?

[Martin Koistinen]

Does IB offer one?  If so, I've been unable to locate it.

[MKoistinen]

Sounds like we need a 'reliable' source of time that all JBT users
could 'sync' to. Why not use IB here?

JBT could request a 15 minute or 1-hour bar on the XYZ, for example.
And, on each received bar, JBT would re-sync its 1-sec interval for
market book data. Certainly IB guarantees that everyone subscribing
to a 1 hour bar on XYZ receives the same bar (although there will be
slight variances in how quickly it is sent). If I understand
correctly, at the moment, JBT relies on the system clock of the host
computer, this should certainly get us all much more in sync.

[ecthx]

I completely forgot that we had a few of these discussions in the past
and the general consensus was that it would not work. Even if our
clocks are in perfect
sync it still takes different amount of time for data packets to reach
every user
and therefore perfect synchronization is not possible.

[Martin Koistinen]

Perfect synchronisation is of course, not possible, but right now we have arbitrarily defined second boundaries and I would expect a significant improvement if we all had at least the same yard stick.

[ecthx]

I also believe there is a room for improvement and since we never
actually
tried to synchronize our clocks we might do some testing.
As I said before NTP is probably the best way to go if we want to do
this.
Some people mentioned Dimension4 software in the past but it's a very
basic tool
and it's only slightly better than default Windows time tool. NTP not
only takes into
account variable network latency but it also deals with clock drift
which on Windows machines
can be very large.

On Aug 20, 2:11 pm, Martin Koistinen <mkoisti...@gmail.com> wrote:
> Perfect synchronisation is of course, not possible, but right now we have
> arbitrarily defined second boundaries and I would expect a significant
> improvement if we all had at least the same yard stick.
>

[Dyno Brium]

On Wed, Aug 19, 2009 at 10:53 PM, nonlinear5 <eugene....@gmail.com> wrote:

Here are a couple of ideas that I have to improve the multi-user
consistency:

1. Instead of the midpoint depth balance, use a 1-second average (or
an exponential average) of the depth balance.

Averaging can alleviate the discrepancy. I have tested with RAW data last year applying artificial time delay of hundreds of milliseconds. The results showed that averaging has better consistency than high-low method.

 

 

2. Instead of using the PC clock, obtain the atomic time from a time
server (such as the US Naval Observatory time server), and trigger
depth balance sampling based on the changes in that atomic clock. This
would involve continuous polling of the time server, multiple times
per second. This is easy to do in Java, but it may not be reliable, as
the server may go down. The server may also deny service, if the
requests are deemed to be too frequent.

This probably is not going to help. IB wouldn't send you exactly the same data at exactly the same time. I have compared data collected from two computers and found that the data sequences are correct most of the time, but the updates were truncated randomly. So if you use any time-based averaging method, you are not going to get perfect synchronized. You probably can get perfect synchronized with event-based averaging method, such as averaging every update, or start a new averaging sequence after some identifiable events, such as the change of best bid/ask.

[Eugene Kononov]

I also believe there is a room for improvement and since we never
actually
tried to synchronize our clocks we might do some testing.
As I said before NTP is probably the best way to go if we want to do
this.
Some people mentioned Dimension4 software in the past but it's a very
basic tool
and it's only slightly better than default Windows time tool.  NTP not
only takes into
account variable network latency but it also deals with clock drift
which on Windows machines
can be very large.

Ok, before we involve other people, let's run the consistency test tomorrow just between you and me. I've installed the same NTP client, and I am running JBT version as it's in SVN. Let's record the ES data tomorrow, and compare our data files at the end of the day. Ok?

[Eugene Kononov]

ecthx, can I take a look at your ntp.conf file? I simulated the clock drift by setting my PC clock 10 seconds back, and NTP doesn't seem to do anything about it, so I assume I am missing something in the configuration.

[Eugene Kononov]

Oh, now I see the effect. My clock has been synced, it just took a few minutes. Please send me your ntp.conf anyway.

[ecthx]

The first time you start ntpd it looks for ntp.drift file which
contains information
about your clock frequency. If it doesn't find this file it will go
into a measuring
mode for about 15 minutes and then it will adjust the time. Other
thing is that whenever
ntp finds that your clock is set forward it will not just set it to
correct time but it will
sort of 'drag' the clock until it matches the correct time which also
takes a while.

I can run the test tomorrow but can you clarify one thing for me.
Let's say I start two JBT instances
on two computers that have perfectly synchronized clocks. Will both
data files be mostly
identical even though forward-test on the second instance was started
0.5 seconds after the first one?

On Aug 20, 3:39 pm, Eugene Kononov <eugene.kono...@gmail.com> wrote:
> Oh, now I see the effect. My clock has been synced, it just took a few
> minutes. Please send me your ntp.conf anyway.
>

[ShaggsTheStud]

My question is "does it matter?"  Just because our data all looks the same, doesn't make it any better.   It's all correct, just a different sampling.  If it effects the outcome if your strategy that much, then your strategy has a problem.  In fact, it would be better if everyone had different versions of the same data, and these seperate data sets could be used to test how sensitive the strategy is to the sampling.

[Eugene Kononov]

I can run the test tomorrow but can you clarify one thing for me.
Let's say I start two JBT instances
on two computers that have perfectly synchronized clocks. Will both
data files be mostly
identical even though forward-test on the second instance was started
0.5 seconds after the first one?

I didn't think about this initially, but now that you mentioned it, yes, two computers with pefectly synchronized clocks would not sample the same 1-second periods if one started 0.5 seconds after another. That's because the line

scheduler.scheduleAtFixedRate(new SnapshotHandler(), 0, 1, TimeUnit.SECONDS);

in StrategyRunner.java starts the time immediately. This should be easy to fix by changing the second parameter, which is "initialDelay", to the value which would start the timer "on the second":

long initialDelay = 1000 - System.currentTimeMillis() % 1000;
        scheduler.scheduleAtFixedRate(new SnapshotHandler(), initialDelay, 1000, TimeUnit.MILLISECONDS);

Can you make that code adjustments for tomorrow's test?

[Eugene Kononov]

My question is "does it matter?"  Just because our data all looks the same, doesn't make it any better.   It's all correct, just a different sampling.  If it effects the outcome if your strategy that much, then your strategy has a problem.  In fact, it would be better if everyone had different versions of the same data, and these seperate data sets could be used to test how sensitive the strategy is to the sampling.

I actually have the same take on this. However, it would be nice to know if there is anything else, besides the sampling, which may cause differences in the trading performance. This could include the possible differences in how IB disseminates market information. By eliminating the sampling differences, the OTHER differences would come to light.

[nonlinear5]

ecthx, I've set my NTP config to be the same as yours.

[ecthx]

After making adjustments to the code I did some preliminary testing
and it didn't
look too good. When I ran the test on two Linux machines two data
files were almost
identical (3 different lines over 30 minute period). During WindowsXP
vs Linux test discrepancies
were much larger however. I think it was caused by the fact that I
started the test right after
booting into Windows so maybe clock wasn't synchronized yet. The other
reason for discrepancies
might be the fact that the scheduler fires every 1000 milliseconds and
doesn't
take actual time into account. Because of that every initial time
difference between two clocks would be
replicated over and over.

[ShaggsTheStud]

I don't know if it would help, but I'm interested if setting the priority of the process in windows a tick higher changes anything?

[Ecthx]

I ran the test today and I'm not happy with the results. I guess it's pointless to compare our data
files but I will post them anyway.  File ES_Lin_a.txt was recorded under Linux and ES_Win_b.txt under
Windows XP. One of the screenshots shows differences between two files ( you can also notice few
duplicated timestamps). I also ran the test on two Linux machines for about 30 minutes and got much better results,
on Monday I will try to repeat the Linux vs Linux test. Files ES_Lin_a_2.txt and ES_Lin_b.txt were recorded under Linux.

Here's interesting blog post that I think explains reason behind discrepancies:
http://www.simongbrown.com/blog/2007/08/20/millisecond_accuracy_in_java.html

[nonlinear5]

Ok, I thought about this and here is my plan for the upcoming release:

1. Instead of using the midpoint between the minimum and maximum
balances, 1-second depth balance will be represented by an exponential
moving average of market depth. I have not yet decided what multiplier
to use for that EMA, though. During every second, market depth changes
somewhere between 10 and 70 times, with the average of about 30, so my
guess is that the best multiplier would be half the average number of
updates, i.e, ema multiplier = 2 / (15 + 1). An alternative approach
is to make the multiplier dynamic:
multiplier = 2 / (N / 2 + 1), where N is the number of depth changes
in the last second.
Intuitively, the ema of depth balance is a better representation of
the current depth balance than the midpoint between min and max. I am
also guessing that this new representation will also be much less
sensitive to sampling differences.

2. In the current implementation, the resulting 1-second balance is
rounded to the next integer:
int balance = (int) Math.round((lowBalance + highBalance) / 2d)
In the next release, I'll make it a double, and round it to two
decimal places. This should also improve consistency.

One big downside to all this is that it is unknown how it would affect
the strategies optimized on the data in the old format. That is, if
you have a strategy with a parameter which has a depth balance
threshold as a criteria for entry/exit, the value of this parameter
would probably need to change with the new depth balance
representation. To put it in other words, the previously recorded data
may need to be discarded.

[Martin Koistinen]

This sounds like an indicator to me.  I don't like the idea of effectively smoothing the data beneath the 'user space' in JBT.  It sounds to me like we're placing multi-user consistency higher than everything else -- potentially above profitability even, and I'm not convinced this is in the best interest of JBT.

With Double depth balance, couldn't we obtain a similar result in user-space by using DepthBalanceEMA(period) or similar where ever we require DepthBalance()?  I know it isn't exactly the same thing, but I wonder if there would be any, real practical difference with this approach.  Using this approach would allow a) existing strategies to continue working as they are or perhaps with very tiny changes, b) existing data would be more relevant and c) strategies implementing the smoothed DB might already see better, if not excellent, multi-user consistency.

Even if I haven't completely convinced everyone, can I at least ask that we take this only one step at a time and implement the Double depth balance first and see if we can get the consistency we require before making more fundamental changes that will render existing strategies and collected data irrelevant.

I do like and welcome the idea of going to Double for the depth balance in general though and I would also like to see us retain the total size of the book as well as the balance (or just retain the bid-size and the ask-size so that both total size and depth balance can be recomputed as required).

[Eugene Kononov]

This sounds like an indicator to me.  I don't like the idea of effectively smoothing the data beneath the 'user space' in JBT. 

The existing implementation also performs smoothing. Every second, it takes the minimum and the maximum of the last 30 or so market depths and records the midpoint. The new approach simply replaces that average with the new type of average. I expect these two different averages to be very close to each other, within a fraction of a point.

[nonlinear5]

> I don't believe that any time server would allow you to check the time
> that often, it would probably consider it as a  DoS attack or some
> sort of abuse.

There seems to be a way around this. Instead of a particular time
server, you can specify an NTP server pool. Some of these pools
contain literally thousands of time servers, and they use "round
robin" to select a server when you request time. So, you can request
time multiple times per second without hitting any particular server
too often. I am using Apache NTPUDPClient.java to get time every 100
milliseconds, and it seems to be running smoothly.

[ShaggsTheStud]

every 100ms?  What kind of ping time do you get to these servers?

[Eugene Kononov]

every 100ms?  What kind of ping time do you get to these servers?

About 37ms. As it turned out, requesting time every 100ms is not necessary. Instead, I do it every 500ms, and then Thread.sleep until the beginning of the next second.

[ecthx]

Yeah, I remember the days before the pool was created. Back then the
best solution was to
hand pick few local time servers from closest universities and such,
www.pool.ntp.org makes
it much easier. I don't know where are you going with this
NTPUDPClient.java client but
I think it's better to run ntp daemon on your system and query your OS
clock. During normal operation
ntpd keeps track of the time on few different servers ( around 5 on my
system). It monitors these servers
and tries to determine which one provides the best time based on
network latency and other factors.
Now, I don't know how NTPUDPClient.java works, but if it doesn't take
network latencty into account and
is not picky about the server it is connected to then it probably it
is far from accurate.

[Eugene Kononov]

Ok folks, all my changes are in SVN. In addition to using a new type of average to represent the 1-second depth balances, I've also introduced an NTP timer. Basically, this means that Strategy runner is now driven by atomic clock, which should put us all on the same page. This should also make it much easier to compare our data files in case of discrepancies. You don't have to run any external time sync software.

Before I roll this stuff into next release, I'd like to run a multi-user test on Monday. If you'd like to participate, please follow these steps:

1. Sync your code with SVN. Don't forget to grab a new jar from the /lib directory (commons-net-2.0.jar), and to update your JBT startup script.

2. On Monday, Aug 24, start JBT any time before 8am EDT, and run strategy Test (it's also in SVN) in the "forward test" mode. As a reminder, no actual trade orders would be transmitted in the "forward test", and you should use a real (not simulated) IB account. The parameters for Test are set to make a significant enough number of trades (I estimate it would be around 10), so that our differences, if any, would be amplified. I am not concerned with profitability for this test, only with consistency between multiple users.

3. Any time after 15:25 EDT, please report your results (number of trades and net profit).

We need as many people as we can get for this test, so please participate.

[Eugene Kononov]

Now, I don't know how NTPUDPClient.java works, but if it doesn't take
network latencty into account and
is not picky about the server it is connected to then it probably it
is far from accurate.

NTPUDPClient implements RFC 1305 spec, so yes, all the bells and whistles are in the returned NTP message. I am not making any special adjustments for network latency yet.

[Eugene Kononov]

I found that I was not using NTPUDPClient correctly. The fix is in SVN, so please re-sync if you are participating in Monday's test.

[Eugene Kononov]

I made some more modifications to the NTP clock:

-- Switched from the NTP pool to the <ntp2.usno.navy.mil> time server, courtesy of the U.S. Navy. This a stratum 1 server, which means that its accuracy is 10 microseconds. With the NTP pool, it's much less predictable what we would get.

-- Reduced the frequency of time requests to one per minute.

Please re-sync with SVN if you are participating in Monday's test.

[skunktrader]

It will be interesting to see if synching the time fixes the
fundamental problem that we observed and confirmed earlier -- that the
TWS data stream is NOT the same for each client.

[Eugene Kononov]

Is anyone participating in the test today? Test made the first trade, entering short at 10:02:25.

[ecthx]

EOD results:
4 trades, Net Profit: -119

[Eugene Kononov]

EOD results:
4 trades,  Net Profit: -119

I have 4 trades, net profit -94. Can you please zip up your data and post it (or email it to me). Thanks.

[Ecthx]

I started the test around 8:00am eastern, couldn't start it earlier because
I collect data from Europe at night.

[Eugene Kononov]

Thanks, ecthx. I compared your data file with mine, and it is a huge improvement over the previous experiments. The lines are nicely synchronized with timestamps. The differences between 1-sec prices are virtually nonexistent, while the 1-sec depth balance differences are mostly fractional (as opposed to whole points differences before). I attached a pic showing a representative sample.

Your file has a gap between 13:57:04 and 13:57:09. I assume you lost connectivity during this time. Can you please check your event report to confirm? Other than that, there are no missed or duplicate snapshots.

I am happy with the results, although I wish that more people participated. Perhaps they were deterred by having to sync with SVN. We'll repeat this test after I roll these changes into release 7.03.

[ecthx]

>Your file has a gap between 13:57:04 and 13:57:09. I assume you lost
>connectivity during this time. Can you please check your event report to
>confirm? Other than that, there are no missed or duplicate snapshots.

There's nothing unusual in the log but around that time I was running
few cpu heavy
applications and I noticed a lot of memory paging.

[Eugene Kononov]

There's nothing unusual in the log but around that time I was running
few cpu heavy
applications and I noticed a lot of memory paging.

Ok, yeah, that would explain the gap.

[nonlinear5]

> Now, I don't know how NTPUDPClient.java works, but if it doesn't take
> network latencty into account and
> is not picky about the server it is connected to then it probably it
> is far from accurate.

I've confirmed that NTPUDPClient does indeed account for network
latency, so we all should have nearly perfect time synchronization
with JBT 7.03.

[Gab]

I am happy with the results, although I wish that more people participated. Perhaps they were deterred by having to sync with SVN. We'll repeat this test after I roll these changes into release 7.03.

I was in a northern forest without internet for the last days, but I'll sure participate in your next test.