Problems with JBT

[Klaus]

Hi, 

for quite some time, I am using JBT to gather data. For this purpose, it is running on a AWS micro-instance.

Once in a while I found it stuck (actually , but was not sure where it was coming from). Usually the happened also after a 

full week of data.

However, today it happened again and I found some strange things (I start recording usually some time Sunday evening and let it run till Friday). Perhaps someone has an idea where it might come from. The fact is, it seems not data itself gets stuck only the price information is kept the same: the volume does continue to change.

This is how the end of the recording looks like: then there is a 30sec gap where I restarted JBT.

100112,101425,-11.6,1.28335,14

100112,101426,-11.6,1.28335,6

100112,101427,-11.6,1.28335,0

100112,101458,9.95,1.29335,0

100112,101459,9.16,1.29335,2

100112,101500,8.24,1.29335,1

The gap is not due to a change in the market, but because the data before was no longer correct (btw, this is the EUR)

This is the part of the data where it seemed to happen:

100112,001556,-9.99,1.2833,0

100112,001557,-4.27,1.28325,0

100112,001558,-2.62,1.28325,0

100112,001559,-11.47,1.28335,16

100112,001600,-9.75,1.2833,1

100112,001601,-10.32,1.2833,0

100112,001602,-10.26,1.2833,0

100112,001603,-13.42,1.28335,1

100112,001604,-13.61,1.28335,0

100112,001605,-12.64,1.28335,0

100112,001606,-12.04,1.28335,0

100112,001607,-11.83,1.28335,0

100112,001608,-11.71,1.28335,0

100112,001609,-11.71,1.28335,0

100112,001610,-11.71,1.28335,0

100112,001611,-11.71,1.28335,0

100112,001612,-11.93,1.28335,0

100112,001613,-11.98,1.28335,0

100112,001614,-11.04,1.28335,7

the price keeped changing until 001602, then was stuck - as far as I could see in the file all the way until I restarted.

The following is an excerpt from the Eventreport file - it contains the end before the restart (the were no further events recorded)

09/30/12 23:59:05.196 IB API 2103: Market data farm connection is broken:usfuture

09/30/12 23:59:21.450 IB API 2104: Market data farm connection is OK:usfuture

10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please empty deep book contents before applying any new entries. (for id 3)

10/01/12 00:14:04.896 My JBookTrader Market data for book 3 has been reset.

10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please empty deep book contents before applying any new entries. (for id 2)

10/01/12 00:14:04.896 My JBookTrader Market data for book 2 has been reset.

10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please empty deep book contents before applying any new entries. (for id 1)

10/01/12 00:14:04.896 My JBookTrader Market data for book 1 has been reset.

10/01/12 00:14:04.896 IB API 2103: Market data farm connection is broken:usfuture

10/01/12 00:14:42.166 IB API 2104: Market data farm connection is OK:usfuture

10/01/12 00:15:48.860 IB API 1100: Connectivity between IB and Trader Workstation has been lost.

10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and Trader Workstation has been restored - data maintained.

So, it looks like connectivity was lost and restored, but from this point forward no price updates were recorded..

Any idea what is going on, would be helpful.

Cheers 

  Klaus

[Klaus]

Just had the same again. This time at 3:25 Monday morning. 

Registered a temporary connection loss in the event report. Book and price data did not change for any of the collected data from thereon, but volume does.

In addition, I found the following messages in the terminal window. Perhaps somehow can make sense of all this?

(I include one message before and after for context)

---

03:08:11:547 JTS-CCPPing-269: Warning: not received heartbeat at 20904000 elapsed:11000

03:25:30:728 JTS-PostAuthenticate-513: pre-logon message successfully sent

03:25:36:225 AWT-EventQueue-0: Warning: hot restart not equal to quick restart

03:25:36:226 AWT-EventQueue-0: Unsubscribe MD before routing table rerequest

03:25:36:227 AWT-EventQueue-0: Unsubscribe HMDS before updating routing table

03:25:36:227 AWT-EventQueue-0: Could not find route table entry for path:exch=NEWS secType=STK currency=USD dataType=DayChart

03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available for the exchange 'NEWS' and the security type 'Stock'

03:25:36:228 AWT-EventQueue-0: Could not find route table entry for path:exch=RTRSFND secType=STK currency=USD dataType=DayChart

03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available for the exchange 'RTRSFND' and the security type 'Stock'

03:25:36:228 AWT-EventQueue-0: Could not find route table entry for path:exch=RTRSFND secType=STK currency=USD dataType=DayChart

03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available for the exchange 'RTRSFND' and the security type 'Stock'

03:25:36:229 AWT-EventQueue-0: Reconnected!

03:25:37:151 JTS-ForwardRouting-HISTORICAL_DATA-532: handleCompatibleRouting HISTORICAL_DATA compmode=true ready=false farm=ushmds

03:25:37:939 JTS-ForwardRouting-MARKET_DATA-536: handleCompatibleRouting MARKET_DATA compmode=true ready=false farm=usfarm

03:26:36:399 JTS-Fuse-sd-533: Reset transferrable socket:ushmds gw1.ibllc.com:4000

03:26:37:187 JTS-Fuse-sd-537: Reset transferrable socket:usfarm gw1.ibllc.com:4000

05:20:18:052 JTS-CCPPing-516: Warning: not received heartbeat at 28813000 elapsed:11000

------

Data that is collected is only for the three instruments: EUR, NQ, ES

(thus I am puzzled also by the stock request above)

Cheers

  Klaus

[Eugene Kononov]

Thanks for reporting the problem, Klaus. With respect to the midnight market data interruptions, this has always been the case, as IB servers restart at that time. The expected behavior is that once the restart is completed, the market data resumes and TWS and JBT would resume receiving the data stream without the need to restart. However, I noticed that occasionally this fails. My remedy is restart both TWS and JBT every day in the morning before regular trading starts. With respect to the interruptions during the other times of the day, I belive it all has to do with the quality of your network connection. The better quality, the less frequent is the loss. In my experience, both JBT and TWS recover from these interruptions (except for the occasional non-recoverable midnight resets). The typical pattern in the log is the message "market data lost", followed by in a minute or two by message "market data restored". Now, with respect to the stock data that you see in the TWS log, it's probably your standard subscription. I'd suggest visiting your "market subscriptions" page associated with your account on the IB we site.

--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.

To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/O4d4rFkJ3F0J.

To post to this group, send email to jbook...@googlegroups.com.
To unsubscribe from this group, send email to jbooktrader...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.

[BJ]

I'm running JBT on an AWS Small instance and tried switching to a Micro instance over the weekend. I use IB Gateway instead of TWS (it doesn't shutdown itself and uses less memory). After starting IB Gateway on the micro instance it used all the memory. I started another IB Gateway instance (my JBT sends orders to two different IB accounts). The result was, that while Ubuntu was freeing the memory for the second instance, the first one was shutdown by the OS. 

I then started only one IB Gateway and JBT and monitored the memory. Over the weekend without any market data processing the memory was always 99% full. I didn't feel comfortable running JBT in such a low memory environment so I switched back to a small instance.

What I'm trying to say is switch to a small instance for a week and see if that solves your problems.

[Klaus]

Dear nonlinear, 

what irritates me most is that it partially recovers: some data (for the same instrument) is reconnected and correctly logged (volume = at least I assume it is correct, it is at least reasonable), while price is not correctly logged. To me this looks more like a SW issue, as both types of data are coming from the same stream. 

Regarding network connection, I assume there is probably few better connections than AWS. It certainly is way ahead of private

connections I dealt with.  It might also be interesting to note that I do not use TWS, but IB Gateway.

Cheers 

   Klaus

[Klaus]

Dear BJ 

thanks for reacting and reflecting on this. 

I am not sure what environment you are using. I admit, your description is not fully clear to me (e.g., the part of JBT sending to two differnt accounts - this probably implies you are using an adapted version as this is no standard JBT behavior).

I use Ubuntu, on top of this I even use the full window interface and I then start JBT and IB-Gateway in it. 

What I then have is the following: (data with some additional shells open and all this)

579072 (out of 609480K) used, 31400K free, 5312 buffers and 0 swap usage (103824K cached)

This looks a pretty comfortable setup to me, memory-wise.

Nevertheless in a VM environment there might occur issues like once the VM is migrated, this might lead to an interruption of the 

data connection (but this can also happen with larger VMs of course).

So, I am not concerned by the interruption per se. This actually happens several times a week. As nonlinear wrote: this is  s.th. JBT is supposed to deal with. The problem here is that it recovers the connection, but somehow not fully, which would be difficult to explain from a connection or hardware side, but seems to me (without deep JBT-knowledge) more like a SW-issue.

Of course, I can now try it in a standard VM. - probably I will for next week. But actually this is does not prove much. This runs for 10months now in the microinstance, including all updates and everything. Once in a while I get disconnects, but it was only by chance (I assume) that it now happened in two consecutive weeks. So, if there is no problem with a standard instance, this does not mean it avoided a problem that would have happened with a micro, as this is ok most of the time as well. 

I actually also did not think, about it stalling to much. Glitches happen. - What made it so surprising to me is that some data

for the Future is still correctly tracked, while other not at all: this seems very weird to me.. - and I am looking for a good hypothesis to understand this.

Cheers

 Klaus

[Eugene Kononov]

One thing that may be worth trying is for JBT to detect when the price feed stops, and when this happens, unsubscribe from the market data and subscribe again. This should be just a few lines of code. Klaus, have you tried pairing both TWS and IBGateway with JBT? Does it affect the frequency and the severity of the market data interruptions? Like BJ, I've been trading using JBT with IBGateway, and as I mentioned before, I don't see any unrecoverable interruptions in the market data feed. Then again, I restart IBGateway and JBT every morning, which I don't see as a big deal, but I also ackwnoledge that it would be nice to be running continuously. Actually, you do have to restart once in a while, simply because the future's contract expires, and there is no auto-rolling in JBT.

To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/JlHTQ1N14wAJ.

[Klaus]

Dear Eugene

I usually do not use TWS for two reasons: it needs more memory and while it currently is not a problem, TWS is simply heavier. 

Further, TWS auto-logsout after 24h and while there are ways around it, they are all rather messy. 

If you say, it is just a few lines of code: if you point me to the critical aspects, I can probably fix it on my version, dry-run it for some weeks and if problems seem solved, this would just go in the regular version. How does this sound?

Cheers

 Klaus

[Judson Wilson]

I thought price was not coming from the same stream as volume, i
thought it comes from the book stream (isn't it the midpoint of the
best bid/ask on the book?)

> https://groups.google.com/d/msg/jbooktrader/-/tEYk3_95L5EJ.

[Eugene Kononov]

Correct, Judson.

The book data is requested via this line in TraderAssistant.java:
socket.reqMktDepth(ticker, contract, 10);
The corresponding callback method in Trader.java:
updateMktDepth();
This is where the price is determined as the midpoint between the current best bid and current best ask.

The volume data is requested via this line in TraderAssistant.java:
socket.reqMktData(ticker, contract, "", false);
The corresponding callback method in Trader.java:
tickSize()
Price data (such as the "last price") can also be handled here, but JBT only uses the volume data here.

The subscription to market depth and volume happens here:
        if (!subscribedTickers.contains(ticker)) {
            subscribedTickers.add(ticker);
            socket.reqContractDetails(strategy.getContract().m_conId, strategy.getContract());
            eventReport.report(JBookTrader.APP_NAME, "Requested contract details for instrument " + instrument);
            socket.reqMktDepth(ticker, contract, 10);
            eventReport.report(JBookTrader.APP_NAME, "Requested book data for instrument " + instrument);
            socket.reqMktData(ticker, contract, "", false);
            eventReport.report(JBookTrader.APP_NAME, "Requested market data for instrument " + instrument);
        }


What I suggested to Klaus is that we can detect the loss of market data via the error() method, and resubscribe if this happens. Klaus, I'll see if I can make that code change and I'll send it over to you to test. Are you OK with it?

[Klaus]

Dear Eugene, Dear Judson, 

first thanks to all for clarifying this. This was puzzling me and now I understand how it can technically

happen. I was mistaken to believe it would be the same stream.

Certainly, if you give me a proposed patch, I would be happy to analyze the relevant effects.

Cheers

 Klaus

[Dyno Brium]

I have similar problem when I tried amazon EC2 back in August. I have micro instance, windows 2008 with IB gateway. The update of price information stopped for NQ, volume update continued. ES was still good at that time. but it could not get price update several hours later. IB gateway was good. I restarted JBT again and got normal data. I gave up the idea of running JBT more than a day after the trial. I wonder if some one can actually run JBT for several days without any incident.

I think that the data request connection between JBT and IB gateway got lost some how. Resend the data request should solve the problem.

[nonlinear]

Klaus, let's go back to your original post where you posted this log:

09/30/12 23:59:05.196 IB API 2103: Market data farm connection is broken:usfuture

09/30/12 23:59:21.450 IB API 2104: Market data farm connection is OK:usfuture

10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please empty deep book contents before applying any new entries. (for id 3)

10/01/12 00:14:04.896 My JBookTrader Market data for book 3 has been reset.

10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please empty deep book contents before applying any new entries. (for id 2)

10/01/12 00:14:04.896 My JBookTrader Market data for book 2 has been reset.

10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please empty deep book contents before applying any new entries. (for id 1)

10/01/12 00:14:04.896 My JBookTrader Market data for book 1 has been reset.

10/01/12 00:14:04.896 IB API 2103: Market data farm connection is broken:usfuture

10/01/12 00:14:42.166 IB API 2104: Market data farm connection is OK:usfuture

10/01/12 00:15:48.860 IB API 1100: Connectivity between IB and Trader Workstation has been lost.

10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and Trader Workstation has been restored - data maintained.

Between 23:59:05 and 00:16:16, there is some activity indicating IB restart processes kicking in, which ultimately end at 00:16:16. The message says "Connectivity between IB and Trader Workstation has been restored - data maintained", which means that all market data subscriptions are still valid. But you are saying that the price data never changed after 00:16:16. Correct?

[Klaus]

Dear Eugene, 

right. I did not check every line, but it was exactly the same price at the end and all checks in between gave the same data

(It had run by this time for 10 hours). It was only afterwards that I found that exactly at the same time these events happened.

Klaus

[Eugene Kononov]

One important thing I noticed in your log is that the message:

10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and Trader Workstation has been restored - data maintained.

was NOT followed up with this the message:

IB API 2104: Market data farm connection is OK:usfuture

So, my plan for implementing the "re-subscribe" functionality is to detect this condition and resubscribe.

To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/VQInXSoseSUJ.

[Klaus]

Dear Eugene, 

right. This was the last event that was registered. Then till I stopped it 10 hours later nothing happened. 

Regarding the re-subscribe I am wondering what this means: is the instrument also reselected?

I am wondering because of the following case:

- out of a type of contract (e.g, ES) always the most liquid one is chosen. Now let's assume

  a position is held, while the problem occurs: if the resubscribe happens (and the very unfortunately it is exactly when 

   the handover between instruments occurs, later the other duration of ES is sold, when the attempt is made to close

   the position. Is this a relevant scenario?

Cheers

 Klaus

[Eugene Kononov]

Regarding the re-subscribe I am wondering what this means: is the instrument also reselected?

I am wondering because of the following case:

- out of a type of contract (e.g, ES) always the most liquid one is chosen. Now let's assume

  a position is held, while the problem occurs: if the resubscribe happens (and the very unfortunately it is exactly when 

   the handover between instruments occurs, later the other duration of ES is sold, when the attempt is made to close

   the position. Is this a relevant scenario?

 

Yes, that's relevant, so we'll have to deal with it eventually. At this point in time, however, I'd like to just be able to see if the re-subscribe feature would work.

[Klaus]

Hi, 

started the new week and it already had a lot of broken connections this morning. 

(never experienced so many), but reliably always had the market data farm connection is ok message next.

Only exception:

10/15/12 06:30:40.607 IB API 2103: Market data farm connection is broken:usfuture

10/15/12 06:31:19.531 IB API 1100: Connectivity between IB and Trader Workstation has been lost.

10/15/12 06:31:36.420 IB API 1102: Connectivity between IB and Trader Workstation has been restored - data maintained.

10/15/12 06:31:37.416 IB API 2104: Market data farm connection is OK:usfuture

but even here, up and running. 

So, this might be another indicator: in few situations the 2104 does not happen within a minute or two and then

there is no answer again.. 

(However, I am somewhat concerned, checking the other broken data farm connections, there were more where it needed about a minute to reestablish the connection.)

Cheers

 Klaus

Am Montag, 1. Oktober 2012 10:42:52 UTC-4 schrieb Klaus:

[Klaus]

so, it seems, this is not the culprit. 

It just had the same problem again and the last entries in the event log are:

10/16/12 00:14:06.167 My JBookTrader Market data for book 1 has been reset.

10/16/12 00:14:06.167 IB API 2103: Market data farm connection is broken:usfuture

10/16/12 00:14:10.752 IB API 2104: Market data farm connection is OK:usfuture

10/16/12 00:16:31.996 IB API 1100: Connectivity between IB and Trader Workstation has been lost.

10/16/12 00:17:18.089 IB API 1102: Connectivity between IB and Trader Workstation has been restored - data maintained.

So, according to the eventreport everything should be fine - only it isn't..

Klaus

[Eugene Kononov]

Klaus,
the message

2104: Market data farm connection is OK:usfuture

should appear *after* the message

1102: Connectivity between IB and Trader Workstation has been restored - data maintained.

In your log, this is not the case: there is no follow up message 2104 after 1102. What I suggested before is that we should detect this condition and resubscribe. Does it make sense?

--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.

To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/qipYEMsGO2UJ.

[Klaus]

Eugene, 

I did not know this detail. So, if you believe that this actually reinforces the problem, 

then I am certainly willing to test run any patch that aims to address it..

Klaus

[Eugene Kononov]

Klaus (and everyone who would like to test this):

Attached is the archive with 3 java files where the "resubscribe" functionality is implemented. Here is how this works. When connectivity between IB and TWS is restored (after the loss of connectivity), JBT spawns a thread which monitors the market data status. If this monitoring thread detects that market data became active within 60 seconds from the moment of re-connection, it doesn't do anything (that is, JBT works the way it works now). If market data is not active after 60 seconds from the moment of re-connection, the monitoring thread unsubscribes all running strategies from the market data, and subscribes them to the same market data again. 

Please give it a spin to see if this works as you'd expect. I'll be running the same version as well. From my past experience, I would need to run JBT 24/7 for about a week to encounter this situation (since in almost all the cases, both JBT and TWS recover nicely from the connect/disconnect events).

Thanks for testing!

To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/75cC8vJa-FwJ.