TorQ-CME

[sergio]

I’ve downloaded TorQ-CME tool from here: https://github.com/AquaQAnalytics/TorQ-CME, followed installation and execution instruction and I managed to import in the “book” table for the hdb the wide book of the ES instrument  for the 1/3/19 trading day (as taken from the CME MDP 3.0 FIX historical data file purchased from https://datamine.cmegroup.com/).

I am concerned about the correct interpretation of the historical file by the tool. Please find below the very first lines of the historical file and the related first rows in the book table of the "hdb" kdb database. Please find my comments as well. Please confirm whether the tool is misinterpreting the FIX protocol or if I'm mistaken. 

1128=9 9=132 35=f 49=CME 34=16039853 52=20190228224500008712862 60=20190228224500000000000 75=20190301 5799=10000000 1151=ES 326=21 327=0 1174=4 10=115
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------
2019.03.01 ESH9 2019.02.28D22:45:00.285000000 `float$()      `float$() `float$()                           `float$()       16039863 16185241 04  
1128=9 9=183 35=X 49=CME 75=20190301 34=16039863 52=20190228224500285182203 60=20190228224500285002187 5799=00000100 268=1 279=0 269=1 48=18720 55=ESH9 83=16185241 270=278675.0 271=1 346=1 1023=1 10=010
1128=9 9=170 35=X 49=CME 75=20190301 34=16039863 52=20190228224500285182203 60=20190228224500285002187 5799=10001000 268=1 279=0 269=4 48=18720 55=ESH9 83=16185242 270=278675.0 286=5 10=212
1128=9 9=183 35=X 49=CME 75=20190301 34=16039928 52=20190228224500993635962 60=20190228224500993501589 5799=10000100 268=1 279=1 269=1 48=18720 55=ESH9 83=16185243 270=278700.0 271=9 346=2 1023=2 10=052
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                 
2019.03.01 ESH9 2019.02.28D22:45:00.993000000 `float$()      `float$() ,278700f                            ,9f             16039928 16185243 84  

According to the FIX message with tag 34=16039863, the first level (1023=1) of the ASK(269=1) has been added (279=0) with price 278675.0 (270=...). Why isn't this addition reflected in the db? The following line (34=16039863) has not deleted this addition!

1128=9 9=184 35=X 49=CME 75=20190301 34=16039982 52=20190228224502548650124 60=20190228224502548497797 5799=10000100 268=1 279=1 269=1 48=18720 55=ESH9 83=16185244 270=278775.0 271=27 346=9 1023=5 10=122
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                 
2019.03.01 ESH9 2019.02.28D22:45:02.548000000 `float$()      `float$() 278700 278775f                      9 27f           16039982 16185244 84  
1128=9 9=258 35=X 49=CME 75=20190301 34=16039986 52=20190228224503373931175 60=20190228224503373797469 5799=10000100 268=2 279=1 269=1 48=18720 55=ESH9 83=16185245 270=278700.0 271=10 346=3 1023=2 279=1 269=1 48=18720 55=ESH9 83=16185246 270=278750.0 271=22 346=9 1023=4 10=056
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                 
2019.03.01 ESH9 2019.02.28D22:45:03.373000000 `float$()      `float$() 278700 278750 278775f               10 22 27f       16039986 16185246 84  
1128=9 9=185 35=X 49=CME 75=20190301 34=16040212 52=20190228224515445748590 60=20190228224515445632771 5799=10000100 268=1 279=1 269=0 48=18720 55=ESH9 83=16185247 270=278600.0 271=29 346=11 1023=4 10=130
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                 
2019.03.01 ESH9 2019.02.28D22:45:15.445000000 ,278600f       ,29f      278700 278750 278775f               10 22 27f       16040212 16185247 84   
1128=9 9=186 35=X 49=CME 75=20190301 34=16040219 52=20190228224515526743995 60=20190228224515526638985 5799=10000100 268=1 279=1 269=1 48=18720 55=ESH9 83=16185248 270=278900.0 271=26 346=11 1023=10 10=202
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                
2019.03.01 ESH9 2019.02.28D22:45:15.526000000 ,278600f       ,29f      278700 278750 278775 278900f        10 22 27 26f    16040219 16185248 84   
1128=9 9=185 35=X 49=CME 75=20190301 34=16040221 52=20190228224515557664306 60=20190228224515557549953 5799=10000100 268=1 279=1 269=0 48=18720 55=ESH9 83=16185249 270=278550.0 271=28 346=12 1023=6 10=147
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                
2019.03.01 ESH9 2019.02.28D22:45:15.557000000 278600 278550f 29 28f    278700 278750 278775 278900f        10 22 27 26f    16040221 16185249 84   
1128=9 9=185 35=X 49=CME 75=20190301 34=16040223 52=20190228224515578558691 60=20190228224515578444361 5799=10000100 268=1 279=1 269=1 48=18720 55=ESH9 83=16185250 270=278875.0 271=27 346=10 1023=9 10=154
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                
2019.03.01 ESH9 2019.02.28D22:45:15.578000000 278600 278550f 29 28f    278700 278750 278775 278875 278900f 10 22 27 27 26f 16040223 16185250 84    
1128=9 9=185 35=X 49=CME 75=20190301 34=16040224 52=20190228224515598668454 60=20190228224515598550505 5799=10000100 268=1 279=1 269=1 48=18720 55=ESH9 83=16185251 270=278875.0 271=28 346=11 1023=9 10=159
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------               
2019.03.01 ESH9 2019.02.28D22:45:15.598000000 278600 278550f 29 28f    278700 278750 278775 278875 278900f 10 22 27 28 26f 16040224 16185251 84   
1128=9 9=555 35=X 49=CME 75=20190301 34=16040225 52=20190228224515618725707 60=20190228224515618559823 5799=00000100 268=6 279=0 269=0 48=18720 55=ESH9 83=16185252 270=278725.0 271=15 346=1 1023=1 279=2 269=1 48=18720 55=ESH9 83=16185253 270=278675.0 271=1 346=1 1023=1 279=2 269=1 48=18720 55=ESH9 83=16185254 270=278700.0 271=10 346=3 1023=1 279=1 269=1 48=18720 55=ESH9 83=16185255 270=278725.0 271=26 346=8 1023=1 279=0 269=1 48=18720 55=ESH9 83=16185256 270=278925.0 271=28 346=9 1023=9 279=0 269=1 48=18720 55=ESH9 83=16185257 270=278950.0 271=35 346=11 1023=10 10=152
date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------                
2019.03.01 ESH9 2019.02.28D22:45:15.618000000 `float$()      `float$() `float$()                           `float$()       16040225 16185257 04  

Please compare the above last 2 lines of the DB: 

date       sym  time                          bprice         bsize     aprice                              asize           msgseq   rptseq   matchevent underlying
------------------------------------------------------------------------------------------------------------------------------------------------------------------               
2019.03.01 ESH9 2019.02.28D22:45:15.598000000 278600 278550f 29 28f    278700 278750 278775 278875 278900f 10 22 27 28 26f 16040224 16185251 84             
2019.03.01 ESH9 2019.02.28D22:45:15.618000000 `float$()      `float$() `float$()                           `float$()       16040225 16185257 04  

Why has the entire book been deleted???? The only 2 deletions (279=2) found in the FIX message are related to the first 2 levels of the ASK(269=1) side of the book and a total of 11 shares (271=1 and 271=10). Why has the BID side and the remaining levels of the ASK side been deleted? Why have the remaining additions (279=0) been ignored?

[Jonathan Foster]

Hi Sergio,

Thanks for your question. We are looking into it and we will get back to you with an answer.

Regards

Jonathan

[sergio]

Thank you.