Which function is used to feed realtime data

[hzadonis]

Hi, Masters:

  I wrote a my owned program to feed realtime data into the TorQ platform.

  But which function I should use to feed data in? I read tickerplant.q, how about the functin "upd", but it doesn't work.

  Do you have any suggestion?

Regards

hzadonis

[hzadonis]

I found the tickerplant port which is 6000, can't be connected.

Such as, I start the TorQ platform with start_torq_demo.bat, and the 6000 port is under listening:

E:\>netstat -ano|findstr "6000"

  TCP    0.0.0.0:6000           0.0.0.0:0              LISTENING       26940

But it can't be connect within a new q window:

q)h:hopen 6000

'access

  [0]  h:hopen 6000

What's the reason?

在 2019年1月7日星期一 UTC+8下午4:15:07，hzadonis写道：

[hzadonis]

Should I connect with username:password

在 2019年1月7日星期一 UTC+8下午4:43:40，hzadonis写道：

[hzadonis]

Yes, username:password is necessary for connection.

But, I still failed to insert realtime data. neither "upd" nor ".u.upd" works.

在 2019年1月7日星期一 UTC+8下午5:01:46，hzadonis写道：

[Fiona]

Hi hzadonis,

You were right with the username:password needed for connection. As for publishing data from feed to tickerplant you can do so by calling

h(`.u.upd;t;x)

in your feed code. This is assuming h is your handle to the tickerplant. For example

 h(`.u.upd;`trade;(2019.01.07D00:00:02.536524000;`MSFT;33342.87;7i;0b;"B";"N"))

would feed the data given to the trade table as defined in your .u.upd.

If you encounter more issues and need assistance with debugging please include error messages. 

[hzadonis]

Hi, Fiona:

  Thank you so much!

  Yes, your sample works fine. So, I can see "`.u.upd" is the entrance.

  Now, I'm trying to modify the schema to create a new table and feed data with my developed C++ FeedHandler. Please give me some time to try with the "`.u.upd" function.

  By the way, do you konw why the rdb log file always says: "could not replay the log file: type"? Is it necessary to replay log file? How can I turn-off replay?

Regards

hzadonis

在 2019年1月7日星期一 UTC+8下午8:09:24，Fiona写道：

[Fiona]

Hi hzadonis,

It is not necessary to replay the rdb log but by disabling it, if you were to restart the rdb intraday, then all that days previous data would not be loaded back in and would have to be manually called using

 -11!pathtologfile

in your rdb q session. You can disable auto replay by either defining .replay.auotreplay to be 0b within a configuration file or alternatively you can edit your tickerlogreplay.q function within TorQ to define

autoreplay:@[value;`autoreplay;0b]

I would recommend the configuration file route to avoid any overwrite issues if you were to update TorQ. The type error is likely coming from your change of schema. Data from the tickerplant is saved in log files according to the current schema and so trying to reload this data into a schema it doesn't fit will likely be the cause of your type error.

[hzadonis]

Hi, Fiona:

  Thanks. I found another way to shutdown replay log by set  "autoreplay:0b" in tickerlogreplay.q file.

  On the other hand, I think I should grant read/write permission to my owned table in default.q file.

  

  One more question, I modifid the structure of logmsg and heartbeat, renamed field name of "sym" to "Name" and put it to the first field,  renamed field name of "time" to "Time" to the second field.  Because my trade and quote tables are these names and order. But the err log files give me message:

  "q)2019.01.09D05:52:46.453304000|adonis|discovery|discovery1|ERR|timer|timer ID 3 failed with error time.  The function will not be rescheduled"

  I found in the .timer namespace, there's a variables "id". do you have any suggestion about how to fix the "timer ID 3 failed"?

Regards

hzadonis

在 2019年1月8日星期二 UTC+8下午5:33:06，Fiona写道：

[hzadonis]

Fiona, Please forget my fields name. let me modify my table to suit it.

在 2019年1月9日星期三 UTC+8下午2:07:31，hzadonis写道：

[hzadonis]

I created my table as:

q)h:hopen`:10.34.5.66:9000:tickerplant:pass

q)h"meta AShare_Intraday_Trade"

c     | t f a

------| -----

sym   | s   g

time  | p

Price | e

Volume| j

and insert data with:

q)h(`.u.upd;`AShare_Intraday_Trade;(`10057011;2019.01.09D00:00:02.536524000;89;1000))

but feedback message is:

'length

  [0]  h(`.u.upd;`AShare_Intraday_Trade;(`10057011;2019.01.09D00:00:02.536524000;89;1000))

I cant see why there is "'length" error, do you know why?

在 2019年1月9日星期三 UTC+8下午2:37:21，hzadonis写道：

[Fiona]

In TorQ the tickerplant appends a timestamp to incoming data if there is not already a time column in the first column. I think this may be where your length error is coming from. Could you try and make your table with column ordering `time`sym`Price`Volume instead?

[hzadonis]

"`time`sym..." works, no "`length" error.

Then, which file I should modify if the fields is "`sym`time...", because I hardly can modify the data format, or do you have other good suggestion? thanks!

在 2019年1月9日星期三 UTC+8下午5:11:47，Fiona写道：

[hzadonis]

I found a way to convert my historical database as the "`time`sym..." format. So it needn't modify the TorQ platform. 

Hope it works. 

Fiona, thank you!

在 2019年1月9日星期三 UTC+8下午5:46:56，hzadonis写道：

[Simon Wai Ling Tsang]

Hi hzadonis,

 

.u.upd in ticketplant doesn’t return any value.  Is anything subscribing from the tickerplant?  You can check with .u.w, which list out the handle for each subscriber, and which table they are subscribing to.

 

Regards,

Simon Tsang | Programmer Analyst Advisory | Global Fixed Income Development

Scotiabank | Global Wholesale & Risk Technology (GWRT)
320 Bay St, 13^th floor, Toronto, Ontario, Canada   M5H 4A6

T  1.416.860-1764

simon...@scotiabank.com

www.scotiabank.com

Scotiabank is a business name used by The Bank of Nova Scotia

 

From: kdb...@googlegroups.com [mailto:kdb...@googlegroups.com] On Behalf Of hzadonis
Sent: January-10-19 1:25 AM
To: AquaQ kdb+/TorQ
Subject: [Aquaq kdbtorq] Re: Which function is used to feed realtime data

 

Hi, Fiona:

  One more question.

  The processes are working, but the insertion form q client doesn't success.

q)h:hopen`:10.34.5.66:9000:tickerplant:pass

q)h"meta AShare_Intraday_Trade"

c     | t f a

------| -----

time  | p   

sym  | s   g

Price | e

Volume| j

 

and insert data with:

q)h(`.u.upd;`AShare_Intraday_Trade;(2019.01.10D00:00:02.536524000;`10057011;89.90;1000))
q)

no any feedback but a normal "q)" prompt.

 

then, I select data from rdb/wdb, the table is empty. do you have any advice? Thanks!

在 2019年1月10日星期四 UTC+8上午11:31:39，hzadonis写道：

--
www.aquaq.co.uk
www.aquaq.co.uk/blog
www.aquaq.co.uk/training
---
You received this message because you are subscribed to the Google Groups "AquaQ kdb+/TorQ" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kdbtorq+u...@googlegroups.com.
To post to this group, send email to kdb...@googlegroups.com.
Visit this group at https://groups.google.com/group/kdbtorq.
To view this discussion on the web, visit https://groups.google.com/d/msgid/kdbtorq/401839e9-a021-4ab5-bb15-4c0ece9f940b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

To unsubscribe from receiving further commercial electronic messages from The Bank of Nova Scotia in Canada, please click here: https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK

To unsubscribe from receiving further commercial electronic messages from certain other senders set out in the attached list, please click here: https://www.unsubscribe.gwm.scotiabank.com?page=gwm

This email may contain confidential information the use of which by an unintended recipient is unauthorized. This email may also contain important disclosure information for the records of the intended recipient(s). For details please click here: http://www.scotiabank.com/email_disclaimer/email_english.html

Pour vous désabonner et cesser de recevoir des messages électroniques commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici : https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK

Pour vous désabonner et cesser de recevoir des messages électroniques commerciaux de certains autres expéditeurs figurant dans la liste ci-jointe, veuillez cliquer ici : https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr

Cette transmission peut contenir de l'information confidentielle et son utilisation par toute personne autre que la personne à laquelle cette transmission est destinée est interdite. Le présent courriel peut aussi contenir des renseignements importants pour les dossiers du ou des destinataires prévus. Pour plus de détails, veuillez vous diriger vers: http://www.scotiabank.com/email_disclaimer/email_francais.html

Para dejar de recibir mensajes electrónicos comerciales de The Bank of Nova Scotia en Canadá, haga clic aquí: https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK

Para dejar de recibir mensajes electrónicos comerciales de otras entidades incluidas en la lista adjunta, haga clic aquí: https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es

Este correo electrónico puede contener información confidencial cuyo uso por parte de personas distintas de los destinatarios del mismo está prohibido. El mensaje puede tener también datos importantes para los archivos del(de los) destinatario(s) a quien(es) está dirigido. Para más detalles, por favor dirigirse a: http://www.scotiabank.com/email_disclaimer/email_espanol.html

[hzadonis]

Hi, Simon:

 .u.w output is like:

q)h:hopen`:10.34.5.66:6007:tickerplant:pass

q)h".u.w"

heartbeat| 604i `

logmsg   | 604i `

Is it looks OK?

在 2019年1月11日星期五 UTC+8上午12:37:54，Simon Wai Ling Tsang写道：

[hzadonis]

Hi, Fiona:

  Yesterday, I did some experiments, but something I can't explain. For example:

   Scenario 1:

      I create a table(tmptrade) which as same fields and field type as trade tabel. The insert operation works fine for both tmptrade and trade.

    Scenario 2:

      Create a table(tmptrade) which fields and field type does not like trade table, such as:  tmptrade:([]time:`timestamp$(); sym:`g#`symbol$(); price:`float$(); size:`int$(); vwap:`real$(); bs:`symbol$(); pctchg:`real$()) 

      The insert operation for trade works. But data can't be insert into tmptrade, and there's no any error feedback after I execute "h(`.u.upd;`tmptrade;(.....))" , the tmptrade keeps empty while I select it.

    Scenario 3:

      Create a table(tmptrade) which fields and field type partially same as trade table, such as:   tmptrade:([]time:`timestamp$(); sym:`g#`symbol$(); price:`float$(); size:`int$(); stop:`boolean$(); cond:`char$(); bs:`symbol$())

      The last field of tmptrade is bs:`symbol$(), other fields are same as trade. The insert operation works fine for both tmptrade and trade.

    Scenario 4:

       Create a table(tmptrade), which stucture is: tmptrade:([]time:`timestamp$(); sym:`g#`symbol$(); price:`float$(); size:`int$())

       The insert operation for trade works. But data can't be insert into tmptrade, and there's no any error feedback after I execute "h(`.u.upd;`tmptrade;(.....))" , the tmptrade keeps empty while I select it.

    I did 2 times for every scenario, and got the same result, which baffled me. Does the .u.upd or other functions have limitation? Such as only trade and only the required table structure?

Best Regards

Zheng

   
在 2019年1月11日星期五 UTC+8上午9:37:15，hzadonis写道：

[Fiona]

Hi hzadonis,

.u.upd will work for any table if you've defined them properly in the schema. I think your problem is your tmptrade schema has different types from the data you are trying to insert. You see an error if the tickerplant is in immediate mode. That is, if you start the tickerplant without -T x (where x is the number of milliseconds) then it doesn't do any type checking (i.e. it doesn't insert the data into a local table and then publish, it just sticks the headers on it and publishes). If you are using TorQ-FSP on top of TorQ then immediate mode is set as default for the tickerplant.

[hzadonis]

Hi, Fiona:

  I'm based TorQ3.3.0 and TorQ-Finance-Starter-Pack1.6.0, is it OK?

  I used to doubt wrong field type I make. Let me just define a simple tmptrade table.

在 2019年1月11日星期五 UTC+8下午5:52:17，Fiona写道：

[Fiona]

They are the most up-to-date versions and are OK.

[hzadonis]

Hi, Fiona:

  Why I was always failed?

  Because for `real$() type field, the parameter in `.u.upd must be defined explicitly with an "e" behind . Such as  h(`.u.upd;`tmptrade;(2019.01.11D09:18:09.536524000;`abc;89.88e;1000;2.45e...)   But for `float$() type field, that's OK even without "f" defined.

  Hope the FeedHandler doesn't need the definiation.

Thanks

hzadonis

在 2019年1月11日星期五 UTC+8下午6:32:46，Fiona写道：

[Fiona]

Hi hzadonis,

In q the default integer type is long and as you've probably noticed to define type int or short you need to specify "i" / "h". This is similar with the default decimal type being float (thus no "f" necessary) while real numbers need to be defined with "e".  I would recommend using float above reals if you can since floats allow a higher level of precision within your data and handle rounding to a better standard.

[Simon Wai Ling Tsang]

Hi Hzadonis,

You can add a wrapper function to convert the type before calling .u.upd

In TP:
typedUPD:{[t;x] .u.upd[t;] (exec t from meta t) {x$y}'x }

Example :

q)tbl:([]sym:`symbol$();rnum:`real$();fnum:`float$();inum:`int$();date:`date$();time:`timespan$())

q).u.upd[`tbl] (`abc;123;123;123;2018.01.01;0D0)

'type

q)typedUPD[`tbl]  (`abc;123;123;123;2018.01.01;0D0)

,0

To view this discussion on the web, visit https://groups.google.com/d/msgid/kdbtorq/a8f612e0-3a94-4c55-b936-de300809afcd%40googlegroups.com.

[hzadonis]

Hi, Simon:

  Thank you for the smart function to do convertion!

Thanks

hzadonis

在 2019年1月15日星期二 UTC+8上午1:41:32，Simon Wai Ling Tsang写道：

[hzadonis]

Hi, Fiona:

  Thanks for your advice and continuance help!

Thanks

hzadonis

在 2019年1月14日星期一 UTC+8下午7:52:26，Fiona写道：