Deep neural networks and kdb

[Rob Smith]

Hello,

Has any used one of the popular deep learning frameworks like TensorFlow
for training neural networks with data from kdb? If so, what's your
work flow like? Do you export data to CSV files? Interface directly
with C libraries? Use PyQ? Any suggestions would be helpful before I
start experimenting myself.

Thanks,
Rob

[Kim Tang]

>>> Has any used one of the popular deep learning frameworks like TensorFlow for training neural networks with data from kdb?

No, usually it is hard to find someone who knows kdb+ and machine learning.

>>> Do you export data to CSV files

No, don’t do it. CSV is not reliable and it shows that you are not an expert in kdb+.

>>> Interface directly with C libraries
Yes, please check Qml and Bml as an example.

>>> Use PyQ?
Yes, you can do it. There are a lot of plugins outside to execute python code inside kdb+.

>>> Any suggestions would be helpful before I start experimenting myself.

It really depends what you want to do or achieve. IMHO if your dataset is small like say smaller than 100mb you don’t need to use kdb+. You can easily interface kdb+ with python or R and use the libraries in the language. These libraries are usually from high quality and they are implemented from experts. Moreover both languages also support visualization something you cannot find in kdb+ yet. (You cannot do machine learning without visualization.) And kdb+ does not support visualization directly, not yet. There will be something awesome in the future where you can do visualization directly from kdb+ but it is not ready yet. Don’t think in dashboard but a nice library in kdb+.

So if your dataset is big, say more than 12gb like the dataset from https://www.kaggle.com/c/bosch-production-line-performance then you cannot use the libraries in python or R natively. Then it makes sense to implement a machine learning libraries in kdb+ that can be used directly in a hdb. Check xgboost https://github.com/kimtang/xgb as an example.

Kim

-----Ursprüngliche Nachricht-----
Von: personal...@googlegroups.com [mailto:personal...@googlegroups.com] Im Auftrag von Rob Smith
Gesendet: Donnerstag, 5. Januar 2017 17:56
An: Kdb+ Personal Developers
Betreff: [personal kdb+] Deep neural networks and kdb

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.
Visit this group at https://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[kc3...@columbia.edu]

Follow up on this question... I can't import tensorflow or keras in pyq. Is there any solution that I can use Python deep learning packages from q?

Thanks,

Kaili

[Science Student]

Use the python interface instead of PyQ

--

[pierre]

sorry, should've been:

https://code.kx.com/q/ml/embedpy/

On 26 June 2018 at 23:16, pierre <k.pie...@gmail.com> wrote:

https://code.kx.com/preview/q/ml/embedpy/

On 26 June 2018 at 22:58, Science Student <newstud...@gmail.com> wrote:

Use the python interface instead of PyQ

On 6/26/18 4:55 PM, kc3...@columbia.edu wrote:

Follow up on this question... I can't import tensorflow or keras in pyq. Is there any solution that I can use Python deep learning packages from q?

Thanks,

Kaili

On Thursday, January 5, 2017 at 10:57:21 AM UTC-6, Rob Smith wrote:

Hello,

Has any used one of the popular deep learning frameworks like TensorFlow
for training neural networks with data from kdb?  If so, what's your
work flow like?  Do you export data to CSV files? Interface directly
with C libraries?  Use PyQ?  Any suggestions would be helpful before I
start experimenting myself.

Thanks,
Rob

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsubscribe@googlegroups.com.
To post to this group, send email to personal-kdbplus@googlegroups.com.

Visit this group at https://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsubscribe@googlegroups.com.
To post to this group, send email to personal-kdbplus@googlegroups.com.

[Kaili Chen]

I was trying:

\l p.q

.p.e "import tensorflow"

It didn't work...ModuleNotFoundError...

You received this message because you are subscribed to a topic in the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/personal-kdbplus/41dGzYkgUx8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to personal-kdbplus+unsubscribe@googlegroups.com.

[Flying]

Is the module usable with your python environment (not via q invocation) in the first place?

https://code.kx.com/preview/q/ml/embedpy/

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.

Visit this group at https://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.

Visit this group at https://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/personal-kdbplus/41dGzYkgUx8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.

[cmcca...@kx.com]

Kaili, if you're using embedPy, the reason for your error is likely the syntax, give this a try for importing Tensorflow;

\l p.q

tf:.p.import[`tensorflow]

If you want to take a look at an example of a Neural Network developed using embedPy and JupyterQ which uses Keras you might be interested in these notebooks and associated blog series. The notebook outlines an installation procedure for tensorflow if that's potentially an issure. I've attached the Neural Network blog as a sample;

https://github.com/KxSystems/mlnotebooks/tree/master/notebooks

https://kx.com/blog/neural-networks-in-kdb-2/

If you're still having issues with the installation of embedPy and JupyterQ there is an outline of how to install both in both a docker environment and via anaconda, which can be found here.

https://code.kx.com/q/ml/setup/

Conor