Tokenizer in Tensorflow.js

[Mayur Bhole]

Hi, I am using tensorflow.js for text classification. We can't use word embedding for some reason, and would like to use bag-of-words approach. Wondering if we have a tokenizer in tensorflow.js for this purpose? Is it possible to use keras tokenizer?

[Edoh Kodjo]

Hi,

You have this answer on stackoverflow that can help you to start: https://stackoverflow.com/questions/51663068/tensorflow-js-tokenizer

El lun., 6 ago. 2018 9:05, Mayur Bhole <mayurb...@gmail.com> escribió:

Hi, I am using tensorflow.js for text classification. We can't use word embedding for some reason, and would like to use bag-of-words approach. Wondering if we have a tokenizer in tensorflow.js for this purpose? Is it possible to use keras tokenizer?

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/0b226055-0899-4f48-8540-0027ac65bcdf%40tensorflow.org.

[Mayur Bhole]

Thanks Edoh Kodjo for your quick reply. I am also using 'natural' package and have written my own text to vector generator. I wanted to check if keras tokenizer is available in js.

[Edoh Kodjo]

For now, I think that it is not available yet

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/a33ea632-7c46-4347-a42d-150374994913%40tensorflow.org.

[Stan Bileschi]

Hi Mayur,

As I'm sure you understand, the python version of the tokenizer can conceptually be broken down into three phases: first to break a string into a sequence of substrings, second to determine the common tokens, and third to convert the substring sequence into an integer sequence using the known list of tokens.

I'm working on developing tooling to support these types of preprocessing actions in JS, but I'm curious, which, if any, of these three steps do you need for your application?

Thanks,

Stan

[Edoh Kodjo]

Hi Stan,

Can you please share your GitHub project as I would like to contribute and add the natural language processing to tensorflowJs?

Thanks

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/e7828349-081e-4c3b-bc67-356000556c37%40tensorflow.org.

[Stan Bileschi]

Hi Edoh, 

Thanks for your interest!  You can get an idea what we're looking at this branch on my private fork.  Right now I'm playing around with designs that would work in both a python and JS context.  I don't think the plan & direction are clear enough at this time to accept external contributions, but I can notify you when we have buy-in from the owners of the python keras that the plan is good to go.

Cheers,

Stan

 

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/e7828349-081e-4c3b-bc67-356000556c37%40tensorflow.org.

--

Stan Bileschi Ph.D.  |

SWE  |

bile...@google.com |

617-230-8081

[Edoh Kodjo]

Okay, It sounds good.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/e7828349-081e-4c3b-bc67-356000556c37%40tensorflow.org.

[Mayur Bhole]

Hi Stan,

I was looking for 2 functions from keras-prerprocessing text.py: texts_to_matrix() and fit_on_texts(). For now, I have implemented these functions for my needs. However, Please let me know when you start accepting external contributions. I will be happy to contribute.