Op Registration and Kernel linking Mechanism

333 views
Skip to first unread message

Abhishek Aggarwal

unread,
May 31, 2016, 9:03:59 AM5/31/16
to Discuss
Hi,

I started looking into how to build custom ops. For that, I am looking into word2vec example here -

which uses "skimgram" custom Op. I see that the registration of the Op is done here -

And corresponding kernel implementation here -

I am trying to understand how this C++ code gets linked with Python to get this working. For that I did following -

1) Built individual targets required to build py_binary (word2vec)
bazel build -c opt tensorflow/models/embedding:word2vec_ops
bazel build -c opt tensorflow/models/embedding:word2vec_kernels
bazel build -c opt tensorflow/models/embedding:word2vec_kernels:gen_word2vec

The first two of those resulted in generated of shared object files (.lo) in $(bazel-bin) and the last one resulted in the gen_word2vec
python file in $(bazel-genfiles)

2) Build the py_binary word2vec target
bazel build -c opt tensorflow/models/embedding:word2vec
As expected, this results in .runfiles folder which symbolic link structure required to run the stub python file word2vec. 

3) Run the word2vec. It runs perfectly. Moreover, I am able to run word2vec (in .runfiles) even without installing Tensorflow either the binary or from the source.

Now I can understand it was able to import all the python stuff to run the wrod2vec given the symbolic structure in maintained in .runfiles folder. 

However, I still don't see where and how the op registration and kernel code got linked to this python binary. That is, how did the this custom op got registered in Tensorflow in the first place. I am sure some of you have clear and detailed understanding of the process happening behind the scenes here. I would very much appreciate if you can 
explain that process in detail. 

P.S. - I have already read the build custom op tutorial.

Thanks.

Abhishek Aggarwal

unread,
May 31, 2016, 9:08:31 AM5/31/16
to Discuss

typo in 3rd bazel build command, corrected below.
"bazel build -c opt tensorflow/models/embedding:gen_word2vec"

Abhishek Aggarwal

unread,
May 31, 2016, 6:04:19 PM5/31/16
to Discuss
I still haven't been able to figure this out. I don't imagine it's a difficult question to answer for the developers of TensorFlow. Please help.

Derek Murray

unread,
May 31, 2016, 6:27:16 PM5/31/16
to Abhishek Aggarwal, Discuss
I just posted an answer to (what I assume is) the same question on Stack Overflow:


If anything in the answer is unclear, let's continue the discussion over there.

Derek.

On Tue, May 31, 2016 at 3:04 PM, Abhishek Aggarwal <abhig...@gmail.com> wrote:
I still haven't been able to figure this out. I don't imagine it's a difficult question to answer for the developers of TensorFlow. Please help.

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
 To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/7d3a1296-50d8-4a7d-a24e-a1d084415e45%40tensorflow.org.

Abhishek Aggarwal

unread,
May 31, 2016, 6:36:58 PM5/31/16
to Discuss
The answer you posted is very detailed and helpful. Thanks a lot. I might ask again if I have anything unclear as I continue exploring TF.

Reply all
Reply to author
Forward
0 new messages