I need help to convert PyTorch nn modules to Keras

[alekd...@gmail.com]

Hello Guys

I am willing to convert PyTorch based NN code to Keras based code.

I am not sure how I can convert them to Keras

My current focus is to implement SQLova in Keras, and SQLova written in PyTorch.

There is a file with PyTorch:

https://github.com/naver/sqlova/blob/master/train.py

Please give me help on this.

Thank you

[alekd...@gmail.com]

Hi.
I am very concerned how I need to convert a class based code(PyTorch)

import torch
import torch.nn as nn

from torch.nn import CrossEntropyLoss

class BERTEmbeddings(nn.Module):
    def __init__(self, config):
        super(BERTEmbeddings, self).__init__()
        """Construct the embedding module from word, position and token_type embeddings.
        """
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
        # any TensorFlow checkpoint file
        self.LayerNorm = BERTLayerNorm(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, input_ids, token_type_ids=None):
        seq_length = input_ids.size(1)
        position_ids = torch.arange(seq_length, dtype=torch.long, device=input_ids.device)
        position_ids = position_ids.unsqueeze(0).expand_as(input_ids)
        if token_type_ids is None:
            token_type_ids = torch.zeros_like(input_ids)

        words_embeddings = self.word_embeddings(input_ids)
        position_embeddings = self.position_embeddings(position_ids)
        token_type_embeddings = self.token_type_embeddings(token_type_ids)

        embeddings = words_embeddings + position_embeddings + token_type_embeddings
        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)
        return embeddings

As you can see, BERTEmbeddings is a class based on torch.nn.Module, but how I can convert it with keras??

Please help me to implement this.

Thank you 

[François Chollet]

The "correct" way is to convert the class-based code to the Keras functional API. It will end up a lot shorter and more readable.

This model you're mentioning appears to be roughly:

word_input = Input(shape=(None,))

position_input = Input(shape=(None,))

token_type_input = Input(shape=(None,))

words = Embedding(config.vocab_size, config.hidden_size)(word_input)

positions = Embedding(config.max_position_embeddings, config.hidden_size)(position_input)

token_types = Embedding(config.type_vocab_size, config.hidden_size)(token_type_input)

embeddings = layers.add([words, positions, token_types])

embeddings = LayerNorm()(embeddings)  # You'll have to implement your own LayerNorm layer

embeddings = Dropout()(embeddings)

bert_model = Model([word_input, position_input, token_type_input], embeddings)

The "easy" way is to use tf.keras and subclassing to write the exact same code as before. Like this: https://colab.research.google.com/drive/1DQ12zpi-QMC5HzL-D0SRkfGD_HFWyMSV

You could also do something in between -- use subclasses for some blocks, the functional API for other blocks.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/9383b343-0220-46fb-8deb-20389abfac96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aleksander Bylaew]

Thanks a lot, François

You gave me a lot of help, but I hope you help me more.

In the past, I have used Tensorflow a lot of time, but I don't have many experiences with Keras and PyTorch.

I've tried to convert existing PyTorch module using keras, but it is mess.

Please check my attached two files(one is original PyTorch module, and another one is converted version).

I am a student who started to learn about machine learning, and it seems you are a senior machine learning developer.

I would like to learn about machine learning more deeply through this chance.

Thank you

PS. My goal is to convert this repo code(PyTorch based) to Keras/Tensorflow based code.

https://github.com/naver/sqlova

[gurtovoi...@gmail.com]