Re: How to make WIT work with sklearn models

[James Wexler]

Thanks for reaching out.

You'll need to create your own python function to act as the custom prediction function. This function will take in a list of inputs in the format you provided to the WitConfigBuilder constructor, and will need to return a list of predictions (one per example it was passed). For regression models, a prediction is a single number, and for classification models, a prediction is a list of class scores for each possible class (such as [0.2, 0.8] for a binary classifier that is returning the positive class with a score of 0.8, and therefore a 0.2 score for the negative class).

One example of a custom predict function can be found in https://colab.research.google.com/github/pair-code/what-if-tool/blob/master/WIT_Toxicity_Text_Model_Comparison.ipynb in the "Define custom prediction functions so that WIT infers using keras models" cell. In that example, the custom predict function converts the data from the format passed to WIT, to the format expected by the model, then calls the model's predict function, and returns those results (as that keras model in that example already formats the predictions in the correct format described above.

Let me know if this helps you out or if you have more questions.

-James

On Wed, Mar 11, 2020 at 12:03 PM <linlin...@gmail.com> wrote:

Hi,

I have been exploring WIT, which is a very interesting for me. I really would like to make it work with sklearn models locally. The tutorial notebook on Sklearn is for deploying on the cloud. I am not sure how to configure the builder locally.

So I used the following dummy data and model to play around:

from sklearn.model_selection import train_test_split
iris = load_iris()
X = iris.data[:, :2]  # we only take the first two features.
Y = iris.target
train_x, test_x, train_y, test_y = train_test_split(
    X, Y, test_size=0.2
)

logreg = LogisticRegression(C=1e5)

# Create an instance of Logistic Regression Classifier and fit the data.
logreg.fit(train_x, train_y)


test_examples = np.hstack((test_x,test_y.reshape(-1,1)))


config_builder = WitConfigBuilder(test_examples.tolist(), ['petal length', 'petal width', 'sepal length', 'sepal width', 'Species'])\
.set_custom_predict_fn(logreg).set_model_type('classification')
WitWidget(config_builder, height=1000)

WIT starts nicely, I can see all the datapoints and group them. But I cannot do predictions within it, it says

TypeError("'LogisticRegression' object is not callable",)

From the documentation, it reads

You can also use What-If Tool with a custom prediction function that takes Tensorflow examples and produces predictions. In this mode, you can load any model (including non-TensorFlow models that don't use Example protos as inputs) as long as your custom function's input and output specifications are correct.

How do we specify the input and output correctly? Can you please elaborate a bit more on this?

Thanks a lot in advance!

--
You received this message because you are subscribed to the Google Groups "What-If Tool" group.
To unsubscribe from this group and stop receiving emails from it, send an email to what-if-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/what-if-tool/98a3d68f-d6a8-4b2b-b406-9e5d79786ab0%40googlegroups.com.

[Linlin Zhao]

Hi James,

thanks a lot for your useful hints, I understand a bit more under the hood. Could you please help me look at the following code snippet 

iris = load_iris()
X = iris.

data  
Y = iris.target
train_x, test_x, train_y, test_y = train_test_split(
    X, Y, test_size=0.2
)

logreg = LogisticRegression(C=1e4)
logreg.fit(train_x, train_y)


def pred(testset):
    probs = logreg.predict_proba(testset[:, :-1])
    return prob.tolist()


test_examples = np.hstack((test_x,
                           test_y.reshape(-1,1)))


config_builder = WitConfigBuilder(
    test_examples.tolist(),
    feature_names=['petal length', 'petal width', 'sepal length', 'sepal width', 'species'])\
.set_custom_predict_fn(pred).set_target_feature('species')
WitWidget(config_builder, height=600)

I thought I configured the constructor properly, but it still reports error within the tool interface:

TypeError('list indices must be integers or slices, not tuple',)

Is there a way to track error? Or my config is still not correct?

Sorry for the dummy questions.

On Wednesday, March 11, 2020 at 5:41:45 PM UTC+1, James Wexler wrote:

Thanks for reaching out.

You'll need to create your own python function to act as the custom prediction function. This function will take in a list of inputs in the format you provided to the WitConfigBuilder constructor, and will need to return a list of predictions (one per example it was passed). For regression models, a prediction is a single number, and for classification models, a prediction is a list of class scores for each possible class (such as [0.2, 0.8] for a binary classifier that is returning the positive class with a score of 0.8, and therefore a 0.2 score for the negative class).

One example of a custom predict function can be found in https://colab.research.google.com/github/pair-code/what-if-tool/blob/master/WIT_Toxicity_Text_Model_Comparison.ipynb in the "Define custom prediction functions so that WIT infers using keras models" cell. In that example, the custom predict function converts the data from the format passed to WIT, to the format expected by the model, then calls the model's predict function, and returns those results (as that keras model in that example already formats the predictions in the correct format described above.

Let me know if this helps you out or if you have more questions.

-James

To unsubscribe from this group and stop receiving emails from it, send an email to what-i...@googlegroups.com.

[Linlin Zhao]

There is a typo, it should be

return probs.tolist()

But still the same error.

[James Wexler]

A good way to test would be to pass test_examples.tolist() (or some smaller subset of that, like test_examples.tolist()[0:10]) to your pred method directly in a notebook cell and see what the output is. Then let me know what you get.

Also, it looks like you're doing iris detection which is multiclass, so you'll want to call .set_multi_class(True) on your WitConfigBuilder to let WIT know that it isn't binary classification (which is the default).

--
You received this message because you are subscribed to the Google Groups "What-If Tool" group.
To unsubscribe from this group and stop receiving emails from it, send an email to what-if-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/what-if-tool/9de72c25-d1c2-44a3-9f4f-0d8575d5fbb7%40googlegroups.com.

[Linlin Zhao]

Thanks a lot! It worked:)

I was slicing 2-D list with numpy-style indexing, which caused the error. Actually slicing the target out is not needed when set_target_feature is True. By removing the indexing it worked.