Is it possible to not have a "TEST" split?

[Lucas Zhang]

When we have a relatively small dataset, we are not interested in seeing the test set metrics but just want to train the best possible model and perform batch prediction on another dataset later, is it possible that we don't have a "TEST" split?

A related question is that how is the "TEST" split used when training the final model using TRAIN, VALIDATE and TEST splits together. Is it simply merged into the training set using the best hyperparameters obtained before, or is it used like a validation set for things like early stopping? If the data is very small (or in our case, it's chronological and the most recent data is very precious) and it is best if we only have a single validation set besides training, what is the best recommendation for this?

Thanks,

Lucas

[Andrew Leach]

Hello Lucas!

See the Controlling Data Splits section and select forecasting, the answer to your question is in part there.

The TRAIN set is used for training during the model search, and evaluated on VALIDATE.
Once the top model architectures are selected the top 10 models architectures are selected, trained on TRAIN + VALIDATE, ensembled, and the predictions are generated for TEST.

Finally, the same architectures are selected and trained on TRAIN + VALIDATE + TEST, ensembled, and then the model is deployed for prediction.

It is not possible to omit the TEST split completely, but you can make it very small if you are not using it to maximize data for the model search (ex. hold just the last date of data). The final model exposed for batch prediction still uses all data available including TEST to train, so hopefully there is no concern in losing valuable training data. If your data set has a very brief history, one option is to split TRAIN and VALIDATE by time series, the final model may be less sensitive to non-stationarity, but it may be worth the tradeoff if the context and horizon are long relative to the history.

@Dawei, hopefully what I've said above is true, but please correct me if I'm mistaken :)

Best,

Andrew

--
You received this message because you are subscribed to the Google Groups "cloud-automl-tables-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-automl-tables...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/CADJWYjJJZKNDROZsXERk_RB0WGipYyiivjv_-WXuJttuWP27VQ%40mail.gmail.com.

[Dawei Jia]

Yes, I think Andrew answered your question. I will add the following:

1) You can make test split very small for regression problem. For classification, we do need test split to have all labels. so I'm not sure how small you can make to pass the validation in this case.

2) As Andrew said, the final model is also trained by test split.

3) We have a new solution called Tables on ML pipeline, that runs AutoML as a white box in user project. This is separated from the current Vertex Tables API and UI. In this solution we don't require test split. If you are interested we can give you more information.

Thanks

[Lucas Zhang]

Thanks Dawei and Andrew! This is clear.

Do you know if what you said (No problem if there's just a very small TEST split) applies to the forecasting model as well?

And Dawei, yes I would like to learn more about your new solution. If you have a link or doc I would like to take a look. Thanks!

[Andrew Leach]

I saw that this was for Adeo and assumed it was for Forecasting, so my answer should apply :)

[Lucas Zhang]

We do both classification/regression and forecasting in Adeo engagement. One of the classification use cases is very similar to the Home Depot work you did earlier, Andew. We may need your advice when testing it in store and let's sync up at a later time :)

I was asking this mainly for classification/regression but glad to know it applies to all models. Thanks again!