log parameters in databricks

[nadine ben harrath]

Hello everyone , can someone please tell me how can i log the best parameters of crossvalid and paramgrid using mlflow , pyspark in databricks 

thank you 

[Jules Damji]

Hello everyone , can someone please tell me how can i log the best parameters of crossvalid and paramgrid using mlflow , pyspark in databricks 

I assume that your best metric is the consequence of your best parameters in the crossvalid and paragrid. So once you find the best

model, use that model's parameters to log using MLflow APIs.  

# Run cross-validation, and choose the best set of parameters.
cvModel = crossval.fit(training)

Hope this direction helps

cheers

Jules

––

The Best Ideas are Simple

Jules S. Damji

Sr. Developer Advocate

Databricks, Inc.

ju...@databricks.com

(510) 304-7686

   

On Tue, May 4, 2021 at 3:18 PM nadine ben harrath <nadinebe...@gmail.com> wrote:

Hello everyone , can someone please tell me how can i log the best parameters of crossvalid and paramgrid using mlflow , pyspark in databricks 

thank you 

--
You received this message because you are subscribed to the Google Groups "mlflow-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlflow-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mlflow-users/08c379bd-3969-492c-a6c7-7cb81650e7e5n%40googlegroups.com.

[nadine ben harrath]

yes i know but i should log every parameter of crossvalidation using mlflow 

[nadine ben harrath]

you didn't get my problem , when i write with mlflow start_run() using paramgrid and crossvalidation , i don't find the best parameter of crossvalidation logged in mlflow 

[Jules Damji]

Nadine,

Sorry for the confusion. Are you a Databricks customer? If so I would urge you to file a ticket and provide the notebook that's not

logging your parameter. Surely, the support person can follow up with the issue if he/she can reproduce it on the Databricks cluster.

(Note: If you use the new autolog feature then all parameters, metrics, and artifacts are automatically logged for you during the model.fit() call; if you don't use autolog() then you will

have to explicitly log the parameters yourself, using the tracking APIs. If you doing the latter, and it's now showing in the MLflow UI run's page, then would be worthwhile to investigate)

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/mlflow-users/25115d0b-283a-45a2-aaf4-90a491d81764n%40googlegroups.com.