trials = SparkTrials() + algo = atpe.suggest NOT WORKING

[llbrunoll llbll]

Dear friend, here I just took the SparkTrials example from:

https://hyperopt.github.io/hyperopt/scaleout/spark/

All dev dependences were installed to make both SparkTrials and ATPE to work. 

Note1: trials = Trials()  works fine for me, for both algo = tpe.suggest  and algo = atpe.suggest

Note2: trials = SparkTrials() works fine for me, when algo = tpe.suggest     but crash when algo = atpe.suggest

there comes the error:       

" Because the requested parallelism was None or a non-positive value, parallelism will be set to (2), which is Spark's default parallelism (2), or the current total of Spark task slots (2), or 1, whichever is greater. We recommend setting parallelism explicitly to a positive value because the total of Spark task slots is subject to cluster sizing.

0%| | 0/32 [00:01<?, ?trial/s, best loss=?]

Total Trials: 1: 1 succeeded, 0 failed, 0 cancelled.

--------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-8-c4cfed799842> in <module>() 74 algo=algo, 75 trials=spark_trials, ---> 76 max_evals=32) 77 best_hyperparameters

7 frames

/content/hyperopt/hyperopt/atpe.py in convertTrialsToResults(self, hyperparameterSpace, trials) 1551 "trial": trialIndex, 1552 "status": trial["result"]["status"], -> 1553 "loss": trial["result"]["loss"], 1554 "log": "", 1555 "time": abs( KeyError: 'loss'  " 

Please, let me know if there is any work around to make atpe.suggest work in multiprocessing framework

Best regards,

||bruno||