Orca's tf2 estimator: Read spark dataframe in data creator function

33 views
Skip to first unread message

AP

unread,
Jul 5, 2021, 9:37:19 AM7/5/21
to User Group for BigDL and Analytics Zoo
Hi Team,
I need to provide sample weights in Keras model, and orca's tf2 estimator provides this functionality. 
Input training data is in spark dataframe format, but for providing sample weights, data creator function needs to be defined. 
So how to read spark data-frame inside data creator function and also prepare sample weights based on input data. 
Any ideas on how should I proceed.
Thanks

yang wang

unread,
Jul 5, 2021, 10:51:26 PM7/5/21
to User Group for BigDL and Analytics Zoo
Hi AP,

I do not think there is an easy way to read spark dataframe inside data creator function as data creator function runs on spark worker.

We are investigating how to support providing sample weights through spark dataframe. You can check the status here https://github.com/intel-analytics/analytics-zoo/issues/4224.

-Yang
Reply all
Reply to author
Forward
0 new messages