Actual taking 10+ hours for training when I set 4 training node hours

[Shijun Liu]

Hi, experts,

I noticed that when I set 4 training node hours in AutoML Tables, the training process will take total 10+ hours to finish. 

Why it takes 10+ hours?

And how's about the charge? The charge is for 4 training node hours or for the actual 10+ hours?

If I have a 1M lines with 1K features per line, and I set 4 training node hours, how do I estimate the actual training time? Any formula?

Thanks,

Shijun

[Helin Wang]

Hi Shijun,

The 10+ hours is due to infra setup time and other non model training related computation time.

If you are using AutoML Tables black box version (e.g., not Tabular Workflows), you will be only billed for the 4 training node hours.

> If I have a 1M lines with 1K features per line, and I set 4 training node hours, how do I estimate the actual training time? Any formula?

As mentioned, the extra time for training is related to infra setup and other non model training related computation. I would expect the additional time is proportional to dataset width (# of feature) and height (# of rows). So if you have a dataset with similar width and height, the additional time should be roughly similar. Are you seeing very inconsistent actual training time for the same dataset?

Thanks,

Helin

--
You received this message because you are subscribed to the Google Groups "cloud-automl-tables-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-automl-tables...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/CAK5fZ7FfJs7A2A3RwGXcYnaD6w1Ju4wtkObUBsRiLvgrrwKYyA%40mail.gmail.com.

[Shijun Liu]

Hi, Helin,

I can see the similar training time each time. I think you have a clear explanation. Thanks very much!

Thanks,

Shijun