It can be done successfully using azurerm_lb...
azurerm_virtual_machine_scale_set.network_profile.ip_configuration.load_balancer_backend_address_pool_ids = ["${azurerm_lb_backend_address_pool.bpepool.id}"]
But if I swap out the backend pool id for one used by azurerm_application_gateway I get the following error:
ag-beap is referencing resource of a wrong type. The Id is expected to reference resources of type loadBalancers/backendAddressPools.
Should there be another attribute under azurerm_virtual_machine_scale_set.network_profile.ip_configuration - Like application_gateway_backend_pools?
...or maybe there is another way to do this that I'm missing?