Error 500 when trying to create a Dataproc cluster with a history server

[Leoul Daniel]

Hello,

I am trying to recreate a dataproc cluster that was previously created with the following configuration ( retrieved from the command 'gcloud dataproc clusters export'):

virtualClusterConfig:
  auxiliaryServicesConfig:
    sparkHistoryServerConfig:
      dataprocCluster: projects/***/regions/europe-west2/clusters/dpgke-hs-stg
  kubernetesClusterConfig:
    gkeClusterConfig:
      gkeClusterTarget: projects/***/locations/europe-west2/clusters/dpgke-stg
      nodePoolTarget:
      - nodePool: projects/***/locations/europe-west2/clusters/dpgke-stg/nodePools/dp-default
        roles:
        - DEFAULT
      - nodePool: projects/***/locations/europe-west2/clusters/dpgke-stg/nodePools/dp-driver
        roles:
        - SPARK_DRIVER
      - nodePool: projects/***/locations/europe-west2/clusters/dpgke-stg/nodePools/dp-executor
        roles:
        - SPARK_EXECUTOR
    kubernetesNamespace: dpgke-staging
    kubernetesSoftwareConfig:
      componentVersion:
        SPARK: 3.5-dataproc-17
      properties:
        ( ... )
  stagingBucket: staging-bucket-dataproc-stg

I have tried creating the cluster using the UI Console, using the terraform resource "google_dataproc_cluster" ( terraform Google provider version 6.7) and by CLI (Gcloud SDK 511.0.0). In every case, i get an error 500 from dataproc.googleapis.com with a "Unknown Error" message ( see screenshots). 

When removing the Spark history server config, the cluster is created without any problems though.

I have tried the suggested actions in the "Troubleshoot cluster creation issues" doc without any success.

Any suggestions or help would be much appreciated.

Thank you