Availability of Dataproc with Spark 3.1.1image needed

[Mich Talebzadeh]

Dear Dataproc team,

We have tested Spark 3.1.1 with Spark Structured Streaming (which we had an issue making it work with in Dataproc built on the image 3.1.1-RC2.).

By test, I mean we set-up a 3-node spark cluster sending streaming data from on-premise to GCP BigQuery. Obviously this cannot be used in anger because of latency incurred sending data from Spark cluster on-premise.

In short to get back to using Dataproc nodes, we need an image of Spark built on Spark 3.1.1. 

Spark only offers 3.1.1 plus two other releases as STABLE versions

Release Notes for Stable Releases

• Spark 3.1.1 (Mar 02 2021)
• Spark 3.0.2 (Feb 19 2021)
• Spark 2.4.7 (Sep 12 2020)

Please bear in mind that Spark 3.0 and 3.1 are not available for download, meaning IMO they are defective releases Also few times Databricks in Spark user group stated that "3.1.1-RC2 is not a release" whatever that means. But taht is a moot point now.

Anyway to get in position, can you please build an image on 3.1.1 and I would be more than happy to test it as I am sure other users would agree with such release.

Finally I attach a spark GUI with structured streaming for version 3.1.1 

Thanks,

Mich

[Mich Talebzadeh]

Hi,

Any update on this request please?

Thanks

[Mich Talebzadeh]

Hi Dataproc team,

Can you please provide a response about when Dataproc images will use Spark 3.1.1.

Currently we cannot work on 3.1.1-RC2 with Spark Structured Streaming.

Thanks,

Mich

[Daniel Solow]

3.1.1 is available under 2.0.6-debian10/-ubuntu18/-centos8

[Mich Talebzadeh]

Thanks Daniel.

In creating dataproc cluster I cannot see 3.1.1 (see the attached image).

Please advise

regards,

Mich

[Mich Talebzadeh]

I can also see The announcement in this page. 

Release notes  |  Dataproc Documentation  |  Google Cloud

Important Announcements

Dataproc 2.0 image version will become a default Dataproc image version on March 15, 2021 (was February 22, 2021).

March 16, 2021

Announcing the General availability (GA) release of CentOS 8 based 2.0 images.

New sub-minor versions of Dataproc images: 1.3.87-debian10, 1.3.87-ubuntu18, 1.4.58-debian10, 1.4.58-ubuntu18, 1.5.33-centos8, 1.5.33-debian10, 1.5.33-ubuntu18, 2.0.6-centos8, 2.0.6-debian10, and 2.0.6-ubuntu18.

Image 2.0: Upgraded Spark to version 3.1.1

Does the Dataproc 2.0 image by default provides Spark 3.1.1?

Thanks

[Mich Talebzadeh]

Yes it is version 3.1.1 if you choose 2.0-debian10 from create cluster image

I just confirmed

echo $SPARK_HOME

/usr/lib/spark

spark-shell

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Spark context Web UI available at http://ctp-311-m.europe-west2-c.c.axial-glow-224522.internal:44721

Spark context available as 'sc' (master = yarn, app id = application_1616075327443_0001).

Spark session available as 'spark'.

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 3.1.1

      /_/

Using Scala version 2.12.13 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)

Thanks

[Mich Talebzadeh]

plus the existing clusters on 3.1.1-RC have been upgraded to 3.1.1

HTH