read avro file using pyspark dataproc

[R1]

I am using a Dataproc cluster (2.0 ubuntu ) with hadoop 3.2 and spark 3.1. I have a python code to read avro files from GCS. So I have used 'spark-avro_2.12-3.1.0.jar' but its give some error like method not found etc (java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.DataSourceUtils$.creteDateRebaseFuncInRead ). How to decide which library is compatible to use ?

I am reading using :
df = spark.read.format("avro").load([ 'file1.avro' , 'file2.avro' ])

[Nicolas Porter]

To find the versions, use this page:

https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-version-clusters

Find your image version, click on the link, and you'll see the scala / spark versions to use for the JAR. You can cross reference against:

https://mvnrepository.com/artifact/org.apache.spark/spark-avro

Side note, you can pull the jar at runtime using this property:

spark:spark.jars.packages=org.apache.spark:spark-avro_{{scalaVersion}}:{{sparkVersion}}

For the latest 2.0-ubuntu18 image, you would use:

spark:spark.jars.packages=org.apache.spark:spark-avro_2.12:3.1.3

Notice that the scala version is missing the last number of the version, and the spark version has all the numbers.

Cheers,

- nick