Import dataset into BigTable using PySparkConnector.

[venkata suresh gummadillli]

Hello, 

I am trying to write a dataframe to BigTable using PySpark Connector. 

Here are my settings.

bigtable_spark_connector_jar="gs://spark-lib/bigtable/spark-bigtable_2.12-0.1.0.jar"
create_new_table = "true"

spark = SparkSession.builder \
  .appName('df-to-bigtable')\
  .config("spark.jars", bigtable_spark_connector_jar) \
  .config("spark.jars.packages", "org.slf4j:slf4j-reload4j:1.7.36") \
  .getOrCreate()

spark.version=3.3.2

Scala code runner version 2.12.18
I am getting error:
Py4JJavaError: An error occurred while calling o212.save. : java.lang.ClassNotFoundException: Failed to find data source: bigtable. Please find packages at https://spark.apache.org/third-party-projects.html at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToFindDataSourceError(QueryExecutionErrors.scala:586) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:675) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:725) at org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:864) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:256) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: bigtable.DefaultSource at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
Can someone help me with this error?
Thanks,Venkat