How to pass Scala Option types from Python (Py4j) when calling Scala function?

[Hemant Kadam]

Hello, 

I have a Scala function that takes multiple parameters and one of the parameter is Option[String], I want to know how to pass Option[String] parameter from Python (using Py4j to call Scala function).

When None is used – Function call fails with scala.MatchError: null

When String is passed – Function signature does not match, so getting error function does not exists.

When Optional[str] is passed - getting error “AttributeError: '_SpecialForm' object has no attribute '_get_object_id'”

Any help is appreciated, thanks,  

Here is a sample function with Option[String] parameter,

def decideMapping(colsToMap : Option[String] = None): String = {

  val defaultColMap = "sourceDfMap"

  val mapColNamesToUse = colsToMap match {

    case Some(x) => x

    case None =>  defaultColMap

  }

  mapColNamesToUse

}

[Hemant Kadam]

Passed Option[String] as sc._jvm.scala.Some('str') works but not sure how to pass None, any help is appreciated. 

[Yitao Li]

Hey just for future reference for anyone running into this -- after thinking about this for long enough, I found out `sc._jvm.scala.Option.apply(None)` would work.

It goes thru a 2-step process though: first "translating" `None` in Python to `null` in Java, and then subsequently mapping `null` to `None` in Scala. But anyhow, it works fine.