[JIRA] (SPARKR-233) Replace dot with underscore in column names

22 views
Skip to first unread message

Shivaram Venkataraman (JIRA)

unread,
Mar 16, 2015, 1:31:03 PM3/16/15
to spark...@googlegroups.com
Shivaram Venkataraman created an issue
 
SparkR / Sub-task SPARKR-233
Replace dot with underscore in column names
Issue Type: Sub-task Sub-task
Assignee: Davies
Created: 16/Mar/15 10:30 AM
Priority: Major Major
Reporter: Shivaram Venkataraman

SparkSQL does not support having dots in column names (see https://issues.apache.org/jira/browse/SPARK-2775) . When we convert a local R data frame to
Spark DataFrame we should replace dot with underscores. We could also print a warning message about this

To reproduce the problem try something like

df <- createDataFrame(sqlCtx, iris) # iris has columns Sepal.Width, Petal.Width etc.
df$Petal.Width # Throws an error
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Davies (JIRA)

unread,
Mar 16, 2015, 3:19:01 PM3/16/15
to spark...@googlegroups.com
Davies commented on Sub-task SPARKR-233
 
Re: Replace dot with underscore in column names

We can have a workaround in user's code by providing the names of schema.

SPARK-2775 targets Spark-1.4, so I'd like to do not put this in createDataFrame(), because it will be incompatible to change back once SPARK-2775 is fixed.

Does this sounds good to you?

Shivaram Venkataraman (JIRA)

unread,
Mar 16, 2015, 3:22:02 PM3/16/15
to spark...@googlegroups.com

I'm not sure I understand the workaround – Could you elaborate more ? What I had in mind was a change in SQLContext.R [1] where we replace all dots with underscore, with a TODO remove this in Spark 1.4.

[1] https://github.com/amplab-extras/SparkR-pkg/blob/6f95d49c3d49dcb97d677369188d974cecafd34c/pkg/R/SQLContext.R#L96

Davies (JIRA)

unread,
Mar 16, 2015, 3:52:01 PM3/16/15
to spark...@googlegroups.com
Davies commented on Sub-task SPARKR-233

You could workaround it as

df <- createDataFrame(sqlCtx, iris, list("Petal_Width", ...))

So you want a temporary solution only for developer (before 1.4 is released) ?

Shivaram Venkataraman (JIRA)

unread,
Mar 16, 2015, 4:14:02 PM3/16/15
to spark...@googlegroups.com

Yeah I think it'll be good to have mostly because it should be a two line function with something like `gsub(".", "_")`. And similar to the agg column thing we can remove it if / when the Spark side issue gets fixed.

Shivaram Venkataraman (JIRA)

unread,
Mar 16, 2015, 8:17:03 PM3/16/15
to spark...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages