[JIRA] (SPARKR-230) support nested type in DataFrame

14 views
Skip to first unread message

Davies (JIRA)

unread,
Mar 14, 2015, 1:23:03 AM3/14/15
to spark...@googlegroups.com
Davies created an issue
 
SparkR / Sub-task SPARKR-230
support nested type in DataFrame
Issue Type: Sub-task Sub-task
Assignee: Unassigned
Created: 13/Mar/15 10:22 PM
Priority: Major Major
Reporter: Davies

ArrayType, MapType and StructureType

We can infer the correct schema for them, but the serialization can not handle them well.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 14, 2015, 2:56:02 AM3/14/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Sub-task SPARKR-230
 
Re: support nested type in DataFrame

Davies I can see why StructType won't work because of mixed types, but I guess ArrayType and MapType should work ?

AFAIK ArrayType should work as its a list of same type. Similarly I think map type should work but only for string keys.

Davies (JIRA)

unread,
Mar 14, 2015, 3:01:04 AM3/14/15
to spark...@googlegroups.com
Davies commented on Sub-task SPARKR-230

ArrayType could be c("ab", "ba"), we will serialize it as character.

We does not support deserialize env in R (serialize Map[] in Scala).

So all the three do not work.

Shivaram Venkataraman (JIRA)

unread,
Mar 14, 2015, 3:04:03 AM3/14/15
to spark...@googlegroups.com

Cool thanks for clarifying. Its good to know what needs to be fixed !

Shivaram Venkataraman (JIRA)

unread,
Apr 9, 2015, 5:05:02 PM4/9/15
to spark...@googlegroups.com
This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014-sha1:e47e3fa)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages