[JIRA] (SPARKR-168) bug fix in Kmeans.R

21 views
Skip to first unread message

Yangxiaoqiang (JIRA)

unread,
Jan 9, 2015, 2:01:03 AM1/9/15
to spark...@googlegroups.com
Yangxiaoqiang created an issue
 
SparkR / Bug SPARKR-168
bug fix in Kmeans.R
Issue Type: Bug Bug
Assignee: Unassigned
Created: 08/Jan/15 11:00 PM
Environment:

ubuntu 12.04, R 3.1.1, Hadoop 2.4.1, Spark 1.1.0

Priority: Major Major
Reporter: Yangxiaoqiang

In kmeans.R,
1. Line 9, in function parsevectors(), list(matrix(as.numeric(unlist(lines)), ncol = length(lines[[1]]))) should better be list(matrix(as.numeric(unlist(lines)), ncol = length(lines[[1]]), byrow = TRUE)).
2. Line 41, takeSample() doesn't work as expected, it returns the entire matrix rather than K random rows. A simple solution would be take the first K rows of the matrix by adding kPoints <- kPoints[1:K,] next to it.
3. Line 56, in reduceByKey(), t(colSums(rbind(p1, p2))) could be replaced by rbind(p1, p2) .
4. Line 64, in collect(), point.sum <- tup[[2]][, -1] should be point.sum <- colSums(tup[[2]][, -1]).

Thank you for your wonderful work of SparkR, looking forward to more convenient and stable release.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-12-026#64007-sha1:039f45b)
Atlassian logo

Zongheng Yang (JIRA)

unread,
Jan 11, 2015, 6:19:02 AM1/11/15
to spark...@googlegroups.com
Zongheng Yang commented on Bug SPARKR-168
 
Re: bug fix in Kmeans.R

[+Yangxiaoqiang]
Sounds like you already have a detailed plan; feel free to submit a pull request on Github: https://github.com/amplab-extras/SparkR-pkg

Zongheng Yang (JIRA)

unread,
Jan 11, 2015, 6:20:00 AM1/11/15
to spark...@googlegroups.com
Zongheng Yang edited a comment on Bug SPARKR-168
[+Yangxiaoqiang]
Sounds like you already have a detailed plan; feel free to submit a pull request on Github: https://github.com/amplab-extras/SparkR-pkg :)

Felix Cheung (JIRA)

unread,
Mar 4, 2015, 12:48:02 AM3/4/15
to spark...@googlegroups.com

Yangxiaoqiang - thoughts? I'd take this and incorporate your suggestions, if you'd like.

This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 4, 2015, 1:33:02 AM3/4/15
to spark...@googlegroups.com
Shivaram Venkataraman assigned an issue to Felix Cheung
Change By: Shivaram Venkataraman
Assignee: Felix Cheung

Yangxiaoqiang (JIRA)

unread,
Mar 5, 2015, 1:40:02 AM3/5/15
to spark...@googlegroups.com

The main problem here is that (the second item in my post) the "takeSample()" function doesn't work as expected. It's either because of the argument we pass to the function is a matrix, and takeSample() works for list rather than matrix; or it's because there is something wrong with this function that I'm not aware of.

Other suggestions listed is just some R programming bugs.

Reply all
Reply to author
Forward
0 new messages