[JIRA] (SPARK-733) Add documentation on use of accumulators in lazy transformation

4 views
Skip to first unread message

Josh Rosen (JIRA)

unread,
Apr 15, 2013, 2:00:53 AM4/15/13
to spark-...@googlegroups.com
Issue Type: Bug Bug
Assignee: Unassigned
Components: Documentation
Created: 14/Apr/13 10:59 PM
Description:

Accumulators updates are side-effects of RDD computations. Unlike RDDs, accumulators do not carry lineage that would allow them to be computed when their values are accessed on the master.

This can lead to confusion when accumulators are used in lazy transformations like `map`:

    val acc = sc.accumulator(0)
    data.map(x => acc += x; f(x))
    // Here, acc is 0 because no actions have cause the `map` to be computed.

As far as I can tell, our documentation only includes examples of using accumulators in `foreach`, for which this problem does not occur.

This pattern of using accumulators in map() occurs in Bagel and other Spark code found in the wild.

It might be nice to document this behavior in the accumulators section of the Spark programming guide.
Project: Spark
Priority: Major Major
Reporter: Josh Rosen
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Anonymous (JIRA)

unread,
Oct 24, 2014, 3:14:51 AM10/24/14
to spark-...@googlegroups.com
Anonymous stopped work on Bug SPARK-733
 
Change By: Anonymous
Status: In Progress Open
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-07-004#64005-sha1:29a0bea)
Atlassian logo

Anonymous (JIRA)

unread,
Oct 24, 2014, 3:14:51 AM10/24/14
to spark-...@googlegroups.com
Anonymous started work on Bug SPARK-733
 
Change By: Anonymous
Status: Open In Progress

Anonymous (JIRA)

unread,
May 20, 2018, 4:07:04 AM5/20/18
to spark-...@googlegroups.com
Anonymous started work on Bug SPARK-733
 
Change By: Anonymous
Status: Open In Progress
Add Comment Add Comment
 
Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS
This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100085-sha1:2007996)
Atlassian logo

Anonymous (JIRA)

unread,
Apr 15, 2019, 1:18:09 PM4/15/19
to spark-...@googlegroups.com
Anonymous stopped work on Bug SPARK-733
 
Change By: Anonymous
Status: In Progress Open
Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS
This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099-sha1:27643ba)
Atlassian logo

Anonymous (JIRA)

unread,
Jun 30, 2019, 7:10:31 AM6/30/19
to spark-...@googlegroups.com
Anonymous started work on Bug SPARK-733
 
Change By: Anonymous
Status: Open In Progress
Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS
This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100105-sha1:327840d)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages