Understanding vega.changeset() versus .remove() and .insert()

Alex

unread,

Jun 15, 2018, 11:25:10 AM6/15/18

to vega-js

The specification is attached here, as well as the CSV file of data. I haven't included the jquery and D3 libraries since they're pretty standard everywhere.

I have a visualization of some stocks data where I'd like to filter by date dynamically. I'm using HTML and Javascript to trigger the filtering rather than a Vega signal approach because I need to automatically find the extent of the time data in order to have an initial default (i.e. I don't want to manually type in the default values for the signal). The filtering is very buggy at the moment when I use the changeset code, but when I use the view.remove and view.insert instead it works (although it does delete the axis titles when I do it this way). I have included a screenshot below of the weird behavior that occurs when I use the changeset code, which I thought does the same thing as the .remove and then .insert on separate data pulses. Can someone explain to me what the difference is between the two methods in the highlighted code above? And/or any modifications I can make to the changeset code to make it work?

Thanks in advance.

<!DOCTYPE html>
<head>
  <meta charset="utf-8">
  <script src="https://cdn.jsdelivr.net/npm/ve...@4.0.0-rc.2"></script>
  <script src="jquery.min.js"></script>
  <script src="d3.js"></script>
  <style>
    body {
      font-family: sans-serif;
    }
    
  </style>

</head>

<body> 
  <h3>Load in the stocks dataset with d3, but then do the filtering using JS and update using Vega.</h3>
  <p>This filters for the data to be added and removed on the fly using Javascript, not D3. But it still uses D3 to load in and parse the stocks.csv file for the default values.</p>
  <div class="form-group">
    <label>Time Period: </label>
    <input type="date" class="form-control" id="time-period-min" placeholder=""> - 
    <input type="date" class="form-control" id="time-period-max" placeholder="">
    <p><button type="button" id="time-period-filter" class="btn btn-default" onclick="filterData()">Filter</button></p>
  </div>
  <div id="vis"></div>
  <script type="text/javascript">
    loadData();

    // create a d3 parser for dates: from string to date in expected format Jan 1 2000
    var parseDate = d3.timeParse("%b %e %Y");
    // create a d3 parser for dates: from dates to string in outputted format Jan 1 2000
    var formatDate = d3.timeFormat("%Y-%m-%d");

    // variable to store the loaded data
    var data;

    // Load CSV file
    function loadData() {
      d3.csv("stocks.csv", function(error, csv) {

        // ensure numbers are read as numbers and dates as dates
        csv.forEach(function(d){
          d.date = parseDate(d.date);
          d.price = +d.price;
        });

        // Store csv data in global variable
        data = csv;

        // Get full time period
        var originalTimePeriod = d3.extent(data, function(d) { return d.date; });

        d3.select("#time-period-min").attr("value", formatDate(originalTimePeriod[0]));
        d3.select("#time-period-max").attr("value", formatDate(originalTimePeriod[1]));

      });
    };

    var view;

    vega.loader()
      .load('vega-line-chart-d3-datefilter.json')
      .then(function(spec_data) { 
        render(JSON.parse(spec_data)); 
      });

    function render(spec) {
      view = new vega.View(vega.parse(spec))
        .renderer('canvas')  // set renderer (canvas or svg)
        .initialize('#vis') // initialize view within parent DOM container
        .hover()             // enable hover encode set processing
        .run();
    };

    function filterData() {
      console.log("Filter function has been triggered");
      var parseDate_input = d3.timeParse("%Y-%m-%d");
      // get the newly selected options for time period
      var timePeriodMin = document.getElementById("time-period-min").value;
      var timePeriodMax = document.getElementById("time-period-max").value;
      console.log("The string versions of the time period I am filtering for is: " + timePeriodMin + " to " + timePeriodMax);
      // timePeriodMin and timePeriodMax are both strings

      var newData = data.filter(function(d){
        return d.date >= parseDate_input(timePeriodMin) && d.date <= parseDate_input(timePeriodMax);
      });

      // The following code does not work, and I do not know why
      view.change('source', 
        vega.changeset()
          .remove(view.data('source').slice())
          .insert(newData)
        )
        .run()

      // I thought the below code did the same thing as above 
      // but the below works while the above (changeset) does not
      // view.remove('source', view.data('source').slice())
      //   .run();

      // view.insert('source', newData)
      //   .run();

    };

  </script>
  
</body>

stocks.csv

vega-line-chart-d3-datefilter.json

Dominik Moritz

unread,

Jun 15, 2018, 1:02:30 PM6/15/18

to Alex, vega-js

I don't know why these two methods would behave differently but the easiest way to replace all data is to use `.remove(function() {return true})`. I often use `view.change(vega.changeset().remove(()=>true).insert(data)).run()` in my applications.

--
You received this message because you are subscribed to the Google Groups "vega-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vega-js+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alex

unread,

Jun 15, 2018, 1:10:03 PM6/15/18

to vega-js

I just tried that, but got no difference in behavior. I think that's to be expected since ".remove(view.data('source').slice())" and ".remove(function() {return true})" should theoretically do the same thing. Thanks for the tip though.

Dominik Moritz

unread,

Jun 16, 2018, 11:41:38 AM6/16/18

to Alex, vega-js

Using a function that returns true will be faster since no copy of the data is necessary. The different behavior sounds like a bug to me but I haven't tried it.

Roy I

unread,

Jun 19, 2018, 1:37:06 PM6/19/18

to vega-js

Alex, Take a look at:

Vega v4.0.0-rc.3 (2018-06-18)

"Fix changeset by adding safeguards for input data: ensure existing tuples are not inserted multiple times and ignore tuples that are both added and removed in the same set."

https://github.com/vega/vega/releases

Vega Issue#1318 Add safeguards to changeset processing?

"When performing streaming data updates, Vega expects the add / remove / modify sets to be distinct. However, clients might easily violate this expectation (e.g., using vega.truthy as the remove predicate and including existing tuples in the add set, as in #1305)."

https://github.com/vega/vega/issues/1318

Note: Vega View API documentation does not mention this "expectation".

https://vega.github.io/vega/docs/api/view/#view_change

Vega Issue#1305

"...vega-dataflow 4.0.2, which includes new safeguards for changeset processing ... will be included in the Vega 4.0.0 release."

https://github.com/vega/vega/issues/1305

-------------------------

When testing your example using Vega View API changeset() with Vega v4.0.0-rc3, the chart is rendered incorrectly after the user interactively changes the Time Period several times:

view.change('source',

vega.changeset()

.remove(function(){return true;})

.insert(newData)

)

.run();

This works properly:

view.remove('source', function(){return true;})

.run()

.insert('source', newData)

.run();

Yuri Astrakhan

unread,

Jun 19, 2018, 1:48:40 PM6/19/18

to Roy I, vega-js

There was some discussion about https://github.com/vega/vega/issues/912 - including observable implementation (#893). Observable support would allow integrators to implement custom data sources that push new data snapshots when they become available.

BTW, if possible, please update main vega docs with clarifications if you think they would benefit others. Searching forums is fairly difficult. Thanks for all the hard work on this!

Alex

unread,

Jun 19, 2018, 1:56:21 PM6/19/18

to vega-js

Thanks for flagging those Github issues for me Roy, I hadn't seen those. I agree that the remove and insert code works, I just wanted to see if I could understand why the changeset code wasn't working. I have tried running the changeset code while using the latest release of Vega (v4.0.0-rc.3) (with no additional changes to my code) and it works now, with the added benefit that the changeset code doesn't remove axis titles like the remove and insert sequence does. I couldn't replicate the problem I'm seeing in your screenshot.

Roy I

unread,

Jun 19, 2018, 6:04:16 PM6/19/18

to vega-js

Alex, changeset() seems to works now with Vega 4.0.0-rc3. Not sure why I experiences the issue when testing earlier.

On Friday, June 15, 2018 at 11:25:10 AM UTC-4, Alex wrote:

Reply all

Reply to author

Forward