Plotting woes

[Nathan Van Ymeren]

Hello,

I'm trying to learn how lisp-stat plotting works and I'm running into a lot of difficulty.  As a simple test, I thought I would try to plot some time-series S&P 500 data.  I have a .csv containing daily Open, High, Low, and Closing prices for the S&P 500 index going back to 1990.

I'm following along with this article:  https://lisp-stat.dev/docs/tasks/plotting/

I note in particular the following passage: 

"Writing data

For larger data sets, you probably want to save the data to a file or network location and use the Vega-Lite ‘url’ property in the specification. You can write data frames to streams or strings in Vega-Lite format using the dfio:df-to-vl function. You can also use the inverse of this function: dfio:vl-to-df to read a Vega-Lite data array into a data-frame. This is useful for obtaining sample data sets from the Vega-Lite ecosystem."

Seems pretty straightforward, so I created a data file for the plot like so:

(with-open-file (s #P"./spx.json" :direction :output
                       :if-exists :overwrite
                       :if-does-not-exist :create)
  (dfio:df-to-vl spx s))

I can check that file and it's some 900kB of json gobbledegook, looks good so far.  The next section says we need to use vglt:save-plot to save the plot grammar and then call the browser to view it, and here's where I run into trouble.

Following along with the docs it seems we should render the plot like so:

(plot:plot-from-file (vglt:save-plot (vglt:scatter-plot #P"./spx.json" "SPXDATE" "CLOSE")))

Where "SPXDATE" is the date column, and "CLOSE" is the daily closing price.  However, I tried for several hours and no matter what I've tried I receive the following error:

There is no applicable method for the generic function
  #<STANDARD-GENERIC-FUNCTION YASON:ENCODE (13)>
when called with arguments
  (#P"./spx.json"
   #<SB-SYS:FD-STREAM for "file /var/folders/bq/9ngn0wj96zn26hfj1rmypxvm0000gn/T/plot/g995.html" {10053A03C3}>).
   [Condition of type SB-PCL::NO-APPLICABLE-METHOD-ERROR]

I'm assuming there's some problem with the way I specify the file path, but the hyperspec is vague on what a filespec actually needs to look like so I'm at a complete dead end.  Hoping someone can shine some light on this situation!

N

[Nathan Van Ymeren]

Okay, well, some progress.

Reading the source I realized that vglt:scatter-plot and friends have data types hard-coded, which is to say that scatter-plot expects both X and Y to be quantitative variables.  I'm not sure if that's what's blocking me but I wrote a function to generate a line plot conforming to this example: https://vega.github.io/vega-lite/examples/line.html

(defun line-plot (data x y &key (title nil) (description nil))
  "Return a Vega-Lite JSON specification for a line plot"
  (let ((spec '(("$schema" . "https://vega.github.io/schema/vega-lite/v5.json"))))
    (when title (setf spec (acons "title" title spec)))
    (when description (setf spec (acons "description" description spec)))
    (if (typep data 'df:data-frame)
    (setf spec (acons "data" `(("values" . ,(df-to-alist data))) spec))
    (setf spec (acons "data" `(("url" . ,data)) spec))) ;data is an url
    (setf spec (acons "mark" "line" spec))
    (setf spec (acons "encoding" `(("x" ("field" . ,x) ("type" . "temporal"))
                   ("y" ("field" . ,y) ("type" . "quantitative")))
              spec))
    (reverse spec)))

Secondly, I decided to futz around some more with plot calls and I realized that dropping the #P from the arguments avoids the error I was getting, so I guess using pathnames is incorrect.  Issuing the following kinda-sorta works:

(vglt:plot (line-plot "file:///Users/nathan/Code/prototyping/spx.json" "SPXDATE" "CLOSE"))

As does

(vglt:plot (line-plot "spx.json" "SPXDATE" "CLOSE"))

Both of those result in chrome opening with an empty vega-lite plot and no visible data.  I'm not getting any errors, it's just silently failing.  Anyone run into this issue?

[Steve Nunez]

Hi Natan,

 

Welcome to the group.

 

Plotting is undergoing a major refactoring at the moment, and you might be better off using the newer stuff. I’ll make a point to push up a branch of the work-in-progress later today.

 

That said, what’s publicly visible now should work, at least with embedded data. The new version has been tested with URLs and embedded data, and the stream writing has been improved.

 

Have you got a link to the spx data set? I’ll try to reproduce what you’re doing.

 

• Steve

 

--
You received this message because you are subscribed to the Google Groups "Common Lisp Statistics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-stat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lisp-stat/78d7621a-244b-45ae-a4a1-9d49366351f6n%40googlegroups.com.

[Nathan Van Ymeren]

Hi Steve, thanks for your reply.

The data set is nothing special; just a .csv I downloaded from MarketWatch purely for test purposes.  I will try to attach a copy but who knows if google groups will let it through.

N

[Nathan Van Ymeren]

Hi again Steve,

In the interest of you being able to reproduce my issue, attached is a quick distillation of what I was attempting.  I'm on macOS running SBCL, if it matters.

[Nathan Van Ymeren]

Just want to give a public thanks and shout-out to Steve for his helpful replies off-list.

Thanks Steve!