[ANN] Oz 1.4.0 - Interactive data visualizations and scientific documents with Vega/Vega-Lite

[Christopher Small]

Greetings!

I'm happy to announce today the release of Oz 1.4.0.

https://github.com/metasoarous/oz

If you're on the Slack #datascience channel, you may have already caught wind of some earlier versions. But in the interest of introducing it more broadly, I'm posting an overview here for those of you who aren't familiar. If you are familiar, you may still wish to scroll down to the bottom as there are some new features available in the latest release.

Vega & Vega-Lite

Oz is based on the fantastic Vega & Vega-Lite data visualization JS libraries, and so to really understand what Oz has to offer, it's best to start here. Vega & Vega-Lite are based on the seminal Grammar of Graphics, an approach to data visualization which emphasizes writing declarative descriptions of how properties of data should translate to aesthetic attributes of a visualization. This approach guided the design of the R's popular ggplot2 library, and has since influenced numerous libraries in other languages.

Vega & Vega-Lite take this vision further in two important ways:

1. In Vega & Vega-Lite, data visualizations are described using pure data. This makes it more declarative, and confers all the benefits we know and love about data-driven programming in Clojure. For instance, you can send a chunk of Vega or Vega-Lite data over the wire from one program to another effortlessly (as Oz does), and load it up in another process without having to worry about the security concerns of executing someone else's code. The bottom line is that Vega & Vega-Lite are philosophically and technically compatible with "the Clojure way" (IT'S. JUST. DATA.).
2. Vega & Vega-Lite take the Grammar of Graphics one step further by introducing a Grammar of Interaction. You can declaratively describe the addition of controls (dropdowns, checkboxes, etc) and interactive properties of the visualization itself (click, hover, etc), and use the data from these interactions to inform other parts of a visualization. For example, you might highlight a set of points in one part of a visualization, and display summary statistics about that selection in another. This is facilitated in part by a general purpose dataflow language as part of the greater spec.

Vega itself is highly customizable and flexible, but somewhat verbose and not suitable for day to day visualization tasks. Vega-Lite steps in as a somewhat higher level and more automated flavor which itself compiles down to Vega. I have been using them together for a better part of a year now, and can say without reservation that they are amazing. For years I've longed for a ggplot2 from Clojure, and at long last I've found something that to my surprise has not only matched, but truly surpassed the standard bearer. In short, I'm sold.

If you want to get a better sense of Vega, and Vega-Lite in particular, I'd recommend this great talk from the creators at the Interactive Data Lab at the University of Washington in Seattle: https://www.youtube.com/watch?v=9uaHRWj04D4

If you're interested in a (mostly) more philosophical look at Vega & Vega-Lite, and their connections to Clojure philosophy, I did a little talk at a local Clojure meetup which you may find interesting: https://www.youtube.com/watch?v=hXq5Bb40zZY&t=815s

Oz

Oz itself is a very small and focused library, as most of the work falls on Vega & Vega-Lite. It offers the following features:

• A REPL API for for pushing vega and vega-lite data to a browser window over websockets, for REPL-based data science workflows
  
• Client side vega and vega-lite Reagent components, for more dynamic usage from ClojureScript apps
  
• A grammar for composing Vega & Vega-Lite together in the context of html as hiccup, for document and dashboard generation
• Plot/document publishing/sharing features via GitHub gists, the IDL's live vega editor, and the new http://ozviz.io

The last two features in particular are where Oz really brings some interesting value to the table beyond the role of a minimal wrapper. I have found the ability to create and quickly share visualizations and scientific documents from the comfort of my favorite text editor and REPL a godsend. While the first several years of my programming experience were in notebook environments (Mathematica, MATLAB, Sage, etc), I now find the experience of writing and executing code from a web application a burden. Part of my goal with Oz was to create a viable alternative to this workflow, and so far I've been very pleased. The last piece to this now in place (the ability to share hiccup+vega documents via http://ozviz.io), I'm excited to put this work out more broadly and hear what the community thinks about this approach to the creation and sharing of scientific documents.

There are some other updates and improvements which those of you familiar with Oz may wish to take a look at in the changelog, included updated Vega* libs, and some smoothing out of the API and UI (https://github.com/metasoarous/oz/blob/master/CHANGELOG.md). Otherwise, please see the project README for up to date information on how to use the library: https://github.com/metasoarous/oz.

Thanks for your time!

Chris

[Colin Yates]

Looks great! The link to examples (https://vega.github.io/examples) 404s.

Sent from my iPhone

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christopher Small]

Good catch! The examples page was split up into separate pages for Vega vs Vega-Lite, and I just updated the README to reflect this.

Thanks for taking a look and letting me know!

Chris

[Juraj Martinka]

I'd like to try this but got stuck pretty early:

(ns clojure-repl-experiments.visualizations.oz
  (:require [oz.core :as oz]))


(oz/start-plot-server!)


(defn group-data [& names]
  (apply concat (for [n names]
                  (map-indexed (fn [i x] {:x i :y x :col n}) (take 20 (repeatedly #(rand-int 100)))))))


(def line-plot
  {:data {:values (group-data "monkey" "slipper" "broom")}
   :encoding {:x {:field "x"}
              :y {:field "y"}
              :color {:field "col" :type "nominal"}}
   :mark "line"})


;; Render the plot to the
(oz/v! line-plot)

It has opened a new browser window at http://localhost:10666/ but I see nothing only errors in the JS console:

socket.cljs?rel=1502542805393:64 WebSocket connection to 'ws://localhost:3449/figwheel-ws/dev' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
figwheel$client$socket$open @ socket.cljs?rel=1502542805393:64
10:10:10.089

Does it require some special setup (figwheel)?

[boris....@gmail.com]

Odd. The exact same code works for me. This is clojure 1.10/oz 1.4, and evaluating the whole blob from lighttable. 

I had to call (oz/v! line-plot) again to get it to show the figure, rather then the opening text. And you can leave out the (oz/start-plot-server!). It will start a server if it needs one.

I guess I have a similar workflow as Christopher, and similar needs in terms of visualization. I have used vega-lite through vizard and now oz for about a year now, after trying so many different visualization packages for clojure (Incanter/JFreechart, C2, quil, gyptis, quil/grafica, rojure->ggplot2, vizard). Really happy that oz takes vizard further. vega/vega-lite works really well with clojure.

[Christopher Small]

Hi Juraj

The code you pasted should work. If you're still having trouble with this, please raise a GH issue with all pertinent system/setup details so we can sort it out there: https://github.com/metasoarous/oz/issues.

I can say that sometimes it takes a while for the websocket connection to establish, and the exact time may depend a bit on machine details. This is what Boris is getting at about having to to call `(oz/v! ...)` again (Thanks for chiming in Boris!). The first thing to try is to just wait a bit longer after calling `(oz/start-plot-server!)`, and try reevaluating `(oz/v line-plot)`.

Oh... one last thing to check. If you had tried an older version of oz prior to this and are just hoping to check out the updates (which I'm wondering about in part because I see you're running the old example code, though you could have gotten that from the video), it might be worth running `lein clean` in case this is a problem with a stale build target.

Thanks!

Chris

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/l-uSMBgI8Ko/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Alan Thompson]

Looks very nice.  I will definitely be using this in the future.

Alan

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Juraj Martinka]

Hi Chris and Boris,

Thanks for the quick response.

Here's the minimal example: https://github.com/jumarko/oz-sample

It looks like a dependency conflict with something in my ~/.lein/profiles.clj

If I run it via `lein repl` it fails.

If I run it only with proflie then it works!

lein with-profile base repl

I did some aggressive cleanup of my profiles.clj and I can run it now.

Unfortunately, I can't really tell what was the offending dependency since I removed lots of them

(I tried to exclude several of them one-by-one but didn't find the problematic one).

Now when it's working, I'm really looking forward to playing with this. 

Would you mind sharing more info about your workflow and how you typically use it?

Is it something you'd use for debugging for example?

Thanks again for your help.

[Christopher Small]

Hi Juraj

Glad you have it working now!

I mostly use oz from the Clojure REPL as an exploratory tool in my work on Polis (see https://www.youtube.com/watch?v=2tBVMAm0-00). I have yet to use the Reagent API in a dynamic front end, but I have used Vega & Vega-Lite in a vanilla React app (see https://github.com/matsengrp/olmsted), so I know what's possible and look forward to being able to do the equivalent in ClojureScript.

I'm a vim user, so I use vim-fireplace to connect my REPL to my text editor. This lets me write and execute plot building code directly from the comfort of my editor, and have a separate browser window open for looking at the results. This gives me a pretty tight feedback loop for visualizing and exploring data. If you're using the Reagent API with Figwheel, you should get a pretty similar feedback loop for development of

To be perfectly honest, debugging can be a bit challenging when first using Vega & Vega-Lite. The IDL's Vega Editor has some helpful tools for analyzing specs for problems, and a tight feedback loop of tweaking and re-evaluating plot code can help keep you on the right path till you get your bearing. They're also working on some really neat debugging tools to help visualize the compiled dataflow topology in order to better diagnose issues, so hopefully this area will improve.

The best way to get started is to go to the Vega & Vega-Lite Examples pages (https://vega.github.io/vega/examples/ & https://vega.github.io/vega-lite/examples/), which helpfully showcase a panapoly of specs available for use as starting points. Once you find a related spec, it's usually only a few changes to a get a "bare bones" adaptation to your data. From there, you can generally stitch in spec code from other examples as needed to get the plot you want. If some "stitching" doesn't work, look for other examples combining similar functionality if possible, and if not raise an issue. The IDL is super responsive and active in their development and release cycle, so bugs generally get solved very quickly; In fact on rare occasions where I've found a Vega bug or something I wanted to do but couldn't easily, I've frequently discovered that there's already been a fix released, or had them specifically address the problem within a matter of weeks.

Thanks again

Chris

[Christopher Small]

EDIT: ...for the development of dynamic web applications.

[Christopher Small]

Great to hear. Thanks for checking it out!

[Juraj Martinka]

Great, thanks for sharing!

[Christopher Small]

Quick note for posterity: Jaraj found a bug in the default credential path code (didn't work on mac) which he submitted a patch for and which I've just released as 1.4.1. So if you're on a mac and want to check out the `publish!` functionality, please update to 1.4.1.

Thanks again Juraj!

Chris