D3.js editor for faster development

[Luca Matteis]

Hello d3 community!

I've been working with d3 for a bit now and I've been thinking about an editor that would facilitate d3-specific code development. For this reason I thought I would first ask the community to understand whether something like this already exists. The editor I'm thinking about would have things such as instant preview side-panel, API auto-complete, highlighting of elements when code-hovering (ex. hovering `d3.selectAll('rect')` would highlight those elements on the side-panel), automatic state recording for replaying interactions (imagine building interactions and having to replicate them every time your code changes. The editor could record interactions and replay them to facilitate changes for each specific state), skeleton projects to facilitate the creation of new visualizations using standard templates, etc...

Any thoughts would be greatly appreciated.

Best,

Luca

[Thom]

I started making a low level SVG editor that I plan to eventually give similar functionality to this, but more geared to SVG generally.

It's currently in very early stages (I first made it so i could quickly throw together some SVG and tweak viewboxes) but you can get a general idea of where I'm going.

http://svg.rocks/

[nick]

Luca:

This is a very cool idea. I've kind of always wanted to do a plugin for Inkscape that supported d3. However, having done a bit of hacking there, the browser is probably the right place for this kind of work... unless one went down the Electron route.

THE UI
I have been doing work with d3 for a while in the Jupyter ecosystem. The Jupyter Notebook, a websockety JavaScript SPA that talks to arbitrary backend "kernels", lets you get to the "good stuff" of d3 very rapidly, without thinking too much about that "other part" of data work: the backend! Python, (backend) javascript, haskell, scala and others are all available, and each supports a low-friction way to get stuff out to the front-end. Built with codemirror, backbone, jquery, etc. it can support things like autocomplete (of both the backend language and javascript!). Integrating directly into this environment might be a pretty good idea!

An interesting use of this is bqplot, a d3-based charting library that uses the Jupyter "widgets," reactive stateful models, synced between the browser and the backend. There are backends for this in python, julia and haskell, and maybe more! This *really* solves the backend-frontend issue, but hides much of the d3... and only works for python!

For the forthcoming 5.0 release (JupyterLab), a new library is being built called PhosphorJS, a framework for building IDE-like applications: demo. It is pretty easy to use d3 in this environment, as you have pretty good low-level access to DOM nodes, but the APIs are deliberately open such to encourage other approaches. Another, indeed frontend, based on Electron (and react) is nteract, and works with all the underlying stuff.

Core to the mainline 5.0 work is a bunch of stuff related to a reusable IDE (think more atom than codemirror, with command palettes, menus, draggable panels, etc.) called Phosphide: demo. This might be a really good place to start, adding the other concepts (generated SVG introspection, etc) as needed, even if the full notebook stuff isn't used. I basically built my own d3-based UI layer for nbpresent, but I think a lot of things in the "editor" family would benefit from the phosphide opinion.

THE STATE
A great library, also used for nbpresent, is Baobab. It is an implementation of Om cursors, and supports an immutable, evented tree. Writing UI with d3 with this is awesome, and it supports history, computed data, etc. I've also done some work with d3 + baobab + typescript, which is a great way to reduce `function(d){}` fatigue. Whether you make the user deal with this is another story... but it would be a great example for a skeleton...

THE SKELETON
Well, depending on your acceptance of having a backend, either Yeoman or Cookiecutter would be good places to start, rather than building an entire new concept. Neither has a web UI today, but either probably could. I've considered in the past using Jupyter widgets to make visual the bootstrapping process, but something bespoke would be cool as well.

THE DOM TRACKING
To get to Inventing On Principal-level DOM<--> source introspection, it would likely be necessary to introduce some overloaded stuff, perhaps, to the __data__ key that d3 uses... or just go the pre-processor route. Probably the hardest problem you mentioned!

[Ian Johnson]

Hi Luca,

I don't think something like this exists and it would be great to see it! I did start down this path a bit with Tributary http://tributary.io

It is focused on the live reloading and a little bit of augmented programming (number sliders and color picking) as well as inline-console logs. I explored "rewriting" user code to add functionality with ASTs (abstract syntax trees)

Here are some examples where I used tributary to prototype these concepts (meta!) 

Inline console logs (hijack the functionality of console.log in user code): http://tributary.io/inlet/5564948

Modifying a for/while loop to set a max # of iterations (show the AST in a jsoneditor as well): http://tributary.io/inlet/5597237

I went a different direction with blockbuilder.org, where I wanted to replicate local development as much as possible and speed up the creation of d3 examples (rather than augment the coding process). For this reason one of my design principles is that a user should be able to copy paste code out of blockbuilder into their local editor and expect it to work exactly the same. This makes it a little harder to introduce helpful magic without a lot more engineering. There are enough technical challenges mirroring browser functionality in a sandboxed iframe :)

You mentioned on twitter that I could make blockbuilder smartly update just the parts that need updating (style, js etc) which is exactly the way tributary worked. I wish tributary was a better codebase to start with, but I used it as my personal playground for learning app development so it got pretty crufty over its few years of existence. I'm happy to talk about the architecture lessons I learned though, I'd love for the augmented programming aspect to be pursued in a deeper way!

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ian Johnson - 周彦

http://enja.org

[Tito]

this is a great idea. I am working on an extension for Brackets http://brackets.io/ ide. Work in progress. not sure if it will help but its gonna be a beast of a project.

[Curran]

Awesome ideas! I was going for something similar with the Chiasm project, which completely separates the state (configuration) from the visualization components, which are completely separate from the data. Please a look at these slides, perhaps our ideas are overlapping on some of this. One of my dreams is also to have history navigation like in photoshop, where you can undo and redo any actions, kind of like scrubbing back and forth in a video editor - all without re-evaluating the whole code, just propagating the minimal changes through the system.

The ultimate thing would be to make all of this real-time collaborative, which Ian touched upon in this really cool project https://github.com/enjalot/cypher . I saw him give a demo of this once and the concept is amazing - people can edit the code and the text is real-time synchronized with everyone else like Google Docs. There are little previews of all the running programs too, and these are also updated in real time. It would be so cool to make something like this that included the notion of history navigation and a collection of, as you say, "standard templates" for various visualizations.

Here's a simple IDE that just passes JavaScript through "eval" when the code changes https://github.com/curran/evalide .

Another approach might be to use LocalStorage to track the state, but re-evaluate the entire page like Blockbuilder does.

All the Best,

Curran

[Ole Kröger]

Any news? I just started with D3 and I'm using Brackets for a while and build several extensions.

Unfortunately I wasn't able to find any brackets extension for d3 yet.

[Curran]

IMO blockbuilder.org is the best for this. What features is Blockbuilder missing that you are looking for?

[Ole Kröger]

Thx, I was looking for a normal (non-online) option ;) I am working a lot in brackets because I implemented several extensions so an extension for Brackets would be nice.

Anyway I'll have a look on blockbuilder.org and the option to have several examples there is definitely worth a look, I guess.