Hey all,
I've opened up a thread about ways to make working with Jupyter + Spark a much more rich environment to make users more informed, productive and happy while enabling library authors (e.g. PySpark, Spark, Dask, Pandas, Presto, ...) primitives in Jupyter kernels to provide this user experience:
Some of these may be better explained with some mockups, which I can certainly provide.
--