So what's happening with Pedestal 0.3.0?

[Brenton Ashworth]

A bunch of people have asked about the timeline for Pedestal 0.3 and
we want to fill everyone in on what we're doing and why.

Pedestal is made up of two parts, Service and App. Service is a
library for building HTTP endpoints that supports routing and async
processing in a Ring-compatible way. App is a library for building
rich client apps in the browser.

The main idea in Pedestal App is that it makes sense to split
application state from rendering state, that events should be queued
up and dispatched to code that updates an application's state using
dataflow, and that as the application's state changes, instructions
are sent to the rendering layer to update the DOM. This approach gets
you out of having to decide what to do to the DOM by looking at its
current state, which gets very complex for sophisticated UIs. And it
updates the DOM incrementally, making only the necessary changes.

Since the original version of Pedestal App shipped in last March, two
other libraries have really change the landscape. The first is
core.async, which provides standard plumbing for event processing
using channels. The second is ReactJS.

At a high-level, the ReactJS model and the Pedestal model are similar:
application state is separate from rendered state. You process events
by modifying application state, then having UI components re-render
based on the updated state. The main difference in model is that,
where Pedestal App tracks changes in your application model and maps
them (via code you write) to a list of small DOM changes, ReactJS lets
you naively re-render entire portions of the UI as if you were
rendering them for the first time. Then it calculates what needs to
change in the UI and makes only the changes that are necessary. ReactJS
does this by giving your rendering code a pseudo-DOM to work with. All
actual DOM updates are managed by React itself. As with Pedestal App,
you don't have to decide what to do to the DOM by looking at its
current state - you just re-render. But unlike in Pedestal App, you
don't have to worry about how fine-grained changes to your App model
map to the UI.

So there is clearly conceptual overlap between Pedestal and ReactJS.
Further, David Nolen's work on Om has shown that ReactJS and
ClojureScript work very well together. So how do Pedestal and ReactJS
fit together, if at all? One possibility is that Pedestal starts using
ReactJS as the rendering layer, which would remove the need for
generating fine-grained UI updates (because ReactJS makes them
unnecessary). Another possibility is that React + core.async + some
atoms are sufficient to do what most people need. There are other
possibilities too - like ReactJS is useful for some types of UIs but not
others and Pedestal App should focus on those. There is only one way
to answer these questions: we need to build some real UIs with ReactJS
to gain experience with it so we can assess how it fits with the work
we've been doing.

To that end, we are doing the following:

- Splitting Pedestal App and Pedestal Service in two. Pedestal Service
  will continue under the Pedestal name. Most of the work to integrate
  channels with Service to support async operation is complete and so
  0.3.0 of Service should be released relatively soon.

- Moving Pedestal App resources to a separate repo, and pausing
  work on it at least until we gain experience with ReactJS (and
  possibly some related projects like
  https://github.com/cgrand/enliven). Based on that experience, we'll
  decide when and how to proceed with work on Pedestal App.

These changes will be made over the next few days.

If you are putting effort into learning Pedestal App and wondering
what to do, look at ReactJS. The key concepts from Pedestal App
translate almost directly to the ReactJS world. If you dig into it,
share your experiences - as we will. We remain completely committed to
there being a great end-to-end Clojure story, including ClojureScript
UIs, wherever the solution(s) come(s) from.

[rob...@cognician.com]

Thanks, Brenton. I was close to posting on the list to ask where you guys are at :-)

My gut says that the dataflow capability - routing many inputs through rules to many outputs - is still going to be of great value for large apps, particularly for those that require client-side coordination between data off the network and user input, like any multi-user app.

What is your opinion on the utility of the work you've done in 0.3.0 App so far? By that I mean the stuff you walk through here: https://github.com/pedestal/pedestal/blob/master/app/examples/walkthrough.clj

Do you consider it to be usable, or lacking in some way?

Thanks again!

[vje...@gmail.com]

If you need any help with React, feel free to ask questions on irc #reactjs on freenode. The team is hanging out there and we'd love to help :)

[Peter Christensen]

Sort of a bummer to hear about App being tabled for now, but it's worth waiting for the right design. Integrating React with Pedestal App was the next project I was going to try and work on ,but it sounds more complicated than I originally intended.

Will the libs still be available via lein? e.g. 0.2.2-SNAPSHOT was the last one I used - will I be able to create a new App project? I saw the documentation links got pulled, not sure about the repo or libs.

I think the dataflow, queue/message based separation and communication, distinction between data- and app-models, etc are all valuable concepts. Regardless of the overlap between React and the app model/rendering part of Pedestal App, App provided a ton of other instructive features.

Big thanks to the Pedestal team and I'm looking forward to seeing what's next!

[Ryan Neufeld]

Hey Peter,

You’ll be able to find app source in https://github.com/pedestal/pedestal-app. Previously released artifacts (incl. Leiningen plugins) will remain available via https://clojars.org/pedestal. App documentation can be found at https://github.com/pedestal/docs and https://github.com/pedestal/app-tutorial.

-Ryan 

--
You received this message because you are subscribed to the Google Groups "pedestal-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pedestal-user...@googlegroups.com.
Visit this group at http://groups.google.com/group/pedestal-users.

[Yew-wei Tan]

Some thoughts ...

I actually set out to try and use ReactJS via Om together with Pedestal a few weeks ago. This started out as a toy project which I put up here: https://github.com/TanYewWei/pedestal-om-todo

The README of that project goes through some gotchas and miscellaneous thoughts about using the two libraries together.

---

Ultimately, I didn't end up using pedestal in production, mainly because I was unable to adequately test dataflow if clojurescript code was involved.

What I did end up using was precisely the React + core.async + some atoms combination that you mentioned. I could still informally model changes to my app's information model as pure transformations. I ended up using Prismatic's Schema library (https://github.com/Prismatic/schema/) in lots of places to have some form of dynamic enforcement of information model transformations.

(Sidenote: I still have no formalised testing methodology for this setup, but it does afford more flexibility to bolt on testing libraries as needed)

---

One thing that I also struggled with in pedestal-app was how to best go about performing side effects. Many a time, I find that the "destination" of the side effect matters, and that destination is best placed within the calling code; I found dispatching events from the effects queue to disperse my programming logic all over the app.

This is where I personally found having globally visible core.async queues to be conceptually simpler. For example, I was working on some document editor, whereby the user is working on a document, and the app should save the user's work on a periodic basis. But to prevent excessive network burden, the app should only phone home to the server when significant changes have been made (the app decides what "significant" means), yet continue to save all data if possible to localStorage for safe persistence over browser refresh, and as a side benefit, to support persistent and generic undo snapshotting (you can apply whatever historical decay logic you want to this information).

It was conceptually simpler to create 2 core.async queues: one that sent updates to the server, and one that persisted information to localStorage, and then simply call the same "generator" function to create a tree representing the user's data, and then have different functions calling on different intervals to dispatch changes to each of the different queues as needed.

That was a simple example, but I think we can see how this could be scaled up to more complex examples like a type-ahead search widget which dynamically fetches results from some server as the user starts typing, or a code editor that dynamically gives autocomplete suggestions, performs the correct syntax highlighting, etc ...

---

I also found client-side routing to be simpler using just Om and a library like secretary rather than together with pedestal. My toy app from above successfully gets routing done while using pedestal, but I personally feel it's not that idiomatic, and has more moving parts than needed.

With Om and secretary, you would simply turn on a goog.History listener, give it a callback that calls your secretary dispatcher, and then swap out Om roots accordingly (which to me is conceptually similar to sending a :navigate message in pedestal). This generic method could also potentially allow us to create protocols like IWillRoute or IDidRoute, so that you could specify data to be fetched before routing to a specific view/focus, persist needed data, etc .... (but my needs have not required me to do such things yet)

---

Anyway, those were just some thoughts, and while I have potentially more experiences to share I'm still trying to figure out whether what I've done actually works in the long run.

Hope that was interesting for some people.

[Murtaza Husain]

Hi,

My two cents - 

I have tried out a few combinations, and these are my thoughts -  

I like reagent (another ReactJs wrapper in cljs). Two features that really stand out - change in atoms, lead to rerendering of DOM, and ability to specify DOM in hiccup. 

One sore point - On any UI event, I specify a fn which does some calculation and then update an atom. However I like the javelin (hoplon) FRP style as opposed to one in reagent. The javelin style says that on any UI event I directly update the cell (javelin reference type), which then updates other formula cells. 

The BMI example on the hoplon site and reagent site illustrate the different approaches. 

So my preferred approach reagent + FRP (javelin cells). 

So where does pedestal-app come in, it is trying to do exactly what javelin is achieving, however through a dataflow paradigm. The dataflow paradigm may be useful for complex UI's, however for my needs a FRP paradigm would suffice. 

So pedestal could offer a combination of reactJs + hiccup + dataflow.

Thanks,

Murtaza 

[Dan Dillinger]

I'm presently working with a small team to bring a full-stack (both -service and -app) Pedestal application to production, based on version 0.2.2, and with the rendering handing off to Om.

Technically, it's been totally fine. We have been productive with it, figuring out how things work and delivering on our new features.

My concerns are focused almost entirely around non-technical issues. The announcement of shelving -app development was troubling to us here, because of why we specifically selected it. We didn't expect it to be perfect; what we thought we were getting was activity, development, a supported framework from a respected source. We had a need to Get Things Done, and there is value to be found in building pragmatic tools using software that is a living thing. Ending up on something that has been [abandoned, shelved, paused, deprioritized, choose your adjective] leaves us with the hard choice of either taking on maintenance ourselves (which is simply not our business), just living with it knowing this may well be all we're ever going to get, or embarking on an effort to port off of it to something else and start the cycle all over again.

So from that point of view, I almost don't care what the details of the great end-to-end Clojure story looks like as long as it is some sufficient value of great. But to get real work done, I care quite a bit that there IS one at all. Prior to Pedestal, it was the Wild West out there. Embarking on a new project was always fraught with a phase of figuring out what is even going on with the Clojure ecosystem that particular week, and then stitching all of these together, learning how they work (or don't) with each other, and generally taking an amount of time that felt like way too much to get the project off the ground.

I would be kind of sad to see a return to those bad old days. It makes arguments for using Clojure for web development ring hollow in my own ears, when other tools have established ways around them of getting work done now, and I definitely would rather be using Clojure. Cognitect in particular is uniquely qualified to bring the kind of vision to the community that would prevent that.

In short, it seems like perfection has become the enemy of the good, or something. We like Pedestal here! I don't want to see it starve off because someone else had an idea too. Use it if it's better, then! We did, and it's turning out great.

[matd...@gmail.com]

I am about to start a project with a small team, where I am planning on using Pedestal-app for the client. The news that Pedestal-app might be discontinued is sad, as I think the dataflow works really well even for small apps. I have already created a couple of prototypes using Pedestal and enjoy the tooling as well as the ability to get up and running quickly.

As for React, Om and Enliven, as Dan Dillinger wrote I think the best effort would be to use those libraries if they really are better than what is in Pedestal now.

There are some parts of Pedestal that are not working for me. The templating system is one, and so I have been using something else for rendering so far and it works perfectly. For our project the plan is to look into React and Om and use those as the rendering layer if appropriate; but I definately want the dataflow of Pedestal-app at the core of our application.

To summarize, please don't abandon Pedestal-app. If anything, split out the parts that are not central to the dataflow model and let the users choose which libraries to use for e.g. templating.

[Daniel Kersten]

I'm in the exact same boat as Matthias - starting a new project with a small team where I had planned to use Pedestal-app for the client.

When I heard about Om, I was thinking of using it for the rendering (and not using pedestals rendering at all, besides perhaps to render the skeleton HTML into which Om roots are mounted). The deltas from the pedestal dataflow were going to be commands sent to the state managed by Om. That way, I would be duplicating the minimum amount of data: pedestal manages the logical state of the app, Om manages the state to be represented visually. Still some overlap, but acceptable to me (for now).

I absolutely love the dataflow system in pedestal app and how it fits with input, output, postprocessing and services. I think this should be kept.

What needs to be looked at is how the renderer should be implemented to give the best of Enliven/Om as needed for a particular project (I imagine by being pluggable: use only one of the two, both or neither as appropriate) and how pedestal dataflow state and state changes fit in with Om's state.

Maybe the solution is that the state is shared between pedestal dataflow and Om. Pedestal updates the state, Om renders it. Deltas can then be (optionally) configured to be omitted if desired. The use cases for deltas would be:

1. When one doesn't want to use Om for the renderer. This would look similar to what we have in pedestal-app now.

2. To send data to services.

----

On the pedestal-service side, I would like to see an implementation of the dataflow system where the renderer is replaced with routing/interceptors and the services would be database access.

[Nathan B]

Another +1 for dataflow programming being the core reason we use Pedestal. The other pieces are all the less interesting parts. The dataflow aspect is what makes it unique and also really powerful once you build around it. If the rendering was pluggable which it sort of already is and the server side just went to some default implementation using http-kit  in order to get the long running connections, that would be fine as long as the dataflow part was what was uniquely "Pedestal" and was kept around and expanded out even further. I also think that would make it easier to digest the framework if it was just one thing versus, all the stuff it tries to do right now.

Just my 2 cents

[fmj...@gmail.com]

+1 to abstract the dataflow aspects into a nice cljs library

+1 to keep it agnostic towards any rendering technology, be it reactjs or any other

In other words, refactor for more composability, and don't just take ReactJS/Om/Reagent as the only rendering tech.

In my case I'm considering the use of Enyo for building a cross-platform application targetting mobile+browser+desktop.

ReactJS and Enyo are very similar in their approach as explained in this post.

I'm also considering async processing on the server-side, so Pedestal does seem to be a good fit, but then http-kit and immutant also become possible solutions. Therefore I would add to the list:

+1 to better use of async technology like core.async, websockets, http-kit, sente.

The project is still on the starting block so I have yet to find out how all these building blocks can fit together and what makes the most sense . Any opinion appreciated BTW.

[François Rey]

I just watched this interesting video on reinventing the REPL which can offer some good inspiration or even some requirements for Om/Pedestal/graphical REPL kind of projects as discussed in the recent threads about the future of Pedestal and interactive REPL development such as the Gorilla REPL and InstaREPL of LightTable. The same subject was  further developed at Clojure/con 2012, the project is called Session on github.
I certainly agree that notebook/graphical/interactive REPL is the way forward, as attested by projects such as Session, Gorilla REPL, and LightTable. Immutability and dataflow seem to be key ingredients there so I guess whatever Pedestal-app/Om become, part of the requirements should be to make it easier to develop such tools. The recently demoed Wolfram Language will probably increase the pressure towards such evolution in the computing community.

[Taylor Sando]

Any news for the pedestal-app/dataflow portion of the project?  It seems like a dataflow engine that produced a stable tree each time would work well with om rendering.  You'd just need the initial om view, which could produce all its sub views/components based on the current tree state.  You'd only really need one global channel for all the om views that you could place messages on.  Those messages would then go back to the dataflow input, which would then spit out a new stable tree, causing a re-render.

I think the problem with om right now is that the view seems like it has a little too much knowledge of the underlying model.  I've used the flux architecture http://facebook.github.io/react/docs/flux-overview.html with facebook react, which splits the view from the underlying data model.  

The problem with the old pedestal-app was that I think it was too fine-grained for most use cases.  You had to come up with rendering functions for every node-create, destroy and transform-enable.  It seems for many web apps, you could get away with using something like om for your rendering engine, which can figure out what's changed in your app tree, and render accordingly.

Also, if you structure your tree app state like this:

{:active-key :key1

 :key1 {:active-key: :key1

           :key1 {:state {:status :loading

                                :errors []}}}

Your main views would act as a kind of controller.  They would have a multi method that would dispatch based on the :active-key value, which would point to the next key/path.  The next view in turn, would then look at its :active-key, and pass it to the next controller-view, or if it was a leaf view, would simply render what was left.  I've found that it's easy to go from state->route, and from route-> state using this kind of structure.

[Kyle Cordes]

On Thursday, July 10, 2014 at 1:44 AM, Taylor Sando wrote:
> Any news for the pedestal-app/dataflow portion of the project? It seems like a dataflow engine that produced a stable
>

From this thread here a while back:

https://groups.google.com/forum/#!topic/pedestal-users/jODwmJUIUcg

… it sounds like the whole of pedestal-app is on ice.



--
Kyle Cordes

http://kylecordes.com

[Ryan Neufeld]

That thread and this are one and the same. I don’t work as closely with the Cognitect folks as I used to when I worked there, but I haven’t heard any news of further development.

I know Luke worked a little on https://github.com/levand/quiescent, which was a slightly different approach than Om.

[man...@gmail.com]

I'm a bit dismayed. It seems to me that given an atom that represents a virtual DOM, and 1 or more atoms that represent the application data model, the whole stack falls out neatly with queues of messages between the bits, and functions registered to listen to queues.

Pedestal-app shouldn't die. Why can't we produce a virtual dom -> real dom loop that fires x times per second and rehook existing APIs to a virtual dom atom? There are animation issues, but that's simply a matter of registering functions that emit a sequence of dom mutations. That sequence could be registered to another event loop that wakes up 30 times a second to play animation sequence transformations.

When the next render loop fires to transform virtual dom -> real dom that animation transform surfaces in the real dom. I'd like to see how that would perform. With so much less real manipulation of the DOM it would seem to be a no-brainer. Also, the real DOM rendering engine could hide all kind of efficient object reuse under the covers. (Of course that hole process could move inside the browser's rendering engine also...)