Language Synergy

[Eagon]

Really cool how Rama leverages all sorts of colocation (computation, memory, network, etc.) and simplicity you get by building on a unified abstraction for purpose-built data index structures. Reactivity, simple partition access patterns, scalability, and synergy with event-sourcing seem to all fall out of choosing to build on top of this abstraction, rather than needing to be engineered and shoe-horned in piece by piece. The freedom to partition on usage patterns (such as account ID for statuses) not even present in the index is very compelling, and it's heartening to see that an intentional balance was struck between ease of use and exposing enough operational control.

Is there a story for synergy with Rama for languages other than Java? From following previous work at RPL, I recall there was mention of an initial DSL developed in Clojure. A Java-first API makes the most sense for many reasons, but the Java DSL at first glance does not seem to lend itself very well to code analysis (and perhaps metaprogramming) for the purpose of reasoning about extremely involved ETLs, PStates, etc. Personally I'm excited for considering the possibility of building on top of Rama as the abstraction for representing scalable data manipulation and storage in the space of data science. An introspectable DSL would enable tools built in this space to serve:

1. Auto-generated never out of sync documentation for data transforms and stored indices (PStates)
2. Visualizations for dynamic exploration of application/data pipeline, particularly useful for investigating edge cases and "what-ifs" for complicated data flow diagrams
3. Metaprogramming to generate speculative dataflow Rama code for the purposes of testing, comparatively evaluating performance of functionally equivalent PState/Query setup pairs, or compiling domain specific queries to Rama primitives

A Clojure API or DSL would lend itself very well to these example use cases as one possibility, but I'm just fairly excited about these potential avenues in general. Thanks again for all the work on Rama!

[Nathan Marz]

These are interesting ideas. You've already gone beyond basic Rama usage into advanced territory :)

We do have a Clojure API to Rama, and the Java API is actually just a thin wrapper around that. It's unlikely we'll expose that anytime soon, but it's also not a big project to make a Clojure wrapper around the Java API. It would work well even though Clojure -> Java -> Clojure is a bit unusual.

For your first two ideas, the Rama implementation is well set up to add first-class support for these. It would be something like "ReifiedModule reified = RamaModule.reified(new MyModule())", and the "ReifiedModule" would let you inspect and walk each declared depot, PState, and dataflow graph. On top of that we could build a visual debugger like in your second idea. 

I like this idea a lot and added it to our roadmap. That said, it is definitely doable with a Clojure wrapper, although it's a non-trivial amount of work, especially the debugger.

As for your third idea, generating dataflow code from a domain-specific representation is actually straightforward as-is, and we do things like that in the Mastodon implementation. The "KeyToLinkedEntitySetPStateGroup" abstraction, for example, generates dataflow code to implement that data structure on top of multiple, more primitive data structures. For instance, it uses a Rama macro to implement "adding to a linked set" which expands to a series of operations on multiple PStates.

More generally, since you're defining dataflow code from within a general purpose language, you can dynamically generate that dataflow code.

I'll be interested in hearing what other ideas you have in these areas or others, especially after playing with the build or Rama we're releasing next week.

--
You received this message because you are subscribed to the Google Groups "rama-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rama-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rama-user/c6015752-483a-4957-aa3f-94ea55d671e3n%40googlegroups.com.

[etza find]

I would add a vote for work on a Clojure wrapper.

To view this discussion on the web visit https://groups.google.com/d/msgid/rama-user/CANRumhLW24m5RnKuFXSRCTJnr7or5KNda7o4otKnD2XRVD9DzA%40mail.gmail.com.

[Marc Brooks (IDisposable)]

Personally, I'm going to look at the source as soon as it drops so I can see how the ideas can be ported to .Net Core

[Pawel Ceranka]

+1 here as well. Definitely would be interested in Clojure API / wrapper: as we are full stack Clojure: datomic -> clj -> cljs.

[Vedang Manerikar]

+1, full-stack Clojure here as well, and it would just be easier to work with a Clojure API. (sorry for the spam, but adding my vote in the hopes the RPL team prioritises releasing the Clojure API)

[Patrik Sundberg]

+1 - it's probably safe to assume there'll be a decent size group of clojure devs in the early adopter bucket :)

[Daniel Jomphe]

+1 - after years of REPL-driven development, can't imagine losing that. But still, willing to try Rama as it is and, obviously, with any clojure wrapper too, hoping, though, that an official clojure API could be made available instead of having an unofficial (or fragmented tries thereof from different people) clojure over java over clojure wrapper.

[Christopher Genovese]

+1 on an official clojure API.  I suspect there will be substantial interest in the clojure community, and it would be good, I think, to avoid the churn of a zillion wrappers being written. In any case, I'm looking forward to learning more about Rama and trying it!

[Mike Potts]

+1 on a clojure API: that was my first thought when I read about it!

[Oskar Boëthius Lissheim]

+1 for working on this from a REPL, so please reconsider not exposing the Clojure API "anytime soon" :) 

[Daniel Jomphe]

Surprised how much clojurians whose names I don't know (I've been closely watching the clojure space since 2008/9) come up here. I suppose Nathan can see this as a strong interest signal from us clojurians.

OTOH, I guess Red Planet Labs hopes and needs to see Rama reach critical mass adoption not only from us, but especially from many java/kotlin/scala/etc shops. I'd further guess that Nathan hopes Rama will not be seen as "the clojurey thingy best used in clojure shops" and wouldn't hope for us clojurians to publish articles like "see how Rama is so much better used with clojure than with java". Based on these guesses, I suppose Nathan would like to give Rama at least a 2-3 year head start before starting to pitch an official clojure API. Am I painting us in a dark corner? :-)

Concurrently, I'm curious... how did the RPL people used to clojure who implemented the Java-based mastodon demo feel while developing it? Did they miss their REPL a lot? Obviously, if my above guesses are right, I suppose we shouldn't expect a detailed response. RPL's attention would be better spent supporting what they're currently putting out for now.

About my tone:, please be gracious with me: English is not my main language. I hope what I wrote won't come out as potentially rude or patronizing. I feel enthusiastic and prudent :-) and I wouldn't want any clojurian to feel forbidden about writing enthusiastically. Well, this comes as a long disclaimer...! :-)

[Nathan Marz]

I suppose I shouldn't be surprised that Rama would resonate more immediately with Clojure programmers, given the emphasis on data structures and it's origins in functional programming. So exposing the Clojure API may come sooner rather than later – no promises though. I'm curious how everyone's experience is next week using Rama through Java interop.

We did feel the pain of not having a proper REPL while developing Mastodon in straight Java, which brought me back to my pre-Clojure days. It's annoying to have to reload everything from scratch when wanting to test a small change. We did a little bit of work to try to be able to reload Java classes dynamically, but we never could get it working robustly especially when there were changes across multiple files. You won't have any of these issues developing modules from Clojure with Java interop. 

--

You received this message because you are subscribed to the Google Groups "rama-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rama-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rama-user/b392d60c-b130-40ed-9b3f-e96394d22474n%40googlegroups.com.

[Henrik Eneroth]

1+, throwing my hat in