Tour of our 250k line Clojure codebase

[natha...@gmail.com]

I published a post today giving on overview of our codebase at Red Planet Labs and the many ways it leverages the unique qualities of Clojure: https://tech.redplanetlabs.com/2021/06/03/tour-of-our-250k-line-clojure-codebase/

Clojure has been a huge reason why we've been able to tackle such an ambitious project as a small team. Please give the post a read, and I'm happy to answer any questions.

[Derek Troy-West]

Hi Nathan (& team), thanks for the post/update it's filled with interesting info on a mature Clojure codebase.

One quick question from me - do you have any open-source projects planned? In a purely selfish sense I would love to take a look at the custom language!

I look forward to seeing the product when it's available, this is very much in my area of interest.

Best,

 Derek

[Leandro Doctors]

On Thu, 3 Jun 2021, 15:06 natha...@gmail.com, <natha...@gmail.com> wrote:

Please give the post a read, and I'm happy to answer any questions.

Nice article, Nathan. Definitely, it would be nice to see the code :-)

Just curious...

1) You mention using Schema for data definition and validation. Did you consider using other options for this, such as clojure.spec? What's your experience with it/them?

2) You mention using "deterministic simulation". Did you consider using other options for this, such as test.check? What's your experience with it/them?

Best,

Leandro

[Nathan Marz]

Derek – we have a bunch of open-source on our Github. I'd like to release our new language one day, but that won't be for a long time. When I release it I want to do it the right way – extensive documentation, academic papers, and a commitment to supporting the language in broader usage. At our early stage we just don't have the bandwidth for that as we're working hard to get our first product out the door. Plus at the moment out language is our "secret weapon" :)

Leandro – I started working on this codebase well before spec was released. Had spec existed when I started I would have explored it more thoroughly, but Schema has met our needs very gracefully. As for deterministic simulation, it's orthogonal to techniques like test.check. I suggest you check our our earlier blog post on the subject. 

[Robert Levy]

Great post on the technical choices made in developing this platform.  Do you plan on writing a post that describes in detail the system architecture that is the bread and butter of the product/platform itself?

The website intriguingly states, "Red Planet Labs is pioneering a radically new kind of software tool. It's not just for the initial construction of an application, but also encapsulates deployment, monitoring, and maintenance. It implements the first truly cohesive model for building software applications – a set of abstractions that can build any application at any scale with greatly reduced engineering cost."

It seems like a full article expanding on the infrastructure-level design, and the approach to generalizing and abstracting infrastructure, would be very interesting.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/70353370-6dd2-4ddc-af82-2fce32ee6bden%40googlegroups.com.

[Nathan Marz]

Probably not in the near future. Once we're close to being ready for production usage, we'll be releasing a preview post demonstrating how our product works through an example of using it to replicate a widely-used service in a comparatively minuscule amount of code. Though we're already able to build and run arbitrary applications at scale, we still have a lot of work to do on operational features to enable the fault-tolerance necessary for production usage.  

[Brent Millare]

Haven't read the article yet but thanks for the work, will definitely give it a read asap. After hearing (already a while ago) that Storm moved to Java, I'm happy to hear Clojure is still actively used (and even instrumental) in your current work.

[Leif]

Hi Nathan.  Interesting post.  Here are some questions that I wrote down while
reading it.      

First-class continuations + distributed programming sounds like a nightmare that

would wake me up in a cold sweat.  Like if a void pointer was bitten by a

radioactive goto.  Does your language have restrictions that make such a pairing

comprehensible to average programmers like myself?  I assume so, so my real

question is: what distributed programming languages / paradigms is your language
inspired by?

The usage of leiningen and deps.edn is interesting.  How are they combined?   
Do you handle all deps with deps.edn, and use lein as a task runner, or...?
(After reading the post comments, it seems somewhat more complicated.
Complicated build processes: I am shocked.  Shocked, I tell you.)

Using with-redefs + concurrency has bitten me before.  Is it not a problem
because your tests are high-level and run sequentially, so the per-test redefs
don't interfere with each other? (I think this is mostly answered by the
"Deterministic simulation" section, but answer if you'd like.)

The "no-op functions as a structured event log" pattern is interesting.  Did you
try out any other methods, like implementing the event logging as separate

components, or directly redef'ing the side-effecting code?  If so, what are the

pros & cons you considered?

Are these testing methods just for internal use, or are you planning to make

them easy to use by the end-users of your language? I have a lot of questions

about your language, actually, but I'll just wait for your demo.

It is an ambitious project, and I hope it succeeds.  (Since most software

nowadays becomes part of "distributed systems" and they are, frankly, a PITA.)

Cheers,
Leif

[Nathan Marz]

Continuations in our language are expressed very differently than has existed before (e.g. like in Scheme). They fit intuitively within the overall paradigm our language implements. Far from being complex or hard to comprehend, continuations are the key construct that enables us to avoid mountains of complexity that exist otherwise in distributed systems. I know this from personal experience building distributed systems in the past. The degree to which continuations help write asynchronous, reactive, and parallel code is huge. It would be clear if you saw the language in action, but we're keeping it under wraps for now.

We use deps.edn just for dependency resolution via lein-tools-deps.plugin/resolve-dependencies-with-deps-edn. We use lein as a test runner and for launching repls from the command line. Our build's not too complicated, but it's not ideal.

That's right. Since tests run sequentially, using with-redefs is safe. We parallelize our build by running it as multiple Jenkins stages, where each stage runs a subset of the tests sequentially in separate processes. Any with-redefs we do are at the beginning of tests and thus global to its execution. 

with-redefs + no-op functions is extremely lightweight. We do use component overrides for other things, but that requires: a) the component system map is accessible where you want to capture events, b) non-trivial overhead in adding a new component and threading it to the spot in the code you care about. For events we want to capture in low-level implementation functions the with-redefs + no-op functions approach is simple, easy, doesn't affect production, and avoids bloating our component system maps with tons of things which are irrelevant to production.

The only testing method we use internally that might be relevant to users of our product is deterministic simulation. We're unlikely to expose that for v1, but we may expose it in the future. For v1 we'll be exposing an "in process cluster" abstraction for testing which runs things in parallel on multiple threads.

[Timothy Baldridge]

On Wed, Jun 9, 2021 at 6:14 PM Nathan Marz <natha...@gmail.com> wrote:

Continuations in our language are expressed very differently than has existed before (e.g. like in Scheme). They fit intuitively within the overall paradigm our language implements. Far from being complex or hard to comprehend, continuations are the key construct that enables us to avoid mountains of complexity that exist otherwise in distributed systems. I know this from personal experience building distributed systems in the past. The degree to which continuations help write asynchronous, reactive, and parallel code is huge. It would be clear if you saw the language in action, but we're keeping it under wraps for now.

Could you expound on that for those of us who are familiar with continuations in many forms, and languages? While delimited, multi-prompt, multi-shot continuations are certainly helpful in reducing complexity compared to traditional full continuations, they still result in spaghetti code unless coupled with some sort of typing and/or algebraic effect system. Most research in this space shows some promise, so I’m interested in seeing how you’ve solved the many well documented problems with these approaches.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Nathan Marz]

It all comes down to the programming paradigm in which the continuations are expressed. In our language continuations fit intuitively within the normal ways in which computation happens, whereas in a language like Scheme continuations are "abnormal" relative to run-of-the-mill Scheme concepts. Continuations were never an original goal for me and implementing them was sort of an accident – while looking at the compiler code one day I realized if I switched a couple fields I had continuations. It was only later that I learned what a powerful primitive they are for building distributed systems. 

That's about as much illumination as I can give without showing example code.