deps.edn support ideas

[Luke Burton]

Just wanted to issue a vote of support for:

https://github.com/cursive-ide/cursive/issues/1910

I've started some experiments with … I guess you might call them microframeworks? And IntelliJ / Cursive is kinda getting in the way of taking this idea to extremes :) I'm having to open Emacs and yak shave my init.el once again, like the good old days.

Sorry for the long-winded explanation, but I figure the Cursive community is full of smart people who might want to point out I'm crazy :)

With microservices, you might build a "payments service" that has its own database, business logic, and API to the outside world. It's a little self contained stack. It's probably a single project.clj, maybe a couple of things spun into libs.

With a "microframework", I'm finding more subdivision is possible. So for example, let's say you have a basic pedestal app. It lifts some data out of PostgreSQL, does interesting transformations on that data, then vends a few API routes. We can subdivide that in ways that are almost along namespace boundaries, but not quite.

We can have separate deps.edn projects for:

- the domain of lifting things out of the database. Anything to do with SQL, connection pools, clojure.spec on data.

- the domain of doing business-stuff on data. Functions that expect to be given some data and return some data.

- the domain of pedestal interceptors. Here's where you can take an incoming request for data, make a DB request, and pass it through the appropriate business function.

- the domain of web server crap. Here's where you beat your head against the wall, starting Jetty, mounting your interceptors into pedestal routes, handling the awful world of HTTP :)

I've found breaking these things into separate deps.edn projects really changes the way you approach them. The cognitive load is reduced, because the responsibilities for each project are tiny. The namespaces rarely exceed what's visible on a single screen. The projects at the leaves have tiny sets of dependencies and so have fast REPL startup times. 

Reuse of projects becomes much easier because it discourages hacks - e.g. if you want to introduce Kafka you will find that you've not accidentally polluted your business logic code with direct references to databases or HTTP. 

And of course each deps.edn project can depend on any other deps.edn project by git SHA, no more heavyweight release process to slow all this down.

One of the things I love about Cursive is the lein checkouts support, which makes factoring out code into libraries, or hacking on dependent libraries alongside a project, much easier than ever before. It would be great to have something like this for deps.edn, that let me consolidate multiple deps.edn projects together.

I guess one approach would be to start an IntelliJ project in whatever deps.edn project forms is at the top of the dependency tree. Then you could navigate the dependent deps.edn projects, and if they're on filesystem paths, just have them show up similarly to how the checkouts/ folder works today.

Another approach would be to have a meta-project that has just a deps.edn file and no src/ directory. Its only job is to provide a place for the IntelliJ project to live and the right list of deps.edn projects to pull in as "checkouts"

Happy to hear any other ideas folks have!

Luke.

[Colin Fleming]

I’m travelling right now so I can’t reply to all of this but I do have one question - if the checkouts support works well for what you want to do, what would deps support add? I’m not debating whether deps support is required in general, clearly it is, but I’m not seeing what it would add in this particular use case over just using checkouts? The git support is nice, but in a multi-module project Cursive doesn’t require a build + release step anyway.

--
You received this message because you are subscribed to the Google Groups "Cursive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cursive+u...@googlegroups.com.
To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/E10CF8B5-9D18-4A14-9DAC-80E7169528C8%40icloud.com.
For more options, visit https://groups.google.com/d/optout.

[Wilker]

One reason for deps.clj is that checkouts don't work well with cljs. I'm staging many projects now using shadow + deps.edn, and I have to keep a project.clj just to make cursive understand dependencies.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/etPan.5acef040.ae5ba37.1a54%40cursive-ide.com.

[Luke Burton]

On Apr 11, 2018, at 10:35 PM, Colin Fleming <cur...@cursive-ide.com> wrote:

I’m travelling right now so I can’t reply to all of this but I do have one question - if the checkouts support works well for what you want to do, what would deps support add? I’m not debating whether deps support is required in general, clearly it is, but I’m not seeing what it would add in this particular use case over just using checkouts? The git support is nice, but in a multi-module project Cursive doesn’t require a build + release step anyway.

Hi Colin,

It's subtle, but I think for all the same reasons deps.edn exists in the first place, which is to give us dependency management options broader than slinging around artifacts.

lein checkouts are really coupled to the artifact - you create a project, give it a name and a version, symlink it into position in checkouts, update dependencies, and start hacking. There are a few gotchas, I believe, which is that the project you've linked in checkouts must already be available in a repo somewhere (so you must have done lein install for the project to get it into ~/.m2). So even to get started locally there's a ton of ceremony. 

The moment this project moves off my local machine is where things get more than just inconvenient. My team members are obligated to do exactly the same setup steps for checkouts on their machines. Or, if they are only hacking on some parts, I'm obligated to produce maven artifacts just so they can get started. Where I work, that involves wiring it up to our CI/CD system that pushes artifacts into a private artifactory instance. It's pretty smooth but way heavier than just pointing at a git SHA.

With deps.edn we can be a lot more flexible. There's really no difference from a project perspective as to whether the dependency is hosted in a git repo, a local dir, or a real maven artifact, and the source can be seamlessly overridden for any other with a minimum of fuss. So we can develop something and share it with git repo dependencies, but on our local machines override the dependency sources a la carte. That's where IDE tooling would be great: "clone this dependency and work on it locally". Or tooling that pushes your :local/root deps in the correct order, replacing your :local/root deps with git SHAs that point to the new version (kinda like clojure -Sresolve-deps does today for git tags).

Another key difference is that we may never produce an artifact for some of this stuff at all. It's possible that we'd take a single project, which pulls in its other deps.edn dependencies via git, build that in a Dockerfile, and push it. No maven. That feels like a natural evolution with deps.edn, whereas it wouldn't really make sense with checkouts (you'd build each checkout, install in ~/.m2, build your main project, etc)

There might be more things I'm missing, I'm certainly not an authority on lein checkouts or multi-module projects. But I think the additional artifact related ceremony of checkouts ends up such that we only make checkouts if we think "yes, we're gonna publish this as library in artifactory." With deps.edn we can split projects up into multiple repositories simply because it makes sense for the project, without being obligated to officially publish anything.

Luke.

[Alan Thompson]

I'm confused.  The whole point of checkouts is what you re independent of ~/.m2 (or the maven ecosystem in general).  In ./checkouts, you just create a symlink to the root dir of another project dir.  No?

Alan

--
You received this message because you are subscribed to the Google Groups "Cursive" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+unsubscribe@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/DA66CB6C-DA7D-41DE-B413-168FFC2BF6D9%40icloud.com.

[Luke Burton]

Nope, direct from the leiningen docs (emphasis mine):

After you've updated :dependencies, lein will still need to be able to find the library in some repository like clojars or your ~/.m2 directory. If lein complains with a message like "Could not find artifact suchwow:jar:0.3.9", it's possible that project.clj and suchwow/project.clj use different version numbers. It's also possible that you're working on the main project and suchwow at the same time, have bumped the version number in both project files, but still have the old version in your local Maven repository. Run lein install in the suchwow directory. That is: the suchwow version number must be the same in three places: in suchwow's project.clj, in the main project's project.clj, and in some repository the main project uses.

If you change the dependencies of a checkout project you will still have to run lein install and restart your repl; it's just that source changes will be picked up immediately.

Checkouts are an opt-in feature; not everyone who is working on the project will have the same set of checkouts, so your project should work without checkouts before you push or merge.

To unsubscribe from this group and stop receiving emails from it, send an email to cursive+u...@googlegroups.com.

To post to this group, send email to cur...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAN67zA2xy-X9pFYxuAGshM650-ZVk8CUoL1_ff%2BNhFbnr5mu9Q%40mail.gmail.com.