Splitting up core.typed into smaller repos

180 views
Skip to first unread message

Ambrose Bonnaire-Sergeant

unread,
Nov 9, 2018, 10:42:51 PM11/9/18
to Clojure Dev
Hi,

For various reasons, core.typed consists of several Maven modules that should really
be their own repositories. Clojure CLI's fantastic git deps has inspired me to take the plunge, and I'd
like to split them up, effectively deprecating the original repo.

There's a few things to address.

==== Start ======
1) Correctly splitting wrt Licensing/attribution
I recently made sure that each src/ file has a license header. I then split up the
repositories by cloning core.typed and running
   git filter-branch --prune-empty --subdirectory-filter module-{$THIS}
Then I copied the licensing info into the root of each repo.
Does that sound reasonable?

2) New org.clojure repos needed
Splitting the core.typed repo as-is gives 5 repos:

All above linked org.typedclojure repos are derived using the method above, so they
should be fine to use in org.clojure.

In the near future I'll be splitting them further into:
- core.typed.runtime
- core.typed.runtime.js
- core.typed.checker
- core.typed.analyzer
- core.typed.analyzer.js

Seems a lot, but it reflects the separation I've been moving towards over the
last few years. If 10 repos is too much to create right now, prioritising
the first 5 is fine.

3) Jira
I'd prefer using CTYP as an umbrella for all these projects, so no new Jira needed.

4) Maven coordinate renames (FYI, I'm handling the details)
org.clojure/core.typed => org.clojure/core.typed.checker.jvm
org.clojure/core.typed-pom => <deleted>
org.clojure/core.typed.infer => org.clojure/core.typed.annotator.jvm
org.clojure/core.typed.rt => org.clojure/core.typed.runtime.jvm
org.clojure/core.typed.analyzer.jvm => <same>

====End ======

I'm happy to discuss any other details. And thanks to gitlibs, this migration isn't
urgent and I can continue development over at org.typedclojure until the time comes.

Thanks!
Ambrose

Alex Miller

unread,
Nov 11, 2018, 9:14:34 AM11/11/18
to cloju...@googlegroups.com
This is probably fine, but let me get back to you soon.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

Daniel Compton

unread,
Nov 12, 2018, 4:50:46 PM11/12/18
to cloju...@googlegroups.com
Hi Ambrose

I don't have any strong feelings about this, but 5-10 Git repos seems like it may get a bit unwieldy for you and for other contributors. What are the downsides of having all of the modules in one Git repo like they currently are?

Just curious, not an objection.

Thanks, Daniel.

Alex Miller

unread,
Nov 12, 2018, 5:41:17 PM11/12/18
to Clojure Dev
I think the main advantages would be

a) Things could change and release at different rates
b) External users could more easily depend on some subset by not being tied into a multi-module project? Maybe that's not even an issue.

Ambrose Bonnaire-Sergeant

unread,
Nov 12, 2018, 8:02:05 PM11/12/18
to cloju...@googlegroups.com
Hi Daniel,

I don't have any strong feelings about this, but 5-10 Git repos seems like it may get a bit unwieldy for you and for other contributors. 

I wrote/rambled a bit about my motivations here (excuse the Patreon). I will definitely miss
the big "release everything" button on build.clojure.org, but there are many components
that deserve their own home.

What are the downsides of having all of the modules in one Git repo like they currently are?

Alex nailed the advantages of multiple repos (I think core.specs <==> spec.alpha or tools.analyzer <===> t.a.jvm
are great examples).

I can mention some disadvantages of a single repo (in the context of a contrib with Ambrose BS as the lead developer :P):
- it's been very difficult to tease apart platform-dependent implementation details (dozens of files in the clojure.core.typed.* namespace).
  This has hindered both moving to .cljc files and making (much) progress on the CLJS checker.
- apparently it's irresistible for me to introduce dynamically resolved cyclic dependencies to give a
  "batteries included" feel, which means the modules are actually tightly coupled
- Maven modules has been a pain from the beginning. core.typed is the only org.clojure lib
  with modules, and there are lots of little pain points (eg. I have no idea how to generate autodoc
  or how to even use Maven)
- mirroring the nested pom.xml's with nested deps.edn linked by relative paths worked fine for local dev
  but didn't (immediately, at least) work with gitlibs, since the relative paths would now be relative to the new project root.
- little incentive to document or nail down an API for the modules
- lack of incentive for good names for things! eg. it took me 3 years to come up with the name core.typed.annotator, and
  now that it's here, it so obviously should be in its own repo. "module-infer", not so much. Who knows, people might
  actually use it once it has a good name and its own home (easier to understand/market).

The reality is that core.typed is huge (2800+ commits) and is largely a port of Typed Racket. I've had
to reinvent some things that Racket has for free (eg. I see core.typed.analyzer as similar to Syntax Objects,
a subtle observation that deserves further investigation). There are many Racket-ty idioms in core.typed
and might be more extensible and decoupled if designed with Clojure in mind.

But really, I shouldn't be blaming anything for the state of core.typed except my sense of design.
90% of my entire dev experience has been developing core.typed, and I've made lots of sloppy
decisions and mistakes. This repo split represents the way I want to think about core.typed,
before it tangles itself up and becomes immovable.

Hope that satisfied your curiosity!
Ambrose

Alex Miller

unread,
Nov 15, 2018, 4:30:08 PM11/15/18
to Clojure Dev
Sounds good. I don't have rights to make the repo so will need to wait for someone else to get that done, hopefully soon.

Alex Miller

unread,
Nov 15, 2018, 4:36:36 PM11/15/18
to Clojure Dev
Actually, I do have the rights and the repos now exist - when you get to a point where you need the build box, let me know so I can do some more steps.

Ambrose Bonnaire-Sergeant

unread,
Nov 15, 2018, 10:36:58 PM11/15/18
to cloju...@googlegroups.com
Thanks Alex!

These are ready for the build box:
- core.typed.runtime.jvm
- core.typed.analyzer.jvm
- core.typed.checker.jvm
- core.typed.annotator.jvm

--

Alex Miller

unread,
Nov 15, 2018, 10:53:54 PM11/15/18
to Clojure Dev
Ready on the build box...


On Thursday, November 15, 2018 at 9:36:58 PM UTC-6, Ambrose Bonnaire-Sergeant wrote:
Thanks Alex!

These are ready for the build box:
- core.typed.runtime.jvm
- core.typed.analyzer.jvm
- core.typed.checker.jvm
- core.typed.annotator.jvm

On Thu, Nov 15, 2018 at 4:36 PM Alex Miller <al...@puredanger.com> wrote:
Actually, I do have the rights and the repos now exist - when you get to a point where you need the build box, let me know so I can do some more steps.

On Thursday, November 15, 2018 at 3:30:08 PM UTC-6, Alex Miller wrote:
Sounds good. I don't have rights to make the repo so will need to wait for someone else to get that done, hopefully soon.

On Sunday, November 11, 2018 at 8:14:34 AM UTC-6, Alex Miller wrote:
This is probably fine, but let me get back to you soon.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

Ambrose Bonnaire-Sergeant

unread,
Jan 17, 2020, 4:07:30 PM1/17/20
to Clojure Dev
I've decided to move back to the core.typed
monorepo this week. I don't think this requires any action from
Clojure core since Jenkins/Git/Jira is already setup and working
fine, but I wanted to touch base.

Firstly, I'm really grateful to those who put effort into helping split
core.typed up, especially Alex for setting up the infrastructure (and I assume Rich had to approve it!).
The process certainly helped crystalize the essence of core.typed for
my PhD and its future future directions -- thank you, and I'm in your debt.

Unfortunately --- as Daniel was concerned about --- it was too much work to maintain
and pretty much impossible to contribute to if you weren't me.
The main friction was keeping both the deps.edn Git SHA's and Maven
snapshots in sync for all the repos. I found this much easier
via a monorepo using `:local/root` and auto-bumping via `maven release`.

My decision to move back to a monorepo is primarily because:
1. I need a more streamlined dev environment since core.typed is now my side
    project rather than my primary focus.
2. tools.deps has improved its monorepo support since 2018
   (especially :deps/root support! I think TDEPS-74 was the missing piece).
   So, massive thanks to Alex for that!

I documented the process I used to merge the 6 polyrepos back into core.typed.
It got messy, and now `clojure/core.typed`'s master branch has 9 orphaned "first commits"!
But it seemed to do the trick.

I did some investigation, and accounted for all the orphaned commits.
Of course, they were all my fault, for various reasons ("only" 7 happened this week).
The write-up explains them all and explains the specific `git subtree` commands I used
to preserve history. I was careful to ensure everything I merged back was 
by a contributor with a CA. Hopefully the write-up is enough
to help someone verify that if they'd like.

I'll post further updates on the core.typed Google Group, as usual.

Thanks for reading, happy to answer questions.

Thanks,
Ambrose


On Thursday, November 15, 2018 at 10:53:54 PM UTC-5, Alex Miller wrote:
Ready on the build box...

On Thursday, November 15, 2018 at 9:36:58 PM UTC-6, Ambrose Bonnaire-Sergeant wrote:
Thanks Alex!

These are ready for the build box:
- core.typed.runtime.jvm
- core.typed.analyzer.jvm
- core.typed.checker.jvm
- core.typed.annotator.jvm

On Thu, Nov 15, 2018 at 4:36 PM Alex Miller <al...@puredanger.com> wrote:
Actually, I do have the rights and the repos now exist - when you get to a point where you need the build box, let me know so I can do some more steps.

On Thursday, November 15, 2018 at 3:30:08 PM UTC-6, Alex Miller wrote:
Sounds good. I don't have rights to make the repo so will need to wait for someone else to get that done, hopefully soon.

On Sunday, November 11, 2018 at 8:14:34 AM UTC-6, Alex Miller wrote:
This is probably fine, but let me get back to you soon.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloju...@googlegroups.com.

Alex Miller

unread,
Jan 17, 2020, 4:38:50 PM1/17/20
to cloju...@googlegroups.com, ilike...@gmail.com
The build box has separate build projects for all the sub types - do these need to get collapsed back to one?
 
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure-dev/ca70839e-a8ac-44dc-9427-a2d56fb47d52%40googlegroups.com.

Ambrose Bonnaire-Sergeant

unread,
Jan 17, 2020, 4:48:03 PM1/17/20
to Clojure Dev
Yes, let's just keep core.typed and core.typed-test-matrix builds. All the
other core.typed.* build projects can be disabled.

Thanks,
Ambrose
To unsubscribe from this group and stop receiving emails from it, send an email to cloju...@googlegroups.com.

Alex Miller

unread,
Jan 17, 2020, 4:59:16 PM1/17/20
to cloju...@googlegroups.com
Done.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure-dev/000ae3c7-4a6c-4bfa-91a2-9a278a2209d0%40googlegroups.com.

Ambrose Bonnaire-Sergeant

unread,
Jan 17, 2020, 5:01:44 PM1/17/20
to Clojure Dev
Thanks Alex!
Done.

Reply all
Reply to author
Forward
0 new messages