Splitting up core.typed into smaller repos

[Ambrose Bonnaire-Sergeant]

Hi,

For various reasons, core.typed consists of several Maven modules that should really

be their own repositories. Clojure CLI's fantastic git deps has inspired me to take the plunge, and I'd

like to split them up, effectively deprecating the original repo.

There's a few things to address.

==== Start ======

1) Correctly splitting wrt Licensing/attribution

I recently made sure that each src/ file has a license header. I then split up the

repositories by cloning core.typed and running

   git filter-branch --prune-empty --subdirectory-filter module-{$THIS}

Then I copied the licensing info into the root of each repo.

Does that sound reasonable?

2) New org.clojure repos needed

Splitting the core.typed repo as-is gives 5 repos:

- core.typed.runtime.jvm

- core.typed.checker.jvm

- core.typed.checker.js

- core.typed.annotator.jvm

- core.typed.analyzer.jvm

All above linked org.typedclojure repos are derived using the method above, so they

should be fine to use in org.clojure.

In the near future I'll be splitting them further into:

- core.typed.runtime

- core.typed.runtime.js

- core.typed.checker

- core.typed.analyzer

- core.typed.analyzer.js

Seems a lot, but it reflects the separation I've been moving towards over the

last few years. If 10 repos is too much to create right now, prioritising

the first 5 is fine.

3) Jira

I'd prefer using CTYP as an umbrella for all these projects, so no new Jira needed.

4) Maven coordinate renames (FYI, I'm handling the details)

org.clojure/core.typed => org.clojure/core.typed.checker.jvm

org.clojure/core.typed-pom => <deleted>

org.clojure/core.typed.infer => org.clojure/core.typed.annotator.jvm

org.clojure/core.typed.rt => org.clojure/core.typed.runtime.jvm

org.clojure/core.typed.analyzer.jvm => <same>

====End ======

I'm happy to discuss any other details. And thanks to gitlibs, this migration isn't

urgent and I can continue development over at org.typedclojure until the time comes.

Thanks!

Ambrose

[Alex Miller]

This is probably fine, but let me get back to you soon.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Daniel Compton]

Hi Ambrose

I don't have any strong feelings about this, but 5-10 Git repos seems like it may get a bit unwieldy for you and for other contributors. What are the downsides of having all of the modules in one Git repo like they currently are?

Just curious, not an objection.

Thanks, Daniel.

[Alex Miller]

I think the main advantages would be

a) Things could change and release at different rates

b) External users could more easily depend on some subset by not being tied into a multi-module project? Maybe that's not even an issue.

[Ambrose Bonnaire-Sergeant]

Hi Daniel,

I don't have any strong feelings about this, but 5-10 Git repos seems like it may get a bit unwieldy for you and for other contributors. 

I wrote/rambled a bit about my motivations here (excuse the Patreon). I will definitely miss

the big "release everything" button on build.clojure.org, but there are many components

that deserve their own home.

What are the downsides of having all of the modules in one Git repo like they currently are?

Alex nailed the advantages of multiple repos (I think core.specs <==> spec.alpha or tools.analyzer <===> t.a.jvm

are great examples).

I can mention some disadvantages of a single repo (in the context of a contrib with Ambrose BS as the lead developer :P):

- it's been very difficult to tease apart platform-dependent implementation details (dozens of files in the clojure.core.typed.* namespace).

  This has hindered both moving to .cljc files and making (much) progress on the CLJS checker.

- apparently it's irresistible for me to introduce dynamically resolved cyclic dependencies to give a

  "batteries included" feel, which means the modules are actually tightly coupled

- Maven modules has been a pain from the beginning. core.typed is the only org.clojure lib

  with modules, and there are lots of little pain points (eg. I have no idea how to generate autodoc

  or how to even use Maven)

- mirroring the nested pom.xml's with nested deps.edn linked by relative paths worked fine for local dev

  but didn't (immediately, at least) work with gitlibs, since the relative paths would now be relative to the new project root.

- little incentive to document or nail down an API for the modules

- lack of incentive for good names for things! eg. it took me 3 years to come up with the name core.typed.annotator, and

  now that it's here, it so obviously should be in its own repo. "module-infer", not so much. Who knows, people might

  actually use it once it has a good name and its own home (easier to understand/market).

The reality is that core.typed is huge (2800+ commits) and is largely a port of Typed Racket. I've had

to reinvent some things that Racket has for free (eg. I see core.typed.analyzer as similar to Syntax Objects,

a subtle observation that deserves further investigation). There are many Racket-ty idioms in core.typed

and might be more extensible and decoupled if designed with Clojure in mind.

But really, I shouldn't be blaming anything for the state of core.typed except my sense of design.

90% of my entire dev experience has been developing core.typed, and I've made lots of sloppy

decisions and mistakes. This repo split represents the way I want to think about core.typed,

before it tangles itself up and becomes immovable.

Hope that satisfied your curiosity!

Ambrose

[Alex Miller]

Sounds good. I don't have rights to make the repo so will need to wait for someone else to get that done, hopefully soon.

[Alex Miller]

Actually, I do have the rights and the repos now exist - when you get to a point where you need the build box, let me know so I can do some more steps.

[Ambrose Bonnaire-Sergeant]

Thanks Alex!

These are ready for the build box:

- core.typed.runtime.jvm

- core.typed.analyzer.jvm

- core.typed.checker.jvm

- core.typed.annotator.jvm

--

[Alex Miller]

Ready on the build box...

On Thursday, November 15, 2018 at 9:36:58 PM UTC-6, Ambrose Bonnaire-Sergeant wrote:

Thanks Alex!

These are ready for the build box:

- core.typed.runtime.jvm

- core.typed.analyzer.jvm

- core.typed.checker.jvm

- core.typed.annotator.jvm

On Thu, Nov 15, 2018 at 4:36 PM Alex Miller <al...@puredanger.com> wrote:

Actually, I do have the rights and the repos now exist - when you get to a point where you need the build box, let me know so I can do some more steps.

On Thursday, November 15, 2018 at 3:30:08 PM UTC-6, Alex Miller wrote:

Sounds good. I don't have rights to make the repo so will need to wait for someone else to get that done, hopefully soon.

On Sunday, November 11, 2018 at 8:14:34 AM UTC-6, Alex Miller wrote:

This is probably fine, but let me get back to you soon.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

[Ambrose Bonnaire-Sergeant]

I've decided to move back to the core.typed

monorepo this week. I don't think this requires any action from

Clojure core since Jenkins/Git/Jira is already setup and working

fine, but I wanted to touch base.

Firstly, I'm really grateful to those who put effort into helping split

core.typed up, especially Alex for setting up the infrastructure (and I assume Rich had to approve it!).

The process certainly helped crystalize the essence of core.typed for

my PhD and its future future directions -- thank you, and I'm in your debt.

Unfortunately --- as Daniel was concerned about --- it was too much work to maintain

and pretty much impossible to contribute to if you weren't me.

The main friction was keeping both the deps.edn Git SHA's and Maven

snapshots in sync for all the repos. I found this much easier

via a monorepo using `:local/root` and auto-bumping via `maven release`.

My decision to move back to a monorepo is primarily because:

1. I need a more streamlined dev environment since core.typed is now my side

    project rather than my primary focus.

2. tools.deps has improved its monorepo support since 2018

   (especially :deps/root support! I think TDEPS-74 was the missing piece).

   So, massive thanks to Alex for that!

I documented the process I used to merge the 6 polyrepos back into core.typed.

It got messy, and now `clojure/core.typed`'s master branch has 9 orphaned "first commits"!

But it seemed to do the trick.

I did some investigation, and accounted for all the orphaned commits.

Of course, they were all my fault, for various reasons ("only" 7 happened this week).

The write-up explains them all and explains the specific `git subtree` commands I used

to preserve history. I was careful to ensure everything I merged back was 

by a contributor with a CA. Hopefully the write-up is enough

to help someone verify that if they'd like.

I'll post further updates on the core.typed Google Group, as usual.

Thanks for reading, happy to answer questions.

Thanks,

Ambrose

On Thursday, November 15, 2018 at 10:53:54 PM UTC-5, Alex Miller wrote:

Ready on the build box...

On Thursday, November 15, 2018 at 9:36:58 PM UTC-6, Ambrose Bonnaire-Sergeant wrote:

Thanks Alex!

These are ready for the build box:

- core.typed.runtime.jvm

- core.typed.analyzer.jvm

- core.typed.checker.jvm

- core.typed.annotator.jvm

On Thu, Nov 15, 2018 at 4:36 PM Alex Miller <al...@puredanger.com> wrote:

Actually, I do have the rights and the repos now exist - when you get to a point where you need the build box, let me know so I can do some more steps.

On Thursday, November 15, 2018 at 3:30:08 PM UTC-6, Alex Miller wrote:

Sounds good. I don't have rights to make the repo so will need to wait for someone else to get that done, hopefully soon.

On Sunday, November 11, 2018 at 8:14:34 AM UTC-6, Alex Miller wrote:

This is probably fine, but let me get back to you soon.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloju...@googlegroups.com.

[Alex Miller]

The build box has separate build projects for all the sub types - do these need to get collapsed back to one?

 

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure-dev/ca70839e-a8ac-44dc-9427-a2d56fb47d52%40googlegroups.com.

[Ambrose Bonnaire-Sergeant]

Yes, let's just keep core.typed and core.typed-test-matrix builds. All the

other core.typed.* build projects can be disabled.

Thanks,

Ambrose

To unsubscribe from this group and stop receiving emails from it, send an email to cloju...@googlegroups.com.

[Alex Miller]

Done.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure-dev/000ae3c7-4a6c-4bfa-91a2-9a278a2209d0%40googlegroups.com.

[Ambrose Bonnaire-Sergeant]

Thanks Alex!

Done.

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure-dev/000ae3c7-4a6c-4bfa-91a2-9a278a2209d0%40googlegroups.com.