Proposal: Improvements for Contributors/Screeners

[Nathan Fisher]

Hola,

I would like to use this as a discussion for a set of proposals that I think would enhance user workflows for both contributors and screeners. This does not include a challenge to the CA signing process. I've seen it come up in recent contributor discussions for other OSS projects so I'd like to be explicit: I have zero interest in this thread being used as a forum to debate the CA as I both understand and agree with it. Phew now that's outta the way... :)

While I would eventually like to see Clojure accept PR's as common place I very much understand it's not going to happen overnight. I do however think there's a few changes/integrations that could subtly enhance the broader user experience for contributors and screeners alike.

Those changes are:

- Convert README.txt to README.md.
- Add Jira service to GitHub.
- Publish Jenkins build status to GitHub.
- Slack build notifications.

I'm sure there are other integrations/changes that I'm not thinking of but these are the ones that immediately come to mind and would aid in discoverability and workflow. I'd be willing to assist in these changes by either providing step-by-step documentation or doing some of the work. Given some of the access that would be required is near and dear to the heart of Clojure I won't be offended if the latter offer is turned down at this point.

## Convert README.txt to README.md

Benefits:

- Provide ability to support some of the other recommendations below.
- Improved styling/readability.

## Add Jira service to GitHub

Benefits:

- Cross-link between Jira and GitHub for related commits.
- Should PR's be used it would make it easier for Screeners to track changes both in Jira and via GitHub.

### Supporting Material

- http://atlassianblog.wpengine.com/2014/04/connecting-jira-6-2-github/
- https://confluence.atlassian.com/jirasoftwarecloud/viewing-the-development-information-for-an-issue-777002795.html

## Publish Jenkins build status to GitHub

Benefits:

- Cross-link between Jenkins and GitHub for related commits.
- Visibility on a per-commit level of the build status for that commit.
- Provide high-level build status via "badges" located in the README (would require migrating README.txt to README.md).

An example can be seen here for what commit-level build status looks like in GitHub (albeit linked to Travis CI) https://github.com/nfisher/mig31/commits/master.

Note the Green checkmarks for success, red x for failed, and yellow circles for other. Those status link through to the specific build on the CI box.

### Supporting Material

- https://wiki.jenkins.io/display/JENKINS/Embeddable+Build+Status+Plugin
- https://jenkins.io/solutions/github/
- https://stackoverflow.com/questions/14274293/show-current-state-of-jenkins-build-on-github-repo/26910986#26910986

## Slack build notifications

Benefits:

- Increased visibility of CI build status (e.g. create channel in Clojure slack - #clojure-builds).
- provides another avenue to notify developers of CI build status.

### Supporting Material

- https://github.com/jenkinsci/slack-plugin

## Summary

None of the above would require a change to the current Jira patch workflow but I think it would enhance visibility and cross linking between GitHub, Jira, and Jenkins. Something which I expect would be of benefit to everyone.

# Accepting PR's

I expect at a minimum two things would need to happen before accepting PR's becomes common place:

1) understanding the screeners current workflow to ensure any changes are a net benefit. 2) create a persuasive argument to use PR's as a new code review workflow that incorporates and improves on the existing Screeners workflow.

Accepting PR's is predicated on GitHub being the source of truth and not a read-only shadow repo to a private repo hosted by Cognitect. Given GIT is a DVCS I could see ways of making it work but it would be outside the commonly accepted PR workflow. This would only be a concern if GitHub is a shadow repo/mirror. Can anyone of the Screeners comment if that is that the case?

Should there be a desire/willingness to shift to accepting PR's I would also suggest these additional changes:

## PR Builder

This would automatically build PR's giving Screeners and Contributors alike confidence that nothings been broken.

Some consideration would need to be given to how the build Infra handles PR's as Jenkins doesn't have a great story regarding build isolation and malicious PR's could be used to impact the CI box in various ways. There's two approaches I could see to provide isolation of CI builds:

1. use docker containers.
2. transition Clojure, ClojureScript, etc to Travis CI or Google Container Builder in a way that supports all current Java builds.

### Supporting Material

- https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
- https://github.com/jenkinsci/docker-plugin
- https://cloud.google.com/container-builder/

## PR Documentation

Documentation to get Contributors/Screeners up to speed on using a GitHub based code review process.

----

All of the above is probably better done as a Wiki but I don't appear to have access at the moment and haven't tracked down what I need for access.

I've done it as Markdown so I can hopefully cut and paste it into the Wiki when I do have access.

Saludos!
Nathan

[Alex Miller]

Hi Nathan,

On Monday, September 25, 2017 at 10:41:19 PM UTC-5, Nathan Fisher wrote:

Hola,

I would like to use this as a discussion for a set of proposals that I think would enhance user workflows for both contributors and screeners. This does not include a challenge to the CA signing process. I've seen it come up in recent contributor discussions for other OSS projects so I'd like to be explicit: I have zero interest in this thread being used as a forum to debate the CA as I both understand and agree with it. Phew now that's outta the way... :)

While I would eventually like to see Clojure accept PR's as common place I very much understand it's not going to happen overnight. I do however think there's a few changes/integrations that could subtly enhance the broader user experience for contributors and screeners alike.

Those changes are:

- Convert README.txt to README.md.
- Add Jira service to GitHub.
- Publish Jenkins build status to GitHub.
- Slack build notifications.

These seem to be changes for the sake of changes, whereas we typically prefer to work in terms of problems -> alternatives -> tradeoffs -> solution. What problem are you trying to solve?

 

I'm sure there are other integrations/changes that I'm not thinking of but these are the ones that immediately come to mind and would aid in discoverability and workflow. I'd be willing to assist in these changes by either providing step-by-step documentation or doing some of the work. Given some of the access that would be required is near and dear to the heart of Clojure I won't be offended if the latter offer is turned down at this point.

## Convert README.txt to README.md

Benefits:

- Provide ability to support some of the other recommendations below.
- Improved styling/readability.

What problem are we trying to solve? I think there are some text issues with the README and it could be doing a lot better job at pointing to other resources. To me, that's a 10x more important problem than "styling".

 

## Add Jira service to GitHub

Benefits:

- Cross-link between Jira and GitHub for related commits.
- Should PR's be used it would make it easier for Screeners to track changes both in Jira and via GitHub.

What problem are we trying to solve?  I think the problem here is how to understand how a resolved jira relates to an actual commit in the github repo, which is, I agree, a non-obvious link that would be useful to have visible. So, I'm on board with the goal (although I can't say it's a burning issue).

 

### Supporting Material

- http://atlassianblog.wpengine.com/2014/04/connecting-jira-6-2-github/
- https://confluence.atlassian.com/jirasoftwarecloud/viewing-the-development-information-for-an-issue-777002795.html

Unfortunately, we are currently on an ancient version of jira and I don't think we can do this until its upgraded. Doing this upgrade is quite difficult. I spent a few weeks working on it last year and have not (yet) succeeded. My infrastructure budget this year has gone into efforts like moving all of the Clojure infrastructure under https, Hudson->Jenkins, etc, which I think was all higher priority. So, interested, but it's gonna be a while.

 

## Publish Jenkins build status to GitHub

What problem are we trying to solve? 

Getting CI visibility in github is super useful on PRs but less useful on commits (where its already happened) and much less uesful when using our workflow where we are only committing heavily screened patches that have already been (locally) tested. The number of times those fail a CI build is very low. We currently build on a 1 hour poll too, not a post-commit hook, so you typically get a whole batch of changes in one build. (Maybe a separate issue.)

However, this might be more useful on contrib projects where some work is done by project leaders directly on master where it could be good feedback. Getting build feedback to contrib maintainers is something I have on my list and I have that scoped but haven't gotten around to it yet.

 

Benefits:

- Cross-link between Jenkins and GitHub for related commits.
- Visibility on a per-commit level of the build status for that commit.
- Provide high-level build status via "badges" located in the README (would require migrating README.txt to README.md).

An example can be seen here for what commit-level build status looks like in GitHub (albeit linked to Travis CI) https://github.com/nfisher/mig31/commits/master.

Note the Green checkmarks for success, red x for failed, and yellow circles for other. Those status link through to the specific build on the CI box.

### Supporting Material

- https://wiki.jenkins.io/display/JENKINS/Embeddable+Build+Status+Plugin

This seems to be something different - readme badge images (which I personally find annoying)

 

- https://jenkins.io/solutions/github/
- https://stackoverflow.com/questions/14274293/show-current-state-of-jenkins-build-on-github-repo/26910986#26910986

I'm willing to play with this on a contrib project and try it out at some point. But not till post-Conj. 

## Slack build notifications

What problem are we trying to solve? Why do you want to increase visibility of CI builds? Who will use this information? (I honestly have no idea.) On core, I don't see much need for it. Might be useful on contrib projects where multiple devs are collaborating (not many of those).

 

Benefits:  

- Increased visibility of CI build status (e.g. create channel in Clojure slack - #clojure-builds).
- provides another avenue to notify developers of CI build status.

### Supporting Material

- https://github.com/jenkinsci/slack-plugin

If there's some call by a contrib project owner that would use this, I'm happy to investigate it further (post Conj).

 

## Summary

None of the above would require a change to the current Jira patch workflow but I think it would enhance visibility and cross linking between GitHub, Jira, and Jenkins. Something which I expect would be of benefit to everyone.

# Accepting PR's

Again, what problem are you trying to solve? Presumably that problem is: contributing is too hard. I understand that (these days) creating patches is *unfamiliar* to people compared to PRs, but as someone that makes both on a regular basis, I don't find patches to be hard.

There are many reasons why Rich prefers patches for core changes. I will simply refer you to: https://groups.google.com/forum/#!msg/clojure/jWMaop_eVaQ/3M4gddaXDZoJ

Linus Torvalds has cited some of the same reasons not to use PRs in Linux:

https://github.com/torvalds/linux/pull/17#issuecomment-5654674

Both posts are old and some of the points have been addressed since, but this should give you some background. Personally, I don't really care. Patch vs PR is far less interesting than the review of the actual change itself as my time is dwarfed by the latter.

 

I expect at a minimum two things would need to happen before accepting PR's becomes common place:

1) understanding the screeners current workflow to ensure any changes are a net benefit.  2) create a persuasive argument to use PR's as a new code review workflow that incorporates and improves on the existing Screeners workflow. 

Accepting PR's is predicated on GitHub being the source of truth and not a read-only shadow repo to a private repo hosted by Cognitect.

The Github repo is the source of truth. There is no read-only private shadow repo. I don't how this idea started but I've seen it a few places now. We have occasionally done work on large new features (like spec) in an independent repo (partially because we thought it might not end up in core at all) but that's a rare special case and only applies to the initial development.

 

Given GIT is a DVCS I could see ways of making it work but it would be outside the commonly accepted PR workflow.  This would only be a concern if GitHub is a shadow repo/mirror. Can anyone of the Screeners comment if that is that the case?

Should there be a desire/willingness to shift to accepting PR's I would also suggest these additional changes:

## PR Builder

This would automatically build PR's giving Screeners and Contributors alike confidence that nothings been broken.

Some consideration would need to be given to how the build Infra handles PR's as Jenkins doesn't have a great story regarding build isolation and malicious PR's could be used to impact the CI box in various ways. There's two approaches I could see to provide isolation of CI builds:

1. use docker containers.
2. transition Clojure, ClojureScript, etc to Travis CI or Google Container Builder in a way that supports all current Java builds.

There are enormous amounts of work implied in moving anything. There is no consideration here for issues like the dozens of contrib builds we currently have, or the automated template process we used to generate Jenkins jobs, or Maven artifact deployment (auth keys). This is literally weeks or months of work for very marginal benefit imo.

 

### Supporting Material

- https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
- https://github.com/jenkinsci/docker-plugin
- https://cloud.google.com/container-builder/

## PR Documentation

Documentation to get Contributors/Screeners up to speed on using a GitHub based code review process.

----

All of the above is probably better done as a Wiki but I don't appear to have access at the moment and haven't tracked down what I need for access.

I've added access to Confluence and jira editing for your id.

[Sameer Rahmani]

Hi Nathan

I don't have any opinion on most of the points made by you ( because simply it's not my call ), But I think it's good idea to revise the README file. 

The github repo is the public face of clojure for developers. Right now It's not so good and feels like an old deprecated repo on github but with

a better read me specially with some nice styling it would have better impression on people and it's really to do.

The second thing I wanted to point out is the my opinion on Patch vs PR. I think patches are alright and I can totally understand the wisdom behinds 

them. The only thing which makes a contributor life hard about the patches is all the tailing white spaces in the code. in both CLJ and CLJS.  basically 

for each patch I have to manually remove lots of patch diffs or shutdown my editor configurations and stuff like that. So it would be a good idea 

to once and for all get rid of those white spaces and other problematic stuff like indentations on the source code. This way creating a patch would be 

really straight forward. I can understand that core members might have a different configuration for their editor and environment and they don't have 

such problem, but as a new contributor it's really hard to take care of these really simple stuff. Also it wouldn't hurt any one to do so.

Cheers

Sameer 

[Nathan Fisher]

Hi Alex,

Let’s catch-up after the conj. thanks for the links was already aware of Tolvolds opinion on PRs. sadly won’t be at conj but hope it goes well!

btw for me a shadow repo isn’t a dirty word it’s merely a design decision with benefits and trade offs like any other.

I would file all of these items under:

- increased discoverability.

- reduced friction.

- increased visibility/transparency.

I’ll expand on that more when I have some time just off to meet a friend for lunch.

Cheers!

Nathan

On Tue, 26 Sep 2017 at 06:04, Alex Miller <al...@puredanger.com> wrote

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

[Sean Corfield]

So it would be a good idea 

to once and for all get rid of those white spaces and other problematic stuff like indentations on the source code.

 

When you create a patch, you can ignore all whitespace differences, which helps to some extent (see the git format-patch docs for the various options).

 

Part of me feels like saying “What? Wait! There’s more than one way to format Clojure code??” 😊

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

--

[Nathan Fisher]

Hi All,

Circling back on this apologies for the delay. Hopefully this doesn't come across "wrong" I sincerely appreciate Clojure and the work that everyone puts into it.

Ok so the long-play of my initial e-mail is to get to a point of accepting some or all classes of PR's. A lot of the items I listed in my original e-mail are contribute to my superficial health indicators of a project (e.g. test coverage, commit frequency, green vs. red build, etc) any one of which individually is a weak indicator but multiple combined provide a stronger indication of health. In some instances they help support a PR process in others they're simply nice to haves.

My argument for PR's;

1. code locality (e.g. code living where code is arguably best placed, in SCM).
   
2. community familiarity (generally speaking I'm not one for bandwagons but...).
   
3. reduced overhead for code reviewers (a successful build would be the minimum bar for peer review).
4. PR's make feedback on specific lines of code super easy whereas a patch is cut-and-paste (example hover over the columns here and click the "+" https://github.com/clojure/clojure/pull/72/files).

If I understand at this point the main objections to PR's are;

1. preference for patch files.
   
2. throttle to drive-by/poor quality contributions (not sure that was explicitly stated but could be implicitly derived from Rich's statement on his appreciation for people willing to learn the contributors process).
   
3. reasons specified by Torvalds in his rant.
4. issue locality and related workflow - assumed on my part (e.g. Confluence/Jira).

Unpacking this and addressing each point in turn;

1. patch files can easily be extracted from the pull request using the following shell script or simply adding .patch to a PR;

#!/bin/sh -eu
# example usage: ./pr.sh 72

PR_NUM=$1

curl -L "https://github.com/clojure/clojure/pull/${PR_NUM}.patch" \
  -o "pr${PR_NUM}.patch"

Github's PR's will identify that these patches have been merged even if it was done outside the GH PR web UI.

2. quality control I understand from a hammock time/design point of view for a big change. Has person X demonstrated they understand the problem space, however I'm not sure I agree with it for all classes of changes (everything in moderation et al). Simple fixes to documentation (e.g. README or docstrings) at a minimum should be pretty easy to accept/reject and are an easy barrier to entry for people new to the community that want too start contribution.

3. feels like an appeal to authority fallacy that doesn't apply to the Clojure community. GitHub has addressed a few of the concerns and/or the concern isn't applicable to the current state of Clojure and/or the workflow can be addressed by item 1 above.

Torvalds arguments were primarly as follows;

1. real explanation.
   
2. proper email addresses (linux uses PGP signing for commits/tags and proper e-mails are a cornerstone for that).
   
3. proper shortlog (e.g. expected output for `git shortlog HEAD~10..HEAD`).
   
4. proper diffstat (e.g. `src/clj/clojure/core.clj                               |   174 +-`).
   
5. signed tags.
   
6. web UI commit messages
   

3.1. relates to item 2 above but the kernel contribution community is enormous whereas Clojure has 141 contributors bigger than many but arguably could be larger to the benefit of the community through a more "mainstream" contributors workflow.

3.2. relates to PGP identities and signing. Clojure doesn't use PGP signing so I question it's relevance.

3.3. proper shortlog addressed by 1.

3.4. addressed by 1.

3.5. ditto of 3.2.

3.6. not fully addressed but more of an issue regarding commits/patches in general and it's unlikely someone will code a large commit via the Web UI anyway. 

4. Whether to comment on PR's is a topic for further discussion.

I'll respond to my initial suggestions for changes in separate e-mails that can be addressed individually.

Happy Holidays!

Nathan

[Andy Fingerhut]

Nathan:

Disclaimer: I'm just a Clojure contributor since about 2012, not one who makes decisions on Clojure's contribution process.  I've seen lots of discussion on the contribution process during that time, and before (only the parts of it done in public on-line fora).

Have you read the link at the bottom of this page, with Rich Hickey's comments in a similar discussion?  https://dev.clojure.org/display/community/Contributing+FAQ

Here is the direct link, for convenience: https://groups.google.com/forum/#!msg/clojure/jWMaop_eVaQ/3M4gddaXDZoJ

I do not speak for anyone but myself, but I haven't heard anything that suggests Rich has changed his mind on anything he said there.

By stating the following statistics, it might sound like incitement to discontent, but please, all gentle readers, take it simply as the statement of fact it is intended to be.  The average commit rate into the Clojure core project, averaged over any 12-month period since 2012, is from 5/month up to 15/month.  That includes the simple doc string fixes, as well as big feature additions like spec.

My guesses: The limiting factor has probably always been the Clojure core team's available time.  If they say that the current process helps conserve their time spent on it, I'm inclined to believe that they mean it.

Andy

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

--

You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

[Sean Corfield]

Further to the links provided by Andy, I would also point to the recently revamped Clojure web site that has further enshrined the contribution process: https://clojure.org/community/contributing

 

This has come up over, and over, and over, and over again over the years. It’s Rich’s project – let’s just respect his choices on how to contribute to it. The process is clearly documented and not onerous. Clojure is critical infrastructure for a number of companies these days. Rich and the Cognitect team take great pains to ensure that it remains stable at all times – to the point where running alpha and beta builds in production is “safe” and upgrades are “easy”, and we can all rely on Clojure for our livelihood. The contribution and review process are, in large part, responsible for that, as far as I’m concerned.

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: cloju...@googlegroups.com <cloju...@googlegroups.com> on behalf of Andy Fingerhut <andy.fi...@gmail.com>
Sent: Thursday, December 21, 2017 5:42:20 PM
To: cloju...@googlegroups.com

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

[Nathan Fisher]

Thanks Andy!

I am familiar with both, my first point on how to extract a patch from a PR was based on that. I guess for me being a bit of a command-line jockey the workflow of pulling the patch from a URL vs downloading a patch from Jira (in a very microbenchmark sense) is more efficient.

I (try) to subscribe to the idea of "strong opinions, weakly held" (e.g. nothing is written in stone). So while it's probably tiresome to hear it brought up, I think the status quo is worth questioning periodically. What are the obstacles? Do they still exist? Are they still relevant? All seem like a worthwhile check-point from time to time.

Based on your response the immediate question that came to mind is how can the process be scaled? The immediate answer appears to be it can't and won't be. Which given Rich took off 2 years to develop the language to begin with is perfectly within his purview.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

--

You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Nathan Fisher]

Hi Sean,

I've been waivering on resuming this thread for this exact reason;

"It’s Rich’s project – let’s just respect his choices on how to contribute to it."

I also kind of giggled at the quote you have in your signature from Margaret Atwood. Does this thread mean I'm alive? :)

To your point part of me is yes agree! However I have this ongoing niggle so not 100%. It's still gestating but I'll try to list the salient points that are creating tension;

1. Clojure is more than a project it's a community and ecosystem.
   
2. An OSS developer is a difficult and undervalued role don't make it unnecessarily difficult.
3. In operational terms the outlined release process has multiple single PoF, a low bus-factor, and is outwardly slow.
4. Not Java committee slow but not confidently tested and out the door in 15m either.
5. 4. might be an unreasonable expectation for a language but does it need to be?
6. A slow end-to-end release process is a potential risk wrt security (e.g. 0-days).
7. The more thriving a community the better for libraries, job opportunities, and employers willing to take a risk.
8. Community health is dependent on a sense of belonging (ingroup/outgroup).
9. The last two points are feedback loops in both the negative and positive direction depending on experience.
   
10. Ingroup in terms of OSS can be reinforced by contribution in filling a community need.
11. That gap can be documentation, articles, code, etc.
12. The closer it is to the core of the communities benefit the more attachment an individual will have.
13. There are gaps particularly in documentation.
14. Those gaps in the absence of an efficient submission process have been filled elsewhere.
15. "Elsewhere" is fragmentation which makes it difficult for people new to the community to orientate themselves.

There's probably more but I'll leave it at that for now.

Kind regards,

Nathan

[Sean Corfield]

Nathan,

 

I’m not going to address all your points (I’m not entirely sure what you mean by some of them anyway) but I will observe that I’ve been an open source software developer across a range of technologies for about 25 years now. The projects I’ve contributed to have had a range of processes from “nothing” to heavily structured workflows and from small libraries to fairly large systems (Mono and OpenOffice, for example). Many have been covered by paper-based Contributor Agreements and some have required very specific formats and processes for code submission.

 

Many newer developers have only known GitHub as an OSS ecosystem and, yes, GitHub has helped casual OSS flourish. I don’t want to sound like one of “The Four Yorkshiremen” but that doesn’t mean it’s automatically an improvement nor a one-size-fits-all system…

 

In the Clojure world, I maintain a couple of the Contrib projects that operate under the CA/JIRA/patch process and I also maintain a number of projects on GitHub that operate under the PR process. I’ve also contributed to both types of projects. So I’ve seen this from both sides, in both types of projects.

 

Here are a few of your points I do, specifically, want to respond to:

 

An OSS developer is a difficult and undervalued role don't make it unnecessarily difficult

 

Remember that there are two groups of OSS developers and they’re not inherently equal: there’s the group that runs the project and there’s those folks in the broader community that contribute. For the folks running the project, they spend a lot of their time on the code, documentation, tooling, whatever. They have a right – and a reasonable expectation – that their role is not made difficult by outside contributors. They determine how the project is run to best suit their needs, they determine the contribution process, they determine guidelines for code that is submitted, and so on. In addition to all the work they do, planning, documenting, building, and testing the features of the project, they also have to deal with bug reports and enhancement requests. Bug reports are rarely accompanied by small, self-contained, complete examples so reproducing a reported bug can be a time-consuming process. Enhancement requests may or may not fall within the project’s goals so there is often a justification process required when they reject an enhancement, and they often come without tests and without documentation. I can tell you, as a long-time project maintainer, that some community members get really bent out of shape when their bug report is dismissed for a lack of a repro case or their enhancement is rejected as not in line with the project goals! When asked for tests or documentation updates to accompany their enhancement request, a common response is “Sorry, I don’t have time for that” and they expect the project team to do that work for them. That all means that the core project team can get dragged into public flame wars over how they run a project. That can be extremely frustrating and time-consuming and does no one – either on the project team or in the broader community – any good at all.

 

Points 3, 4, 5, and 6 re: “slowness”

 

As I said in a previous post, many of us value Clojure’s stability for our critical production infrastructure. My company has about 75,000 lines of Clojure and we regularly run prerelease builds in production so that we can leverage new features as they are being developed and made available in alpha and beta releases. We’re able to do that because the core Clojure team are extremely careful about what they change and what they add. They care about the design of features, they care about the language’s overall coherence, and they care about performance and robustness. That takes time. Do they make mistakes sometimes? Sure, but more often those mistakes come from rushed decisions (and I’m talking about large, critical projects in general in this case: rushed decisions are usually the ones that lead to changes that many people consider mistakes in hindsight!).

 

One of the great things about Clojure – about a Lisp – is that the core language is small and very powerful so folks can easily build new “language features” themselves using functions and macros. At World Singles we have a worldsingles.clojure.extensions namespace with a handful of functions that we’d love to see in core but we know are unlikely to be accepted. That’s fine. We can treat them just like core functions by referring them into our application namespaces. We all have the power to extend Clojure to suit our needs. Many things that would need to be built-in features in other languages can be libraries in Clojure (core.async is a stunning example of this), and those libraries can move at a different speed, and have different contribution processes if they want.

 

Ingroup in terms of OSS can be reinforced by contribution in filling a community need

 

There are thousands of projects out there that Clojurians can contribute to. There are community-run documentation sites that accept PRs without any ceremony and even sites that can be directly edited by the community (the examples on ClojureDocs, for instance). The reality is, however, that the number of actual contributors is tiny compared to the size of any community. Then there are people who complain about the contribution processes on projects but don’t contribute even when the process is made simpler (and before anyone says #notallcomplainers – be honest: there is, in every community, a vocal group of folks who kvetch about how the project is run and that’s the totality of their “contribution”).

 

An example I like to bring up in discussions like these is the clojure.java.jdbc documentation. It used to be in the GitHub repo with the code. I had a steady stream of complaints about how “ridiculous” it was that contributing to documentation required both a signed CA and a JIRA-based patch process. So I moved it all to the clojure-doc guides site instead, where anyone can contribute changes via simple Pull Requests with no CA needed. How many of those complainers do you think have contributed to that documentation since then?

 

So, forgive me for being less than sympathetic when this topic comes up (and thank you for reading this far!). 😊

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: cloju...@googlegroups.com <cloju...@googlegroups.com> on behalf of Nathan Fisher <nfi...@junctionbox.ca>
Sent: Thursday, December 21, 2017 11:27:49 PM
To: cloju...@googlegroups.com

[Nathan Fisher]

On Fri, 22 Dec 2017 at 15:36 Sean Corfield <se...@corfield.org> wrote: 

Many newer developers have only known GitHub as an OSS ecosystem and, yes, GitHub has helped casual OSS flourish. I don’t want to sound like one of “The Four Yorkshiremen” but that doesn’t mean it’s automatically an improvement nor a one-size-fits-all system…

I've committed to other ecosystems including Sourceforge, SVN, and CVS. I'm not suggesting Github PR's is the one true way in the slightest. To be honest in a commercial setting I hate the PR model.

 

 

In the Clojure world, I maintain a couple of the Contrib projects that operate under the CA/JIRA/patch process and I also maintain a number of projects on GitHub that operate under the PR process. I’ve also contributed to both types of projects. So I’ve seen this from both sides, in both types of projects.

 

Here are a few of your points I do, specifically, want to respond to:

 

An OSS developer is a difficult and undervalued role don't make it unnecessarily difficult 

"There are only two kinds of languages: the ones people complain about and the ones nobody uses."

I meant maintainer here... as in Rich et al. As in their job is often thankless with a lot of complaints. The tension I was describing was what's fair as I acknowledge my moaning is a PITA. I understand being an OSS maintainer is a *huge* undertaking. I've made financial donations to projects that I get regular value from where they accept payments to an amount similar-ish to an IntelliJ license because of that understanding.

Points 3, 4, 5, and 6 re: “slowness”

 

As I said in a previous post, many of us value Clojure’s stability for our critical production infrastructure. My company has about 75,000 lines of Clojure and we regularly run prerelease builds in production so that we can leverage new features as they are being developed and made available in alpha and beta releases. We’re able to do that because the core Clojure team are extremely careful about what they change and what they add. They care about the design of features, they care about the language’s overall coherence, and they care about performance and robustness. That takes time. Do they make mistakes sometimes? Sure, but more often those mistakes come from rushed decisions (and I’m talking about large, critical projects in general in this case: rushed decisions are usually the ones that lead to changes that many people consider mistakes in hindsight!).

I'm not suggesting a sacrifice of Quality on the alter of speed. On the contrary I'm advocating the exact opposite... things like downstream builds as sanity checks that integrate and validate execution of "core libs/tools" like boot, lein, clojurescript, etc. Manual inspection is not scalable nor is it terribly repeatable. I was sort of surprised to see the issues in the 1.9 release relating to Clojure/ClojureScript builds. Again there's only so many hours in the day and it's not meant as a criticism. It's merely an observation and raises a question... how can we as a community efficiently and effectively spread the burden?

For further reference on my view that Quality and Speed to market aren't necessarily in opposition please see the talk below by Nicole Forsgren and Jez Humble. I've worked in both move fast and break thing environments (it sucked) and move fast without a hiccup environment (it was awesome not having to hold our collective breaths on deployment).

https://www.youtube.com/watch?v=gsjCWrCUjNg

 

I know and very much appreciate/understand everyone at Cognitect are likely slotting this in around project work which keeps the lights on. I'm not trying to be a pain in the ass about it although I understand that is probably the net impact of my continued nattering.

 

One of the great things about Clojure – about a Lisp – is that the core language is small and very powerful so folks can easily build new “language features” themselves using functions and macros. At World Singles we have a worldsingles.clojure.extensions namespace with a handful of functions that we’d love to see in core but we know are unlikely to be accepted. That’s fine. We can treat them just like core functions by referring them into our application namespaces. We all have the power to extend Clojure to suit our needs. Many things that would need to be built-in features in other languages can be libraries in Clojure (core.async is a stunning example of this), and those libraries can move at a different speed, and have different contribution processes if they want.

 

Ingroup in terms of OSS can be reinforced by contribution in filling a community need

 

There are thousands of projects out there that Clojurians can contribute to. There are community-run documentation sites that accept PRs without any ceremony and even sites that can be directly edited by the community (the examples on ClojureDocs, for instance). The reality is, however, that the number of actual contributors is tiny compared to the size of any community. Then there are people who complain about the contribution processes on projects but don’t contribute even when the process is made simpler (and before anyone says #notallcomplainers – be honest: there is, in every community, a vocal group of folks who kvetch about how the project is run and that’s the totality of their “contribution”).

This is where I was referring to fragmentation in terms of community-run documentation. I've worked across a number of languages by far the current "Gold-standard" for documentation in my view is Go. Everything's available under godoc.org, examples are validated (see https://godoc.org/testing#hdr-Examples), all 3rd party libraries are co-located and searchable.

I admit to currently being a vocal complainer however in my initial e-mail I mentioned I am willing to contribute/collaborate to the changes I proposed inline with the existing process.

 

An example I like to bring up in discussions like these is the clojure.java.jdbc documentation. It used to be in the GitHub repo with the code. I had a steady stream of complaints about how “ridiculous” it was that contributing to documentation required both a signed CA and a JIRA-based patch process. So I moved it all to the clojure-doc guides site instead, where anyone can contribute changes via simple Pull Requests with no CA needed. How many of those complainers do you think have contributed to that documentation since then?

 

So, forgive me for being less than sympathetic when this topic comes up (and thank you for reading this far!). 😊

Thank-you for the thoughtful response I understand it's easy to complain from the arm-chair. :) I hope I'm not ruffling any feathers too thoroughly or causing migraines.

I'm happy to call the thread at a close at this point. I am looking to make some of the changes Alex mentioned for the README doc if I can find them outlined in Jira.