ClojureScript experience report: writing CouchDB views

[Chas Emerick]

For the past week or two, I've been picking away at using ClojureScript to write CouchDB views (and filters, and validators, etc). I can report success:

http://cemerick.com/2011/10/11/writing-couchdb-views-using-clojurescript

Very sweet!

However, I wanted to provide some feedback re: the issues I ran into while making this work. They're outlined in the blog post above and mentioned in the clutch-clojurescript README, but I'll repeat here for purposes of discussion:

(1) ClojureScript is not yet available as a proper library. This forces me to include some binary version of it in this git repo (a hefty 8.3MB!…which includes various google JavaScript UI bits that I’d hope would be broken out eventually), and bundle the necessary bits into the clutch-clojurescript jar. I would very much like to roll clutch-clojurescript’s functionality into Clutch proper, but I'll not do so until the latter can rely upon a ClojureScript dependency. I totally understand the core devs not wanting to taint what is not strictly a JVM codebsae with JVM build-and-deploy apparatus, but there is surely room for a release mechanism off to the side that would provide a way for developers far and wide to embed ClojureScript. Bootstrap scripts and binaries in git repos are OK when tinkering, experimenting, and establishing a beachhead, but they must fade (or at least coexist with more regularized process) for sane mortals to use the bits.

(I've heard that Google's lack of releases is one concern behind not providing ClojureScript releases to date. To that, I'd say, don't let junk run downhill — say that ClojureScript vX.Y.Z includes gClosure rABC. Sounds solved to me…?)

FWIW, I'll happily volunteer to produce a standalone release/deployment process to interface with and reuse whatever is in use now to build/test ClojureScript.

(2) ClojureScript / Google Closure produces a very large code footprint, even for the simplest of view functions. I seem to recall Rich saying at the first announcement that this is/was a implementation limitation — that ClojureScript somehow prevents some of gClosure's optimizations from being applied. What is the current state of this? I don't see any relevant outstanding tickets.

(In any case, the code size of a view function string should have little to no impact on runtime performance of that view. The only penalty to be paid should be in view server initialization, which should be relatively infrequent. Further, the vast majority of view runtime is dominated by IO and actual document processing, not the loading of a handful of JavaScript functions.)

(3) To my surprise (and shock/horror), the rev of Spidermonkey that is used by CouchDB (and Couchbase Single, and Cloudant) does not treat regular expression literals properly — they work fine as arguments to e.g. string.match, but e.g. /foo/.exec() fails. Using the RegExp() function with a string argument *does* work. This was reported a long time ago, but has had little attention (though I'm trying to stir it up a bit): https://issues.apache.org/jira/browse/COUCHDB-577

I'm hoping to get to the bottom of this sooner or later, but I wonder if it'd be worthwhile to change the ClojureScript reader to emit (js/RegExp "foo") calls instead of /foo/ literals (and hope that gClosure doesn't optimize the former into the latter)? After all, there's lots of CouchDB deployments out there with apparently broken spidermonkey installs/configurations, and likely lots of other apps/servers/environments in similarly dire straits.

Thanks for helping me kick Javascript out of a part of my life,

- Chas

[Brenton Ashworth]

> but there is surely room for a release mechanism off to the side that would provide a way for developers
> far and wide to embed ClojureScript. Bootstrap scripts and binaries in git repos are OK when tinkering,
> experimenting, and establishing a beachhead, but they must fade (or at least coexist with more regularized
> process) for sane mortals to use the bits.

I agree 100%. We don't have to commit to numbered releases right now.
Just having automated, timestamped builds would be an improvement.

[Chris Granger]

+1

If this happens I'll commit to building a lein plugin for cljs as well as cljs compilation as middleware.

Cheers,
Chris.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.
To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.

[Chas Emerick]

On Oct 11, 2011, at 12:32 PM, Brenton Ashworth wrote:

>> but there is surely room for a release mechanism off to the side that would provide a way for developers
>> far and wide to embed ClojureScript. Bootstrap scripts and binaries in git repos are OK when tinkering,
>> experimenting, and establishing a beachhead, but they must fade (or at least coexist with more regularized
>> process) for sane mortals to use the bits.
>
> I agree 100%. We don't have to commit to numbered releases right now.
> Just having automated, timestamped builds would be an improvement.

Minor suggestion: it really should be an incrementing version like 0.0.853. It's the equivalent of timestamps, but will have the benefit of working with version ranges in lein, maven, etc. I think such version numbers are clearly not "numbered releases" per se, just convenient usage of the version number format.

- Chas

[Stuart Halloway]

We had talked about git-tagging head on every commit to source these numbers. (I think it is important to get this right in git and then let other tools consume it, because we don't want to presume any other tools in consumers' build chains.)

Does anybody know the performance implications of tagging every commit in a git repo?

Stu

[Kevin Downey]

how does tagging address the dependency issue at all?

what is the difference between tagging every commit and just trading shas?

> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>
>

--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

[Paul Stadig]

Well, there's the problem that git is distributed, so two people can commit to the repo at exactly the same time. I assume that you're talking about having a "canonical" repo which would do this tagging when you pushed to it. You can have jenkins tag each build and push the tags back to git.

It seems like a better solution would be to just do snapshot releases, and a real release every once in a while. Why are we trying to avoid releases again?


Paul

[Stuart Halloway]

> how does tagging address the dependency issue at all?
>
> what is the difference between tagging every commit and just trading shas?

Humans can compare tags for ordering.

[pmbauer]

Thoughts on keeping the three dependencies (goog.js/goog.compiler/clojurescript) as separate artifacts rather than have one giant ClojureScript artifact to rule them all?

FWIW, I'm using a set of lein projects to deploy the three artifacts to clojars for my own use, and I see others have as well.

[Chas Emerick]

We're just talking about coming up with a commit number, right? If so, no need to tag:

git rev-list master | wc -l

Ignores branches of course. If that's not desirable, then perhaps:

git rev-list --all | wc -l

…which mostly behaves like revision number in svn.

- Chas

[Stuart Halloway]

> Well, there's the problem that git is distributed, so two people can commit to the repo at exactly the same time. I assume that you're talking about having a "canonical" repo which would do this tagging when you pushed to it. You can have jenkins tag each build and push the tags back to git.

Yes. Jenkins (or similar) could do this. And Clojure and ClojureScript already have canonical repos.

> It seems like a better solution would be to just do snapshot releases, and a real release every once in a while.

Simple building blocks are better. If every build from the canonical repo has an incrementing tag, then you can easily build everything else on that. And anybody can target anything as a release. If, on the other hand, you only expose a catered set of releases, then anybody who wants more flexibility is back to square one.

> Why are we trying to avoid releases again?

Since I am always pushing for releases, I must assume that there is some "we" that you and I are allied against. :-)

Stu

[Chas Emerick]

On Oct 11, 2011, at 2:05 PM, pmbauer wrote:

> Thoughts on keeping the three dependencies (goog.js/goog.compiler/clojurescript) as separate artifacts rather than have one giant ClojureScript artifact to rule them all?
> FWIW, I'm using a set of lein projects to deploy the three artifacts to clojars for my own use, and I see others have as well.

Beyond that, I'd love to see the "core" goog.js stuff split off from the UI/DOM/browser-related bits; the latter has no use in e.g. nodejs, couchdb, etc.

- Chas

[Stuart Halloway]

>>>>> but there is surely room for a release mechanism off to the side that would provide a way for developers
>>>>> far and wide to embed ClojureScript. Bootstrap scripts and binaries in git repos are OK when tinkering,
>>>>> experimenting, and establishing a beachhead, but they must fade (or at least coexist with more regularized
>>>>> process) for sane mortals to use the bits.
>>>>
>>>> I agree 100%. We don't have to commit to numbered releases right now.
>>>> Just having automated, timestamped builds would be an improvement.
>>>
>>> Minor suggestion: it really should be an incrementing version like 0.0.853. It's the equivalent of timestamps, but will have the benefit of working with version ranges in lein, maven, etc. I think such version numbers are clearly not "numbered releases" per se, just convenient usage of the version number format.
>>
>> We had talked about git-tagging head on every commit to source these numbers. (I think it is important to get this right in git and then let other tools consume it, because we don't want to presume any other tools in consumers' build chains.)
>>
>> Does anybody know the performance implications of tagging every commit in a git repo?
>
> We're just talking about coming up with a commit number, right? If so, no need to tag:
>
> git rev-list master | wc -l

That's quite cool. It seems like the tradeoff with tags is convenience for writers vs. readers. Inferring the number simplifies writing out of the picture, as no tag need be written. But tags, once written, will be visible in all kinds of useful places in tools.

Stu

[Chas Emerick]

Since these readers have the git repo that they'd be reading tags from, they by definition have a shell…so are you thinking of people using graphical tools, e.g. gitx et al.?

Also, what performance implications worry you? A tag is just another ref, 40 bytes in a file. It looks like git's git repo has ~400 tags…and gitx shows them all and the full commit tree nicely FWIW.

- Chas

[Rich Hickey]

Other ideas here:

http://stackoverflow.com/questions/514188/how-would-you-include-the-current-commit-id-in-a-git-projects-files/514668#514668

Rich

[Rich Hickey]

On Oct 11, 2011, at 10:50 AM, Chas Emerick wrote:

> For the past week or two, I've been picking away at using ClojureScript to write CouchDB views (and filters, and validators, etc). I can report success:
>
> http://cemerick.com/2011/10/11/writing-couchdb-views-using-clojurescript
>
> Very sweet!
>
> However, I wanted to provide some feedback re: the issues I ran into while making this work. They're outlined in the blog post above and mentioned in the clutch-clojurescript README, but I'll repeat here for purposes of discussion:
>
> (1) ClojureScript is not yet available as a proper library. This forces me to include some binary version of it in this git repo (a hefty 8.3MB!…which includes various google JavaScript UI bits that I’d hope would be broken out eventually), and bundle the necessary bits into the clutch-clojurescript jar. I would very much like to roll clutch-clojurescript’s functionality into Clutch proper, but I'll not do so until the latter can rely upon a ClojureScript dependency. I totally understand the core devs not wanting to taint what is not strictly a JVM codebsae with JVM build-and-deploy apparatus, but there is surely room for a release mechanism off to the side that would provide a way for developers far and wide to embed ClojureScript. Bootstrap scripts and binaries in git repos are OK when tinkering, experimenting, and establishing a beachhead, but they must fade (or at least coexist with more regularized process) for sane mortals to use the bits.
>
> (I've heard that Google's lack of releases is one concern behind not providing ClojureScript releases to date. To that, I'd say, don't let junk run downhill — say that ClojureScript vX.Y.Z includes gClosure rABC. Sounds solved to me…?)
>
> FWIW, I'll happily volunteer to produce a standalone release/deployment process to interface with and reuse whatever is in use now to build/test ClojureScript.

I think the key thing is not to equate

release mechanism == JVM build-and-deploy apparatus

JavaScript/web tools are not generally distributed via Maven (nor, BTW, are lots of Java things. I'd peg our success at finding artifacts corresponding to lib versions we want to use at < 50%)

Also note, ClojureScript is not in fact a library but a language tool. Using it as a library from Clojure apps is likely a minority case.

IMO, release numbers for something at this early stage convey almost nothing. It is just manual effort to produce them, produce readmes etc, all of which duplicates information in the repos.

The ideas behind the monotonically increasing, automatically generated revision numbers scheme are:

SHAs have no order discernible by humans or tools
- but order matters for useful semantics like before/after
Automatic trumps manual
Provide ingredients for packaging, but not packages
- a key to which is useful names
Relate names to some git truth (an artifact should be no better/different than a checkout)
Source and git are primary
- the further you move away from them the more complex things become

These are going to be our first steps. IMO, any packaging strategy should remain subservient to these ideas, rather than become the driver of all things as Maven has for us with Clojure.

Rich

[Chas Emerick]

On Oct 11, 2011, at 9:24 PM, Rich Hickey wrote:

> I think the key thing is not to equate
>
> release mechanism == JVM build-and-deploy apparatus
>
> JavaScript/web tools are not generally distributed via Maven (nor, BTW, are lots of Java things. I'd peg our success at finding artifacts corresponding to lib versions we want to use at < 50%)
>
> Also note, ClojureScript is not in fact a library but a language tool. Using it as a library from Clojure apps is likely a minority case.

I'm not sure I understand what a "language tool" is here, or what that implies in contrast to the library concept. ClojureScript isn't just a javascript file I can add into or refer to from my web project, and I don't want to mentally categorize it with other "language tool" novelties that I can download, install, play with, learn from, and not use in real applications. Concretely, there's a critical JVM, or, non-JavaScript component (the compiler & userland macros) whose most obvious home for the most obvious use case (interactive web development) is integrated into one's (Clojure!) web backend. The way into that environment is through dependency management, and maven repositories in particular.

> IMO, release numbers for something at this early stage convey almost nothing. It is just manual effort to produce them, produce readmes etc, all of which duplicates information in the repos.
>
> The ideas behind the monotonically increasing, automatically generated revision numbers scheme are:
>
> SHAs have no order discernible by humans or tools
> - but order matters for useful semantics like before/after
> Automatic trumps manual
> Provide ingredients for packaging, but not packages
> - a key to which is useful names
> Relate names to some git truth (an artifact should be no better/different than a checkout)
> Source and git are primary
> - the further you move away from them the more complex things become

This sounds like a recipe for author/developer/maintainer bliss, so I can understand the appeal. I know you're just talking about the numbering scheme here, but:

(a) Providing packages is arguably the function of a project, and they are a key abstraction in any sane software development process. What produces those packages, from a user perspective, is firmly in "I don't know, and I don't want to know" territory.

(b) Source and git can be *extraordinarily* complex, and is usually something that maintainers abstract away from for good reasons. Insofar as good packaging ensures monotonically-increasing version numbers, and insofar as users of those packages care not a whit about source, the subtleties of the particular structure of any particular git tree can be impenetrable. Good 'ol CVS and subversion were masters at this, where r127 was on this branch, r128 was from that branch, and r129 was from crazy Larry's branch. That dynamic can be recreated with git, and carried through whichever dependency management system(s) you drop packages into the top of, but I don't know that it's an unabashed good.

> These are going to be our first steps. IMO, any packaging strategy should remain subservient to these ideas, rather than become the driver of all things as Maven has for us with Clojure.

Quite tangentially, maven is whatever you make of it (little-m maven dep mgmt, not speaking about mvn the tool). You can stuff single-segment-versioned artifacts into its repos with nary a complaint, and you can do so without manual effort to produce those version numbers, readmes, and so on. The notions of semantic versioning, documentation, stable releases, etc. are things demanded by users, not maven.

- Chas

[Rich Hickey]

On Oct 12, 2011, at 6:57 AM, Chas Emerick wrote:

>
> On Oct 11, 2011, at 9:24 PM, Rich Hickey wrote:
>
>> I think the key thing is not to equate
>>
>> release mechanism == JVM build-and-deploy apparatus
>>
>> JavaScript/web tools are not generally distributed via Maven (nor, BTW, are lots of Java things. I'd peg our success at finding artifacts corresponding to lib versions we want to use at < 50%)
>>
>> Also note, ClojureScript is not in fact a library but a language tool. Using it as a library from Clojure apps is likely a minority case.
>
> I'm not sure I understand what a "language tool" is here, or what that implies in contrast to the library concept.

gcc, Java, Python, Ruby etc are language tools. Which are released via Maven?

> ClojureScript isn't just a javascript file I can add into or refer to from my web project,

Exactly, it's a compiler.

> and I don't want to mentally categorize it with other "language tool" novelties that I can download, install, play with, learn from, and not use in real applications.

Note to everyone - did you just add an adjective/qualifier like 'novelty' or 'real'? Is it adding value or is it just rhetoric? If the latter, please leave it out.

> Concretely, there's a critical JVM, or, non-JavaScript component (the compiler & userland macros) whose most obvious home for the most obvious use case (interactive web development) is integrated into one's (Clojure!) web backend. The way into that environment is through dependency management, and maven repositories in particular.
>

That's your perspective, and I granted the use case, but I think it is important ClojureScript not focus on merely being an adjunct to Clojure development.

>> IMO, release numbers for something at this early stage convey almost nothing. It is just manual effort to produce them, produce readmes etc, all of which duplicates information in the repos.
>>
>> The ideas behind the monotonically increasing, automatically generated revision numbers scheme are:
>>
>> SHAs have no order discernible by humans or tools
>> - but order matters for useful semantics like before/after
>> Automatic trumps manual
>> Provide ingredients for packaging, but not packages
>> - a key to which is useful names
>> Relate names to some git truth (an artifact should be no better/different than a checkout)
>> Source and git are primary
>> - the further you move away from them the more complex things become
>
> This sounds like a recipe for author/developer/maintainer bliss, so I can understand the appeal. I know you're just talking about the numbering scheme here, but:

It goes far beyond that. People have to stop seeing only the benefits of things and not the costs. Have we gotten some benefits from Maven? - sure. What has it cost us? - plenty.

> (a) Providing packages is arguably the function of a project, and they are a key abstraction in any sane software development process. What produces those packages, from a user perspective, is firmly in "I don't know, and I don't want to know" territory.
>

If you are going to insist on that, I can assure you now our project-approved official artifacts for ClojureScript will not be Maven. More likely zip files (or make! :-) like everyone else.

> (b) Source and git can be *extraordinarily* complex, and is usually something that maintainers abstract away from for good reasons. Insofar as good packaging ensures monotonically-increasing version numbers, and insofar as users of those packages care not a whit about source, the subtleties of the particular structure of any particular git tree can be impenetrable. Good 'ol CVS and subversion were masters at this, where r127 was on this branch, r128 was from that branch, and r129 was from crazy Larry's branch. That dynamic can be recreated with git, and carried through whichever dependency management system(s) you drop packages into the top of, but I don't know that it's an unabashed good.
>

This makes no sense. If artifacts rule, yet *don't* have identifiable connections to some canonical source version, then what do they represent? Whatever Larry decided to jar up? Sure, people could make a mess in CVS - how is that relevant?

You certainly don't want SHAs instead of 1.2.3. Why prefer Larry's 1.2.3 to the system-generated, directly-connected 123? Blessing particular points in time is orthogonal. It can be done periodically, but in the interim we suffer greatly from SHAs and snapshots with no ordering, no connection to source, nor ability to back out.

>> These are going to be our first steps. IMO, any packaging strategy should remain subservient to these ideas, rather than become the driver of all things as Maven has for us with Clojure.
>
> Quite tangentially, maven is whatever you make of it (little-m maven dep mgmt, not speaking about mvn the tool). You can stuff single-segment-versioned artifacts into its repos with nary a complaint, and you can do so without manual effort to produce those version numbers, readmes, and so on. The notions of semantic versioning, documentation, stable releases, etc. are things demanded by users, not maven.

Ah, user demands, those great sources of creativity and productivity. Let's be clear - I'm not trying to negate the needs, but Maven cannot be the only acceptable answer, and it's time for some fresh thinking in this area.

Maven simply isn't used for language tools, period. Language-as-library is a separate use case. ClojureScript is a perfect opportunity to do something different. IMO, canonic artifact repos are redundant with canonic source repos, as human assigned numbers are redundant with vcs identities, and the indirections only add complexity. I would much prefer a system that was git aware and integrated, and I challenge the community to help us create one for ClojureScript.

Rich

[Chas Emerick]

On Oct 12, 2011, at 7:52 AM, Rich Hickey wrote:

>>> I think the key thing is not to equate
>>>
>>> release mechanism == JVM build-and-deploy apparatus
>>>
>>> JavaScript/web tools are not generally distributed via Maven (nor, BTW, are lots of Java things. I'd peg our success at finding artifacts corresponding to lib versions we want to use at < 50%)
>>>
>>> Also note, ClojureScript is not in fact a library but a language tool. Using it as a library from Clojure apps is likely a minority case.
>>
>> I'm not sure I understand what a "language tool" is here, or what that implies in contrast to the library concept.
>
> gcc, Java, Python, Ruby etc are language tools. Which are released via Maven?

Three of those are languages that, generally speaking, establish and operate islands of their own.

I see obvious corollaries with gcc: multiple compilation targets, etc. And, I can obviously see ClojureScript being an installed part of a non-JVM toolchain. I don't see either fact alleviating the pressures for release numbering, canonical packaging, etc.

>> Concretely, there's a critical JVM, or, non-JavaScript component (the compiler & userland macros) whose most obvious home for the most obvious use case (interactive web development) is integrated into one's (Clojure!) web backend. The way into that environment is through dependency management, and maven repositories in particular.
>
> That's your perspective, and I granted the use case, but I think it is important ClojureScript not focus on merely being an adjunct to Clojure development.

No, not merely, and not just Clojure — but the JVM affinity isn't nothing, and neither is the reach of the distribution lingua franca there.

>>> IMO, release numbers for something at this early stage convey almost nothing. It is just manual effort to produce them, produce readmes etc, all of which duplicates information in the repos.
>>>
>>> The ideas behind the monotonically increasing, automatically generated revision numbers scheme are:
>>>
>>> SHAs have no order discernible by humans or tools
>>> - but order matters for useful semantics like before/after
>>> Automatic trumps manual
>>> Provide ingredients for packaging, but not packages
>>> - a key to which is useful names
>>> Relate names to some git truth (an artifact should be no better/different than a checkout)
>>> Source and git are primary
>>> - the further you move away from them the more complex things become
>>
>> This sounds like a recipe for author/developer/maintainer bliss, so I can understand the appeal. I know you're just talking about the numbering scheme here, but:
>
> It goes far beyond that. People have to stop seeing only the benefits of things and not the costs. Have we gotten some benefits from Maven? - sure. What has it cost us? - plenty.

(Sorry, but this reminded me of the classic, "What has Maven ever done for us?": http://www.wakaleo.com/blog/283-what-has-maven-ever-done-for-us :-)

I don't know that costs aren't being taken into account. More like — again, from many users' perspective, including many/most Clojure programmers — those costs are sunk.

I think very, very hard about every tool I adopt that isn't available through a sane distribution mechanism; that's maven at the app and library level, apt for server bits, etc. Similarly, I think very hard about every library I adopt that doesn't have sane management of state, either via Clojure itself, a Clojure wrapper, or other reasonable mechanisms. Otherwise, in both cases, I know that I'll have to account for the complexity that wasn't accounted for upstream.

>> (a) Providing packages is arguably the function of a project, and they are a key abstraction in any sane software development process. What produces those packages, from a user perspective, is firmly in "I don't know, and I don't want to know" territory.
>
> If you are going to insist on that, I can assure you now our project-approved official artifacts for ClojureScript will not be Maven. More likely zip files (or make! :-) like everyone else.

"Everyone else" being the aforementioned "language tools"?

Well, that's unfortunate. I'd think there would be room for multiple official distribution channels, including tarballs, maven, apt, rpm, etc. Such things will be taken care of by people downstream if you don't take pains to do it yourself (speaking broadly here, of course), but that leaves your project helpless at the hands of whoever is doing the bundling. I've been touched by it myself, but many others can speak with more authority than I can about how particular distributions of libraries, tools, and servers are often totally botched by third-party package maintainers.

>> (b) Source and git can be *extraordinarily* complex, and is usually something that maintainers abstract away from for good reasons. Insofar as good packaging ensures monotonically-increasing version numbers, and insofar as users of those packages care not a whit about source, the subtleties of the particular structure of any particular git tree can be impenetrable. Good 'ol CVS and subversion were masters at this, where r127 was on this branch, r128 was from that branch, and r129 was from crazy Larry's branch. That dynamic can be recreated with git, and carried through whichever dependency management system(s) you drop packages into the top of, but I don't know that it's an unabashed good.
>
> This makes no sense. If artifacts rule, yet *don't* have identifiable connections to some canonical source version, then what do they represent? Whatever Larry decided to jar up? Sure, people could make a mess in CVS - how is that relevant?

Again, named, versioned artifacts are a necessary abstraction; no one can — nor wants to — have their hands in the source of all (or even some) of the things they depend upon. The use of those artifacts represents trust in one direction, and responsibility in the other. Certainly an asymmetrical relationship, but both parties opt into and benefit from it.

And yes, you're Larry in this case. ;-)

> You certainly don't want SHAs instead of 1.2.3. Why prefer Larry's 1.2.3 to the system-generated, directly-connected 123? Blessing particular points in time is orthogonal. It can be done periodically, but in the interim we suffer greatly from SHAs and snapshots with no ordering, no connection to source, nor ability to back out.

No, I don't want SHAs. The system-generated 123 is perfectly fine for what it represents, and can underly the blessed 1.2.3, but I only want to use the latter because of the abstraction, the trust/responsibility relationship, and the (hopefully, hopefully!) useful semantic hints in the version number that of course never exist in the system-generated number. That I sometimes use the former is only a result of circumstance, not preference.

>>> These are going to be our first steps. IMO, any packaging strategy should remain subservient to these ideas, rather than become the driver of all things as Maven has for us with Clojure.
>>
>> Quite tangentially, maven is whatever you make of it (little-m maven dep mgmt, not speaking about mvn the tool). You can stuff single-segment-versioned artifacts into its repos with nary a complaint, and you can do so without manual effort to produce those version numbers, readmes, and so on. The notions of semantic versioning, documentation, stable releases, etc. are things demanded by users, not maven.
>
> Ah, user demands, those great sources of creativity and productivity. Let's be clear - I'm not trying to negate the needs, but Maven cannot be the only acceptable answer, and it's time for some fresh thinking in this area.
>
> Maven simply isn't used for language tools, period. Language-as-library is a separate use case. ClojureScript is a perfect opportunity to do something different. IMO, canonic artifact repos are redundant with canonic source repos, as human assigned numbers are redundant with vcs identities, and the indirections only add complexity. I would much prefer a system that was git aware and integrated, and I challenge the community to help us create one for ClojureScript.

I'd say those users think their demands/expectations help *them* be more productive (and creative, as a result). It's easy to dismiss, but doing so can be a dicey business, especially around bikesheds that have a dearth of hard facts. I'm not sure I'd classify tarballs and system-generated rev numbers as fresh thinking, but I cringe to consider "creative package management" or "fresh thinking in release planning"; aside from being akin to "creative shell scripting" or "fresh thinking in network routing topology" for me, the results would presumably be islandesque.

I've been very careful and explicit about how maven can and should be an adjunct to whatever primary release mechanism/process/methodology is chosen otherwise. Best to keep its brain damage to itself; likewise for rpm, apt, etc. etc.

In any case, I'll certainly help where I reasonably can (only in the JVM arena), and stir up trouble otherwise. :-)

Cheers,

- Chas

[Brenton Ashworth]

> These are going to be our first steps. IMO, any packaging strategy should remain subservient to these ideas, rather than become the driver of all things as Maven has for us with Clojure.

I would like to propose a concrete step forward.

We have all agreed that there is a use case for having a ClojureScript
jar.

We can start by adding an initial version tag to the ClojureScript
repo like v0.0 or v0.1. We can then use 'get describe' to produce
additional versioning information:

=> git describe
v0.1-123-g1f8115b

This tells us that we are 123 commits ahead of the v0.1 tag and the
current SHA starts with 1f8115b.

Next, we add a new script named 'build' to the script dir in
ClojureScript. Running this script will produce a ClojureScript jar
file which has a name like

clojurescript-0.1.123-g1f81115b.jar

This would allow anyone at any time to create a jar which corresponds
to an exact point in the git repo. You can even go back and create
jars for any previous point in the history.

With this script in place, an external system (which ClojureScript
knows nothing about) can checkout and build ClojureScript jars and put
them into a _____ repo somewhere.

P.S.

I am not sure if the number produced by 'git describe' will be
monotonically increasing. If you rewrite git's history wouldn't that
number decrease?

[Chas Emerick]

On Oct 12, 2011, at 11:00 AM, Brenton Ashworth wrote:

> I am not sure if the number produced by 'git describe' will be
> monotonically increasing. If you rewrite git's history wouldn't that
> number decrease?

Yes; git describe, and other routes to systemic revision numbers (e.g. rev-list | wc) all work over the live tree, so you're SOL (or, inconsistent) if you rebase or otherwise rewrite history.

Note that git describe will only see annotated tags (e.g. `tag -a -m "comment" 0.0.1`; "lightweight tags" don't seem to work with it).

- Chas

[Colin Jones]

I'm not sure what the right thing to do is, but does history-rewriting
of the master branch on the canonical git repo actually happen in
practice? Seems like if you rewrite history on your local branch, then
try and push to a remote, it's not going to fly without a force-push,
which would screw anyone pulling it down. And if history-rewriting
doesn't happen on the canonical master branch, then if the system
making the builds could be guaranteed to not have local commits,
things like `git describe` ought to be safe.

-Colin

p.s. `svn checkout http://svn.github.com/clojure/clojure.git` gives an
SVN view of the git repo, which would give the monotonically
increasing version numbers ;)

> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>
>

--
Colin Jones

[David Nolen]

You don't rewrite history on a public branch.

David 

[Chas Emerick]

There's rules, there's edicts, and then there's those things that no one dares speak of, because they can happen quickly with just a couple of absent-minded keystrokes.

Sure, there's the reflog, and multiple copies everywhere.  But there's also git gc, bad luck, and Murphy. :-)

If there's enough riding on the results of `git describe`, it's good to be very clear about what the failure modes are.

- Chas

[Brenton Ashworth]

On Oct 12, 12:27 pm, David Nolen <dnolen.li...@gmail.com> wrote:

Yes, but you *can* rewrite history on a public branch. I just wanted
to point out that the source of these numbers is not guaranteed to
always be increasing.

I don't think this is a problem as it *should* never happen and even
if it did, the whole identifier is still unique because the SHA is
unique.

[David Nolen]

ClojureScript could host its own git repo and disable non-fast-forward pushes:

http://stackoverflow.com/questions/2085871/strategy-for-preventing-or-catching-git-history-rewrite

GitHub would then effectively be a mirror. Given that GitHub is primarily for visibility and nothing else I don't see the problem with this.

David

--

[Chris Granger]

While I agree that CLJS isn't meant for use with just Clojure, the people that are going to be using it at this point are by and large Clojure people that are likely using it with Clojure as well. We *need* to make it easy for those people, because they're our greatest asset in terms of getting others to use it. CLJS goes nowhere without libraries and tools, both of which are greatly facilitated by having a simple way to include the compiler as a service. While perhaps not ideal, tools like leiningen abstract all the pain from maven for the end user and that's the important part in the long run.

I can't tell you how many people I've talked to who have given up on trying ClojureScript because the only way to get it was "go clone this git repo and set up these env vars". On the other hand, people loved when  noir-cljs (which added compilation as middleware) worked, because they added it as a dependency and they were on their way.

 

With this script in place, an external system (which ClojureScript
knows nothing about) can checkout and build ClojureScript jars and put
them into a _____ repo somewhere.

When we do this, the one request that I have is that the compiler and goog.jar's be separate artifacts so that different "distributions" of the closure library can be paired with it. This is especially important given that we don't include everything in the library :)

Cheers,
Chris.

[David Nolen]

On Wed, Oct 12, 2011 at 1:18 PM, Chris Granger <ibd...@gmail.com> wrote:

While I agree that CLJS isn't meant for use with just Clojure, the people that are going to be using it at this point are by and large Clojure people that are likely using it with Clojure as well. We *need* to make it easy for those people, because they're our greatest asset in terms of getting others to use it. CLJS goes nowhere without libraries and tools, both of which are greatly facilitated by having a simple way to include the compiler as a service. While perhaps not ideal, tools like leiningen abstract all the pain from maven for the end user and that's the important part in the long run.

We're not in disagreement. I would love to see Leiningen support ClojureScript.

 

I can't tell you how many people I've talked to who have given up on trying ClojureScript because the only way to get it was "go clone this git repo and set up these env vars". On the other hand, people loved when  noir-cljs (which added compilation as middleware) worked, because they added it as a dependency and they were on their way.

I'm definitely not advocating that git be the way that most users are introduced to ClojureScript.

David

[Rich Hickey]

On Oct 12, 2011, at 10:26 AM, Chas Emerick wrote:

> I don't know that costs aren't being taken into account. More like — again, from many users' perspective, including many/most Clojure programmers — those costs are sunk.
>

That is patently false. The costs are incurred on an ongoing basis. The user's perspective, rather, is that software is free.

Rich

[Rich Hickey]

I've tagged my initial commit 'v0.0' so we can start playing with this idea.

We should produce a zip file, for those who won't know what a .jar is, nor care. Any jar-building tools downstream can consume zips.

We'll need a strategy for gclosure compiler and libs, both of which are svn based with occasional .zips. Not that we should incorporate their bits, but we should, in a single place (Clojure data file), specify with revision of each we depend upon such that the bootstrap script and any downstream tools can pull them.

Rich

On Oct 12, 2011, at 11:00 AM, Brenton Ashworth wrote:

[Daniel Solano Gomez]

On Thu Oct 13 06:48 2011, Rich Hickey wrote:
> I've tagged my initial commit 'v0.0' so we can start playing with this
> idea.
>
> We should produce a zip file, for those who won't know what a .jar is,
> nor care. Any jar-building tools downstream can consume zips.
>
> We'll need a strategy for gclosure compiler and libs, both of which
> are svn based with occasional .zips. Not that we should incorporate
> their bits, but we should, in a single place (Clojure data file),
> specify with revision of each we depend upon such that the bootstrap
> script and any downstream tools can pull them.
>
> Rich

One approach is to use a git submodule. You can create git clones of
the Google Closure repositories using git-svn. These repositories could
be automatically synchronized to the upstream subversion repositories on
a regular basis, i.e. nightly. The ClojureScript repository could then
reference these Git repositories as submodules.

I think the main advantage to this approach is that it reduces the
number of tools you need to use to get all the sources. All you need is
git and you don't have to worry about extra downloads or have scripts
depend on tools such as wget or curl.

It should also make it somewhat harder to mess up the dependencies
between ClojureScript and the upstream tools. Each revision of
the ClojureScript repository will have a reference to a specific commit
of its dependencies.

The main downside to this approach is that git submodules may not be
familiar to all users. If you are using the command line version of
git, you will need to 'submodule init' and 'submodule update' after your
clone.

Overall, I think it is an approach worth considering. I've used it in
some projects with good success.

Sincerely,

Daniel Solano Gómez

[David Nolen]

Perhaps the beginning of another conversation - but does it make sense to remove all dependencies on Google Closure APIs in core ClojureScript? Most of the bits of functionality currently being used are simple helpers we can provide ourselves.

David