math.combinatorics (and other new contrib libraries)

118 views
Skip to first unread message

Sean Corfield

unread,
Sep 9, 2011, 2:35:46 PM9/9/11
to cloju...@googlegroups.com
What are the obstacles to getting builds in place for
math.combinatorics and a JIRA project for it?

It looks like there are no test cases at present. That should be addressed.

It also needs to be listed here as a migration for
clojure.contrib.combinatorics :
http://dev.clojure.org/display/design/Contrib+Library+Names

Can all new contrib maintainers please look at / update / bug
Clojure/core about:
* http://build.clojure.org/ - if you don't see automated builds /
matrix tests, ask for them to be created
* http://dev.clojure.org/jira/secure/Dashboard.jspa - if your project
is not listed, ask for it to be created -
http://dev.clojure.org/jira/secure/BrowseProjects.jspa#all
* http://dev.clojure.org/display/design/Contrib+Library+Names - make
sure your library is listed and explain what it is migrated from

We probably need a page on the wiki that lists all old contrib modules
and explains their status. I'm happy to kick that off but I don't know
enough about many of the old modules so I'll need help. I've started
here: http://dev.clojure.org/pages/viewpage.action?pageId=2687076 I'll
complete a first pass based on what I see in JIRA, github and
build.clojure.org and then I'll post a request for help!

Sean

---------- Forwarded message ----------
Take an example -- even in the relatively simple case of
clojure.contrib.combinatorics: note that it is NOT listed on
http://dev.clojure.org/display/doc/Clojure+Contrib, and if I follow
the github repos I'll end up in https://github.com/clojure/math.combinatorics
which tells me nothing about what I should add to project.clj (what is
the latest released version, for instance).

Mark Engelberg

unread,
Sep 9, 2011, 4:53:17 PM9/9/11
to cloju...@googlegroups.com
Back at the beginning of August, I made a first pass at organizing my three contrib projects into the form required by the new modular contrib, and uploaded them to github.  Aaron Bedra volunteered to look them over to make sure I organized them correctly, and would then "create builds and cut releases".  He has not had time to do that yet.

On a related note, my priority-map module was broken by changes in 1.3:
CompilerException java.lang.VerifyError: (class: clojure/data/priority_map/
PersistentPriorityMap, method: create signature: (Lclojure/lang/IPersistentMap;)Lcloj
ure/data/priority_map/PersistentPriorityMap;) Expecting to find unitialized obje
ct on stack, compiling:(clojure/data/priority_map.clj:164)

Aaron also volunteered to help me make sense out of this error, but has been busy.

If anyone else has time to help me out with either of those two issues (verifying that I organized the github modules correctly, and interpreting the 1.3 error with respect to priority-map), that would be great!

--Mark


--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.
To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.


Aaron Bedra

unread,
Sep 9, 2011, 5:55:47 PM9/9/11
to cloju...@googlegroups.com
Unfortunately with the upcoming 1.3 release, and the upcoming Programming Clojure 2nd Edition release, I have been tight on time.  As soon as one of those things happen I am more than happy to get back to looking at these.  If anyone else has time before that it would be even better.  I would love to get these packaged up and released.

Cheers,

Aaron
-- 
Cheers,

Aaron Bedra
--
Clojure/core
http://clojure.com

Sean Corfield

unread,
Sep 9, 2011, 9:00:23 PM9/9/11
to cloju...@googlegroups.com
On Fri, Sep 9, 2011 at 1:53 PM, Mark Engelberg <mark.en...@gmail.com> wrote:
> Back at the beginning of August, I made a first pass at organizing my three
> contrib projects into the form required by the new modular contrib, and
> uploaded them to github.

Since I'm pushing so hard on documenting / organizing contrib stuff,
I'll be happy to have a look. Can you confirm the following
migrations:
* clojure.contrib.combinatorics -> clojure.math.combinatorics
* clojure.contrib.math -> clojure.math.numeric-tower
* clojure.contrib.priority-map -> clojure.data.priority-map

It looks like you need the epl.html license file adding, as well as
pom.xml and it would be nice to have README in markdown since that's
what the other repos do, and add a CHANGES.txt file perhaps? Since I
have commit rights, I'm happy to add all of that if you're OK with me
committing to your project?

Then we should get the old test/ code moved over from contrib modules
and get it working with maven. Again, happy to help if you're OK with
me committing...

> On a related note, my priority-map module was broken by changes in 1.3:

No idea if I can help with that but, hey, I can try!
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/
Railo Technologies, Inc. -- http://www.getrailo.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

Mark Engelberg

unread,
Sep 10, 2011, 2:01:17 AM9/10/11
to cloju...@googlegroups.com
On Fri, Sep 9, 2011 at 6:00 PM, Sean Corfield <seanco...@gmail.com> wrote:
On Fri, Sep 9, 2011 at 1:53 PM, Mark Engelberg <mark.en...@gmail.com> wrote:
> Back at the beginning of August, I made a first pass at organizing my three
> contrib projects into the form required by the new modular contrib, and
> uploaded them to github.

Since I'm pushing so hard on documenting / organizing contrib stuff,
I'll be happy to have a look. Can you confirm the following
migrations:
* clojure.contrib.combinatorics -> clojure.math.combinatorics
* clojure.contrib.math -> clojure.math.numeric-tower
* clojure.contrib.priority-map -> clojure.data.priority-map


That is correct.
 
It looks like you need the epl.html license file adding, as well as
pom.xml and it would be nice to have README in markdown since that's
what the other repos do, and add a CHANGES.txt file perhaps? Since I
have commit rights, I'm happy to add all of that if you're OK with me
committing to your project?

That would be wonderful.  Yeah, I wasn't quite sure how to structure the license and the pom.xml.  I looked at some examples and they seemed to vary.  I think I found at least one precedent in contrib for omitting the license file and just saying in the README that it follows the same license as clojure.  Also, it wasn't clear to me whether the pom.xml was just some artifact that developers used locally, and just happened to be uploaded as part of the git directory, or whether it had some real significance.  I think that one of the three projects, I tried converting to markdown, but ended up losing out on some formatting that actually made it harder to read, so I abandoned markdown, since again, it didn't seem to be mandatory and was just causing me grief.  But if you see a way to make it nicer with markdown, I'm all for it.

So commit away, and if there are any implications I need to learn for long-term maintenance of the code.

Then we should get the old test/ code moved over from contrib modules
and get it working with maven. Again, happy to help if you're OK with
me committing...


priority-map's tests are in the same file.  The math module had tests in a separate file, and I wasn't sure how to organize them in the module to ensure they get run automatically whenever Clojure is tested for regressions.  Moving them over would be great.

I never wrote test cases for combinatorics, just tested in the REPL.  I intended to write a test file eventually, but held off recently because I was contemplating making some changes to the library once done with the migration, and figured I'd do it all at once.

Thanks in advance for all the help.  This will really help me understand how these modules are supposed to be organized.

--Mark

Sean Corfield

unread,
Sep 10, 2011, 2:42:17 PM9/10/11
to cloju...@googlegroups.com
On Fri, Sep 9, 2011 at 11:01 PM, Mark Engelberg
<mark.en...@gmail.com> wrote:
> That would be wonderful.  Yeah, I wasn't quite sure how to structure the
> license and the pom.xml.

License, pom.xml and README added / updated. All three projects are mvn-able...

> priority-map's tests are in the same file.  The math module had tests in a
> separate file, and I wasn't sure how to organize them in the module to
> ensure they get run automatically whenever Clojure is tested for
> regressions.  Moving them over would be great.

math.numeric-tower tests run with mvn test now. I'll work on
data.priority-map later today to get that into a mvn compatible
structure.

> I never wrote test cases for combinatorics, just tested in the REPL.  I
> intended to write a test file eventually, but held off recently because I
> was contemplating making some changes to the library once done with the
> migration, and figured I'd do it all at once.

OK. You should be able to follow what I've done for math.numeric-tower
but ping me if you have any questions.

Can we get builds for these libraries up on build.clojure.org now?
(Stu / Aaron / ?)

Also can we get JIRA projects created for them? (Stu / Aaron / ?)

And can the contrib git admin disable "issues" on github for these
three? (not sure who admins what in new contrib)

Sean Corfield

unread,
Sep 10, 2011, 2:46:30 PM9/10/11
to cloju...@googlegroups.com
On Sat, Sep 10, 2011 at 11:42 AM, Sean Corfield <seanco...@gmail.com> wrote:
> math.numeric-tower tests run with mvn test now. I'll work on
> data.priority-map later today to get that into a mvn compatible
> structure.

OK. data.priority-map's tests are separated out and run via mvn now.

Tchavdar Roussanov

unread,
Sep 11, 2011, 2:06:22 AM9/11/11
to cloju...@googlegroups.com
On a related note, my priority-map module was broken by changes in 1.3:
CompilerException java.lang.VerifyError: (class: clojure/data/priority_map/
PersistentPriorityMap, method: create signature: (Lclojure/lang/IPersistentMap;)Lcloj
ure/data/priority_map/PersistentPriorityMap;) Expecting to find unitialized obje
ct on stack, compiling:(clojure/data/priority_map.clj:164)


The verification error is triggered by double underscore characters in the third argument of deftype definition: __meta. Simply removing one of the underscores resolves the issue. I am not sure why compiler generates invalid byte codes. Someone with deeper knowledge of how clojure compiler emits byte codes may help here.   

There is a second issue: missing the third parameter (meta) to the calls to PersistentPriorityMap constructor in assoc, without, pop and pm-empty-by methods. 

With attached patch priority-map compiles and runs successfully all tests under clojure 1.3-beta3. The patch is build with the assumption that you want to preserve existing meta data when calling assoc, without and pop methods.

Tchavdar
priority_map.patch

Stuart Sierra

unread,
Sep 11, 2011, 11:13:08 AM9/11/11
to cloju...@googlegroups.com
I have created Hudson jobs on build.clojure.org for math.combinatorics, math.numeric-tower, and data.priority-map.

-Stuart Sierra
clojure.com

Tchavdar Roussanov

unread,
Sep 11, 2011, 1:25:20 PM9/11/11
to cloju...@googlegroups.com
The minimal test to trigger the verification error is :

(deftype t [__f])

It works fine in clojure 1.2. Should I open a ticket for clojure.core?

Tchavdar

Sean Corfield

unread,
Sep 11, 2011, 4:56:00 PM9/11/11
to cloju...@googlegroups.com
On Sat, Sep 10, 2011 at 11:06 PM, Tchavdar Roussanov <trou...@gmail.com> wrote:
> The verification error is triggered by double underscore characters in the
> third argument of deftype definition: __meta. Simply removing one of the
> underscores resolves the issue. I am not sure why compiler generates invalid
> byte codes. Someone with deeper knowledge of how clojure compiler emits byte
> codes may help here.

Interesting. Looks like that was the way to add optional fields at some point:

http://onclojure.com/2010/08/26/reusable-method-implementations-for-deftypes/

I assume that wasn't documented and was just an implementation artifact?

> There is a second issue: missing the third parameter (meta) to the calls to
> PersistentPriorityMap constructor in assoc, without, pop and pm-empty-by
> methods.

Yup, ran into that as I tried to fix __meta :) Thanx for the patch.
It's looks better than my attempt at fixing it so I merged that in on
top of mine. Hopefully we'll see a clean matrix build when Hudson next
fires...

Sean Corfield

unread,
Sep 11, 2011, 4:57:35 PM9/11/11
to cloju...@googlegroups.com
On Sun, Sep 11, 2011 at 10:25 AM, Tchavdar Roussanov <trou...@gmail.com> wrote:
> The minimal test to trigger the verification error is :
> (deftype t [__f])
> It works fine in clojure 1.2. Should I open a ticket for clojure.core?

I'd say yes. It's a clear change of behavior from 1.2 and that
OnClojure blog post indicates it was useful behavior so an official
word from Clojure/core would be valuable here.

Sean Corfield

unread,
Sep 11, 2011, 5:02:00 PM9/11/11
to cloju...@googlegroups.com
On Sun, Sep 11, 2011 at 8:13 AM, Stuart Sierra
<the.stua...@gmail.com> wrote:
> I have created Hudson jobs on build.clojure.org for math.combinatorics,
> math.numeric-tower, and data.priority-map.

Thank you!

data.priority-map had 1.3.0 incompatibilities that should now be fixed.
math.combinatorics has no tests yet so of course it "passes".
math.numeric-tower failed on the 1.2.0 / 1.2.1 tests but it looks like
a build issue?

1.2.0,Sun JDK 1.5 is still in the queue: Waiting for next available executor
Finished: FAILURE

(Can I get permission to run builds on these three so I don't have to
wait for Hudson to get "a round tuit"? :)

Sean Corfield

unread,
Sep 11, 2011, 8:15:04 PM9/11/11
to cloju...@googlegroups.com

I pulled the usage examples out of the comment and used those as a
basic set of seed tests. We can add more later.

Rich Hickey

unread,
Sep 11, 2011, 8:51:52 PM9/11/11
to cloju...@googlegroups.com
Any verification error is a bug, ticket welcome.

OTOH, relying on the undocumented behavior given double underscore is a user bug - users of such code should be prepared to be broken.

Rich

Tchavdar Roussanov

unread,
Sep 11, 2011, 9:03:57 PM9/11/11
to cloju...@googlegroups.com
Done: CLJ-837

Fogus

unread,
Sep 11, 2011, 9:53:07 PM9/11/11
to Clojure Dev
> I'd say yes. It's a clear change of behavior from 1.2 and that
> OnClojure blog post indicates it was useful behavior so an official
> word from Clojure/core would be valuable here.

The privileged names __meta and __extmap are implementation details of
types and records and should probably not be viewed as real. In other
words, since they are not published as part of the type/record docs,
then they may not exist in the future. Having said that, the current
Clojure implementation treats any name starting with __ as a special
field and doesn't count them toward the field count. As a result,
when emitting the MyType.create method (another implementation
detail), the fields marked as __ throw off the field count and as a
result the bytecode is not emitted properly. I know how to change the
code to allow names prefixed with __ (not called by those two special
names listed above), but I hesitate to say that is the solution
without further reflection.

Am I correct that this has been worked around by using different
names?

Sean Corfield

unread,
Sep 11, 2011, 10:57:31 PM9/11/11
to cloju...@googlegroups.com
On Sun, Sep 11, 2011 at 5:51 PM, Rich Hickey <richh...@gmail.com> wrote:
> OTOH, relying on the undocumented behavior given double underscore is a user bug - users of such code should be prepared to be broken.

Thanx. I had a feeling double underscore was "special" and should not
be relied on.

defrecord automatically supports metadata, as I understand I, but
deftype does not unless it implements meta and withMeta from
clojure.lang.IObj (based on my reading of the data.priority-map
source) - is that correct?

Is there a "preferred" way to add metadata support to a deftype that
doesn't force an extra field to be provided? Or is there simply no
other way for a deftype?

Fogus

unread,
Sep 12, 2011, 9:34:51 AM9/12/11
to Clojure Dev
Thanks!

I've taken the technical discussion to the ticket.
http://dev.clojure.org/jira/browse/CLJ-837

Stuart Sierra

unread,
Sep 12, 2011, 7:53:07 PM9/12/11
to cloju...@googlegroups.com

(Can I get permission to run builds on these three so I don't have to
wait for Hudson to get "a round tuit"? :)
--
Sean A Corfield -- (904) 302-SEAN


Done.

-Stuart Sierra
clojure.com

Sean Corfield

unread,
Sep 12, 2011, 8:04:39 PM9/12/11
to cloju...@googlegroups.com
On Mon, Sep 12, 2011 at 4:53 PM, Stuart Sierra
<the.stua...@gmail.com> wrote:
>> (Can I get permission to run builds on these three so I don't have to
>> wait for Hudson to get "a round tuit"? :)
> Done.

Thank you!

I see the failures in math.numeric-tower now (before it failed due the
JVM not being available in the build matrix) so I have something
concrete to work on...


--
Sean A Corfield -- (904) 302-SEAN

Mark Engelberg

unread,
Sep 13, 2011, 2:15:35 AM9/13/11
to cloju...@googlegroups.com
Wow, what an amazing community!  I go away for the weekend, and when I come back, all the hard work of migrating my contrib modules has already been done.  Thanks so much for the help.  I'm such a newb when it comes to git, hudson, maven, jira, etc. that I'm sure you've saved me from dozens upon dozens of hours of floundering.  I continue to welcome any further assistance that anyone wants to provide, but I will try to get up to speed on everything that has been done so I can take responsibility for any further maintenance.

--Mark

Sean Corfield

unread,
Oct 15, 2011, 10:09:51 PM10/15/11
to cloju...@googlegroups.com
On Mon, Sep 12, 2011 at 5:04 PM, Sean Corfield <seanco...@gmail.com> wrote:
> I see the failures in math.numeric-tower now (before it failed due the
> JVM not being available in the build matrix) so I have something
> concrete to work on...

I finally got around to making math.numeric-tower compatible with
Clojure 1.2.x so the matrix test passes now. I made a 0.0.1 release so
that should be up on Maven soon (I've updated the Where Did... page in
anticipation of that).

Reply all
Reply to author
Forward
0 new messages