Proposal for major breaking changes

[Brendan Tracey]

Hello all,

A few of us have been talking, and we would like to make several major breaking changes to Gonum. This should set us on solid ground going forward.

----
The proposal:
1) Change import paths to gonum.org. That is, instead of "github.com/gonum/stat/distuv" the path would be "gonum.org/stat/distuv".

2) Combine gonum/matrix/mat64 and gonum/matrix/cmat128 into just one package. Rename that package to mat. Combined with (1), it would be located at gonum.org/mat.

3) Change most of the gonum packages to be under a single monorepo.

Once these are done, the plan is to fix the remaining issues in gonum/blas, gonum/lapack, and gonum/mat, and make a stability promise for those packages.

-----
Rationale:
1) First, it removes a forced dependence on github, if we ever need to move for some reason. Secondly, it also explicitly directs to a website where we can put more information about the gonum project, including tutorials and other announcements.
2) Real and Complex matrices are entangled, and it's hard to decouple them as the current packages do. Eigenvalue decomposition is an example of a fundamental primitive that needs both kinds of matrices. Circular import paths make it difficult to otherwise interface them well. It was originally separate due to the API surface of a "gonum/mat", but in the end, saying `mat.CDense` is just as easy as `cmat.Dense`, and the other benefits are significant
3) Early on, it was nice to have different repos as we were figuring everything out. Recently though, we've had problems with version skew as breaking changes happen in one package that are unnoticed in a different package. Combining them into one repo makes integration testing simpler. One repo makes it easier to to release versions of gonum (though stability promises will be per-package for the time being). There is also a side-benefit of making the gonum suite easier to use as a `go get` command will grab all of the packages.

------
Discussion:
This clearly breaks everything, and so I'd propose 1-3 all happen at the same time. This would be a painful transition, but code can be fixed once and for all. Additionally, doing all at once allows for a transition where old code would still work (since the new repo would live at github.com/gonum/gonum). After a transition period, we'll delete the old individual repositories, forcing an update.

I don't have a specific timeline in mind, but would like it to be sooner rather than later as Gonum interest and usage has been growing, and we want to make these changes before there is too much code to make them. I have gonum.org registered and working, though there isn't a real website along with it yet.

Some gonum repositories will not be part of the monorepo, and instead will be on their own. There is not yet consensus on what that line is.

-----

Comments appreciated

[Brendan Tracey]

My thoughts on the monorepo:

I think it's clear that most of the gonum libraries should go into the monorepo. matrix, stat, optimize, graph, etc. All of these fit the core goal of gonum, and all are pure go and under our control.

I also think it's clear that the new hdf5 repo should not go in the monorepo. The package is largely a wrapper around a C package, which takes effort beyond `go get` to use and install.

More controversially, I would suggest that perhaps plot should have its own repository. Unlike the rest of the Gonum repositories, its goal is making attractive visuals, not just numeric algorithms. As such, it relies on non-standard library tools (zombiezen/gopdf/pdf, ajstarks/svgo, /x/image/math/f64), and needs to solve problems that aren't encountered in other gonum libraries, such as font choice. It seems like a more independent development project.

I would also suggest that the c-interfaces to BLAS and LAPACK not be in the monorepo, but instead be elsewhere. They aren't pure Go, and Lapack in particular sometimes makes breaking changes to their interface (and so we can't guarantee stability). Moving forward, I think there are many libraries we may want to interface with (fftw, Gurobi), and perhaps we could have a repository for all of those.

Under this suggestion, the repositories are:
github.com/gonum/wrapper (containing hdf5, cblas, clapack)
github.com/gonum/plot
github.com/gonum/gonum   (containing mat, stat, graph, and everything else)

[Dan Kortschak]

I agree with this break down with the exception that cgo wrapper packages that are not related should be in separate repos. The rationale for this is fairly base pragmatism; cgo wrapper packages can be slow to run tests, so only testing related/entangled packages should be the goal. In the case of hdf5 and lapacke, there does not seem to be any real coupling between them that would need to be tested.

--
afk
Dan

---

From: gonu...@googlegroups.com <gonu...@googlegroups.com> on behalf of Brendan Tracey <tracey....@gmail.com>
Sent: Wednesday, January 4, 2017 4:15:36 AM
To: gonum-dev
Subject: [gonum-dev] Re: Proposal for major breaking changes

 

--
You received this message because you are subscribed to the Google Groups "gonum-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gonum-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brendan Tracey]

On Tuesday, January 3, 2017 at 3:08:20 PM UTC-7, kortschak wrote:

I agree with this break down with the exception that cgo wrapper packages that are not related should be in separate repos. The rationale for this is fairly base pragmatism; cgo wrapper packages can be slow to run tests, so only testing related/entangled packages should be the goal. In the case of hdf5 and lapacke, there does not seem to be any real coupling between them that would need to be tested.

Whereas, for example, wrappers for Gurobi, cplex, and cvxopt would be in the same repository since they all relate to convex optimization?

[Dan Kortschak]

I don't know enough about them to make a call on that, I think it
should be decided on a case by case basis.

[Seb Binet]

Hi,

The rationale and splitting sound fine to me as well. 

I agree that cgo-based packages should be in their own segregated packages. 

But I see one possible issue with serving the mono repo like is currently proposed: the 'internal' package. 

Where should it be rooted? 

Also, if we go the mono repo route, with packages hosted at github.com/gonum/gonum/pkg but served by gonum.org/pkg, we loose (I think) the ability to do 'go get gonum.org/...' to install the whole kaboodle on a new machine.

I believe we should insert an /x/ or /pkg/ (or your favorite colored bikeshed).

There's also the issue on how to map the other packages ('hdf5, plot, clapack,...) under the gonum.org/... import path.

Anyways, just my 2c, and kudos to move forward with this :)

-s

sent from my droid

--
You received this message because you are subscribed to the Google Groups "gonum-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gonum-dev+unsubscribe@googlegroups.com.

[Dan Kortschak]

On Wed, 2017-01-04 at 13:20 +0100, Seb Binet wrote:
> But I see one possible issue with serving the mono repo like is
> currently proposed: the 'internal' package.
> Where should it be rooted?

gonum.org/pkg/internal

> Also, if we go the mono repo route, with packages hosted at
> github.com/gonum/gonum/pkg but served by gonum.org/pkg, we loose (I
> think) the ability to do 'go get gonum.org/...' to install the whole
> kaboodle on a new machine.

Why is this different from the current situation? At the moment you
can't do go get github.com/gonum/... on a fresh machine. If we really
wanted to support this we could have a gonum.org/{,pkg/}meta that has
dependencies on all the others and then allow go get
godoc.org/{,pkg/}meta. I think this is a bad idea - one of the nice
things about go packages is their lazy resolution with go get.

> I believe we should insert an /x/ or /pkg/ (or your favorite colored
> bikeshed).

It was my understanding that the pkg category path was going to be
used.

> There's also the issue on how to map the other packages ('hdf5, plot,
> clapack,...) under the gonum.org/... import path.

gonum.org/pkg/plot
gonum.org/pkg/hdf5
gonum.org/pkg/lapacke

The package indirection meta data handles all of this.

[Seb Binet]

Dan,

On Jan 4, 2017 11:17 PM, "Dan Kortschak" <dan.ko...@adelaide.edu.au> wrote:

On Wed, 2017-01-04 at 13:20 +0100, Seb Binet wrote:
> But I see one possible issue with serving the mono repo like is
> currently proposed: the 'internal' package.
> Where should it be rooted?

gonum.org/pkg/internal

Ack.

> Also, if we go the mono repo route, with packages hosted at
> github.com/gonum/gonum/pkg but served by gonum.org/pkg, we loose (I
> think) the ability to do 'go get gonum.org/...' to install the whole
> kaboodle on a new machine.

Why is this different from the current situation? At the moment you
can't do go get github.com/gonum/... on a fresh machine. If we really
wanted to support this we could have a gonum.org/{,pkg/}meta that has
dependencies on all the others and then allow go get
godoc.org/{,pkg/}meta. I think this is a bad idea - one of the nice
things about go packages is their lazy resolution with go get.

OK. In Brendan's email, the package 'stat' was gonum.org/stat so it wasn't obvious to me whether one could use the ellipsis to install it in one go, together with its clients (which is one big advantage of going to the mono repo option, IMHO)

I am still not sure I understand correctly whether, e.g., 'stat' will be a directory as part of the new mono repo 'github.com/gonum/gonum' and served under 'gonum.org/pkg/stat' (which I am not sure is technically possible w/o resorting to shallow git clone + git tree trickeries) or served under 'gonum.org/pkg/gonum/stat'.

(Apologies for being a bit dense.)

> I believe we should insert an /x/ or /pkg/ (or your favorite colored
> bikeshed).

It was my understanding that the pkg category path was going to be
used.

> There's also the issue on how to map the other packages ('hdf5, plot,
> clapack,...) under the gonum.org/... import path.

gonum.org/pkg/plot
gonum.org/pkg/hdf5
gonum.org/pkg/lapacke

The package indirection meta data handles all of this.

Indeed. 

> Anyways, just my 2c, and kudos to move forward with this :)

-s

[Brendan Tracey]

Why is the internal location bad? If the general gonum packages stem from $ROOT, then internal will have to be at $ROOT/internal (or it can’t be used appropriately with go get). $ROOT could be gonum.org (as originally stated), but gonum.org/gonum, gonum.org/pkg, or gonum.org/pkg/gonum could all be reasonable choices.

[Seb Binet]

I wasn't saying it was bad, but merely raising the possible issue of its interaction with packages not part of the mono repo which could break if not 'at the correct place' wrt the new place (in the package hierarchy) of the 'new' internal package. 

-s

sent from my droid

[Dan Kortschak]

I think Seb's email acknowledged that it works (I still read "Ack" as a
synonym for "urk"/"ugh" on first reading with p>0.99 though if that is
the issue).

[Dan Kortschak]

They will be in the correct place when the process has finished.

[Seb Binet]

Sorry, yes. 

In my broken 'angrish', ack stands for acknowledged :)

-s

sent from my droid

[Brendan Tracey]

Do you have a specific concern in mind? That’s already true for other internal packages, github.com/gonum/graph/internal isn’t accessible to the stat packages. 

It seems like Dan had in mind that there is an internal package accessible by everything, which we may need, but we may not. The functionality in internal/asm is provided in other forms (floats).

Just saw the other emails. I, like Dan, read “ack” as “that seems awful”

[Dan Kortschak]

The reward for all of this (other than just personal satisfaction)
could be a few submissions to JOSS http://joss.theoj.org/about

We've talked previously about getting DOIs for the code, but this is
actually a peer reviewed journal. So for those of us where publication
is part of our work performance indicator measures, this will be
useful.

[Seb Binet]

Hi,

One other issue I've just remembered. 

Do you know how we should tackle issues filed against the multi repos?

Do we gather them inside the new mono repo? (There's no real tool to do so, afaik)

Or just leave them where they are? 

-s

sent from my droid

[Dan Kortschak]

This is possibly my biggest concern. I think porting the issues to the new repo is the way to go, but the prices requires thought and discussion - and mechanisation.

Things that need to be considered in my view are:

1. Cross matching between issues.

2. Temporal consistency of issues across the entire suite, not just within each donor repo. This means the whole thing need to be done as one operation.

3. Porting of PRs. I don't even know if this is possible, particularly since the SHAs will be wrong.

The issue of commit history is trivial in comparison.
--
afk
Dan

---

From: Seb Binet <seb....@gmail.com>
Sent: Friday, January 6, 2017 7:34:17 PM
To: Dan Kortschak
Cc: gonu...@googlegroups.com; Brendan Tracey
Subject: Re: [gonum-dev] Re: Proposal for major breaking changes

 

[Seb Binet]

On Fri, Jan 6, 2017 at 10:33 AM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

This is possibly my biggest concern. I think porting the issues to the new repo is the way to go, but the prices requires thought and discussion - and mechanisation.

Things that need to be considered in my view are:

1. Cross matching between issues.

2. Temporal consistency of issues across the entire suite, not just within each donor repo. This means the whole thing need to be done as one operation.

3. Porting of PRs. I don't even know if this is possible, particularly since the SHAs will be wrong.

The issue of commit history is trivial in comparison.

FYI, I have used https://github-issue-mover.appspot.com/ for migrating issues in the past.

it worked ok (even if I didn't like that it closed the original issue. I would have prefered it had added some predefined tag instead...)

-s

PS: code for the above app: https://github.com/google/github-issue-mover

[Dan Kortschak]

I would prefer that take something from that and quite or own that we can experiment with.

--
afk
Dan

---

From: gonu...@googlegroups.com <gonu...@googlegroups.com> on behalf of Seb Binet <seb....@gmail.com>
Sent: Saturday, January 7, 2017 7:10:33 AM

To unsubscribe from this group and stop receiving emails from it, send an email to gonum-dev+...@googlegroups.com.

[Dan Kortschak]

One thing that has not been discussed here is the possibility of
including versioning in the import paths. I am not talking about minor
version changes since we (I think) are agreed that minor changes that
break compatibility are not going to be allowed. However, there may be
major changes elsewhere that force breaking changes on us.

Having import paths like "gonum.org/pkg/v1/stat" from the outset would
allow us to gracefully include versioning. (I don't like the gopkg.in
approach of including the version last.)

Thoughts?

On Tue, 2017-01-03 at 09:29 -0800, Brendan Tracey wrote:

[Seb Binet]

On Thu, Jan 12, 2017 at 11:13 PM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

One thing that has not been discussed here is the possibility of
including versioning in the import paths. I am not talking about minor
version changes since we (I think) are agreed that minor changes that
break compatibility are not going to be allowed. However, there may be
major changes elsewhere that force breaking changes on us.

Having import paths like "gonum.org/pkg/v1/stat" from the outset would
allow us to gracefully include versioning. (I don't like the gopkg.in
approach of including the version last.)

I don't like gopkg.in approach either and prefer yours.

I was about to object that migrating from v1 to v2 would be problematic but actually the "go-source" and "go-import" meta fields can correctly handle tags.

usually one writes:

```

<meta name="go-import"

  content="gonum.org/pkg/{{.Repo}} git \

    https://github.com/gonum/{{.Repo}}">

<meta name="go-source" 

  content="gonum.org/pkg/{{.Repo}} \

    https://github.com/gonum https://github.com/gonum/{{.Repo}}/tree/master{/dir} \

    https://github.com/gonum/{{.Repo}}/blob/master{/dir}/{file}#L{line}">

```

but we could replace the "master" with {{.Version}} if/when switching from v1 to v2.

Thoughts?

I'd be enclined to use this scheme.

depending on how we'd organize the repos back on github (especially the "mono" repo vs the cgo/wrapper ones), we could probably even drop the "/pkg/v1/" part in favor of just "/v1/".

Anybody's seeing a possible issue with "go package management" work[1] ?

(not saying there is, just wondering)

That said, should we move forward and start planning:

- layout of the mono-repo (what should be put in there, and where)

- layout of the cgo-wrappers and plot repos (ditto)

- mapping of git repos to gonum.org/xyz import paths

so we can assess whether the mappings would work in a typical $GOPATH workspace ?

-s

[1]: https://blog.gopheracademy.com/advent-2016/saga-go-dependency-management/

[Sebastien Binet]

hi,

just to get the ball rolling a bit, I've created this simple migration script:

 https://gist.github.com/sbinet/cfa02fa3700e5eba7b42ad4613212b6b

to reduce stuttering, I have put the mono-repo under "gonum.org/v1".

so importing distuv would look like: 

 "gonum.org/v1/stat/distuv",

instead of:

 "gonum.org/pkg/v1/gonum/stat/distuv"

the cgo-based packages could be imported from, say, "gonum.org/x/v1/cgo/lapack"

and "plot", from "gonum.org/x/v1/plot"

I've noticed (or remembered) that "matrix/mat64" has a cgo-based test ("matrix/mat64/cblas_test.go")

not sure if this should be "// +build cgo"-tag guarded or migrated to the cgo-wrappers repo.

I've also put into the script (disabled by default) the ability to rewrite history and put each imported repo (say, "stat") under the "stat" directory (inside the new mono-repo) for every past commit of the old repo, not just for the last merged commit.

not sure if this is really better (as we loose cross-referenced SHA-1).

play with it and let me know.

-s

[Brendan Tracey]

I'm not opposed to the versioning in principle, but I don't know how we'll execute this in practice.

How does the versioning work with our own version skew? Is it going to be golang.org/v1/matrix but golang.org/v0/stat? Or are we going to start by saying everything is v1 at the start (even though it isn't), but then tock everything together after that? 

When we jump to v2, are we going to leave all of the old code? I assume we aren't going to maintain it, but are these "v1" elements going to live in the monorepo (github.com/gonum/gonum/v1/matrix, then becomes github.com/gonum/gonum/v2/matrix).

We don't know how the versioning stuff is going to shake out in Go, so I don't yet understand how we're supposed to play nicely with it. I don't think the version tags are so confusing, but they are more confusing than "gonum.org/pkg/matrix". This is especially true as it relates to future documentation and links. A hypothetical is to keep the most recent major version at "gonum.org/pkg/x", but to then also explicit vendor old versions for those who need it.

[Sebastien Binet]

The way I see it, 'gonum.org/v1/stat' is the directory 'stat' under the mono repo 'github.com/gonum/gonum' which would be (at first) the 'master' branch, then a branch 'v1'.

When 'v2' comes along, it becomes the new master and then a branch 'v2' (where bug fixes land) when 'v3' developments open (and happen in master.)

You don't actually have 'vX' directories inside the mono repo.

Just branches (or tags) which get transferred into directories as part of the import path.

We could start with v0 (or something else) to make it explicit there isn't (yet) a completely set in stone backward compatible API.

-s

sent from my droid

--

[Kunde21]

Have we looked into how this would affect the gonum/internal repo?  If gonum.org/v2/matrix can't import gonum.org/v1/internal, we should address that up front.

To unsubscribe from this group and stop receiving emails from it, send an email to gonum-dev+...@googlegroups.com.

[Dan Kortschak]

I asked Dominik Honnef's opinion on this in light of his recent post to
golang-nuts where he announced a merge of repos. I'm inclined to agree
that keeping the old repos as archives is worthwhile. His position on
the cgo wrappers is also worth considering.

Dan

-------- Forwarded Message --------
From: Dominik Honnef <dom...@honnef.co>
To: Dan Kortschak <dan.ko...@adelaide.edu.au>
Subject: Re: thoughts on approaches to aggregating go package
repositories
Date: Thu, 26 Jan 2017 19:33:24 +0100

> Hi Dan,
> 
> I haven't dealt with migrating issues myself (they weren't that
> important in my case and I'm keeping the old repos around for
> historical reasons) but I'm aware of some strategies.
> 
> They largely depend on how you're going to merge the repositories,
> though. One option is using git-subtree (a git contrib script) to
> merge the old repositories into folders. This will maintain the
> existing history/git SHAs, so none of the existing references to
> commits need to be updated. Unfortunately, these subtree merges
> prevent 'git log -- some/file' from working, as they won't be able to
> follow renames past the merge commit. It will also not allow you to
> rewrite commit messages, which you probably want to do in order to
> update GitHub issue references.
> 
> The other solution, rewriting the histories (as per
> https://honnef.co/posts/2016/04/merging-git-repositories/) could be
> used for a full migration, but it requires multiple passes. You'd
> first need to migrate all the issues, with a tool of your choice,
> then
> rewrite the histories and update all references to issues as well as
> commits, then rewrite all the issues and replace old SHAs with new
> SHAs. Both issue references and SHAs should be easy to string match
> automatically without false positives. I'm not aware of any tools
> doing all these steps for you, but I haven't looked hard, either.
> 
> Neither option will allow you to migrate PRs. If you're concerned
> about losing reviews and discussions, consider keeping the old
> repositories somewhere, as an archive, or to scrape them and check
> them into the new repository, as a form of documentation. Personally
> I'd keep the old repositories around at any rate, for external users
> who have their own references that can't be updated. That way,
> there'll always be a place to look up old commits, issues and PRs.
> 
> 
> By the way, I've read through the thread, and I'm not sure I'm really
> following the rationale for putting C wrappers in separate
> repositories. That seems to go against the idea of having a monorepo,
> for little benefit. A monorepo is, by nature, a collection of related
> projects. They needn't all be treated as one single project. If C
> wrappers need different testing strategies, then adjust CI
> accordingly. Similarly, users will already not want to install every
> package in the monorepo unconditionally, so having _more_ packages
> they may not need doesn't sound like an issue to me.
> 
> Feel free to forward my message to that thread if you think it
> contained any useful/new information.
> 
> Best regards,
> Dominik
> 
> On Thu, Jan 26, 2017 at 8:16 AM, Dan Kortschak
> <dan.ko...@adelaide.edu.au> wrote:
> > 
> > Hi Dominik,
> > 
> > I saw your post notifying of the change in import paths for your
> > code
> > analysis tools and was hoping to get some input in the process of
> > combining repos.
> > 
> > We are planning on moving nearly all the gonum repos into a single
> > repo
> > to aid development (mainly avoiding missing version skew in
> > dependencies). The main issue I am worried about is maintaining
> > links
> > between issues, pull requests and commits - all of which retain
> > important corporate knowledge for the development of the project.
> > 
> > Do you have any suggestions? The discussion we've been having is at
> > htt
> > ps://groups.google.com/forum/#!topic/gonum-dev/4oLmSvtDuZY
> > 
> > thanks
> > Dan

[Sebastien Binet]

hi,

On Thu, Jan 26, 2017 at 9:38 PM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

I asked Dominik Honnef's opinion on this in light of his recent post to
golang-nuts where he announced a merge of repos. I'm inclined to agree
that keeping the old repos as archives is worthwhile. His position on
the cgo wrappers is also worth considering.

FYI, I have done my own mono-repo migration for go-hep:

 https://go-hep.org

with the new mono-repo:

 https://github.com/go-hep/hep

I only had one cgo-based repo (plus a self-contained one that I simply dropped for the moment) so the migration was somewhat easier.

so far, I am happy with it.

(I am still working on the migration of the issues, though)

-s

[Brendan Tracey]

On Wednesday, January 25, 2017 at 3:42:30 PM UTC-7, Kunde21 wrote:

Have we looked into how this would affect the gonum/internal repo?  If gonum.org/v2/matrix can't import gonum.org/v1/internal, we should address that up front.

It is my understanding that the plan is for version skew to go away. At the beginning, some packages will be v0, some will be v1, but all under gonum.org/v1. If we have a v2, all will jump simultaneously.

Keeping around the old repos SGTM. Still not convinced about the cgo in the monorepo. One argument is that we can't guarantee API stability for the C-code, while we can for the Go versions. Another argument is that cgo-wrappers are also different in style. The monorepo can be installed and used with a single call to "go get". The cgo packages require more expertise, understanding how to install C code and using CGO_LDFLAGS among them. They also serve a different purpose. Gonum code seeks to be consistent and transparent, looking at a fresh implementation of these algorithms that is approachable. The cgo wrappers serve the purpose to increase the capability of Go as a language for scientific computing, either because of speed or because there's no spec to implement. Additionally, it gives us more flexibility to "decommission" the cgo wrappers if necessary, for example if hdf5 wanted to create and maintain their own Go wrappers.

On Wednesday, February 8, 2017 at 10:36:11 AM UTC-7, Sebastien Binet wrote:

hi,

Anything that you saw that would prevent us from using a similar procedure? 

[Sebastien Binet]

On Feb 10, 2017 5:21 PM, "Brendan Tracey" <tracey....@gmail.com> wrote:

On Wednesday, January 25, 2017 at 3:42:30 PM UTC-7, Kunde21 wrote:

Have we looked into how this would affect the gonum/internal repo?  If gonum.org/v2/matrix can't import gonum.org/v1/internal, we should address that up front.

It is my understanding that the plan is for version skew to go away. At the beginning, some packages will be v0, some will be v1, but all under gonum.org/v1. If we have a v2, all will jump simultaneously.

Keeping around the old repos SGTM. Still not convinced about the cgo in the monorepo. One argument is that we can't guarantee API stability for the C-code, while we can for the Go versions. Another argument is that cgo-wrappers are also different in style. The monorepo can be installed and used with a single call to "go get". The cgo packages require more expertise, understanding how to install C code and using CGO_LDFLAGS among them. They also serve a different purpose. Gonum code seeks to be consistent and transparent, looking at a fresh implementation of these algorithms that is approachable. The cgo wrappers serve the purpose to increase the capability of Go as a language for scientific computing, either because of speed or because there's no spec to implement.

Wrt the cgo issue: cgo packages tend to make cross compiling "somewhat" more painful... (And make packages or whole projects AppEngine-unfriendly)

Additionally, it gives us more flexibility to "decommission" the cgo wrappers if necessary, for example if hdf5 wanted to create and maintain their own Go wrappers.

On Wednesday, February 8, 2017 at 10:36:11 AM UTC-7, Sebastien Binet wrote:

hi,

On Thu, Jan 26, 2017 at 9:38 PM, Dan Kortschak <dan.kortschak@adelaide.edu.au> wrote:

I asked Dominik Honnef's opinion on this in light of his recent post to
golang-nuts where he announced a merge of repos. I'm inclined to agree
that keeping the old repos as archives is worthwhile. His position on
the cgo wrappers is also worth considering.

FYI, I have done my own mono-repo migration for go-hep:

 https://go-hep.org

with the new mono-repo:

 https://github.com/go-hep/hep

I only had one cgo-based repo (plus a self-contained one that I simply dropped for the moment) so the migration was somewhat easier.

so far, I am happy with it.

(I am still working on the migration of the issues, though)

-s

Anything that you saw that would prevent us from using a similar procedure? 

Nope.

My migration procedure did miss links pointing to the old repos inside, e.g. the readme-s (for embedded images)

But apart from that and the GitHub issues I still need to tackle, I didn't see any show stoppers.

-s

--
You received this message because you are subscribed to the Google Groups "gonum-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gonum-dev+unsubscribe@googlegroups.com.

[Brendan Tracey]

My proposal would be (based on the discussion):

- Keep around old repositories, but make sure there's a note in the package documentation that we have updated ones that are preferred

- Main repository is gonum.org/v1/gonum

- Other repositories are gonum.org/v1/plot, gonum.org/v1/netlib (which contains BLAS and LAPACK), gonum.org/v1/hdf5 

I uploaded a simple splashpage that's at least enough to migrate the code. When we do the migration, the actual URLs can link to the godoc (like rsc.io/benchstat). If someone wants to make a fancier webpage, we can switch over later, but I wanted to make sure that's not the hold-up with the code.

[Brendan Tracey]

I went through and modified all of the issues to have the appropriate prefixes.

Are we settled enough to try and move forward on this?

[Sebastien Binet]

On Thu, Mar 16, 2017 at 6:12 AM, Brendan Tracey <tracey....@gmail.com> wrote:

I went through and modified all of the issues to have the appropriate prefixes.

Are we settled enough to try and move forward on this?

one little issue I noticed afterwards with the issue migration tool I've used is that github issues cross-references (say, in repo gonum/stat issue-42 is referencing #22 somewhere in the text) are not properly translated (it still says #22 instead of gonum/gonum#22)

I didn't have that many issues cross-referencing other issues (so I corrected them by hand) but for gonum repos, that might be an issue (ha!).

-s

[Dan Kortschak]

I think we can avoid this by just keeping the old repos as deprecated
and then nulling out their code after a time, but leaving the issues.

[Kunde21]

Do we have an official drop-dead date on this migration?  I've gotten the asm2plan9s tool up and running as an on-save-hook, so my write/compile/test isn't slowed down by missing instructions anymore.  I'm putting together a pull request for asm dot products as a testing base and I don't want to screw up any plans.

[Brendan Tracey]

I think I have the correct procedure up and we're waiting for the last few PRs to be completed. We'll then have a couple more PRs on the main repo cleaning things up, but it should be ready to go soon.

[Kunde21]

This came up in slack, and I think the merge would be a perfect time to make such a change:

Move gonum/floats functionality into floats/f64 and create floats/f32 as its counterpart.  

Another option to go along with that would be to rename the base package and have it mirror the internal/asm structure (f64, f32, c64, c128) to export everything with input validation.

[Dan Kortschak]

I think I'd be OK with floats/f64 and floats/f32.

[Brendan Tracey]

I don't think floats/f64 and floats/f32 is a good idea. For one, it's going to overlap with internal/asm/f64. Integrating well with goimports is such a bonus to coding, we should avoid overlaps within gonum packages.  I think it's better to make float64 the "default" state, so for instance, I would suggest /floats and /floats/floats32. Normally users will typically want float64, and so it makes sense for those to have clean names.

More generally though, it seems like we should think about how to deal with float32. It seems there is interest in at least float32 blas, lapack, and the ASM routines. If we are also translating floats, it seems inevitable that there will be requests for matrix next. If this isn't just a slippery slope argument, I would suggest maybe gonum.org/v1/pkg and gonum.org/v1/pkg/pkg32, or possibly gonum.org/v1/pkg32. Blas/Lapack would be exceptions given their design.

[Kunde21]

Another option would be to borrow the naming convention from chewxy and go with something like vector/{ vecf64 vecf32 vecc64 vecc128 }

My thought/goal was to export the internal/asm functions with input validation, so there wouldn't be a reason to import both floats and internal in the same package.  With that said, I agree that naming both f64 would make development more difficult since users would be fighting with goimports.  

[Brendan Tracey]

On Wednesday, May 3, 2017 at 12:58:29 PM UTC-6, Kunde21 wrote:

Another option would be to borrow the naming convention from chewxy and go with something like vector/{ vecf64 vecf32 vecc64 vecc128 }

I understand the argument, but I dislike the idea that I'll have to type xxxf64 every time I want to use a gonum function. I feel the common case should have nice names. It's one thing for functions that should be using sparingly and carefully like vecf64 and internal/asm/f64. It's another thing to have floatsf64.Max()

 

My thought/goal was to export the internal/asm functions with input validation, so there wouldn't be a reason to import both floats and internal in the same package.

I'm confused. A part of floats already is this (Add, AddTo). You mean "export the ones that aren't already"? I see that there are some implementations in asm/f64 that haven't been incorporated into floats (L1Norm), but that's an easy fix.

[Kunde21]

On Wednesday, May 3, 2017 at 12:15:48 PM UTC-7, Brendan Tracey wrote:

On Wednesday, May 3, 2017 at 12:58:29 PM UTC-6, Kunde21 wrote:

Another option would be to borrow the naming convention from chewxy and go with something like vector/{ vecf64 vecf32 vecc64 vecc128 }

I understand the argument, but I dislike the idea that I'll have to type xxxf64 every time I want to use a gonum function. I feel the common case should have nice names. It's one thing for functions that should be using sparingly and carefully like vecf64 and internal/asm/f64. It's another thing to have floatsf64.Max()

 

vecf64.Max() and  floats.Max() are the same char-count (I don't think floatsf64 was an option, as it's a bit redundant), so I'm not sure what you're looking for as to better-to-type name.  Maybe vector/{ vec vec32 cvec cvec128 } would be better?  

 

My thought/goal was to export the internal/asm functions with input validation, so there wouldn't be a reason to import both floats and internal in the same package.

I'm confused. A part of floats already is this (Add, AddTo). You mean "export the ones that aren't already"? I see that there are some implementations in asm/f64 that haven't been incorporated into floats (L1Norm), but that's an easy fix.

Yes, I mean export all the asm functions (of all types) that aren't already exposed via floats.  There have been a few requests for the float32 functions, and I'm sure the complex will gain interest, too.  

[Sebastien Binet]

On Wed, May 3, 2017 at 9:37 PM, Kunde21 <kun...@gmail.com> wrote:

On Wednesday, May 3, 2017 at 12:15:48 PM UTC-7, Brendan Tracey wrote:

On Wednesday, May 3, 2017 at 12:58:29 PM UTC-6, Kunde21 wrote:

Another option would be to borrow the naming convention from chewxy and go with something like vector/{ vecf64 vecf32 vecc64 vecc128 }

I understand the argument, but I dislike the idea that I'll have to type xxxf64 every time I want to use a gonum function. I feel the common case should have nice names. It's one thing for functions that should be using sparingly and carefully like vecf64 and internal/asm/f64. It's another thing to have floatsf64.Max()

 

vecf64.Max() and  floats.Max() are the same char-count (I don't think floatsf64 was an option, as it's a bit redundant), so I'm not sure what you're looking for as to better-to-type name.  Maybe vector/{ vec vec32 cvec cvec128 } would be better?  

allow me to put my grain of bikeshedding:

 internal/asm/f64 -> internal/asm/asmf64

 floats -> floats/f64, floats/f32

asmf64 is a low-level, not-to-be-used-by-everybody package.

it's fine if its name is a bit long in the tooth.

-s

[Brendan Tracey]

On Wednesday, May 3, 2017 at 1:37:17 PM UTC-6, Kunde21 wrote:

On Wednesday, May 3, 2017 at 12:15:48 PM UTC-7, Brendan Tracey wrote:

On Wednesday, May 3, 2017 at 12:58:29 PM UTC-6, Kunde21 wrote:

Another option would be to borrow the naming convention from chewxy and go with something like vector/{ vecf64 vecf32 vecc64 vecc128 }

I understand the argument, but I dislike the idea that I'll have to type xxxf64 every time I want to use a gonum function. I feel the common case should have nice names. It's one thing for functions that should be using sparingly and carefully like vecf64 and internal/asm/f64. It's another thing to have floatsf64.Max()

 

vecf64.Max() and  floats.Max() are the same char-count (I don't think floatsf64 was an option, as it's a bit redundant), so I'm not sure what you're looking for as to better-to-type name.  Maybe vector/{ vec vec32 cvec cvec128 } would be better?  

Sorry. I wasn't arguing about vec vs. floats, I was arguing about supporting alternate numeric types in general in gonum. Right now, we don't have addendums to package names for their types (well, we do have mat64, but that's going to become matrix). We have stat, distuv, etc.  I feel `floats` fits into this category of "functions we expect normal users to use". It should thus have the same conventions as the rest of the gonum suite (whereas internal/asm is not something we even allow normal users to use, so it can have differing conventions). 

We then have to ask about the ramifications of decisions on extensions to gonum. If we rename floats to vecf64, etc., then why does "stat" just have the name "stat"? We either have to rename "stat" to "statf64", or somehow decide right now that we will never support a float32 version of stat (or matrix, etc.). 

I'm arguing that the float64 versions should have the privileged name (floats, stat, matrix), and the float32 versions would follow a convention of name and location. One example would be floats/floats32, stat/stat32, matrix/matrix32, another would be just at the base level. I think the discussion if floats should be renamed "vec" is orthogonal -- if we do so it should be vec and vec32, not vecf64 and vecf32.

I don't think we can say anything in general about how complex numbers should work into this. Some places like matrix it makes sense to have them in the same package, places like floats it probably makes sense to have the equivalent cplxes (or something) a la the math package.

[Kunde21]

Ok, that makes sense.  

To throw my opinion into the ring, I'd say we codify the conventions according to the two package groups:  high-level types vs low-level functions.  Packages built around types would have a default/main type, while functions packages designate their associated type.

We're already doing this, in a way, with low-level functions packages of blas and lapack with blas64/blas32/cblas64/cblas128 and lapack64 packages.  Vector/{type} and internal/asm{type} would follow that naming pattern, since their use with raw slices is expected.

Types packages like matrix/stat/plot would have a default (float64) underlying primitive type, with name-designation -- [pkg]32 c[pkg]64 c[pkg] -- on the other types.  The average user is going to be using float64, so these higher-level packages would reflect that with a default name. 

On the complex numbers piece, is there a reason the complex matrix in matrix/conv is implemented using two mat64 matrices?  That design doesn't lend itself to the use of the low-level packages like cblas128 or the future clapack128.  

[Dan Kortschak]

On the basis of Brendan's comments here, I rescind my previous comment.
I agree that keeping package names as simple single words is a thing we
should aim for.

I also agree that the dividing line between high and low level packages
does deserve some gradation (also because it aids code generation).

The conv handling of matrices is for special cases. Routines in matrix
that need a cblas128.General would perform a clone first if needed.

[Kunde21]

The conv handling of matrices is for special cases. Routines in matrix that need a cblas128.General would perform a clone first if needed. 

Ok, that makes more sense.  

On that note, would the native c128 matrix be defined in the gonum/matrix package or a sub-pkg (gonum/matrix/cmat)?

[Dan Kortschak]

That's sort of up in the air-ish (though not completely undecided - it
is likely complex matrices will be in the same package as real
matrices).

You should look at https://github.com/gonum/matrix/issues/308

[Kunde21]

Okay, that makes more sense with respect to gonum/matrix/conv.  Making float64 and complex128 the default primitives fits with the rest of gonum.  

I've been digging through closed issues and kind of feel like an idiot.

[Brendan Tracey]

Alea iacta est!

We have merged into the monorepo, github.com/gonum/gonum.

You can now `go get gonum.org/v1/gonum/[pkg]`, and `go get gonum.org/v1/plot`.

If there are no objections, I am going to create a `netlib` repository so we can move cblas and clapack there.

[Sebastien Binet]

Looks good to me! 

sent from my droid

[Brendan Tracey]

The repository is made, and I started doing the port, but there are quite a few changes necessary actually.

I'm wondering if a better plan is to literally copy the repositories from gonum (and delete them there), and from there make a PR that makes the necessary changes to make the repositories work. That way the changes will be much more clear. 

[Dan Kortschak]

It seems a little odd to move the repos into gonum and then out again.
I would just copy (not github fork) the lapack repo to netlib and then
gut the parts that are not needed there.

[Brendan Tracey]

Sorry, I didn't communicate properly

My plan was for the first commit to copy github.com/gonum/gonum/blas/cgo -> github.com/gonum/netlib/cblas  and github.com/gonum/gonum/lapack/cgo -> github.com/gonum/netlib/clapack

Then the 1st PR would be to get a travis build to pass, by changing import paths, etc.

[Dan Kortschak]

Yeah, I understood that. It just seems like an odd way to proceed.
Copying the cgo directories loses the git history, so an entire repo
should be copied, but you only need to copy the relevant part, which is
the lapack and blas repos (essentially what was done for gonum, but
just those two into netlib).

[Brendan Tracey]

Copy the whole bias and lapack repos, and then the first PR deletes most of that code (leaving just the cgo parts). Agreed?

[Dan Kortschak]

On Tue, 2017-05-23 at 21:10 -0600, Brendan Tracey wrote:
> Copy the whole bias and lapack repos, and then the first PR deletes
> most of that code (leaving just the cgo parts). Agreed?

Yes. So we end up with

gonum.org/v1/netlib/blas
gonum.org/v1/netlib/lapacke

This means that we can then simplify our imports of the Go implementations to

gonum.org/v1/gonum/blas
gonum.org/v1/gonum/lapack

if we want.

[Brendan Tracey]

I would suggest cblas instead of bias to avoid the name clash. Agreed on the bias condensation.

[Dan Kortschak]

SGTM

[Dan Kortschak]

I've had more thought on this and I think that it is unlikely that
people will import both the cgo and native packages, and that lapacke
already exists as the actual interface to C, so probably they should be
just blas and lapack. This makes swapping out code easier too.

On Tue, 2017-05-23 at 21:16 -0600, Brendan Tracey wrote:

[Brendan Tracey]

I agree with that.

There are three reasons to implement blas/lapack. The first is for code we don't support (complex numbers), the second is to call functions not in the blas64 packages, and the third is to use the possibly faster alternate implementation. The first and third don't clash, because both of those require importing the alternative. The second and the third can, but it means you're calling obscure (dlaqr1) and really want it to be in Go, but then want to use other functions in C. That seems pretty unlikely to me. 

[Brendan Tracey]

Okay. The cgo packages are refactored out.

First step is to get travis to pass.

Next step after that is condensing matrix?

I would suggest we rename it to `mat`, so we get `mat.NewDense`, `mat.NewCDense`, etc. It's good to keep the package names short, especially for one so fundamental.

[Dan Kortschak]

On Thu, 2017-05-25 at 20:07 -0700, Brendan Tracey wrote:
> First step is to get travis to pass.

Yep.

> Next step after that is condensing matrix?

I think blas/native => blas and lapack/native => lapack first.

[Brendan Tracey]

SGTM

[Brendan Tracey]

The travis failure is because of  gonum.org/v1/gonum/lapack/internal/testdata/netlib

I guess it’s coveralls trying to compile and run the package, but not having a fortran compiler. I’m not sure what the right solution is. The hack way is just to rename to netlib._go or something to fool it, since it shouldn’t be needed on a normal basis anyway.

[Kunde21]

In it's current state, the monorepo still allows calling `go get github.com/gonum/gonum` since there's no .go files to force the canonical import path.  I'd suggest we put a `doc.go` file in the root directory with the import path and general documentation, so there's no way to end up with the gonum repo at the wrong place in the user's $GOPATH.