goissues.org: seeing the issues for any specific stdlib/subrepo Go package

[Dmitri Shuralyov]

Hi golang-dev,

This weekend, I got an idea that I was eager to try. The idea was making goissues.org, a simple website like godoc.org, except for showing the issues of a Go package. So if you've ever wanted to know what are all the issues for a specific package, say, net/http, it'd be as easy as typing goissues.org/net/http in your browser's address bar and seeing issues only for that package, nothing more.

---

I built a quick prototype. The scope includes only the Go standard library and sub-repositories for now. You can try it out. For example, to see all issues for image/png package:

https://goissues.org/image/png

There are only 4 open ones at this time. All issues for net/http are:

https://goissues.org/net/http

For golang.org/x/crypto/ocsp package:

https://goissues.org/golang.org/x/crypto/ocsp

Here are only the issues marked for x/vgo:

https://goissues.org/golang.org/x/vgo

It's powered by Brad's x/build/maintner work, which made this very easy to implement, and issues are updated in real-time. It scans over all the issues in the github.com/golang/go issue tracker, and parses the issue titles. Whenever it finds a "import/path: " prefix, it considers that issue to belong to import/path if that import path is an existing Go package or a parent directory.

If an issue has comma-separated prefix, e.g:

go/printer, gofmt: line directives specifying columns must stay in place relative to next token

Then it gets added to both go/printer and gofmt packages.

"x/foo" gets mapped to "golang.org/x/foo". "proposal:" is stripped before consideration, since such issues are titled "proposal: path: issue title".

All the issues that are not found to belong to any existing Go package are currently dumped to the other pseudo-package. I need to think more about what to do with these (it currently includes prefixes like "doc:", "wiki:", "spec:", "all:", and all issues without a valid path prefix).

There's a "New Issue" link on top for each package. When clicked, it pre-populates the issue title with the correct path prefix. (Perhaps that'll make it convenient for reporting new issues. You can quickly check it hasn't already been reported because it's easy to Cmd+F the entire list.)

---

I'm not sure how useful this would be (compared to godoc.org, which I use daily), but the "what are issues for this package" question has popped up in my head on various occasions on the past, so I wanted to create this so it'd be available next time I need it. Plus, it was interesting to investigate whether this could work at all or not.

I wanted to share this (on -dev first, because the target audience includes people who work on Go), and I'm very happy to receive any thoughts or feedback, if you have any.

Thanks.

[Brad Fitzpatrick]

This is excellent!

And I imagine this will all work offline just as well? (as I'm about to hop on a long flight.)

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dmitri Shuralyov]

And I imagine this will all work offline just as well? (as I'm about to hop on a long flight.)

It won't, since it's a website. We already have the maintserve command for viewing all Go issues offline, you should use that. :)

https://godoc.org/golang.org/x/build/maintner/cmd/maintserve

P.S. I'll be sending a trivial CL to it soon.

[Brad Fitzpatrick]

On Tue, Mar 13, 2018 at 11:53 AM, Dmitri Shuralyov <shur...@gmail.com> wrote:

And I imagine this will all work offline just as well? (as I'm about to hop on a long flight.)

It won't, since it's a website.

I meant the program running that website, of course. :)

 

We already have the maintserve command for viewing all Go issues offline, you should use that. :)

I do, but I like the UI on goissues.org of filtering down by package easily.

[Dmitri Shuralyov]

I meant the program running that website, of course. :)

Ah, ok. I haven't considered that use case yet... The program itself should be completely standalone, but it links to golang.org/issue/123 for all the individual issues, so that won't work offline.

I do, but I like the UI on goissues.org of filtering down by package easily.

I see. If you like it, it might make more sense to implement the same functionality in maintserve, either as a replacement for its current UI, or as an addition. It wouldn't be hard. Just need to decide on the URL design.

[jimmy frasche]

Very cool.

Noticed a couple edge cases playing around with this:

There are some issues with {} glob expansions for the package name
like https://github.com/golang/go/issues/24379 that don't show up:
https://goissues.org/compress/zlib?state=closed

And looking at that list of closed issues I saw that there's also an
issue with no title in that list:
https://github.com/golang/go/issues/8650

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to golang-dev+...@googlegroups.com.

[Dmitri Shuralyov]

Thanks for finding those edge cases and letting me know!

I'm aware of the {} glob expansions not being supported right now. In the parsePrefixedTitle func, I have a comment:

// TODO: Parse "x/image/{tiff,bmp}" as ["x/image/tiff", "x/image/bmp"], maybe?

I need to decide whether to support those (and therefore encourage them) or not (which would potentially discourage their use). I've seen some discussions somewhere that pointed out it may be better to avoid them, because it makes finding things via grep harder. But, with a tool like goissues.org doing the parsing, that argument may not be as strong.

I'll probably raise this question separately and try to find the best way to proceed. It's currently in NeedsDecision state now.

an issue with no title

Thanks. Not being able to click the issue title because it's empty is bad. I'll fix that. Filed the bug at https://dmitri.shuralyov.com/website/gido/...$issues/1.

[Josh Bleecher Snyder]

This is awesome.

If you're looking for feature ideas:

* Also show CLs for that package
* Filter by milestone (and label?)
* Show "subpackages" at the top. E.g. for https://goissues.org/cmd,
show links to issues for cmd/go, cmd/compile, cmd/link, etc.
* Some url to show issues for a set of packages, so that I have a
single page to bookmark (I am interested in a few different packages)

But even as it is it is delightful. Thanks for making it.

-josh

[Dmitri Shuralyov]

Thanks for feedback and feature ideas Josh! I will consider them.

The 4th point you make is something I've been pondering about. One idea I am considering is supporting import path patterns (as defined at https://golang.org/cmd/go/#hdr-Package_lists). That way, you could visit goissues.org/net/... and it would show all the issues for packages that match the "net/..." import path pattern.

I've already experimented with import path patterns as valid URLs on my personal site (e.g., try visiting dmitri.shuralyov.com/.../image/..., it shows Go packages that match that import path pattern) and I found it to work well (IMO).

Perhaps, in addition to that, I could also make it support comma-separated import path patterns. Commas are not allowed as part of import paths (see https://golang.org/ref/spec#Import_declarations) and they're safe characters that don't need to be escaped in the path part of URL. So, a URL like https://goissues.org/net/...,io/ioutil could display issues for all net/... packages and io/ioutil combined. (Maybe there is a better character than comma to use as a separator... I'd need to consider them all.)

What do you think about that?

> email to golang-dev+unsubscribe@googlegroups.com.

[Ralph Corderoy]

Hi Dmitri,

> So, a URL like https://goissues.org/net/...,io/ioutil could display
> issues for all net/... packages and io/ioutil combined. (Maybe there
> is a better character than comma to use as a separator... I'd need to
> consider them all.)

+1 for comma; it's the most natural, what one would assume, and is
simple to type when forming a goissues URL more quickly than a GUI would
allow.

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

[jimmy frasche]

https://goissues.org/golang.org/x/playground 404s

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Dmitri Shuralyov]

Thanks for reporting Jimmy. It has been fixed now (commit 8ce3fa6b), please try again.

[jimmy frasche]

It works now—thanks!

[Dmitri Shuralyov]

Today, I added a new /-/packages page. It lists all known packages (and their number of open issues):

https://goissues.org/-/packages

(The relevant commit is dcee4de7.)

I added it because I think it's helpful sometimes to be able to see all the packages at a glance, and use browser's search to find one you're looking for (instead of always having to directly type in its import path in the address bar).

Back when I was developing the original prototype, the index page was initially non-existent. Then I added one which listed all packages, which I thought was nice to see. But when I was publishing it, I wanted to add an "About" section, and the huge list of packages seemed too much. I made it show the top 15 packages. But I was missing the list of all packages. That's why I added it today (on a separate page).

If you give it an Accept header value equal to exactly "application/json", it returns JSON.

There are also some anchors on the page to jump to those sections:

https://goissues.org/-/packages#stdlib

https://goissues.org/-/packages#subrepo

https://goissues.org/-/packages#other

Feedback is welcome.

My next priority for goissues.org is to add support for comma-separated import path patterns as proposed above, to make it possible to view issues from multiple packages at once.

[Ralph Corderoy]

Hi Dmitri,

> It lists all known packages (and their number of open issues):
> https://goissues.org/-/packages

Nice.

> If you give it an Accept header value equal to exactly
> "application/json", it returns JSON.

Will it change to conform to the RFC, e.g. case-insensitive, q-values, ..?

> Feedback is welcome.

Toggling the hiding of all those with zero open issues, perhaps by
Javascript and CSS rather than a page reload.

[Anthony Starks]

with the code below,  and dchart (go get github.com/ajstarks/deck/cmd/dchart) 

you can do

$ curl -H "Accept: application/json" -s 'https://goissues.org/-/packages' | go run readissue.go | dchart -wbar  | pdf

to get a sorted list of the top 20 issues (see attached)

package main

import (

"encoding/json"

"fmt"

"io"

"os"

"sort"

)

type issue struct {

Path       string `json:"ImportPath"`

IssueCount int    `json:"OpenIssues"`

}

type Issues []issue

func (a Issues) Len() int      { return len(a) }

func (a Issues) Swap(i, j int) { a[i], a[j] = a[j], a[i] }

func (a Issues) Less(i, j int) bool {

if a[i].IssueCount < a[j].IssueCount {

return false

}

if a[i].IssueCount > a[j].IssueCount {

return true

}

return a[i].IssueCount < a[j].IssueCount

}

func readissues(r io.Reader, w io.Writer, n int) {

var data Issues

if err := json.NewDecoder(r).Decode(&data); err != nil {

fmt.Fprintf(os.Stderr, "%v\n", err)

return

}

sort.Sort(Issues(data))

fmt.Fprintf(w, "# Top %d Go Issues\n", n)

for i, d := range data {

if i <= n-1 && d.IssueCount > 0 {

fmt.Fprintf(w, "%s\t%d\n", d.Path, d.IssueCount)

}

}

}

func main() {

readissues(os.Stdin, os.Stdout, 20)

[Ralph Corderoy]

Hi,

Anthony Starks wrote:
> with the code below, and dchart (go get
> github.com/ajstarks/deck/cmd/dchart) you can do
>
> $ curl -H "Accept: application/json" -s 'https://goissues.org/-/packages' |
> go run readissue.go | dchart -wbar | pdf
>
> to get a sorted list of the top 20 issues

OT... And that's part of the part of the problem with the proliferation
of JSON for simple data, discarding what was learnt from decades of Unix
text-processing tools. Still, I suppose other text/* `Accept's might be
supported in time. :-)

curl -sSH Accept:application/json https://goissues.org/-/packages |
sed 's/.* //' | tr -d '\011[]{},"' |
sed -n '/^$/d; h; n; G; s/\n/ /p' |
sort -k1,1nr -k2 |
sed 42q |
awk 'NR == 1 {h = $1}
$1 {printf "%4d %"($1/h * 100)"s%s\n", $1, "", $2}'

Most of it is undoing the JSON. There's also jq(1), either to extract
the data, or consistently format it for processing as text.
https://stedolan.github.io/jq/ But its later releases have added
datetime, regexps, maths, try/catch, module system, etc., because it's a
JSON silo that can't leverage what already exists.