[Haskell-cafe] Getting the haddocks back (was: documentation build failing in hackage?)

[Rogan Creswick]

On Sun, Oct 12, 2014 at 4:50 AM, Mateusz Kowalczyk <fuuz...@fuuzetsu.co.uk> wrote:

The biggest problem is that there is no official statement as to why the
builds stopped and an estimate of when they are coming back.

I agree!

My understanding (which is at /least/ 2nd or 3rd hand) is that the doc builds were turned off intentionally because it was a security issue, and that they are unlikely to come back.

Now, assuming that is the case, how can we solve the "there is no documentation" issue?  Some ideas to kick-start discussion:

 - Provide an option to include haddocks in the sdist bundle, and extract them on Hackage for display.

 - Add a 'cabal uploadHaddock'

 - Run all haddock builds in a VM/docker container / etc.. that mitigates the security concerns.

 - ???

--Rogan

[Mateusz Kowalczyk]

What I heard is that it's the combination of the recent server migration
and then running out of disk space. It seems that whoever I ask, they
always have heard from someone else…

AFAIK there is some movement towards having ‘cabal uploadHaddock’ already.

--
Mateusz K.
_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Heinrich Apfelmus]

Mateusz Kowalczyk wrote:
> On 10/13/2014 04:25 PM, Rogan Creswick wrote:
>> Mateusz Kowalczyk <fuuz...@fuuzetsu.co.uk> wrote:
>>> The biggest problem is that there is no official statement as to why the
>>> builds stopped and an estimate of when they are coming back.

Apparently, Haskell.org has a blog since yesterday, where they explain
what's up with the documentation builds:

https://blog.haskell.org/post/the_new_haskell_org/


Best regards,
Heinrich Apfelmus

--
http://apfelmus.nfshost.com

[Johannes Waldmann]

Heinrich Apfelmus <apfelmus <at> quantentunnel.de> writes:

> Apparently, Haskell.org has a blog since yesterday, where they explain
> what's up with the documentation builds:
>
> https://blog.haskell.org/post/the_new_haskell_org/

Looks good. This post contains:

> And, the doc builder is now working again,
> and should hopefully stay that way too.

What to do if they don't? E.g., I'm getting this:
https://hackage.haskell.org/package/satchmo-2.8.1/reports/

- J.W.

[Carter Schonwald]

use the curl api to "delete the old docs" to try to kick off a new build

https://hackage.haskell.org/api 

[Carter Schonwald]

OR just upload your own using the fuuzetsu script :) 

[Carter Schonwald]

alternatively: the doc builders could be catching up with the huge backlog :) 

[Mikolaj Konarski]

>> Apparently, Haskell.org has a blog since yesterday, where they explain
>> what's up with the documentation builds:

[...]

> What to do if they don't? E.g., I'm getting this:
> https://hackage.haskell.org/package/satchmo-2.8.1/reports/

Yep, I've seen the doc building failure in another package today, too
I guess, despite the comments section, the blog is
mostly a one-way communication channel, so we are
left with the options I mentioned in

http://thread.gmane.org/gmane.comp.lang.haskell.cafe/113239/focus=113269

[Mikolaj Konarski]

Hmm, actually, if you'd like to be an early adopter,
you can file your tickets about haskell insfrastructure here

https://phabricator.haskell.org/maniphest/query/open/

and here is some documentation

https://phabricator.haskell.org/w/projects/haskell.org_infrastructure/

Contributions are welcome!

[Carter Schonwald]

you have to add your acls

curl -X DELETE  https://$username:$p...@hackage.haskell.org/package/linear-1.10.1.2/docs

something like that, suitably adjusted for the libs in question

On Tue, Oct 14, 2014 at 3:00 PM, Johannes Waldmann <johannes...@htwk-leipzig.de> wrote:

Hi Carter,

> https://hackage.haskell.org/api

Aha. How do I authenticate? I'm getting this (no surprise)

[waldmann@R630 ~]$ curl -i -X DELETE  -H "Accept: application/json"
https://hackage.haskell.org/package/satchmo-2.8.1/docs

HTTP/1.1 401 Unauthorized

- J.

[Sven Panne]

Currently doc building on Hackage seems to be broken again: Three tiny
packages (StateVar, ObjectName, Tensor) I've uploaded more than 2 days
ago are still without documentation. I think there is a way to upload
this manually, but I don't think that's a good idea, because there's
no guarantee that the documentation and the package are in sync.
Furthermore, I can't even find a description on hackage.haskell.org
how to do that (if I wanted to).

I'm well aware that most of the Haskell infrastructure depends on
volunteers, so things can't be perfect and there are no SLAs etc. What
I'm only complaining about is that it's totally unclear what's going
on and what the current status is: Is documentation building totally
broken at the moment? If yes, when will it be available again? Are
there any kind of logs I can see? Are my packages in some hidden
documentation builder queue? If yes, knowing its position and/or ETA
might be nice. The building doesn't have to be lightning fast, but
having some kind of progress indicator would really be nice. Without
any visible documentation basically all packages are useless because
one can't really know what what they contain, a synopsis is not enough
most of the time.

[Carter Schonwald]

1) once they finally finish updating the hackage deply to current hackage-server master, there will be more visibility into the build process

2) i've pinged the person who is owning making sure doc builders are working to ask them to check it out

[Sven Panne]

2014-10-26 18:07 GMT+01:00 Carter Schonwald <carter.s...@gmail.com>:
> 1) once they finally finish updating the hackage deply to current
> hackage-server master, there will be more visibility into the build process
>
> 2) i've pinged the person who is owning making sure doc builders are working
> to ask them to check it out

Still no documentation. :-( All I'm asking for is some kind of
feedback, even a low-tech textual warning on the Hackage starting page
would be fine. As it is, things are very opaque...

[Michael Snoyman]

In case it helps anyone, I've started uploading the Haddocks I generate from Stackage builds to stackage.org (e.g. [1]). I'm still working out a few kinks in the process, for example, I'd forgotten to install the htmldocs PPAs on the Jenkins server, so some of the uploaded packagesets are currently missing docs for GHC-provided packages. However, it mostly works, and feedback is definitely appreciated.

Of course, this only provides docs for package in Stackage, so you can consider this email as a not-so-subtle encouragement to add your packages to Stackage[2] :).

[1] http://www.stackage.org/haddock/21a32ef62e5e60edb49f29e6baa31931fcbaa8dd/index.html
[2] https://github.com/fpco/stackage/wiki/Maintainers-Agreement

[Carter Schonwald]

I spoke with the person owning the doc builder, and seems like the it was a problem with how the doc builder would order its work.  It was getting stuck trying to build docs for very old packages instead of newest first.  So that's getting fixed this week. 

What textual warning are you looking for?

[Yuras Shumovich]

On Wed, 2014-10-29 at 12:04 +0100, Sven Panne wrote:
> All I'm asking for is some kind of
> feedback, even a low-tech textual warning on the Hackage starting page
> would be fine. As it is, things are very opaque...

+1
It is easy to manually upload docs to Hackage, but at least you need to
know that doc builder is not working.

Probably status.haskell.org should reflect doc builder status?

[Simon Michael]

+1. Why all the secrecy ?

[Carter Schonwald]

There is no secrecy.  Just no one as yet volunteering to write the tools needed to create the visibility into the doc builder queue progress :-)

Go on #haskell-infrastructure on freenode if you want to help make things happen.  Nothing else matters :)

[Sven Panne]

2014-10-29 14:17 GMT+01:00 Carter Schonwald <carter.s...@gmail.com>:
> I spoke with the person owning the doc builder, and seems like the it was a
> problem with how the doc builder would order its work. It was getting stuck
> trying to build docs for very old packages instead of newest first. So
> that's getting fixed this week.
>
> What textual warning are you looking for?

Exactly the stuff you wrote in this email should be in big fat letters
on the front page (and probably even on each page header) of Hackage.
As I said, it's totally fine that things take some time (it's an open
source effort relying on volunteers after all), but it's crucial that
the current state of a central piece of the Haskell ecosystem is very
visible.

Regarding status.haskell.org: This is too indirect for this purpose
IMHO. As somebody who is not closely following each and every email on
the various Haskell mailing lists, how should you find out that this
site even exists? It's not linked from Hackage AFAICT.

[Sven Panne]

2014-10-29 17:37 GMT+01:00 Carter Schonwald <carter.s...@gmail.com>:
> There is no secrecy. Just no one as yet volunteering to write the tools

> needed to create the visibility into the doc builder queue progress :-) [...]

Nobody asked for a fully automated tool/status page, a plain visible
note on Hackage done by hand would be enough. Problems with the doc
builders are not new, see e.g.
https://github.com/haskell/hackage-server/issues/244 or
https://github.com/haskell/hackage-server/issues/265.