Getting rid of dependency hell
Michael Snoyman
wastelands of Northern Israel. We came up with this idea in an awesome
Yesod brainstorming session.)
Those of us with a lot of experience with Cabal know how to avoid its
warts. We'll add extra constraints sometimes, or simply wipe out
~/.ghc and hope for the best. Even those of us with experience
sometimes get stumped. But the real problem is for new developers. I'm
afraid to think how many prospective Yesoders get turned off by the
fact that typing "cabal install yesod" blows up in their faces. We
need to fix that.
One route that I've been exploring is using a distribution system like
Nix in place of Hackage. There are still many things that need to be
worked out (such as how to actually install Nix properly), but it's an
interesting route to take. It also means that for some systems,
binaries could be downloaded instead of source, making it much faster
to get a working Yesod install.
But I want to explore another route as well. While working on Nix, I
realized that the reason it doesn't have dependency hell is that there
is a central maintainer who is white-listing specific packages, and
their versions, which work together. The problem with Cabal and
Hackage- besides the idiocy of Cabal's dependency tracker- is that
anyone can upload a new version of a package, and it can wreak havoc
on existing builds. Two recent examples:
* A new version of zlib was released, which wasn't compatible with GHC
7.0. Until the maintainer released a patch, no one could build
anything that depended on zlib. Solution: I added a restrictive upper
bound on zlib-bindings that forced usage of the older zlib.
* base64-bytestring 0.1.1.0 introduced a horrible performance
regression which actually caused a 300x slowdown in Yesod (not
exaggerating...).
So here's my idea: we can create a profile of packages which are
guaranteed to work well together. This will be maintained by the Yesod
team, kept in a Github repo, and used (we'll get into that in a bit)
by every Yesod users. The versioning for these profiles would follow
normal PVP versioning rules: yesod-profile-0.10.0.1 would be a
bugfix/performance upgrade from 0.10.0. 0.10.1 would introduce new
features without breaking backwards compatibility. This profile would
give precise versions- *not* version bounds- for a large number of
"blessed" packages. For non-blessed packages, you would still be on
your own. The idea, however, would be to include a very large number
of common packages within this list of blessed packages.
So how does all this work? The idea would be that, instead of a
scaffolded site having a cabal file, it has a settings file. That file
would contain information like which packages are required and
language extensions used. That we would have a utility that
automatically generates the cabal file from there. It would compare
the list of packages in the settings file with the list of blessed
packages, and assign a specific version in the cabal file. If the
profile is updated in the future to include bugfixes, your cabal file
will automatically be regenerated.
To get into more details, I think we would need to take the following steps:
* We'll include the cabal-generation facility as part of the Yesod executable.
* Separate the Yesod executable from the rest of the yesod package, so
that it has as minimal a set of dependencies as possible. We don't
want dependency hell on the tool fixing dependency hell (yo dawg...).
* I think it makes sense to use cabal-dev by default for this. The
tool could automatically handle downloading of any required
executables, including alex and happy as well.
* We'll also want to create some scripts to automate the management of
the profile definition.
* The new "yesod in five minutes" would go something like:
1. cabal install yesod-exe
2. yesod init
3. yesod devel, which will download the profile information,
generate the cabal file, install dependencies, and whatever else needs
to be done.
Michael
Kirill Zaborsky
Felipe Almeida Lessa
> So here's my idea: we can create a profile of packages which are
> guaranteed to work well together. This will be maintained by the Yesod
> team, kept in a Github repo, and used (we'll get into that in a bit)
> by every Yesod users. The versioning for these profiles would follow
> normal PVP versioning rules: yesod-profile-0.10.0.1 would be a
> bugfix/performance upgrade from 0.10.0. 0.10.1 would introduce new
> features without breaking backwards compatibility. This profile would
> give precise versions- *not* version bounds- for a large number of
> "blessed" packages. For non-blessed packages, you would still be on
> your own. The idea, however, would be to include a very large number
> of common packages within this list of blessed packages.
I like this idea especially since it would be a lot easier to get an
old project to compile (I love how Hackage keeps all versions of all
packages).
> So how does all this work? The idea would be that, instead of a
> scaffolded site having a cabal file, it has a settings file. That file
> would contain information like which packages are required and
> language extensions used. That we would have a utility that
> automatically generates the cabal file from there. It would compare
> the list of packages in the settings file with the list of blessed
> packages, and assign a specific version in the cabal file. If the
> profile is updated in the future to include bugfixes, your cabal file
> will automatically be regenerated.
...but I don't like this approach. Actually, I don't like anything
that creates build step =). So I'd like to convince myself that the
simpler one doesn't work:
Why not create a plain Cabal package names yesod-profile? It would
depend on the exact versions of all blessed libraries. To install the
latest Yesod, you'd just 'cabal install yesod-profile'. To use an
specific profile on your project, just depend on a specific version of
the yesod-profile. Or depend just on the major version (e.g.
yesod-profile == 0.10.*).
There's only one downside to this approach that I can see, which is
that you'd end up compiling a lot of packages that you don't need.
However, I don't know if this would slow down the linker, and I
*think* that it would not affect the final executable size at all
(since unused symbols are thrown away).
OTOH, there are at least a couple of advantages. First of all, we
don't need another tool. Also, we'd get for free Hackage's
infrastructure for both distribution of the profile *and* for keeping
old versions of the profile around should someone need it.
The biggest question is: does this work, or would the dependency hell
continue? I think that 'cabal install yesod-profile' would always
work, even if you had a b0rked set of libraries. As a dependency of
the project, I'm not sure it would work 100% of the time. But 'cabal
install yesod-profile'ing again should fix it.
Cheers,
--
Felipe.
Michael Snoyman
upstream. But I'm not holding my breath. I think cabal is by far the
weakest link in the Haskell toolchain right now, which is why there
are so many tools trying to work around it (cabal-dev, virthualenv,
cabal-src).
Michael Snoyman
You hit the two downsides I was thinking of. As the blessed package
list grows (I'm thinking 300 packages wouldn't be unreasonable), it's
going to be a huge burden to compile all of those. Worse yet, some of
those packages have C library dependencies, which can't be installed
by Cabal, and are a major pain to install on Windows.
I'm also pretty sure that it won't solve the newbie issue: they'll
type "cabal install" in their new package, and due to Cabal stupidity,
the thing won't compile. "I know yesod-profile requires foobar 0.2.3,
but I see 0.2.4 is already installed, so I'm going to barf."
The other issue is that this still doesn't solve the build tools dependencies.
I agree with the principle, however: adding an extra tool to the chain
is a bad idea, so I'm *very* happy to continue brainstorming on better
solutions to the problem at hand.
Michael
>
> Cheers,
>
> --
> Felipe.
Felipe Almeida Lessa
> You hit the two downsides I was thinking of. As the blessed package
> list grows (I'm thinking 300 packages wouldn't be unreasonable), it's
> going to be a huge burden to compile all of those. Worse yet, some of
> those packages have C library dependencies, which can't be installed
> by Cabal, and are a major pain to install on Windows.
Why not having more than one yesod-profile-* package then? Something
like the metapackages of TeXLive: each metapackage gets you a lot of
things and you get only the metapackages that you want. Then
'yesod-profile' itself would contain n00b set with everything that
works everywhere and is useful on the beginning. Yo dawg, I heard you
like metapackages, so I put your metapackages in a metametapackage so
that you can metadepend while you depend.
> I'm also pretty sure that it won't solve the newbie issue: they'll
> type "cabal install" in their new package, and due to Cabal stupidity,
> the thing won't compile. "I know yesod-profile requires foobar 0.2.3,
> but I see 0.2.4 is already installed, so I'm going to barf."
I don't think that will happen at all. When the dependency with exact
version in on the package you told cabal-install to install, IME it
always tries to reinstall it. But this does happens sometimes when
the dependency is buried in another package since it currently doesn't
backtrack. Have seen otherwise?
> The other issue is that this still doesn't solve the build tools dependencies.
This is uglier, specially since this needs a change to the user's
PATH. Would your proposed yesod-exe try to change the PATH as well?
Cheers,
--
Felipe.
Felipe Almeida Lessa
<felipe...@gmail.com> wrote:
>> The other issue is that this still doesn't solve the build tools dependencies.
>
> This is uglier, specially since this needs a change to the user's
> PATH. Would your proposed yesod-exe try to change the PATH as well?
We could still have an easily installable no frills yesod-exe package
that deals with build tools, scaffolding, building, devel server,
etc., while create yesod-profiles as metapackages. Then yesod-exe
would also be able to list what is inside which metapackage and even
tell you which packages you depend that aren't covered by
yesod-profile.
Cheers,
--
Felipe.
Lambda
Greg Weber
night. This was provoked by my inability to get my project to compile
that uses Yesod installed from github source. All cabal-meta does is
let you point to specific packages and install them all at once.
If you run this command, you can easily get a failure:
cabal install foo && cabal install bar
Whereas if you run this command it should almost always work:
cabal install foo bar
cabal-meta facilitates this: installing everything at once.
Right now it looks for a file "sources.txt" where each line is a
directory to install or a folder with a "sources.txt" that it recurses
into. cabal-meta will use cabal-src-install (which isn't necessary if
you only use cabal-meta).
Previously I tried something similar to having just a meta package.
That sort of works, but we have use cases where we want to point to a
directory or a url.
So for my project that wouldn't build locally, I added a source.txt of
./ to build itself, and a pointer to where my yesod install is. My
yesod install has a sources.txt of hamlet, persistent, wai, and yesod.
Each of these has a source.txt of all their cabal repos.
This project is a very simple 30 lines of code, and it works for this
use case. The next step I have in mind is to allow git urls in
addition to just folders: that will solve the beta/release candidate
usage issue. If we added support for downloading cabal packages
directly and automatically unpacking them, I think we could extend
this to make Yesod's installation easier.
I will push these projects later today.
Greg Weber
https://github.com/yesodweb/cabal-meta
cabal-meta can be thought of as an abstraction of our current
installer shell script.
It relies on my just uploaded Shelly library:
https://github.com/yesodweb/Shelly.hs
Shelly is a fork of Shellish for efficiency's sake. Once I get
feedback from others on it, I will release it to hackage. One thing I
did for efficiency was switch to using blaze-builder for the command
return result (it reads one line at a time, which enables efficient
folding over the output). So the library is mostly using Text now. My
main question is what to do about the String/Text clash, particularly
since FilePath is a synonym for String. Note this is not an efficiency
issue, but just one of avoiding the inconvenience of conversion. I am
considering gradually making a Text version of every single FilePath
function (they would all just start as proxies that unpack and pack),
and am wondering if there is already a library that does this.
Greg Weber
There is a GSoC proposal to merge cabal-dev into cabal:
http://hackage.haskell.org/trac/summer-of-code/ticket/1590
I commented that we may want to go the virthualenv route instead of
cabal-dev. Regardless of the final solution I think we should promote
this GSoC project and work on orthogonal installation functionality
until we see if it accepted.
Scott Vachalek
Bryan Richter
> As a relative noob to the world of Haskell, Cabal, and Yesod I applaud all
> of these efforts. I'm getting to the point where I can usually keep my
> blood pressure from spiking and slowly try to solve cabal issues but I can't
> help chanting the mantra "there must be a better way".
>
Sorry to branch further off-topic, but I wanted to chime in with my
personal mantra, "didn't Debian already solve the concept of
packaging? Can we not at least start there?"
-B
(currently reading http://www.debian.org/doc/debian-policy/)
Felipe Almeida Lessa
> Sorry to branch further off-topic, but I wanted to chime in with my
> personal mantra, "didn't Debian already solve the concept of
> packaging? Can we not at least start there?"
One of the reasons I stopped using Debian many years ago was that
unstable had its own dependency hell and testing lagged behind
everytime something a lot of packages depended upon had an upgrade.
That being said, I welcome any suggestions coming from different backgrounds =D.
Cheers, =)
--
Felipe.
Brian Troutwine
<felipe...@gmail.com> wrote:
> On Fri, Mar 2, 2012 at 5:42 PM, Bryan Richter <bryan....@gmail.com> wrote:
>> Sorry to branch further off-topic, but I wanted to chime in with my
>> personal mantra, "didn't Debian already solve the concept of
>> packaging? Can we not at least start there?"
>
> One of the reasons I stopped using Debian many years ago was that
> unstable had its own dependency hell and testing lagged behind
> everytime something a lot of packages depended upon had an upgrade.
Oh man, running Unstable unless you're a Debian dev is nothing but
heartbreak. Even then...
> That being said, I welcome any suggestions coming from different backgrounds =D.
Well, if an OS specific solution were to be taken apt.yesodweb.com
would be a welcome sight. 10gen, the RabbitMQ folks and PuppetLabs,
among others, run package repositories for the most common
distributions. You avoid cabal and negate the need for an end-user to
install/learn any tools other than those they know, increasing, of
course, the complexity of release management on the dev team.
I, personally, would prefer to see OS packages rather than a fancy new
tool, speaking with my Ops hat on. (Though Nix is fantastic.) It's
easier to create new dev, testing and staging environments if as many
extant tools are reused as possible. Less learning overhead as well.
I'll happily pitch in with the Debian and Ubuntu work and would like
to point out that there's already a fairly active Debian Haskell
taskforce which does just this sort of work.
http://lists.debian.org/debian-haskell/
Which Michael's already aware of, now that I think of it.
--
Brian L. Troutwine
Erik de Castro Lopo
> On Fri, Mar 2, 2012 at 3:46 PM, Felipe Almeida Lessa
> <felipe...@gmail.com> wrote:
> > On Fri, Mar 2, 2012 at 5:42 PM, Bryan Richter <bryan....@gmail.com> wrote:
> >> Sorry to branch further off-topic, but I wanted to chime in with my
> >> personal mantra, "didn't Debian already solve the concept of
> >> packaging? Can we not at least start there?"
> >
> > One of the reasons I stopped using Debian many years ago was that
> > unstable had its own dependency hell and testing lagged behind
> > everytime something a lot of packages depended upon had an upgrade.
>
> Oh man, running Unstable unless you're a Debian dev is nothing but
> heartbreak. Even then...
That does not match my experience of running Debian testing (with a
few things from unstable) for the last 10+ years. I am not a Debian
Developer, but I am a Debian Maintainer.
> I, personally, would prefer to see OS packages rather than a fancy new
> tool,
+1
> Less learning overhead as well.
> I'll happily pitch in with the Debian and Ubuntu work and would like
> to point out that there's already a fairly active Debian Haskell
> taskforce which does just this sort of work.
>
> http://lists.debian.org/debian-haskell/
+1. The Debian Haskell group does amazing work.
Erik
--
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/
Brian Troutwine
<mle...@mega-nerd.com> wrote:
> Brian Troutwine wrote:
>
>> On Fri, Mar 2, 2012 at 3:46 PM, Felipe Almeida Lessa
>> <felipe...@gmail.com> wrote:
>> > On Fri, Mar 2, 2012 at 5:42 PM, Bryan Richter <bryan....@gmail.com> wrote:
>> >> Sorry to branch further off-topic, but I wanted to chime in with my
>> >> personal mantra, "didn't Debian already solve the concept of
>> >> packaging? Can we not at least start there?"
>> >
>> > One of the reasons I stopped using Debian many years ago was that
>> > unstable had its own dependency hell and testing lagged behind
>> > everytime something a lot of packages depended upon had an upgrade.
>>
>> Oh man, running Unstable unless you're a Debian dev is nothing but
>> heartbreak. Even then...
>
> That does not match my experience of running Debian testing (with a
> few things from unstable) for the last 10+ years. I am not a Debian
> Developer, but I am a Debian Maintainer.
Hmm? It's entirely possible that I've said something I didn't mean. I
intended to convey my experience that running a pure Sid system was
not pleasant. Running anything other than pure Sid--in all the various
permutations--has always been a very solid experience.
>> I, personally, would prefer to see OS packages rather than a fancy new
>> tool,
>
> +1
>
>> Less learning overhead as well.
>> I'll happily pitch in with the Debian and Ubuntu work and would like
>> to point out that there's already a fairly active Debian Haskell
>> taskforce which does just this sort of work.
>>
>> http://lists.debian.org/debian-haskell/
>
> +1. The Debian Haskell group does amazing work.
>
> Erik
> --
> ----------------------------------------------------------------------
> Erik de Castro Lopo
> http://www.mega-nerd.com/
--
Brian L. Troutwine
Michael Snoyman
There are two problems with going the Debian route:
1. We lose control. Depending on the processes in place with getting
packages into Debian, this may not be a problem.
2. More importantly: many people aren't using Debian. I'm writing from
Fedora right now, and (as much as we may hate it) we need to support
Windows.
For people already on Debian, it might be a good solution, but I
haven't tried it myself yet.
Michael
Michael Snoyman
For that matter, would could just take the existing yesod package and
put strict version bounds on all of its ancestor dependencies. That
may actually not be a bad idea...
Another approach to the separate tool- without doing cabal file
generation- would be to simply call "cabal configure" for us,
automatically passing in the --constraint arguments.
Michael
Greg Weber
broader use case. Right now it only installs other sources, but it
could also do constraints. Constraints can just be specified in a
Cabal file. However, Cabal can't directly handle directories or remote
beta sources. Having a separate file with constraints also allows us
to hand out a blessed constraints file on every release rather than
telling users to modify their Cabal.
This doesn't solve the alex or required binaries issue, but it would
be nice to narrow the problem down.
Brian Troutwine
> There are two problems with going the Debian route:
>
> 1. We lose control. Depending on the processes in place with getting
> packages into Debian, this may not be a problem.
> 2. More importantly: many people aren't using Debian. I'm writing from
> Fedora right now, and (as much as we may hate it) we need to support
> Windows.
>
> For people already on Debian, it might be a good solution, but I
> haven't tried it myself yet.
>
> Michael
I intended to suggest something more like this:
http://www.rabbitmq.com/download.html
Note that there are quick-installations of platform specific packages
and more long-term package repository instructions. The use case is
somewhat different--RabbitMQ's a piece of infrastructure where yesod,
et al. is a software library. That so many programming texts begin
with elaborate instructions merely to bootstrap the build tools is a
sign of some fundamental deficiency in the every language gets its own
peculiar build-tool approach. Even rubygems, arguably one of the more
successful, is incredibly finicky to install correctly and maintain.
--
Brian L. Troutwine
Michael Snoyman
which version of the binary is installed, and if it's not the right
one (or not present) install it. I'm actually a little baffled that
cabal itself doesn't handle this properly.
Greg Weber
missing binaries. If we go that route, we may want to look at existing
tools for platform-independent installers. But the shake library might
help us write our own in Haskell. We should see if we can tie this in
with a framework for installing deployments - it doesn't seem that the
needs (other than working on Windows) are actually different.
A while back I toyed around with a script for installing Yesod from
source. I just updated it to use the latest version of my Shelly
library. https://github.com/yesodweb/install
Michael Snoyman
available on Hackage, like alex and happy. I don't think we really
depend on any other build tools in general (barring GHC itself of
course).
Max Cantor
dag.od...@gmail.com
> So here's my idea: we can create a profile of packages which are
> guaranteed to work well together. This will be maintained by the Yesod
> team, kept in a Github repo, and used (we'll get into that in a bit)
> by every Yesod users. The versioning for these profiles would follow
> normal PVP versioning rules: yesod-profile-0.10.0.1 would be a
> bugfix/performance upgrade from 0.10.0. 0.10.1 would introduce new
> features without breaking backwards compatibility. This profile would
> give precise versions- *not* version bounds- for a large number of
> "blessed" packages. For non-blessed packages, you would still be on
> your own. The idea, however, would be to include a very large number
> of common packages within this list of blessed packages.
So, the equivalent of the Haskell Platform for Yesod development. Why
not call it the Yesod Platform?
Michael Snoyman
* Having a blessed list of packages is a Good Thing. (We may as well
call it the Yesod Platform, I like it.)
* We need a separate solution for
* Adding an extra tool if we don't have to is a Bad Thing.
* We need some way to force Cabal to install packages from our blessed
list. Possible approaches to that:
## My original idea: Augment the yesod executable to generate the cabal file
* Pro: Would definitely solve the problem
* Pro: Users can easily upgrade to the newest Yesod Platform.
* Pro: Can solve the build tool issue
* Con: Makes it more difficult to write special Cabal code
* Con: Fairly complicated to get correct
## Put strict version bounds in the yesod package (or yesod-profile) itself
* Pro: No extra tools needed
* Pro: Users can easily upgrade to the newest Yesod Platform.
* Con: May not solve all of dependency hell (because Cabal really is
that stupid)
* Con: Does not solve build tools
## cabal-meta ++ --constraint rules
* Pro: Would definitely solve the problem
* Pro: Users can easily upgrade to the newest Yesod Platform.
* Pro: Can solve the build tool issue
* Pro: Easy to write custom cabal files
* Con: Also complicated to get correct
---
As a separate discussion, we need to make it easy for someone who's
never used Haskell before to get up-and-running with Yesod in a
minimal amount of time. I think we could write shell scripts for
individual OSes (Debian/Ubuntu, Fedora, Arch, and Mac OS X) that would
do the job. Alternatively, we could provide a virtual machine, but I
think that will be more heavyweight for a newbie. If we want to go for
the gold (like Max keeps pushing me to do :)), we could have an entire
"program in the cloud" setup where you could spin up a new virtual
machine on EC2.
Michael
Greg Weber
start there, see how it goes, and then circle back. The suggested name
is a bad choice though because it could easily mean yesod with
profiling enabled. How about yesod-constraints or yesod-platform?
I think this will actually solve the cabal stupidity issue. The only
problem is that eventually users will want to venture off the
platform. For that you have to take the constarints out of the
yesod-constraints package and put it in your application cabal, and
then start tinkering. Users can share their constraint solutions on
the wiki.
No more shell scripts! Haskell install scripts instead! We should
write an installer for one OS (I will help for debian/Ubuntu) and
release a VM for that OS. This isn't a separate issue, as a VM can
solve or mitigate most of the install issues - some newbies would love
it as a solution. Working from the cloud sounds nice until your
internet connection lags or dies, but it is easy for anyone to do this
once we have an install script.
Darrin Thompson
really. But the nix folks on the other hand...
What happened to the "just use nix" idea?
* Pro: Would get people up and running in five minutes, without the 40
minute build delay.
* Pro: Possibly low-pain Windows support.
* Pro: Masochists can still just use cabal.
* Pro: Users can easily upgrade to the newest Yesod Platform.
* Pro: Can solve the build tool issue
* Pro: Devs can produce an easy to support set of binaries, including ghc.
* Pro: Good deployment story.
* Con: It's a separate tool, not everyone is familiar with it.
* Con: We didn't invent it here. See also: Not written in Haskell.
--
Darrin
Felipe Almeida Lessa
> * Con: It's a separate tool, not everyone is familiar with it.
For me this is a big big con, I actually don't know even how to start
using it! Someone would need to take a stab at teaching everyone else
how to deal with it, or at least pointing out where to start.
Cheers,
--
Felipe.
Greg Weber
for you. Nix is capable of installing multiple versions of a package,
which is why it is touted as better than others. However, the nix
maintainers usually try to package a smaller set of compatible
packages. Nix assumes Unix. AFAIK it is possible to try to use it with
cygwin on Windows, but the installed binaries aren't going to always
work.
Greg Weber
packages and use build flags, so I have a sources.txt file that looks
like this:
./
sphinx -fone-one-beta
/home/gweber/proj/hs/y/yesodweb
cabal-meta solves installing local directories, applying build flags
to dependencies, and in the future installing from github for beta
releases. All of these capabilities are mostly orthogonal to the
discussion here. cabal-meta could be used instead of a
yesod-constraints package, but I think we may as well start with a
yesod-constraints.
Michael Snoyman
1. We should start consolidating our different tools together, and
make one awesome tool that solves all our problems.
2. It doesn't sound like cabal-meta is geared towards the problems of
newcomers at all, but instead for those of us hacking on Yesod itself,
so we should keep it separate.
Maybe what we need to do is clearly delineate what would be our ideal
workflows. For example, I don't think we want newcomers to have to
deal with a sources.txt file. On the other hand, tools like cabal-src
and cabal-meta seem like they fit together very well.
Greg Weber
automatically install cabal-src. At this point I don't see a benefit
to combining with cabal-src. Instead, cabal-src should be combined
into cabal proper because it is clearly always useful for everyone.
I built cabal-meta to solve *my* existing problems. I think we should
consider using it for releasing betas also though.
Using a yesod-constraints package is a better workflow. I have a
feeling that it won't be flexible enough and then we will want to use
a cabal-meta approach, but that doesn't mean we shouldn't try it
first.
I think the best workflow is adding yesod-constraints to the cabal
file: this can automatically be done by the scaffolder. There is a
problem though: one must install yesod first! We should figure out how
to distribute a scaffolder with minimal dependencies. Yesod would not
be installed until the user first installs their application. This
also combines a user's application install from 2 steps into 1 step,
which always plays nicer with cabal.
Michael Snoyman
user's workflow is:
cabal install yesod-exe
yesod init
...
cabal(-dev) install
(sit and wait)
And the very least, it gives a much better feeling. We can also set up
yesod init to automatically install necessary build tools, including
alex and cabal-dev.
Vagif Verdi
yesod install
On Tuesday, March 6, 2012 9:17:34 AM UTC-8, Michael Snoyman wrote:
Well, easy solution here: we'll make a yesod-exe package. Then a new
user's workflow is:cabal install yesod-exe
yesod init
...
cabal(-dev) install
(sit and wait)And the very least, it gives a much better feeling. We can also set up
yesod init to automatically install necessary build tools, including
alex and cabal-dev.
On Tue, Mar 6, 2012 at 3:55 PM, Greg Weber wrote:
> cabal-meta automatically uses cabal-src. I think it should
> automatically install cabal-src. At this point I don't see a benefit
> to combining with cabal-src. Instead, cabal-src should be combined
> into cabal proper because it is clearly always useful for everyone.
>
> I built cabal-meta to solve *my* existing problems. I think we should
> consider using it for releasing betas also though.
>
> Using a yesod-constraints package is a better workflow. I have a
> feeling that it won't be flexible enough and then we will want to use
> a cabal-meta approach, but that doesn't mean we shouldn't try it
> first.
>
> I think the best workflow is adding yesod-constraints to the cabal
> file: this can automatically be done by the scaffolder. There is a
> problem though: one must install yesod first! We should figure out how
> to distribute a scaffolder with minimal dependencies. Yesod would not
> be installed until the user first installs their application. This
> also combines a user's application install from 2 steps into 1 step,
> which always plays nicer with cabal.
>
> On Mon, Mar 5, 2012 at 11:13 PM, Michael Snoymanwrote:
>>>> <wrote:
Michael Snoyman
take a crack at the problem. I've uploaded a new package to Hackage
called cabal-nirvana[2]. I've detailed how it works in a new blog
post[3].
[1] https://plus.google.com/116553865628071717889/posts/WFPfB3M5C1Q
[2] http://hackage.haskell.org/package/cabal-nirvana
[3] http://www.yesodweb.com/blog/2012/03/cabal-nirvana
Anthropornis Nordenskjoldi
Greg Weber
windows. We just need someone to step up and maintain it. We want to
help with providing a Haskell-based install script unless you want to
go a different route. This tool might provide some inspiration for the
general purpose usefulness of a VM solution and sets up things like
ssh forwarding: http://vagrantup.com/
Greg Weber
every constraint that is not in my project:
Resolving dependencies...
cabal: There is no available version of SHA that satisfies ==1.5.0.0
So unfortunately at the moment nirvana is probably going to have to be
limited to a per-project basis with virthualenv and will require
manual intervention to remove constraints unless we are going to
require them to be installed. Perhaps cabal can be patched to ignore
unneeded constraints.
Greg Weber
that it might surprise some users. This can probably be overcome by
documentation. But there are some more subtle dependency issues. A
user will need to keep their yesod-exe and yesod in sync to a certain
extent, although we could mitigate that by maintaining backwards
compatibility in yesod-exe. But if yesod-exe depends on anything at
all common with yesod, cabal could still easily decide to have yesod &
yesod-exe wipe each other out when it installs. We would manually try
to make sure the constraints match, but we again need to worry about
someone who doesn't update their yesod-exe. It seems better to keep
things how they are now unless we have a very clear picture of the
consequences.
Michael Snoyman
makes constraints in the config file completely worthless. This also
applies when using --constraint on the command line. I guess it's time
to file a bug report.
Michael Snoyman
Michael Snoyman
Instead of using Cabal's constraints[1], we'll go a step higher: the
list of available packages. When you run cabal-nirvana start now, it
will just open up your full list of packages, and delete any versions
that aren't blessed. In other words, if your package list shows that
yesod-0.10.1 is blessed, then all other versions of yesod will be
deleted, but packages like wai will remain untouched.
I also simplified usage a bit: simply running cabal-nirvana will
download a new list of packages from Hackage, download the blessed
package list, and then apply the filtering. To undo Nirvana, just run
cabal update.
I've released version 0.2 to Hackage, let me know if (when) anyone
finds another cabal-install bug getting in the way of this working.
Michael
[1] Which, I repeat, are broken.
Greg Weber
that helps out the conversation. Constraints are just defined to
behave the same way as a cabal file constraint, which is perfectly
reasonable. What we want is something different. Preferences seem to
almost match that. You should consider having a nirvana option that
behaves just like the original constraints version but uses
preferences. The preferences behavior could actually be a better
approach than constraints.
The new nuclear option seems problematic if we have to explain to
users that they have to run nirvana after every cabal update.
Darrin Thompson
> The new nuclear option seems problematic if we have to explain to
> users that they have to run nirvana after every cabal update.
>
tldr: I propose that this is a release engineering problem, not a
build tool problem, and we can achieve build nirvana only by first
achieving release nirvana.
caveat: I haven't followed the full discussion/development of nirvana,
so apologies in advance if nirvana is already doing this, but I think
it's not.
The main point of my thinking is this: push the problem of
constraining dependency resolution to Yesod _release_ time and
my-yesod-site _release_ time. Lets look at how the consequences might
shake out. I'm just speculating here, but I've done stuff like this
before with whole Linux distros and it can work very well. I think the
similarities are relevant.
Assumptions:
1. Clearly everyone prefers to use vanilla cabal.
2. Most people (real yesod devs, five-minute setup beginners, and
deployments) should use virtualhenv.
3. Hosting for alternative hackage repos can be found.
4. Tools to publish purpose-built hackage repositories can be found or
are worth building.
5. It's easy to tell cabal to ignore hackage. (Am I wrong here? Never tried it.)
6. The real problem here is hackage, not so much cabal.
Non-assumptions:
1. People really want cabal to pull from directly from hackage. (In
fact, let's stop doing this exact thing.)
Perhaps what we should be blessing is not a set of cabal constraints,
but a hackage repository, a real deployed alternative repository with
alternative http urls, or a pure expression that completely describes
one so it could be exactly reproduced locally. (which is distilled
awesome)
We can start with the yesod 0.10 release repository. It contains yesod
0.10 and all the cabal libraries needed to build yesod.
In addition, we can publish a yesod-dev repository. It contains only
github yesod's dependencies. Yesod itself comes from github.
An extension to this idea:
The "source," which some tool consumes to build a hackage repository
in support of a particular yesod could be checked into github. Users
could consume that to build their repository locally. Cabal would pull
from that local repo. (Admit it, that's kinda cool. Imagine if that
source was created as part of the scaffolding.)
The biggest disadvantage to this in my mind is that users doing
anything non-trivial will soon need to add packages from hackage. We
don't want those users adding hackage back into their cabal config as
that's likely to eventually blow up yesod. What they should be doing
is putting their new packages into a local hackage repository. But
even if that's easy to do, that's a separate tool from cabal, right?
So we've introduced a new tool. That's bad, right?
Maybe that's ok though. Here are two reasons why:
First of all, running a separate third party package repository in
your own control is natural and good release management. You should be
doing it anyway, right? So let's just make nice tools for doing that
and use them earlier.
Second, imagine you are that new user. You set up your new Yesod site
with cabal (not using hackage!) and your site says hello world to you
on the first try. Yay! Then you read the book and learn how to store
data and make pages and forms and stuff. Yay! So far you have only
used cabal. Yay! We don't actually introduce that new tool until you
are at the point of editing your cabal file (Boo!) to add something
outside of the Yesod repo. At this point you are arguably _ready_ for
a new tool. Meh!
A long time ago when I worked at Progeny we built tools to slurp Linux
packages out of existing repositories, like debian stable plus debian
stable security plus our local anaconda packages repo and into a final
repository. One of the tools inputs was essentially and expression
describing how to constrain what ended up in the final repository. In
the end it was pretty nice to work with compared to what anyone else
had. The tools (pdk) are still in use today by some folks doing a
distro called studio64.
The reallly nice thing was you ended up with a single repository. Our
primary consumer was Debian apt or yum, which can install packages on
a running OS. But there are a lot of other tools that consume
repositories. Tools like a third party iso builder (buggy, rarely
used) or an OS installer builder (shudder) could just be their buggy
stupid selves. You didn't have to worry about them suddenly consuming
some oddball package that some nut released to debian unstable. You
always knew when you were changing the inputs to your downstream
tools, so those tools didn't need to be perfect.
That's the key right? Hackage is unsafePerformIO. Creating your own
repo is restoring purity.
It's basically the same problem in my mind. Cabal's depsolver is
complex enough as it is. So help it help you. Take control of its
input.
My $0.02 US.
--
Darrin
Michael Snoyman
Let's add some more constraints to the game:
* We need to release Yesod into Hackage proper, otherwise normal
Haskell users can't experiment with Yesod.
* We want to have the fewest steps for both maintaining the system,
and for a new user to get started.
Maintaining our own repository than contains not only all of Yesod's
packages, but also all of the dependencies, *and* every other package
that someone wants to use is a huge burden. And I don't see it adding
much versus the cabal-nirvana approach of simply disallowing
installation of non-blessed versions of included packages.
Also, having a separate repository for Yesod would require users to
block the main Hackage repository. Not only is this a big requirement,
but I don't want to ask new users to manually edit their
~/.cabal/config files. We could write a program to do it for them...
but then we may as well just do a cabal-nirvana approach.
Perhaps I'm reading you incorrectly, but I think this all comes down
to a whitelist vs blacklist approach. We could create a separate Yesod
repo which only includes the packages we approve of (whitelist), but
that would exclude a lot of other useful packages. Or we could do what
cabal-nirvana does (blacklist package versions we don't want), and
then some of the packages not in our list may not install correctly. I
favor the latter approach, at least until our list of approved
packages grows to encompass a wider range of Hackage.
Michael
Darrin Thompson
vs. current nirvana. I've seen and demonstrated that whitelisting has
solutions which _compose_ and at the scale of whole OS's. Maybe
blacklists compose but I doubt it's as meaningful.
On Thu, Mar 8, 2012 at 4:37 AM, Michael Snoyman <mic...@snoyman.com> wrote:
> Maintaining our own repository than contains not only all of Yesod's
> packages, but also all of the dependencies, *and* every other package
> that someone wants to use is a huge burden. And I don't see it adding
> much versus the cabal-nirvana approach of simply disallowing
> installation of non-blessed versions of included packages.
>
A number of your criticisms are fair. This one is wrong IMHO on a
couple of points.
Maintaining a repository is as hard as the tools at hand make it.
Cabal repositories are source only and architecture independent, so
it's a straightforward to work from some expression describing what
you want to a consumable repository. Working from an expression is
important as I'll describe more below.
In Darrin's world, having new users pull from hackage was a
non-assumption. By introducing it back, which you are possibly right
about, I'll admit that, it recolors the whole problem. I still believe
you will have to subset hackage one way or another. My approach of
separate repos is possibly too heavy handed, but there's going to be
an external tool no matter what, and whatever that tool consumes has
to be maintained. But again, the tools define the maintenance burden.
You're assuming your repo would have everything a user could _ever_
want. I'm saying when a user starts wanting stuff not in the yesod
repo, they need to mirror the subset of yesod they are using and
mirror a small subset of hackage also. This takes tools, the same ones
you use. Again, the benefit of my way is the users see only cabal and
virtualhenv until they actually need stuff outside Yesod, and we all
are trained early to control our cabal inputs with rigor with these
magic tools that don't exist yet.
> Perhaps I'm reading you incorrectly, but I think this all comes down
> to a whitelist vs blacklist approach. We could create a separate Yesod
> repo which only includes the packages we approve of (whitelist), but
> that would exclude a lot of other useful packages. Or we could do what
> cabal-nirvana does (blacklist package versions we don't want), and
> then some of the packages not in our list may not install correctly. I
> favor the latter approach, at least until our list of approved
> packages grows to encompass a wider range of Hackage.
>
Blacklist vs. whitelist is nice way to contrast the approaches so
thanks for introducing it.
I would assert that Whitelisting will ultimately lead to fewer queries
about why yesod didn't build. (Well, whitelist plus virtualhenv.)
You'll get queries about how to add more packages to nirvanna or
whatever tool, but those are _way_ easier. Even I could help with
those questions. You can put that in the book and be done.
Also, and this is important, when you make your whitelist tool, if you
are like most people, you start out with something "simple." In the
first generation solution you say "let's just list the packages and
exact versions we support." But then you say, "well, I want a way to
compare what I have with what's out on hackage so it's easy for me to
upgrade my deps." That works ok but it's a pain to maintain the list.
So in the second generation tool you switch from a simple list to an
_expression_ which does ultimately constrain a dep solution. And you
provide a way to "freeze" the dep solution at a given time. That
solves the problem with less churn on the release engineers part.
That's as far as we ever got at Progeny. So there may be a third
generation solution but, it turns out, not many people are willing to
pay lots and lots of money for what I described. So Progeny started
doing consulting gigs again until it died. T
(And your tools can easily solve again with fresh hackage so you can
diff your solution lock to see what's new. At Progeny I wrote
"semantic diff" to compare the result of two solution locks in terms
of "upgrade," "downgrade," and "unchanged," and probably some other
weird distro stuff I've forgotten. That was pretty helpful. Regardless
of the final shape of nirvanna, you'll probably want that.)
The point is, the shape of that _expression_ becomes an important
description of your application, and a wonderful thing to check into
version control. Also you check in your solution lock and you've found
a new way to control cabal's inputs.
That was what I discovered at Progeny. A customer could come to us and
say "we want a distro based on Debian stable but we want to backport
Apache and Postgres from unstable and add our custom web dsp processor
app." The expression ultimately said _exactly_ that, which was really
cool. Then I could freeze the solution, generate repos, and start
testing the resulting OS. As we found problems we just added things to
the expression which express the little fixes for all the bugs we
found. In retrospect, we were _composing_ requirements and fixes to
find a solution.
That's just my experience. I'm not really willing to contribute much
code so I won't argue this any further. I just want try and describe
my experience to hopefully save you and Greg and the rest of the devs
and lots of new users some migranes.
Good luck!
--
Darrin