- when recompiling a package with ABI changes, does cabal always
update dependent packages? It seems "not always" - it didn't update
itself, nor refuse the breaking upgrade, and the ABI breakage caused
linker errors on Setup.hs. Luckily, cabal was already linked so I
could always use it for queries.
- can cabal extract and check dependencies as specified in the .hi
files? I had a broken dependency which I could see with "grep" but not
otherwise.
- is there a "specification" of which are the "core" packages?

Not sure if this is more useful as a bug report or as discussion, or
if I just misused cabal and it's the only perfect software in the
world; still I wanted to share my experience.

Also I am not sure if this is the best place - but I'm not subscribed
to other lists, and the previous "cabal odissey" was posted here, so I
hope it's fine.

A related idea is that Gentoo had a tool which not only extracted such
dependencies, but recompiled all affected packages. While package
removal is not supported through cabal, it is sometimes needed (and
well, it should be supported at some point). See for instance this
FAQ:
http://www.haskell.org/cabal/FAQ.html#dependencies-conflict
My problem seems a nastier variation of the one described there :-(.

Details follow.

Best regards

OTOH, if you are building a new version of a package on which others
depend it won't build all the others.  Ex: build a new "containers"
package doesn't cause any of the ~1400 packages depending on it to be
rebuilt.

Cheers,
Thomas

"To avoid this problem in the future, avoid upgrading core packages.
The latest version of cabal-install has disabled the upgrade command
to make it a bit harder for people to break their systems in this
way."

I think that's what Paolo meant by "core" package.  Sadly the FAQ
doesn't say what core means.  Nor is that page user editable.  I think
"core" here must refer to packages that ghc is linked to.  For
example, the process package in the example on the FAQ.

I actually had this problem last weekend and I make a habit of never
running 'cabal upgrade' and never installing things globally.  Yet
some how on my system a package that ghc was built with did get
upgraded and installed in my user package db.  It was causing various
things to fail to configure.  If I recall correctly it was the
directory package, but I'll use FOO as a place holder.  I used the
suggested command line:
ghc-pkg unregister --user FOO-X

ghc-pkg said it was ignoring the command because it would break
packages. So then I tried adding --force.   At that point, ghc-pkg
still said it was ignoring me and that I should use --force.  This was
on ghc-6.12.1.  I tried it one more time with the --force option then
ran ghc-pkg list FOO, and all instances of the FOO package were gone.
At that point I could no longer configure any packages needing FOO.
In the end I had to reinstall ghc so I took it as a chance to upgrade
to 6.12.3.

I could have done something not so good before, but the final damage
was cabal's choice.

In my case, the same version of a package (old-time) was recompiled
against a different version of one of its dependencies (old-library),
and that broke the ABI. That's tricky. And Cabal recompiled most of
the package which needed help, except Cabal itself.

In my case, I got cabal proposing the following change while I was
trying to install pandoc-1.5:

$ cabal install -v pandoc-1.5
[snip, see attached inst-pandoc-1.5-without-pref-old-loc-_1.txt for
full output.]
old-time-1.0.0.2 (reinstall) changes: old-locale-1.0.0.1 -> 1.0.0.2
[snip]

When I did that, the result was that a symbol from old-time,
referenced from the Cabal package, disappeared, preventing further
package builds. It was possible because the symbol was
oldzmtimezm1zi0zi0zi2_SystemziTime_a149_{closure,info}, i.e. an
anonymous function. I wouldn't be surprised if the "a149" part were
randomly generated, causing the ABI breakage at every rebuild. But
recompiling against another version of a dependency is surely enough
(I guess that if inlining is disabled these things can't happen).
The net result, when Cabal was compiling the main Setup.hs of another
package (texmath IIRC), was a linker error on that symbol. Luckily,
the old version of old-time was in the system DB, so I could just
uninstall the new version from the user DB (after I decoded the linker
error, which wasn't trivial even knowing more about linking than one
would want to know).

An interesting fact is that cabal had recompiled many other packages
depending on old-time (HTTP, directory, process, zip-archive), just
not Cabal.

Maybe recompiling Cabal could have fixed it, but I avoided trying, and
reverted everything. The interesting part is that one would need to
recompile it after building the new version of its dep ("old-time"
here) but before installing it in place of the old binary, since after
reinstalling old-time-1.0.0.2 and before installing the new Cabal, no
package can be installed, including Cabal itself.
That's rather tricky to do it, and it can't be done by hand. And cabal
is probably unique in having this need - I've used CPAN, Gentoo's
portage, and so on, and for various reasons (stable ABIs, no
compilation going on, etc.) this scenario can't probably happen.

After adding "preference: old-locale == 1.0.0.1", the output of $
cabal install -v pandoc-1.5 became the one in
inst-pandoc-1.5-with-pref-old-loc-_1.txt, where no upgrade of
old-time/old-locale is needed. Interestingly, HTTP, directory,
process, zip-archive were not reinstalled, which confirms that Cabal
had reinstalled them before just because of an upgrade to the
dependencies.

I managed to successfully remove old-locale-1.0.0.2 from my system and
recompile all broken dependencies, even if it was rather tricky - even
when ghc-pkg reported no errors, several dependencies were still
broken, and the error message was crazy (sadly, I lost the log).

Additionally, I could have a security breach in a Haskell library and
want to guarantee an upgrade - not all vulnerabilities are C-specific
(take cross-site scripting, SQL injection, bad handling of temporary
files for setuid programs, bad validation of user input for a mail
server...). Like this:
http://www.amateurtopologist.com/2010/04/23/security-vulnerability-in...

ghc-pkg doesn't uninstall packages, it just deregisters them and
doesn't remove the files. Indeed, there's already a ticket open for
having 'cabal uninstall':

http://hackage.haskell.org/trac/hackage/ticket/234

I wanted to add is that I discovered this with an error similar to:

<command line>: unknown package old-locale-1.0.0.2

where the mentioned module had just been unregistered. BTW, there's no
obvious way to re-register a module because the file to pass to
ghc-pkg register is not saved on disk.

Below is the list of packages that it would have tried to upgrade. You can use
the 'install' command to install the ones you want. Note that it is generally
not recommended to upgrade core packages.

It is noteworthy that "cabal install --help" has no such warning.

Anyway, package manager Gentoo and *BSDs can upgrade everything while
the system is running. It would be nice to do the same here, including
"cabal install ghc", but "if you want, you can fix it" would be an
appropriate response - I kind-of know I won't get to do that, likely,
at least not while starting my PhD.

$ cabal install --dry cabal-install leksah-0.8.0.6
Resolving dependencies...
cabal: cannot configure cabal-install-0.8.2. It requires Cabal ==1.8.*
For the dependency on Cabal ==1.8.* there are these packages: Cabal-1.8.0.2,
Cabal-1.8.0.4 and Cabal-1.8.0.6. However none of them are available.
Cabal-1.8.0.2 was excluded because Cabal-1.6.0.3 was selected instead
Cabal-1.8.0.2 was excluded because ghc-6.10.4 requires Cabal ==1.6.0.3
Cabal-1.8.0.4 was excluded because Cabal-1.6.0.3 was selected instead
Cabal-1.8.0.4 was excluded because ghc-6.10.4 requires Cabal ==1.6.0.3
Cabal-1.8.0.6 was excluded because Cabal-1.6.0.3 was selected instead
Cabal-1.8.0.6 was excluded because ghc-6.10.4 requires Cabal ==1.6.0.3

That's on ghc-6.10.4 with Cabal-1.8.0.6.
However, trying to install cabal-install and leksah separately works quite well.
Indeed, they are already installed, but since they are not tracked by
ghc-pkg, cabal forgot that it installed them, and keeps forgetting
that (as I just discovered).

   http://hackage.haskell.org/trac/hackage/ticket/701
   http://hackage.haskell.org/trac/hackage/ticket/704

When you asked cabal-install to install pandoc, it tried to make a
consistent install plan for all its transitive dependencies.
cabal-install will not touch a package which is not a transitive
dependency of the package you request to be installed. Therefore,
cabal-install will not touch Cabal if you ask it to install pandoc.

To make a consistent install plan, cabal-install has to pick exactly one
version number for each of the transitive dependencies, so that all
version constraints of all transitive dependencies are fullfilled. For
some reason, cabal-install picked old-locale-1.0.0.2 instead of the
already installed old-locale-1.0.0.1, and newer versions of HTTP,
directory etc. too.

I think this is the bug: cabal-install should not be allowed to install
old-locale, because doing so apparantly causes havoc.

Looking at the inter-dependencies of pandoc's transitive dependencies, I
do not see a reason to install a new version of a package instead of
keeping the old. Maybe it's somehow related to the transition from
base-3 to base-4? But I don't know how cabal-install decides which
install plan to follow anyway.

cabal install p1 p2 is supposed to find a single consistent install plan
for p1 and p2 and the transitive dependencies of either of them. This is
useful if you plan to use p1 and p2 in a single project.

   Tillmann

First, sorry for some confusion - I wanted to post further details
needed for the analysis in the mail I lost, and I had excluded them
from my summary to keep it short.

On Sun, Sep 12, 2010 at 15:26, Tillmann Rendel

Recompiling the same version of a package does not always yield the
same result, ABI-wise.

The problem was an ABI difference between two compilation results for
old-time-1.0.0.2, one linked against old-locale-1.0.0.1, the other
against version 1.0.0.2 of the same package. The ABI difference was
_probably_ due to different cross-module inlining decisions: the
disappeared symbols were:

oldzmtimezm1zi0zi0zi2_SystemziTime_a149_{closure,info},

i.e. (I assume) old-time-1.0.0.2-System.Time.a149. That name is
generated by GHC during optimization, and my educated guess is that it
is exported to allow cross-module inlining. In particular, a149 is
mentioned in the .hi interface of old-time's System.Time - <libs
dir>/old-time-1.0.0.2/System/Time.hi

It thus _seems_ that the ABI of a module is a (pure) function of the
versions of all its (transitive) dependencies, and their compilation
options - in particular, the optimization options and the used
compiler version.

More formally, my conjecture is:
- let us represent "package a depends on b" as "a =>> b", where a, b
are package names tagged with a version
- let =>>* be the transitive-reflexive closure of =>>
we need to compute:
DEPS(p) = {q | p =>>* q }
TAGGED_DEPS(p) = { (q, compilation_opts(q)) | q \in DEPS(p) }
where compilation_opts(q) are the compilation options used to compile package q.

Then, the ABI of a module is a pure function of TAGGED_DEPS(p), not
just of p. My experience proves at least that the ABI does not depend
just on p.

And since, after the discussion up-to-now, it turns out that this is
unexpected, I reported this as a bug:
http://hackage.haskell.org/trac/hackage/ticket/738

Another result of the above is that a mere "cabal install --reinstall
-O2 FooPackage", recompiling the same version of FooPackage with -O2
instead of the default (-O1), requires recompiling all packages which
depends on (i.e. use symbols from) FooPackage; recompiling them
implies recompiling packages depending on them, and so on, in a
"transitively closed" way.

<BINARY_LIBRARIES>
A final result is that shipping Haskell libraries in binary form (i.e.
closed source libraries) is not supported without major changes. Not
that I care directly, but somebody might, especially since you choose
the BSD license over the GPL one. And fixing that might also make
package upgrades more "sustainable", i.e. allowing to upgrade a
library while rebuilding less dependent modules. To ship a binary
library, one would need:

- disabling cross-module inlining when building the closed source
library, but that's not enough: one still needs to link to a specific
version of a package, because the version number is mangled into the
name, up to the 4th version level. That's inconvenient, because a
library can't benefit from a change affecting only the implementation,
and not the interface, of a library (e.g., a bugfix or a security
fix).

== On .NET, changes to the last version number must be
binary-compatible, so 1.0.0.1 and 1.0.0.2 are equivalent at link time,
and 1.0.0 and 1.0.1 are incompatible but can coexist.

== On ELF, for libraries which care (say glibc), each symbol is
available (with the same ABI) from a given library version onwards;
client binaries depend on at dynamic linking time on a library version

- resorting to full static linking, if that's supported; even then,
one can't statically link to the runtime system, and typechecking
needs to use versioned typenames.

</BINARY_LIBRARIES>

And of course, cabal should prevent a user from shooting himself in
his foot, but that's a more advanced feature.

A consistent plan for installing pandoc would have been to either
compile it using the older old-locale, or to recompile old-time
against the new version. Sadly, cabal chose the latter alternative,
since it is not obviously wrong, and this broke everything, except
packages on which pandoc depended: they were recompiled because of the
changed dep.
I later forced the other install plan, with some pain, but it works.

So cabal behavior is inconsistent: it knows that recompiling a package
means recompiling its dependencies, but it should either do so also
for packages unrelated to the one being installed, or it should not
recompile any package (as you suggest).

Finally, while old-locale is a core package and one can restrict its
upgrades, if I recompile non-core package A, and non-core package B
breaks, that's still "havoc" - the only difference is that B is not
cabal, and the bug is thus less severe and you can recompile manually
B.
OTOH, fixing this less severe and more complicated bug, via
transitively closed dependency tracking as described above, helps
supporting complete upgrades.

Consider a package pair, which defines an abstract datatype of pairs in
version 1:

   module Pair (Pair, fst, snd, pair) where
     data Pair a b = Pair a b

     fst (Pair a b) = a
     snd (Pair a b) = b
     pair a b = Pair a b

In version 2 of pair, the internal representation of the datatype is
changed:

   module Pair (Pair, fst, snd, pair) where
     data Pair a b = Pair b a
     fst (Pair b a) = a
     snd (Pair b a) = b
     pair a b = Pair b a

Now we have a package foo which depends on pair-1:

   module Foo (foo) where
     import Pair

     foo = pair 42 '?'

And a package bar which depends on pair-2:

   module Bar (bar) where
     import Pair

     bar = fst

Now, we write a program which uses both foo and bar:

   module Program where
     import Foo
     import Bar
     main = print $ bar $ foo

Even with the technical ability to link all of foo, bar, pair-1 and
pair-2 together, I don't see how this program could be reasonably
compiled. Therefore, I think that the notion of consistent install plans
is relevant semantically, not just to work around some deficiency in the
linking system.

On 9/11/10 15:43 , Daniel Fischer wrote:

- --
brandon s. allbery     [linux,solaris,freebsd,perl]      allb...@kf8nh.com
system administrator  [openafs,heimdal,too many hats]  allb...@ece.cmu.edu
electrical and computer engineering, carnegie mellon university      KF8NH
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

The issue is how to express or recognise the distinction.

I had this kind of scenario in mind, and that's why I proposed using
versioned typenames for typechecking - your example program would then
be caught as ill-typed. However, that's not enough, because the
correct solution is to use the same pair version.

- OTOH, Program would probably have its own cabal file, which could
maybe list a dependency on pair. But I don't like this solution - the
developer shouldn't have to do this.

BTW, I just realized that thanks to static linking, the ghc and cabal
binaries should never stop working - only using those libraries for
building packages could break. If you always upgrade locally, you can
always remove packages from the user DB as I did.

But actually, on my system I can't see ghc in cabal's DB; I can see
that in the ghc-pkg database for binaries, but there all dependencies
are tight, probably because they are dependencies among installed
packages.

Have a look:
$ cabal info QuickCheck
[...]
    Dependencies:  mtl -any, base >=4 && <5, random -any, base >=3 &&
<4,
                   random -any, base <3, ghc -any, extensible-
exceptions -any
$ ghc-pkg describe QuickCheck | grep depends
depends: base-3.0.3.1 random-1.0.0.1

I would like to share a new very cool result given by the current
Cabal: if packages FOO, BAR and FOOBAR exist, such that FOO depends on
BAR which depends on FOOBAR, and all three are installed, you just
can't safely upgrade FOOBAR. When the same version of BAR is ever
recompiled against the new FOOBAR, its ABI might change and FOO _will_
thus break if you're not very lucky.
Call it the No Upgrade Theorem, if you want.
Note that upgrades of FOO and BAR can still be safe: the key
hypothesis is a dependency chain of 3 packages, and the thesis is that
the deepest of the three ones in the chain (FOOBAR in the above
example) can't be upgraded safely.

Not that it's necessarily Cabal's "fault", IMHO. Given this kind of
cross-module inlining, proper dependency handling (which includes what
we discussed) seems insanely complicated to get right. Of course,
Cabal can be fixed, but I would call that a major achievement.

On Sep 13, 12:53 am, Brandon S Allbery KF8NH <allb...@ece.cmu.edu>
wrote:

To verify this theory, I upgraded "bytestring" (on which GHC but not
Cabal depend), and I can still configure+build+copy packages. But of
course, bytestring fulfills the hypothesis of the above theorem, so
installing a package with enough dependencies might trigger actual
breakage. That's however not because ghc depends on bytestring.

Only Cabal is special, because if you break that you can't recompile
anything any more, and because recompiling Cabal dependencies and
Cabal takes special effort (as said elsewhere). That's why "core
packages" should include just cabal (recursive) deps.

If I've understood correctly, in this series of emails you're pointing
out two problems:

1. upgrading packages can break dependencies (and Cabal does not do a
lot to prevent/avoid this)

2. cabal ought to allow using multiple versions of a single package in
more circumstances than it does now

Both of these issues are known to the Cabal hackers (i.e. me and a few
other people). I'll share my views on the problem and the solution.

1. This is certainly a problem. The current situation is not awful but
it is a bit annoying sometimes. We do now accurately track when
packages get broken by upgrading dependencies so it should not be
possible to get segfaults by linking incompatible ABIs.

My preferred solution is to follow the example of Nix and use a
persistent package store. Then installing new packages (which includes
what people think of as upgrading) become non-destructive operations:
no existing packages would be broken by an upgrade. It would be
necessary to allow installing multiple instances of the same version
of a package.

If we do not allow multiple instances of a package then breaking
things during an upgrade will always remain a possibility. We could
work harder to avoid breaking things, or to try rebuilding things that
would become broken but it could never be a 100% solution.

2. This is a problem of information and optimisitic or pesimistic
assumptions. Technically there is no problem with typechecking or
linking in the presense of multiple versions of a package. If we have
a type Foo from package foo-1.0 then that is a different type to Foo
from package foo-1.1. GHC knows this.

So if for example a package uses regex or QC privately then other
parts of the same program (e.g. different libs) can also use different
versions of the same packages. There are other examples of course
where types from some common package get used in interfaces (e.g.
ByteString or Text). In these cases it is essential that the same
version of the package be used on both sides of the interface
otherwise we will get a type error because text-0.7:Data.Text.Text
does not unify with text-0.8:Data.Text.Text.

The problem for the package manager (i.e. cabal) is knowing which of
the two above scenarios apply for each dependency and thus whether
multiple versions of that dependency should be allowed or not.
Currently cabal does not have any information whatsoever to make that
distinction so we have to make the conservative assumption. If for
example we knew that particular dependencies were "private"
dependencies then we would have enough information to do a better job
in very many of the common examples.

My preference here is for adding a new field, build-depends-private
(or some such similar name) and to encourage packages to distinguish
between their public/visible dependencies and their private/invisible
deps.

On Wed, Sep 15, 2010 at 18:33, Duncan Coutts

I am even ready to send patches.

At some point I unregistered a package with ghc-pkg
(old-locale-1.0.0.2 probably), without using --force, and I started
getting linker errors mentioning it, in a form like:
<command line>: unknown package: old-locale-1.0.0.2
even if old-locale-1.0.0.2 appeared on no command line (not even
internal ones, I checked everything with -v), but was just mentioned
by a package mentioned on the command line of an internal command.

There is a small possibility that this was due to the older Cabal
which was installed with GHC - but IIRC the new cabal was one of the
first packages (or the first) I installed.

E.g. I just made up a syntax for a regexp library, and built a
function which should check if (useless) trailing spaces are present
in some text:
checkNoTrailingSpace:: String -> String
checkNoTrailingSpace = not . (regexpMatch "\s+$")
allowing inlining of such a function would turn a possibly private
dependency on some regexp package into a public one.

However, automatic checking as I proposed (without extra help from
GHC) does not work either, and I show a counterexample, which is also
about "bad library design".
Suppose that V1 of package Foo has functions:

buildFoo bar baz = (bar, baz)
takeBar (bar,baz) = bar

and that V2 of Foo swaps the order of bar and baz in the underlying
pair. The pair representation should be either encapsulated by a data
constructor (but it is not), or part of the API and ABI and thus not
changeable. Today doing this causes no harm, but if linking multiple
versions of Foo were allowed, this would create a nightmare. If a data
constructor where used, versioned typechecking would catch the
problem.

Since these functions could be fully inlined in module Foo2, it
becomes impossible to infer from .hi files of Foo2 which dependencies
are public, unless GHC stores from which modules come bodies of
functions exposed in .hi files.

So, I propose to:
- depending on the solution, possibly educate library developers about
resulting pitfalls, if they are not supposed to write code like the
above.
- extend GHC to produce needed information (if not done)
- use that for automatically checking which dependencies are public
and which are private (at package installation time)

I think it would help to document (prominently) what Cabal
fundamentally doesn't (try to) do, to avoid optimistic
expectations (and hence the surprises when Cabal doesn't
meet those expectations), and to point out the design choices
behind many bug tickets (even if the choices are temporary
and driven by limited manpower). Such as

- cabal doesn't keep track of what it installs, hence

    - no uninstall
    - no application tracking
    - library tracking and information about installed
        library configurations only via ghc-pkg
    ..

- cabal's view of package interfaces is limited to explicitly
    provided package names and version numbers, hence

    - no guarantee that interface changes are reflected in
        version number differences
    - no detailed view of interfaces (types, functions, ..)
    - no reflection of build/configure options in versions
    - no reflection of dependency versions/configurations
        in dependent versions
    - no knowledge of whether dependencies get exposed
        in dependent package interfaces
    ..

This is just to demonstrate the kind of information I'd like
to see - Duncan&co know the real list, though they don't
seem to have it spelled out anywhere? So users see that
it works to say "cabal install" and build up their own often
optimistic picture of what (current) Cabal is supposed to
be able to do. It might even be useful to have a category
of "core-tickets" or such on the trac, to identify these as
work-package opportunities for hackathons, GSoC and
the like, positively affecting many tickets at once.

On to the specific issue at hand:

Even for monolithic ABI hashes, it might be possible to compute
the package ABI hash a second time, setting the versions of all
dependencies declared to be private to 0.0.0 and seeing if that
makes a difference (if it does, the supposedly private dependency
leaks into the package ABI, right?).

And secondly, how about making it possible for cabal files to
express sharing constraints for versions of dependencies?

To begin with, there is currently no way to distinguish packages
with different flag settings or dependency versions. If I write a
package extended-base, adding the all-important functions car
and cdr to an otherwise unchanged Data.List, that package will
work with just about any version of base (until car and cdr
appear in base:-), but the resulting packages may not be
compatible, in spite of identical version numbers.

If it was possible to refer to those package parameters (build
options and dependency versions), one could then add constraints
specifying which package parameters ought to match for a build
configuration to be acceptable.

Let us annotate package identifiers with their dependencies
where the current lack of dependency and sharing annotations
means "I don't care how this was built". Then

    build-depends:  a, regex

means "I need a and regex, but I don't care whether a also uses
some version of regex", while

    build-depends: a, regex
    sharing: a(regex)==regex

would mean "I need any version of a and regex, as long as a
depends on same the version of regex I depend on" (choose
any syntax that works). And

    build-depends: a, b
    sharing: a(text)==b(text)

would mean "I need any version of a and b, as long as they
both depend on the same version of text".

I can already think of situations where one might want more
complex constraints (e.g. "I want any version of base and mtl,
but either both have to be before the Either-move, or both
have to be after the move" - this part could be expressed
already, but there is currently no way to enforce this
transitively, for dependencies, so if I depend on c, which
depends on a new base, and on d, which depends on an
old mtl, I'm out of luck, I think). Or: "never mix mtl and
its alternatives".

Of course, this scheme depends on whether it is possible
to reverse engineer the information about dependencies of
installed packages from ghc-pkg info or ABI hashes.. but
if the general idea is sound, perhaps that could be made
possible?

Just a suggestion,
Claus