[Haskell-cafe] Current status of Mavericks CPP

[Michael Snoyman]

I still seem to be getting bug reports about the CPP implementation of Mavericks. Last I'd heard, it seemed that general consensus was that packages should *not* be patched to work around the different CPP implementation, and instead Mavericks users should be installing GCC's CPP.

Is this accurate? And is there a wiki page somewhere describing the situation and how to work around it? I'd like to have some authoritative URL to point people to, especially given that I have no access to a Mac system and therefore can't test this myself.

[Roman Cheplyaka]

* Michael Snoyman <mic...@snoyman.com> [2014-04-16 09:05:05+0300]

FWIW, the /topic on #haskell says:

XCode 5 issues? http://www.haskell.org/platform/mac.html

I wonder whether the problem was addressed by GHC 7.8 in any way.

Roman

[Brandon Allbery]

On Wed, Apr 16, 2014 at 2:05 AM, Michael Snoyman <mic...@snoyman.com> wrote:

I still seem to be getting bug reports about the CPP implementation of Mavericks. Last I'd heard, it seemed that general consensus was that packages should *not* be patched to work around the different CPP implementation, and instead Mavericks users should be installing GCC's CPP.

Is this accurate? And is there a wiki page somewhere describing the situation and how to work around it? I'd like to have some authoritative URL to point people to, especially given that I have no access to a Mac system and therefore can't test this myself.

The correct answer is for ghc 7.8 to become established, because it removes the dependency on an external C preprocessor. Some of the reasons for this are:

- with FreeBSD 10 having moved to clang on x86 / x86_64, it is clearly only a matter of time before this becomes more than just a "Mavericks" issue;

- pretty much everyone *except* the Haskell community accepted that expecting cpp to handle anything other than C and derivatives was a mistake back when ANSI C came out;

- even with a K&R-style cpp like gcc -traditional, there are Haskell constructs that can break it; consider that it must know how to parse strings (there are some subtle differences between Haskell and C string syntax) and char constants (did you know that C allows multi-character (char) constants? and pretty much always has?) in order to know when to expand macros. With an ANSI cpp like clang's, it must also know when to interpret # and ## as splices.

--

brandon s allbery kf8nh                               sine nomine associates

allb...@gmail.com                                  ball...@sinenomine.net

unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

[Carter Schonwald]

Further more, once some patches I've cooked up land in 7.8.3, it will be possible to ship ghc with it's own cpp program (or use GCC or clang cpp)
  

Austin's 7.8.2 mavericks build will correctly handle / cope with clangs cpp, but there's a clear concensus to evolve Haskell cpp tooling going forward to avoid these issues in the future 

For those on ghc 7.6 or earlier who can't upgrade to ghc 7.8, the work arounds listed on the platform site are the current options. 

[Kazu Yamamoto]

Hi Michael,

Mavericks users *must* install ghc-clang-wrapper provided by HP:
http://www.haskell.org/platform/mac.html

This scripts passes "-E -undef -x assembler-with-cpp" to CPP (clang)
which solves many problems of CPP.

If ghc-clang-wrapper causes problems with your code, I'd suggest to
fix your code. I'm not sure this actually happens.

Also, Haskell users should install the latest happy and alex.

For more information, please read:

http://www.haskell.org/pipermail/ghc-devs/2013-November/003300.html

--Kazu
_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Brandon Allbery]

On Wed, Apr 16, 2014 at 2:16 AM, Brandon Allbery <allb...@gmail.com> wrote:

- pretty much everyone *except* the Haskell community accepted that expecting cpp to handle anything other than C and derivatives was a mistake back when ANSI C came out;

Having managed to completely omit the point of this observation: the C / C++ / ObjC / etc. communities are under no obligation whatsoever to ensure that their tools do anything sensible with Haskell code. Indeed, strict ANSI compliance prohibits doing so because of lexical differences such as ' being allowed in identifiers and # in operators (or identifiers with -XMagicHash).

[Brandon Allbery]

On Wed, Apr 16, 2014 at 2:28 AM, Kazu Yamamoto <ka...@iij.ad.jp> wrote:

If ghc-clang-wrapper causes problems with your code, I'd suggest to
fix your code. I'm not sure this actually happens.

It does. See my previous messages. Haskell is not C, and some Haskell constructs simply will not work with a program that must lex (and, for ANSI, parse) C code.

[Michael Snoyman]

So if I'm reading this all correctly:

1. Package authors are *not* encouraged to change their code to try and work around this issue.

2. Medium/long term, this won't be a problem at all due to changes in GHC itself.

3. Short term, users need to follow the instructions at [1] if they're using GHC 7.6 (or earlier?).

[1] http://www.haskell.org/platform/mac.html

[Brandon Allbery]

On Wed, Apr 16, 2014 at 2:33 AM, Michael Snoyman <mic...@snoyman.com> wrote:

So if I'm reading this all correctly:

1. Package authors are *not* encouraged to change their code to try and work around this issue.

2. Medium/long term, this won't be a problem at all due to changes in GHC itself.

3. Short term, users need to follow the instructions at [1] if they're using GHC 7.6 (or earlier?).

Pretty much, although practically I suspect some packages *will* be changed to be compliant with clang's cpp simply because the community may not be able or willing to wait until the dependency on a C-parsing tool is removed. And there are some programs on Hackage which are broken even with the clang wrapper script because there is no way to tell clang's cpp not to recognize ANSI C splices (gcc's -traditional disables them) or to accept ' as a possible identifier character (someone earlier today on #haskell pointed out that they need to make sure there's an even number of ' characters on each line when using -XCPP... which makes sense when cpp needs to parse (char) constants).

[Kazu Yamamoto]

Hi Brandon,

>> If ghc-clang-wrapper causes problems with your code, I'd suggest to
>> fix your code. I'm not sure this actually happens.
>
> It does. See my previous messages. Haskell is not C, and some Haskell
> constructs simply will not work with a program that must lex (and, for
> ANSI, parse) C code.

I agree that ghc-clang-wrapper can cause problems and GHC should
provide its own CPP. But have you ever experienced such problems
actually?

I have been using Mavericks just after it was released, and using both
GHC 7.6.3 and GHC 7.8 (head in the past) with ghc-clang-wrapper/new
alex/new happy. I have not experienced such problems so far.

[Brandon Allbery]

On Wed, Apr 16, 2014 at 2:42 AM, Kazu Yamamoto <ka...@iij.ad.jp> wrote:

>> If ghc-clang-wrapper causes problems with your code, I'd suggest to
>> fix your code. I'm not sure this actually happens.
>
> It does. See my previous messages. Haskell is not C, and some Haskell
> constructs simply will not work with a program that must lex (and, for
> ANSI, parse) C code.

I agree that ghc-clang-wrapper can cause problems and GHC should
provide its own CPP. But have you ever experienced such problems
actually?

I personally have not, but that's largely because I haven't really ever needed to use -XCPP. I did earlier today talk with someone on IRC who was running into one lexical incompatibility (Haskell's allowing ' to be an identifier character), and have in the past helped diagnose issues with the clang wrapper not working with some programs on Hackage which are incompatible with ANSI C (that ' issue, plus the use of # and ##).

And I consider it nonsensical that people who need to use -XCPP to deal with different library versions or language extensions are restricted to that subset of Haskell which is lexically compatible with C. The problem here is not those people's code; it is that we are relying on a tool which by its nature lexes C code, and using it on Haskell code.

[Ben Franksen]

Brandon Allbery wrote:
> On Wed, Apr 16, 2014 at 2:42 AM, Kazu Yamamoto <ka...@iij.ad.jp> wrote:
>> >> If ghc-clang-wrapper causes problems with your code, I'd suggest to
>> >> fix your code. I'm not sure this actually happens.
>> >
>> > It does. See my previous messages. Haskell is not C, and some Haskell
>> > constructs simply will not work with a program that must lex (and, for
>> > ANSI, parse) C code.
>>
>> I agree that ghc-clang-wrapper can cause problems and GHC should
>> provide its own CPP. But have you ever experienced such problems
>> actually?
>>
>
> I personally have not, but that's largely because I haven't really ever
> needed to use -XCPP. I did earlier today talk with someone on IRC who was
> running into one lexical incompatibility (Haskell's allowing ' to be an
> identifier character), and have in the past helped diagnose issues with
> the clang wrapper not working with some programs on Hackage which are
> incompatible with ANSI C (that ' issue, plus the use of # and ##).
>
> And I consider it nonsensical that people who need to use -XCPP to deal
> with different library versions or language extensions are restricted to
> that subset of Haskell which is lexically compatible with C. The problem
> here is not those people's code; it is that we are relying on a tool which
> by its nature lexes C code, and using it on Haskell code.

I couldn't agree more. My experience in this area is very bad. I am
maintaining a language that is very C-like (it has *almost* the same lexical
syntax), and still I regularly encounter problems when CPP is used as a
preprocessor for it, due to sudden changes that are made e.g. by the GNU
folks that are compatible with C but not other languages.

Furthermore, I strongly suggest that a suitable replacement for CPP should
be designed in such a way that its specification can be added to a future
Haskell language standard. This is the only way to ensure that
implementations other than GHC can be brought along.

Has nobody ever written a paper about how to extend Haskell with the few
features we actually need from CPP? I think general macro replacement like
CPP offers is not on the list, right? IIUC, CPP is used in Haskell libraries
almost exclusively for conditional compilation. I think the syntax should be
based on pragmas, like in

{-# DEFINE WITH_FEATURE_X True #-}

{-# IF WITH_FEATURE_X && GHC_VERSION >= 708 && GHC_VERSION < 712 #-}
...
{-# ELSIF <some other condition> #-}
...
{-# ENDIF #-}

That's four new pragmas (IF, ELSIF, ENDIF, DEFINE). Expressions (in DEFINE,
IF and ELSIF) must have type Bool (True, False) or Integer (decimal notation
only), identifiers must be ASCII (letter, followed by any amount of '_' or
letter or decimal). Writing an evaluator for such expressions is an exercise
for beginners.

There is no need for such a CPP-replacement to be able to parse (nor even
lex) all of Haskell; it just needs to understand lexing of Haskell string
literals (so a pramga inside a string literal gets ignored) and comments.
(Hmm, what about TH splices?) Existing Haskell implementations like GHC are
of course free to integrate the preprocessing stage with normal compilation,
making implementation even simpler.

It's all very simple and nicely integrated. It would even be possible to
write a tool that automatically converts existing Haskell modules that now
use CPP directives to the new style (assuming the above is really the subset
of CPP features that is actually used in practice).

Cheers
Ben
--
"Make it so they have to reboot after every typo." -- Scott Adams

[Sven Panne]

2014-04-16 12:34 GMT+02:00 Ben Franksen <ben.fr...@online.de>:
> [...]Has nobody ever written a paper about how to extend Haskell with the few

> features we actually need from CPP? I think general macro replacement like
> CPP offers is not on the list, right? IIUC, CPP is used in Haskell libraries

> almost exclusively for conditional compilation. [...]

Nope: https://github.com/haskell-opengl/OpenGLRaw/blob/master/include/HsOpenGLRaw.h

Perhaps some template Haskell magic might work nowadays, bug there was
no TH at all when the bindings started, and I am not sure if I should
tie this package to GHC. The real solution is generating the binding
via the OpenGL XML spec, anyway, but I somehow have to do some real
(non-Haskell) work for real money. ;-)

My point is: Do we really know that conditional compilation is *the*
use case? I remember that in ancient days macros were used to generate
instances etc. Make things as simple as possible, but not simpler.

[Tobias Florek]

hi,

i may certainly be missing something, as i only used cpp in haskell for
conditional compilation. having said that:

i'd like to have (instead of LANGUAGE CPP) the following extension.

LANGUAGE FilterWithExternalProgram my_cpp

that would make it possible to use a cpp implementation implemented
separately from ghc and would leave it to the ecosystem to implement
best practices (i.e. a haskell implementation of cpp or the proposed
declarative extension).

i _guess_ that should also make it easier to use compile-to-haskell
programs like she. maybe a hypothetical template haskell alternative
might also work with it.

regards,
tobias florek

[Roman Cheplyaka]

It's already possible (although I can't tell the exact flag off the top of
my head — look at the GHC docs for details).

* Tobias Florek <has...@ibotty.net> [2014-04-16 13:29:23+0200]

[Tom Ellis]

On Wed, Apr 16, 2014 at 01:29:23PM +0200, Tobias Florek wrote:
> i'd like to have (instead of LANGUAGE CPP) the following extension.
>
> LANGUAGE FilterWithExternalProgram my_cpp

I really don't want LANGUAGE pragmas running arbitrary programs on my
computer. Regrettably there are already ways to get GHC to do this, but the
fewer the better, in my opinion.

Tom

[Mikhail Glushenkov]

Hi,

On 16 April 2014 13:29, Tobias Florek <has...@ibotty.net> wrote:
> hi,
>
> i may certainly be missing something, as i only used cpp in haskell for
> conditional compilation. having said that:
>
> i'd like to have (instead of LANGUAGE CPP) the following extension.
>
> LANGUAGE FilterWithExternalProgram my_cpp

Use custom preprocessor support in Cabal or 'ghc -cpp -pgmP
/path/to/custom_cpp' .

[Mikhail Glushenkov]

Hi,

On 16 April 2014 14:11, Mikhail Glushenkov

<the.dead....@gmail.com> wrote:
>
> Use custom preprocessor support in Cabal or 'ghc -cpp -pgmP
> /path/to/custom_cpp' .

Correction: A better method is to use '-F -pgmF /path/to/custom_cpp' [1].
Together with the OPTIONS_GHC pragma this gives you exactly what you requested.

[1] http://www.haskell.org/ghc/docs/7.0.3/html/users_guide/options-phases.html#pre-processor

[Ben Franksen]

Sven Panne wrote:
> 2014-04-16 12:34 GMT+02:00 Ben Franksen <ben.fr...@online.de>:
>> [...]Has nobody ever written a paper about how to extend Haskell with the
>> [few
>> features we actually need from CPP? I think general macro replacement
>> like CPP offers is not on the list, right? IIUC, CPP is used in Haskell
>> libraries almost exclusively for conditional compilation. [...]
>
> Nope:
> https://github.com/haskell-opengl/OpenGLRaw/blob/master/include/HsOpenGLRaw.h

Yes, I expected there are libraries out there that use more of CPP's
features, just not very many. And note, I'm not proposing to deprecate CPP
any time soon, just to offer a sane(r) alternative ASAP.

> Perhaps some template Haskell magic might work nowadays, bug there was
> no TH at all when the bindings started, and I am not sure if I should
> tie this package to GHC.

Agreed, you should not have to.

> The real solution is generating the binding
> via the OpenGL XML spec, anyway, but I somehow have to do some real
> (non-Haskell) work for real money. ;-)

I can relate to that, believe me. In the meantime, CPP will stay around for
some time, for special use cases like yours. Its use would just no longer be
encouraged for cases where it is not needed.

BTW, I do not think that mixing conditional compilation and general macro
replacements in the same tool is particularly good design. That CPP is used
for both is an unfortunate accident of history and we should not try to
repeat that.

> My point is: Do we really know that conditional compilation is *the*
> use case? I remember that in ancient days macros were used to generate
> instances etc. Make things as simple as possible, but not simpler.

Well, what everybody said whenever the question came up in the last years
was that this is the killer feature we absolutely can't live without.

But you are right, we should find out!

There are people on this list who grep the whole of hackage in a matter of
minutes whenever someone (seriously) proposes some non-compatible change to
see who many libraries would be affected. (Any volunteers?) My guess is that
no more than a few hands full of the hackage packages use more than
conditional compilation, but I may be completely wrong about that.

Cheers
Ben
--
"Make it so they have to reboot after every typo." -- Scott Adams

[Erik Hesselink]

On Wed, Apr 16, 2014 at 2:48 PM, Ben Franksen <ben.fr...@online.de> wrote:
> There are people on this list who grep the whole of hackage in a matter of
> minutes whenever someone (seriously) proposes some non-compatible change to
> see who many libraries would be affected. (Any volunteers?) My guess is that
> no more than a few hands full of the hackage packages use more than
> conditional compilation, but I may be completely wrong about that.

I think some core packages like base and (IIRC) containers use CPP to
define instances and such.

One conditional compilation feature that's used relatively often and
not covered by your earlier proposal is bounds checks on libraries
through cabal. It could be handled the same way as GHC versions, of
course, but I think the triple of version numbers is a bit more
'correct' and flexible.

Erik

[Ben Franksen]

Erik Hesselink wrote:

> On Wed, Apr 16, 2014 at 2:48 PM, Ben Franksen <ben.fr...@online.de>
> wrote:
>> There are people on this list who grep the whole of hackage in a matter
>> of minutes whenever someone (seriously) proposes some non-compatible
>> change to see who many libraries would be affected. (Any volunteers?) My
>> guess is that no more than a few hands full of the hackage packages use
>> more than conditional compilation, but I may be completely wrong about
>> that.
>
> I think some core packages like base and (IIRC) containers use CPP to
> define instances and such.

Ok, these would still have to use CPP, until we have something more
suitable.

> One conditional compilation feature that's used relatively often and
> not covered by your earlier proposal is bounds checks on libraries
> through cabal. It could be handled the same way as GHC versions, of
> course, but I think the triple of version numbers is a bit more
> 'correct' and flexible.

Can you point me to an example or some documentation? I am not too familiar
with advanced cabal usage...

Cheers
Ben
--
"Make it so they have to reboot after every typo." -- Scott Adams

[Brandon Allbery]

On Wed, Apr 16, 2014 at 9:19 AM, Ben Franksen <ben.fr...@online.de> wrote:

Erik Hesselink wrote:
> One conditional compilation feature that's used relatively often and

> not covered by your earlier proposal is bounds checks on libraries
> through cabal. It could be handled the same way as GHC versions, of
> course, but I think the triple of version numbers is a bit more
> 'correct' and flexible.

Can you point me to an example or some documentation? I am not too familiar
with advanced cabal usage...

http://www.haskell.org/cabal/users-guide/developing-packages.html#conditional-compilation

which is actually the most common use of -XCPP in my experience, so it really needs to be supported.

[Brandon Allbery]

On Wed, Apr 16, 2014 at 7:24 AM, Sven Panne <sven...@gmail.com> wrote:

My point is: Do we really know that conditional compilation is *the*
use case? I remember that in ancient days macros were used to generate
instances etc. Make things as simple as possible, but not simpler.

It's the most common one but pretty sure it's not the only one --- and, supporting conditional compilation may involve only a tiny subset of cpp in the user's code, but a fair bit more in terms of what cabal-install has to generate for those triplet macros to exist for user code to use. There are ANSI features we don't need (assertions, token splicing) but everything else probably gets used one way or another.

[Brandon Allbery]

On Wed, Apr 16, 2014 at 10:00 AM, Brandon Allbery <allb...@gmail.com> wrote:

are ANSI features we don't need (assertions, token splicing) but everything else probably gets used one way or another.

And, come to think of it, I can't swear someone's debugging messages module doesn't use token splicing somehow to generate strings for Debug.Trace or error....

[Ben Franksen]

Brandon Allbery wrote:
> On Wed, Apr 16, 2014 at 9:19 AM, Ben Franksen
> <ben.fr...@online.de>wrote:
>
>> Erik Hesselink wrote:
>> > One conditional compilation feature that's used relatively often and
>> > not covered by your earlier proposal is bounds checks on libraries
>> > through cabal. It could be handled the same way as GHC versions, of
>> > course, but I think the triple of version numbers is a bit more
>> > 'correct' and flexible.
>>
>> Can you point me to an example or some documentation? I am not too
>> familiar with advanced cabal usage...
>
>
> http://www.haskell.org/cabal/users-guide/developing-packages.html#conditional-compilation
>
> which is actually the most common use of -XCPP in my experience, so it
> really needs to be supported.

I see. Thanks. I'll have to think some more about how to support that w/o
unduly complicating everything.

[Mike Meyer]

On Apr 16, 2014 5:35 AM, "Ben Franksen" <ben.fr...@online.de> wrote:
> Furthermore, I strongly suggest that a suitable replacement for CPP should
> be designed in such a way that its specification can be added to a future
> Haskell language standard. This is the only way to ensure that
> implementations other than GHC can be brought along.

Do we need a Haskell-specific language preprocessor, or just one that's designed to be a general purpose preprocessor instead of specific to some language that's not Haskell? For instance, m4 should be available on most Unix-like systems, and wouldn't have most of the problems I've seen mentioned here. Not allowing single quotes in its variable names would still happen, but those should be lexically distinct from Haskell variables.

[malcolm.wallace]

My experience with receiving bug reports for cpphs is certainly that Haskell people use every imaginable feature of cpp.  Initially, cpphs supported only conditional compilation.  But then some macros started to appear, so I made that a separate pass.  And then the conditional compilation started to depend on the values of the macros, so I had to fold the passes together somehow.  And so on.  I can confirm that token splicing is definitely used by someone, because I received the feature request.

Regards,
    Malcolm

[Carter Schonwald]

I'd like to point out that cpphs is a thing.  And that the near term plans for ghc are likely to lean on cpphs. 

A huge constraint is that ghc and many other Haskell code bases use cpp, and unless we migrate everything any near term choice will essentially be a Haskell aware cpp.  

[Vincent Hanquez]

On 2014-04-16 15:42, Carter Schonwald wrote:
> I'd like to point out that cpphs is a thing. And that the near term
> plans for ghc are likely to lean on cpphs.
>
> A huge constraint is that ghc and many other Haskell code bases use
> cpp, and unless we migrate everything any near term choice will
> essentially be a Haskell aware cpp.

I don't think anyone want to remove cpp anyhow in a near term.
I think this exercise is about offering another preprocessor that would
be more suited to preprocess haskell (HPP).

Once ghc (or some standard) add this {-# LANGUAGE HPP #-} feature,
projects can start migrating to the saner solution in their own time.

--
Vincent

[Carter Schonwald]

That experimentation will become much easier in 7.8.3 :-)

Because some patches that decouple cpp from the c compiler should land