llvm_mode Makefile CC/CXX version

[cha...@ceriksen.com]

Hi!

I've been playing around more with the new LLVM mode. However on many distros, llvm-config, clang, and clang++ aren't symlinked directly. Instead, you have to use the version-specific one, which is nice. But the makefile for llvm_mode doesn't allow for overriding CC and CXX without editing the make file(It does however for llvm_config, so that you can use llvm-config-3.5, for instance).

Is there any reason for this? Or would it maybe make sense to allow overriding CC/CXX in the make file to make it easier to override the version which it compiles against?

Thanks!

[Michal Zalewski]

No reason, but I'm very surprised that distros don't give you a
working 'clang' or 'clang++' binary - what the heck?

(The reason we no longer query llvm-config for the paths is that some
folks reported problems with llvm-config reporting wrong locations.)

/mz

> --
> You received this message because you are subscribed to the Google Groups
> "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to afl-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michal Zalewski]

Oh, wait, there's a reason - build tess will fail if afl-clang-fast
can't find clang (and it needs clang, not gcc). I'm inclined to just
keep it as-is, TBH...

[cha...@ceriksen.com]

There actually does seem to be a reason:

Even if you build afl-*-fast with CC/CXX overridden, when you execute the resulting binary it'll try and call clang/clang++ directly. So I had to create a symlink myself to get it to compile.

FWIW, the binaries I'm using are: http://llvm.org/apt/. So it's technically not the distro that's not doing it, but it'd still be nice to target a specific version of clang, even if it's not what's symlinked to clang/clang++.

:-)

[cha...@ceriksen.com]

Seems like Google ate up my reply to the previous reply :(

You are right, it does indeed try to use clang when you then invoke afl-clang-fast. Personally I think it makes sense that you can override the version of clang that's used without changing symlinks, since you may want to target different versions of clang without messing up other things on the system that may use clang. But it's not a huge issue I guess.

Managing symlinks sucks though :-)

[Oliver Schneider]

Hi there,

On 2015-04-13 15:03, Michal Zalewski wrote:
> Oh, wait, there's a reason - build tess will fail if afl-clang-fast
> can't find clang (and it needs clang, not gcc). I'm inclined to just
> keep it as-is, TBH...

Why not make it a conditional assignment then? I.e. ?= ... you're using
it in other places inside the make files already. This would allow to
override it from the command line. Better yet, since this uses recursive
make, let it be passed from the parent make file via some "shuttle"
variable.

Also, there are a few things in the make files that seem a bit strange
or might choke on parallel makes (make -j n ...). For example you'll
usually want to declare:

.NOTPARALLEL: clean
.PHONY: clean

and so on.

And for the clean recipe it would be good to say something like

$(MAKE) -C llvm_mode clean

instead of the existing line. Even with --no-print-directory if preferred.

It _will_ make a difference on systems where several make flavors exist
and GNU make is named gmake, for example (and the top-level make file
gets invoked using 'gmake'). This would also be a very good reason to
rename the Makefile to GNUmakefile, so it doesn't get picked up by any
make flavor. The contents of the make files are already using specific
GNU make functionality.

Are patches to the make file accepted? Or would these proposed changes
be deemed unreasonable?

// Oliver

[Michal Zalewski]

> Why not make it a conditional assignment then? I.e. ?= ... you're using
> it in other places inside the make files already.

Because it will still render the build non-functional unless you also
specify AFL_CC. Perhaps AFL_CC in Makefile is the way to go.

> Also, there are a few things in the make files that seem a bit strange
> or might choke on parallel makes (make -j n ...).

I haven't thought about people doing make -j n clean all. Worth fixing.

> $(MAKE) -C llvm_mode clean

Fair point, will fix.

> Are patches to the make file accepted? Or would these proposed changes
> be deemed unreasonable?

I'll make the changes.

Cheers,
/mz

[Michal Zalewski]

OK, this and the __DATE__ stuff should be taken care of.

/mz

[David A. Wheeler]

On Mon, Apr 13, 2015 at 5:47 AM, <cha...@ceriksen.com <javascript:>>:

> > But the makefile for llvm_mode doesn't allow for overriding CC and CXX without editing

> > the make file...

Stupid question: Is there a reason the "make" command line doesn't work, e.g.:
make CC=my_odd_compiler

According to POSIX, command-line macro definition should override the definition within the file.
Note that this is completely different from environment variable manipulation, e.g.:
CC=my_odd_compiler make

--- David A. Wheeler

[Jakub Wilk]

* Oliver Schneider <afl-...@assarbad.net>, 2015-04-13, 17:03:

>>Oh, wait, there's a reason - build tess will fail if afl-clang-fast
>>can't find clang (and it needs clang, not gcc). I'm inclined to just
>>keep it as-is, TBH...
>Why not make it a conditional assignment then? I.e. ?= ... you're using
>it in other places inside the make files already.

Err, but make sets CC and CXX on its own, so this:

CC ?= clang
CXX ?= clang++

is no-op.

--
Jakub Wilk

[Oliver Schneider]

On 2015-04-13 18:41, Jakub Wilk wrote:
> Err, but make sets CC and CXX on its own, so this:
>
> CC ?= clang
> CXX ?= clang++
>
> is no-op.
>

You are right, at least makes no difference compared to =.

Unless the variables get passed via the command line.

That's why the "shuttle" variable approach would not be prone to the
effect you describe (unless the variables would be defined as 'override
CC=clang'). Example for the "shuttle" approach:

llvm_mode/Makefile:

----
ifdef LLVM_CC
CC := $(LLVM_CC)
else
CC = clang
endif

ifdef LLVM_CXX
CXX := $(LLVM_CXX)
else
CXX = clang++
endif
----

Makefile:

----
LLVM_CC = clang
LLVM_CXX = clang++
.PHONY: llm_mode
llvm_mode:
$(MAKE) -C llvm_mode LLVM_CC=$(LLVM_CC) LLVM_CXX=$(LLVM_CXX)
----

and perhaps put that into an ifeq or ifdef block to add as dependency to
'all'?

Just a thought :)

// Oliver

[Oliver Schneider]

On 2015-04-13 18:08, David A. Wheeler wrote:
> Stupid question: Is there a reason the "make" command line doesn't work, e.g.:
> make CC=my_odd_compiler
>
> According to POSIX, command-line macro definition should override the definition within the file.
> Note that this is completely different from environment variable manipulation, e.g.:
> CC=my_odd_compiler make

From all I see this seems to work.

I was under the impression this was due to CC and CXX being set
differently in the Makefile inside the root. But that's not the case.

With GNU make use:

make -np CC=mytest-clang CXX=mytest-clang++

and pipe to a grep instance, looking for 'mytest'.

// Oliver

[David A. Wheeler]

> On 2015-04-13 18:41, Jakub Wilk wrote:
> > Err, but make sets CC and CXX on its own, so this:
> >
> > CC ?= clang
> > CXX ?= clang++
> >
> > is no-op.

Basically, yes. The "?=" only assigns a macro if it has no current value.
Per POSIX "CC" is defined by default, so it's normally a no-op.
In practice "CXX" always has a value too, so it's also a no-op.

By the way, "?=" has been in GNU Make and *BSD makes forever,
and more recently it's been accepted for inclusion into POSIX (blame me for asking):
http://austingroupbugs.net/view.php?id=330
The "+=" operator in make has also been accepted into POSIX.
They also standardized "::=" as having the GNU Make ":=" semantics;
current GNU make accepts either syntax.
My point is that these are actually standard; see the URL for the details.

--- David A. Wheeler

[Michal Zalewski]

> Basically, yes. The "?=" only assigns a macro if it has no current value.
> Per POSIX "CC" is defined by default, so it's normally a no-op.
> In practice "CXX" always has a value too, so it's also a no-op.

I have a fix for this since yesterday, but didn't want to release this
tiny change alone because I'm then given funny looks by package
maintainers. If nothing else comes up before Wed, I'll probably push
it out.

/mz

[David A. Wheeler]

> On 2015-04-13 18:41, Jakub Wilk wrote:
> > Err, but make sets CC and CXX on its own, so this:
> >
> > CC ?= clang
> > CXX ?= clang++
> >
> > is no-op.

The "?=" only assigns a macro if it has no current value.

Per POSIX "CC" is defined by default, and in practice "CXX" always has a default
value, so those constructs would be no-ops.

By the way, "?=" and "+=" assignments, which have been around forever, have
been accepted for next version of POSIX make (blame me for asking):
http://austingroupbugs.net/view.php?id=330

[Oliver Schneider]

On 2015-04-14 17:09, David A. Wheeler wrote:
>> On 2015-04-13 18:41, Jakub Wilk wrote:
>>> Err, but make sets CC and CXX on its own, so this:
>>>
>>> CC ?= clang
>>> CXX ?= clang++
>>>
>>> is no-op.
>
> Basically, yes. The "?=" only assigns a macro if it has no current value.
> Per POSIX "CC" is defined by default, so it's normally a no-op.
> In practice "CXX" always has a value too, so it's also a no-op.
>
> By the way, "?=" has been in GNU Make and *BSD makes forever,
> and more recently it's been accepted for inclusion into POSIX (blame me for asking):
> http://austingroupbugs.net/view.php?id=330
> The "+=" operator in make has also been accepted into POSIX.
> They also standardized "::=" as having the GNU Make ":=" semantics;
> current GNU make accepts either syntax.

What about $(findstring ...)?

I realize several things that used to be introduced by GNU make made it
into other make flavors, but I think the "problem" (if we want to call
it that) runs deeper here.

// Oliver

[David A. Wheeler]

On Tue, 14 Apr 2015 17:43:17 +0000, Oliver Schneider <afl-...@assarbad.net> wrote:
> What about $(findstring ...)?
> I realize several things that used to be introduced by GNU make made it
> into other make flavors, but I think the "problem" (if we want to call
> it that) runs deeper here.

POSIX's make is notoriously impoverished, making it unnecessarily hard to
create portable makefiles. It doesn't specify portable ways to implement a
variety of important capabilities. I've submitted some bug reports to try to improve things.

If anyone is interesting in improving POSIX' make, I encourage you to get
involved in the POSIX Austin group:
https://www.opengroup.org/austin/

Specifically to your question... I submitted comment #512, "Add macro functions to make",
to add support for the $(function_name ...comma-separated-parameters...) syntax and
some specific functions. The proposal specifically includes findstring. Details here:
http://austingroupbugs.net/view.php?id=512
This proposal is pending; has NOT been accepted NOR rejected.

Others pending comments involving "make" include:
* Add pattern rules: http://austingroupbugs.net/view.php?id=513
* Add make conditionals: http://austingroupbugs.net/view.php?id=805
* Add support for "!=": http://austingroupbugs.net/view.php?id=337
- I'd REALLY like to see this one. GNU make and *BSD makes support it, it's easy to add, it's useful.

I have had some successes. Improvements to make that have been accepted include:
* Add support for assignments "::=", "+=", and "?=": http://austingroupbugs.net/view.php?id=330
* Add support for special target .PHONY in make: http://austingroupbugs.net/view.php?id=523
* Add support for silent include, "-include": http://austingroupbugs.net/view.php?id=333

Since POSIX doesn't even include make conditionals, I don't think it's reasonable for AFL's Makefile
to try to strictly limit itself to POSIX. But if AFL only uses a limited number of extensions, POSIX
may slowly add the functions necessary so AFL's Makefile becomes standards-compliant.

I hope that's useful information.

--- David A. Wheeler

[Jakub Wilk]

* Michal Zalewski <lca...@gmail.com>, 2015-04-14, 10:13:

>>Basically, yes. The "?=" only assigns a macro if it has no current
>>value

>>Per POSIX "CC" is defined by default, so it's normally a no-op.
>>In practice "CXX" always has a value too, so it's also a no-op.
>
>I have a fix for this since yesterday,

The makefile now sets CC/CXX only under this condition:

ifeq "$(findstring clang, $(CC))" ""

This should be good enough in practice, but perhaps it would be more
elegant to use the origin function:

ifeq "$(origin CC)" "default"

--
Jakub Wilk

[Michal Zalewski]

> ifeq "$(origin CC)" "default"

Neat-o, thanks!

/mz