Disable assertions?

[Magnus Lie Hetland]

If I read the code right, there's no standard way to disable assertions, right? Given that this is a rather common functionality in many languages, is this something you have explicitly decided not to support (in the default implementation)? It would be easy enough to roll my own – but if there is good reason not to, then I'd rather not :-)

Whether or not there's a performance hit would depend on how costly your assertion expression is, I guess. If, for example, you check some involved invariant over a large structure, that's something you might want to disable if you were to do a benchmark, for example. (I guess this might be an argument for something like a debug flag in general.) This could certainly be handled by using a test external to the code for this sort of thing, and using assertions only for the minor things. That won't give you the same access, though.

[Tim Holy]

I would bet it's just that no one has gotten around to implementing it yet.
Feel free to take the reigns here. You could create a julia startup option
that causes `@assert` to be defined as a no-op, which would completely
eliminate any performance hit.

--Tim

[John Myles White]

I'd love to have this functionality, but it's worth noting that some code (e.g. Optim) has assert being used for error handling right now. So there'd need to be a period where people transition away from using assertions for error-handling rather than for testing.

-- John

[Magnus Lie Hetland]

Tim: The startup option making @assert a no-op was what I had in mind, yes. Either specifically for this, or it could be linked to testing/debugging in general, perhaps? I saw a mention of a @debug macro somewhere on this list, which I assume doesn't exist either? Maybe both could be turned on by default, but one could have a --no-debug switch or something (similar to the common -DNDEBUG in C/C++, which is also used to nix assertions)? (I guess the assert function should have the same fate as the @assert macro, for consistency? Its arguments would still be evaluated, I guess – the compiler won't optimize away those?)

John: Hm, yes. I don't know – @assert for error handling might not be a bad idea in general? (Sort of minimal DbC support? Although that's also a form of testing, I guess.) Perhaps one could have levels of assertions, sort of like levels of logging? Not to over-design anything or make a huge API out of this, but … one could have two levels, perhaps, where @assert is left as it is (for backward compatibility), and another macro/function pair is added – say, @expect – which is intended to be used for testing/debugging, and which is turned off by the --no-debug switch? Hey, one might even have slightly more configurability, and let the user configure what @expect should do – at least a choice betwen error and warn? (Then silencing it would be the third option. One could even link this stuff to logging, but I guess there's no single standardized logging package yet.) Perhaps configuring this globally is ugly. And perhaps I'm overdoing this completely xD

But, yeah, if we just go for an added switch that lets you turn off asserts (and none of the other stuff, except maybe @debug), what would the situation be in the transition period? (No one would force you to run with the assertions turned off, after all.)

I don't know. This isn't frightfully important to me, but if adding a switch/startup option isn't too involved (haven't looked at that part of the source yet), I could have a whack at it (with some input on what functionality is actually desired, here :-).

[Tim Holy]

On Thursday, July 24, 2014 12:09:59 PM Magnus Lie Hetland wrote:
> I saw a mention of a @debug macro somewhere on this list,
> which I assume doesn't exist either?

@debug is alive and well in Toivo's amazing Debug.jl package. But since
debugging will change radically once Keno's work lands, I wouldn't worry
building a common infrastructure for both @assert and @debug.

And right, the assert function would still have to evaluate its arguments,
even if it becomes trivial.

--Tim

[Magnus Lie Hetland]

On Thursday, July 24, 2014 9:51:57 PM UTC+2, Tim Holy wrote:

On Thursday, July 24, 2014 12:09:59 PM Magnus Lie Hetland wrote:
> I saw a mention of a @debug macro somewhere on this list,
> which I assume doesn't exist either?

@debug is alive and well in Toivo's amazing Debug.jl package. But since
debugging will change radically once Keno's work lands, I wouldn't worry
building a common infrastructure for both @assert and @debug.

Ah, right. Awesome stuff :-) But it's quite different from what I meant – I was just thinking of a macro for marking code that would be nulled/removed if we're not running in debug mode (or whatever you want to call it). So that assertions would be just a special case. I guess this is sort of what -DNDEBUG (or, for that matter, -DDEBUG) in C/C++, or the __debug__ constant in Python, is used for.

(Slightly rambling email below…)

The Python setup is basically that __debug__ is true if you start Python without the -O switch. The assert statement, then, is then wrapped in an if __debug__ statement. This seems to me to be a conservative and sensible setup, which could be separate from Debug.jl, though, of course, if we had something like __debug__, it would be possible for Debug.jl to use that to switch things off. I guess the “magic names” of Python aren't that appropriate; perhaps it could simply be Base.debug?

There would really be no reason for the kind of @debug macro I was talking about, as one could just use if statements (as in Python), without any overhead … maybe? Maybe not? Seems OK for simple cases:

julia> const debug = false

false

julia> f() = if debug 42 else 41 end

f (generic function with 1 method)

julia> code_llvm(f, ())

define i64 @"julia_f;18972"() {

top:

  ret i64 41, !dbg !772

}

But if I add some more …

julia> function f()

           if debug

               print("Debug!")

           end

           41

       end

f (generic function with 1 method)

julia> code_llvm(f, ())

define i64 @"julia_f;18980"() {

top:

  %0 = alloca [4 x %jl_value_t*], align 8

  %.sub = getelementptr inbounds [4 x %jl_value_t*]* %0, i64 0, i64 0

  store %jl_value_t* inttoptr (i64 4 to %jl_value_t*), %jl_value_t** %.sub, align 8

  %1 = load %jl_value_t*** @jl_pgcstack, align 8, !dbg !796

  %2 = getelementptr [4 x %jl_value_t*]* %0, i64 0, i64 1, !dbg !796

  %.c = bitcast %jl_value_t** %1 to %jl_value_t*, !dbg !796

  store %jl_value_t* %.c, %jl_value_t** %2, align 8, !dbg !796

  store %jl_value_t** %.sub, %jl_value_t*** @jl_pgcstack, align 8, !dbg !796

  %3 = getelementptr [4 x %jl_value_t*]* %0, i64 0, i64 2

  store %jl_value_t* null, %jl_value_t** %3, align 8

  %4 = getelementptr [4 x %jl_value_t*]* %0, i64 0, i64 3

  store %jl_value_t* null, %jl_value_t** %4, align 8

  %5 = load %jl_value_t** %2, align 8, !dbg !797

  %6 = getelementptr inbounds %jl_value_t* %5, i64 0, i32 0, !dbg !797

  store %jl_value_t** %6, %jl_value_t*** @jl_pgcstack, align 8, !dbg !797

  ret i64 41, !dbg !797

}

I can see that this might be left to LLVM, but the native code for the two is still pretty different. So … it seems the if-statement does give some overhead? In that case, we might need some macro anyway, for this kind of functionality, perhaps. Maybe @ifdebug?

I don't know. Initially, I guess having just a global constant indicating whether we are in a more tentative mode of running the code (whether we call it debug or something else) or if we're running things in a more optimized setting, coupled with a code-expansion-level if-statement for @assert (and a runtime if-statement for assert) would be an OK start.

Judging by the precedence established by the current switches (such as --check-bounds={yes|no}) perhaps making the functionality more specific might even be desired, I guess, and limiting this to only assertions, with --assertions={yes|no}, perhaps? Less stepping on Debug.jl's (or LLVM dbg's) toes, maybe?

[Tim Holy]

I suspect you may be interested in the Expr "meta" proposal. Start reading
approximately here:
https://github.com/JuliaLang/julia/pull/3796#issuecomment-21428185

On Friday, July 25, 2014 03:17:06 AM Magnus Lie Hetland wrote:
> On Thursday, July 24, 2014 9:51:57 PM UTC+2, Tim Holy wrote:
> > On Thursday, July 24, 2014 12:09:59 PM Magnus Lie Hetland wrote:
> > > I saw a mention of a @debug macro somewhere on this list,
> > > which I assume doesn't exist either?
> >
> > @debug is alive and well in Toivo's amazing Debug.jl package. But since
> > debugging will change radically once Keno's work lands, I wouldn't worry
> > building a common infrastructure for both @assert and @debug.
>
> Ah, right. Awesome stuff :-) But it's quite different from what I meant – I
> was just thinking of a macro for marking code that would be nulled/removed
> if we're not running in debug mode (or whatever you want to call it). So
> that assertions would be just a special case. I guess this is sort of what
> -DNDEBUG (or, for that matter, -DDEBUG) in C/C++, or the __debug__ constant

> in Python <https://docs.python.org/3/library/constants.html>, is used for.

>
> (Slightly rambling email below…)
>
> The Python setup is basically that __debug__ is true if you start Python
> without the -O switch. The assert statement

> <https://docs.python.org/3/reference/simple_stmts.html#assert>, then, is

> then wrapped in an if __debug__ statement. This seems to me to be a
> conservative and sensible setup, which could be separate from Debug.jl,
> though, of course, if we had something like __debug__, it would be possible
> for Debug.jl to use that to switch things off. I guess the “magic names” of
> Python aren't that appropriate; perhaps it could simply be Base.debug?
>
> There would really be no reason for the kind of @debug macro I was talking
> about, as one could just use if statements (as in Python), without any
> overhead … maybe? Maybe not? Seems OK for simple cases:
>

> *julia> **const debug = false*
>
> *false*
>
>
> *julia> **f() = if debug 42 else 41 end*
>
> *f (generic function with 1 method)*
>
>
> *julia> **code_llvm(f, ())*

>
>
> define i64 @"julia_f;18972"() {
>
> top:
>
> ret i64 41, !dbg !772
>
> }
>
> But if I add some more …
>

> *julia> **function f()*
>
> * if debug*
>
> * print("Debug!")*
>
> * end*
>
> * 41*
>
> *end*
>
> *f (generic function with 1 method)*
>
>
> *julia> **code_llvm(f, ())*

> might even be desired, I guess, and limiting this to *only* assertions,

[Magnus Lie Hetland]

Looks interesting! Haven't quite absorbed it yet, though :-}

Wrote a simple implementation of a switch (trying to follow the style of what's already in client.jl), and then use the corresponding config value to determine the definition of assert and @assert. However … my straightforward approach would require the code in client.jl to be executed before that of error.jl, which it isn't. Should/could I somehow trigger a redefinition once the code in client.jl has been run?

I'm working on a kind of a solution now, but it seems a bit iffy. A summary:

• I've defined use_assertions and useassertions() – similar to other flags in client.jl – but placed it in base.jl, so it's available when we get to the assertion definition. I've done this rather than place the assertion definitions after client.jl, because assertions are used elsewhere, of course. (Though they may be getting the wrong definitions, now – at least initially).
• I've split out assertions into assert.jl. This is then loaded just after error.jl to keep the parser/compiler happy, but it is then reloaded after client.jl, when useassertions() has its correct value.

Now, I'm guessing this can't work. When @assert is used before client.jl, the first (possibly wrong) definition will be used, and the code will be constructed based on that. That can't be undone when @assert is redefined, I guess. So … what to do? Move the client code earlier? Sneak in a lone option check just for this, early on? Avoid assertions for error checking in the earlier code (if we're talking about using it more for testing purposes anyway), so assertions can be imported only once, after client.jl?

I guess some kind of juggling along these lines (perhaps a combination) would work. Though there are currently (as far as I can see) twenty files that (i) are included before client.jl, and (ii) use assertions (89 uses). Not sure how many of these the client code depend on, though?

Another option: We could have an internal assert that is always used – a renaming of the current assert(s). Then the publicly available one (called assert) would either be a no-op or call the internal one. Then the internal one could be used in the library code. This would mean we couldn't turn off the assertions used in the library, though (or in the code defined before the client runs).

Suggestions/thoughts?

[Ivar Nesje]

I don't think we can really have a runtime flag that affects how Base is parsed, because we cache compiled code to improve startup time. How is theese problems solved with --check-bounds and --int-literals?

[Magnus Lie Hetland]

Ah, right. Yeah, I did look for those in the Julia code, but couldn't find them. I see now that they're in repl.c, where they affect jl_compileropts.

Well, there are two ways of dealing with that part of the issue:

1. Modify repl.c (which I'd need to do anyway, to give the proper help messages, etc.) and do some kind of mojo there.
2. Use a global variable/function (like is_interactive/isinteractive()) that simply modifies assertion behavior – not definition.

This, of course, would add another condition to the assert function, but it shouldn't be a problem for @assert, anyway, if the if-statement is moved inside it. As long as useasserts() has the proper value when the assert macro is executed, it would still generate no code.

Now, assuming I'll want to keep most of this code high-level, and set the value of this variable in client.jl rather than in repl.c, we still have the issue of order of execution, I guess. Here's a proposal to deal with that part: We follow D, which has put some thought into assertions and testing, etc. (They have built-in support for unit test clauses, and design-by-contract clauses with pre- and post-conditions for functions, for example.)

They have two names, like I hinted at before. They have assert for checking program correctness. This is primarily for testing purposes, and although shipping with assertions can be a good idea, it should probably be optional. Then they have enforce, which is for error checking (e.g., for user-supplied values, etc.). It is not possible to turn off enforce – it's basically just an if statement linked to a throw statement.

(They also have static asserts, but I guess we don't need a separate mechanism for that.)

So: How about renaming the current asserts to enforce, and then adding if-statements using the command-line flag to assert? The performance hit would then only be in the version that is designed to be turned off if performance is crucial, and would have zero cost in the macro version (after compilation).

[John Myles White]

I'd be in favor of having both assert and enforce.

 -- John

[Magnus Lie Hetland]

OK, I've got a solution with both assert and enforce, and the --assertions switch working (with all uses of assert replaced with enforce in base, but nowhere else). My version of the specific assert is now:

assert(x) = useassertions() ? enforce(x) : nothing

macro assert(ex,msgs...)

    if useassertions()

        :(@enforce(($ex),($msgs)...))

    else

        :nothing

    end

end

It seems to work, though I'm not sure I got the quoting/unquoting right (or if it's redundant). Seemed necessary to do it like this…? Should I polish this and make a pull request? I guess polishing would include an update to the docs and manpage, at least. Also, I didn't find any tests for assert – just a use of assert() rather than @test in test/core.jl :-} I don't know if I should add tests for this? If so, where? (Testing the command-line switches would be a bit clunky.)

[Magnus Lie Hetland]

Added an issue for this (#7732).