Confusing issues regarding module/struct completeness & accessibility.

[Benjamin Scherrey]

I believe the following code examples are all identically equivalent.
I don't see a place in the spec that would cause me to think that any
of these code examples would fail to compile yet the first two fail
and the last two work. I would also suggest that the first two
(especially the first) are far better in terms of being able to reason
about one's code because it doesn't require one to put a small
submodule/structure which is only meant to be meaningful in the
context of the parent module/structure into an independent file or
define it prior to its parent owning module. I know that modules are
technically flat (which is why v4 works) and that we're really just
talking about something that kinda resembles a namespace but it
doesn't change my impression that these are still equivalent code.

Can anyone tell me where in the language specification would make me
understand why these first two fail?
Also, is there any reason we can't support the first version of this code?

Version 1:
file demo.ex
defmodule Demo do

defmodule Demo.Embed do
defstruct local: 0
end

defstruct name: "", embed: %Embed{}
end

scherrey@(none) ~/develop/elixir/demo $ mix compile

== Compilation error on file lib/demo.ex ==
** (CompileError) lib/demo.ex:7: cannot access struct Demo.Embed, the
struct was not yet defined or the struct is being accessed in the same
context that defines it
(elixir) src/elixir_map.erl:58: :elixir_map.translate_struct/4
(stdlib) lists.erl:1353: :lists.mapfoldl/3
(stdlib) lists.erl:1354: :lists.mapfoldl/3

This: "the struct is being accessed in the same context that defines
it" is a pretty big hint but I disagree with it. By time line 7 is
being compiled the Embed struct is already fully defined.

Version 2:
file demo.ex
defmodule Demo do

alias Demo.Embed

defstruct name: "", embed: %Embed{}
end


defmodule Demo.Embed do
defstruct local: 0
end

scherrey@(none) ~/develop/elixir/demo $ mix compile

== Compilation error on file lib/demo.ex ==
** (CompileError) lib/demo.ex:5: Demo.Embed.__struct__/0 is undefined,
cannot expand struct Demo.Embed
(elixir) src/elixir_map.erl:58: :elixir_map.translate_struct/4
(stdlib) lists.erl:1353: :lists.mapfoldl/3
(stdlib) lists.erl:1354: :lists.mapfoldl/3

Version 3:
file demo.ex
defmodule Demo do

alias Demo.Embed

defstruct name: "", embed: %Embed{}
end

file embed.ex
defmodule Demo.Embed do
defstruct local: 0
end

This compiles and works.

Version 4:
file demo.ex
defmodule Demo.Embed do
defstruct local: 0
end

defmodule Demo do

alias Demo.Embed

defstruct name: "", embed: %Embed{}
end

This compiles and works.

I find this quite counter intuitive. Am I missing something obvious?

thanx,

-- Ben Scherrey

[Peter Hamilton]

Nesting defmodule inside another defmodule prepends the outer module. So in the first example you actually defined Demo.Demo.Embed, and that's why it couldn't find Demo.Embed.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACo3ShgwQ5V1bkRQ7_C4A5Kc7y2OGONE%2Bs6wVh7no1rhwzDn3A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[José Valim]

I believe the following code examples are all identically equivalent.

Well, the compiler is explicitly telling you that they are not. :)

There are two concepts to understand in the examples you posted: one of aliasing and the other one is about compile time expansion. In particular, remember that structs are expanded at compilation time.

 

Version 1:

defmodule Demo do
    defmodule Demo.Embed do
        defstruct local: 0
    end

    defstruct name: "", embed: %Embed{}
end

In this example, you are actually defining a struct named Demo.Demo.Embed. In this case, when you try to expand %Embed{}, it doesn't see an Embed alias and then it tries to expand the "root" Embed which does not exist.  That's *not* the error message you reported though. So I am assuming you mixed something up either when copying the code or the error message.

If we fix the code:

defmodule Demo do
    defmodule Embed do

        defstruct local: 0
    end

    defstruct name: "", embed: %Embed{}
end

Then the error is the one you reported:

== Compilation error on file lib/demo.ex ==
** (CompileError) lib/demo.ex:7: cannot access struct Demo.Embed, the
struct was not yet defined or the struct is being accessed in the same
context that defines it
    (elixir) src/elixir_map.erl:58: :elixir_map.translate_struct/4
    (stdlib) lists.erl:1353: :lists.mapfoldl/3
    (stdlib) lists.erl:1354: :lists.mapfoldl/3

And then:

This: "the struct is being accessed in the same context that defines
it" is a pretty big hint but I disagree with it. By time line 7 is
being compiled the Embed struct is already fully defined.

The compiler is correct, regardless of how much you disagree with it. :) Remember that structs are expanded at compile time so when you try to expand %Embed{}, the defmodule macro was also expanded but the module contents was not yet executed/defined.

You can effectively think of your code as running twice: one goes through all the source, doing all the compiler time work, then when you execute it does another pass, running all of the expanded code.

Version 2 will fail since you can't use a struct before you defined it. The next two examples are correct.

[Benjamin Scherrey]

Sorry I screwed up copying Example 1. The internal stuct,  Embed, should not have been prepended with Demo. The error message reflects the error for the correct code but my cut & paste of the example afterwards forgot to cut it off.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAOMhEnwbM%3DfeN%2BWndyOUnXCO0VToG_uMKnjL-N7tQKPvjzjUDg%40mail.gmail.com.

[Benjamin Scherrey]

Jose,

Thank you for the patient explanatory response. I didn't doubt for
a second that the compiler was "correct". My conundrum was twofold,
however. 1) is there some documentation or specification of the
language that I could have read that would have explained this to me,
and 2) Is it necessary that the compiler behave this way and this the
way that we'd want the language to work at all?

Regarding #1 - I just couldn't find it. I understand Elixir is a
young language but I think such an "annotated reference" of the
language will become more and more important if it doesn't yet exist.
No doubt this is further complicated by it's attachment to Erlang. But
I'm hoping I just missed something that you could point me to.

Regarding #2 - Without a clearer understanding of the formal
specification of the language (ala #1) that tells me otherwise, I
still think (your corrected) Version #1 should be legal Elixir but
perhaps just requires a different mechanism for identifying when the
Module/Struct is fully defined. A two pass system could absolutely
still work this way. More importantly, perhaps, is don't you think
Example #1 is a much clearer way for the programmer to express his
intent and shouldn't that be a strong consideration when balancing the
design of a programming language?

You speak as if this is already a settled issue and I completely
understand if I'm just late in the game. (There are other interesting
bits in the language I note but this is one of the more obvious ones
that I think new devs would encounter.) That's why I'm so interested
in #1 so I can get a more formal understanding of Elixir and be a
better contributor. Language design is a concern near & dear to my
heart and Elixir is fraught with potential, cool insights, and
occasional frustrations. Of course if you can't name 10 things you
hate about your favorite language then you certainly don't know it
well enough. More importantly is then knowing why those warts exist
and what tradeoffs were made that got them that way in the first
place. Languages like C++ have some mind twisting "features" and
syntax but one can absolutely trace each one back to first principles
and clearly understand how they got there (irrespective of whether or
not one agrees with them) by reading existing documentation on the
subject. Java and javascript, for example, have no such core
beliefs/principles and suffer greatly as a result.

I'd like to see Elixir's development be more like the former than
the latter. If documentation that satisfies #1 does not yet exist, how
would you recommend someone start to get familiar with the compiler
(and the most relevant aspects of the Erlang/BEAM target) in order to
most effectively familiarize themselves with how the language is
specified (at least in terms of code, which sometimes confuse intent
with practicalities) and implemented?

I see Elixir as the most promising language introduced since
python (of which I am also a great advocate). I could definitely see
it displacing Java, Python, Ruby, & C# while introducing much stronger
anti-fragile abstractions to the industry. I'd like to help as much as
possible in seeing it reach its full potential.

many thanx,

-- Ben Scherrey

> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JF6BK1MkZ_bOUuXH33ranJrWnJGo6KV9wLwPKy%2BTcfTQ%40mail.gmail.com.

[José Valim]

 

    Regarding #2 - Without a clearer understanding of the formal
specification of the language (ala #1) that tells me otherwise, I
still think (your corrected) Version #1 should be legal Elixir but
perhaps just requires a different mechanism for identifying when the
Module/Struct is fully defined. A two pass system could absolutely
still work this way.

Since this seems to be the contention point, let's try to understand why this is the only possible way for this to work. Imagine you have this code:

variable = System.get_env("DEFINE_BAR")

defmodule Foo do

  if variable in ~w(1 true) do

    defmodule Bar do

    end

  end

end

 

If the module Bar was defined during compilation, Bar would be *always* defined, because compilation does not know the value of the variable nor it would evaluate the if (since it depends on runtime values). Code that is expanded, is always expanded, even if inside a conditional. If the module was defined during expansion then the module would always be defined.

In other words, a struct expansion that happens at compile time, cannot depend on values that are in the same context and will be defined only at runtime. That's the same reason why you can't invoke a function in the same context you define it. The reason why multiple modules work is because we first expand both modules and then we execute them. 

While I personally think this behaviour is documented (because the two passes compilation is documented as part of the macro system as well as the fact that structs are expanded during compile time), I would agree it may take a while to build the intuition around it and we could very likely benefit from more examples and discussions.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[alco]

While I personally think this behaviour is documented (because the two passes compilation is documented as part of the macro system as well as the fact that structs are expanded during compile time)

Where exactly is this documented?

[Benjamin Scherrey]

Perhaps I'm being dense but I don't see how your example relates to
mine. The only difference between my working code and non-working code
is that the working code has the sub-module defined prior to the
primary module. They're both fully closed at compile time in both
cases in that there is nothing more about the Demo.Embed struct to be
defined so references to it could be resolved at both compile and run
time.

Can you point me to where this issue of what is resolved at run time
vs compile time is documented?

I'm especially curious about your example code - is that legal elixir
and, if so, is my understanding correct that Foo.Bar is always
compiled but that its accessibility is still determined at run time?
This implies that you are supporting conditional compiling at run time
by allowing a different definition of Bar depending on the state of a
variable at run time. Is that correct?

thanx,

-- Ben Scherrey

> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Jc6%2BF_6ggP8gLAFVvvjtuvc74RBbd2MaOsZH%2Bc_y3gpg%40mail.gmail.com.

[José Valim]

They're both fully closed at compile time in both
cases in that there is nothing more about the Demo.Embed struct to be
defined so references to it could be resolved at both compile and run
time.

Let me be a bit more emphatic. *No, they are not*. That's the assumption I am trying to break from the beginning of the conversation and it keeps coming back. :) There is no such thing as "fully closed at compile time". Modules are defined at *runtime* because, if they were not, the examples I have posted earlier would not work.

 

Can you point me to where this issue of what is resolved at run time
vs compile time is documented?

Only macros and special forms are expanded at compile-time. We cover it in our guides: http://elixir-lang.org/getting-started/meta/quote-and-unquote.html. Saša also has a good series on the topic: http://theerlangelist.com/article/macros_1

 

I'm especially curious about your example code - is that legal elixir

Yes, it is legal Elixir code. The goal of that example was to show places where your intuition does not match how Elixir actually works.

 

if so, is my understanding correct that Foo.Bar is always
compiled but that its accessibility is still determined at run time?

No. Foo.Bar will only be compiled if the conditional evaluates to true. If it does not evaluate to true, Foo.Bar is never compiled and therefore never available. Once again: *modules are defined at runtime, as the code executes*.

 

This implies that you are supporting conditional compiling at run time
by allowing a different definition of Bar depending on the state of a
variable at run time. Is that correct?

No. It is not a different definition of Bar. In my examples, or Bar exists or it does not exist. If I wanted different definitions of Bar, I would write the "if" inside "defmodule Bar".