Introspect other modules at compile time

[Mark Schmidt]

Hi,

I'm currently trying to "optimize" some of our code. The starting point is the following:

defmodule Schema do
  @types [Type1, Type2...]
  ...
end

defmodule Type1 do
  use MyType
  ...
end

defmodule Type2 do
  use MyType
  ...
end

# some more types

That's all fine and doesn't work too bad, but it's a bit annoying to change two places when we add a new type, so I'm trying to improve that point and got stuck.

I think the easiest way would be while compiling the Schema-Module to grep for the source files which are using the common macro and build the types list from that, but that's really dirty and fragile.

What I would like to do is to trigger some sort of callback from the MyType-Macro and "store" it in the Schema-Module. My best idea so far was to store the individual module names in a plain text file (while compiling the individual types) and read the types-List from a that file while compiling the Schema-Module (marked as external_resource). Apparently changes to external_resources during compile time are ignored, so I tried to trigger a recompile of the Schema-Module after each Type-Module was compiled, which also failed (you can't compile an already compiled module during compile time).

One could also solve it at runtime with the on_load-hook and store the list in an agent or so, but 1) I would like to avoid the performance overhead / potential bottleneck, 2) I need to make sure all the modules are loaded.

I'm fully aware, that I'm probably doing it wrong and misusing the macro features, but I'm really curious if there is a way to solve that at compile time?!

Best regards,

  Mark

[Phani Mahesh]

You need to delay compilation of Schema till all modules are compiled. Detecting when all other modules have been compiled is the trickiest part. If that can be done, the rest is fairly straightforward.

A module is not loaded into the VM at compilation if it has not been required and it has not changed since last compilation. So any tricks using the list of loaded modules will not work.

Adding an after_compile hook that recompiles module Schema might be your best bet. Dirty, but might work.

But are you really "cleaning" up the code by doing that? That's for you to decide, after considering the sacrifice in clarity you have to make and the baggage it introduces vs the benifits it provides.

Mahesh

[Booker Bense]

On Friday, December 18, 2015 at 10:08:38 AM UTC-8, Phani Mahesh wrote:

You need to delay compilation of Schema till all modules are compiled.

There is Code.ensure_compiled? for exactly this problem. 

- Booker C. Bense 

[Phani Mahesh]

That only works if you know in advance the modules you are testing for. Here, we need to detect when all modules other than schema have been compiled,  and their names are not known in advance.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/K209eiTNimA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/7ea9c9b4-f680-4ae0-acfe-18d7c39559c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[José Valim]

For what is worth, Phani Mahesh is 100% correct.

It is something not easily achievable in Elixir due to conveniences like the parallel compiler and partial recompilation. Also because modules are loaded only when required, on_load may not work trivially. You could maybe make some of this work but it would likely make things much more complex.

To put it in another way: how many types will you add per week to justify all the engineering and indirection you want to add to the source?

If you are worried about eventually forgetting a type, just add a test that traverses your project "ebin" directory, get all types, and compare with the ones in the schema. It will be a 5 LOC test and the source will remain clean.

[Mark Schmidt]

Thank you all for your feedback. That's very much appreciated, especially because it's kind of a esoteric problem.

@Mahesh: Yes, exactly the trickiest part is where I got stuck.

@Jose: The effort I already put into it, is not at all justified ;-). Testing the ebin content in a test is a good idea! My main intention was to not change multiple places once I need to add another type, which is kind of a smell to me (but that might be my not yet fully converted/extended OO mind speaking).

I wanted to understand the limitations of what you can do at compile time and make sure I don't miss a good way to archive it. I just take this as not feasible and continue loving elixir :D

Best,

  Mark

[José Valim]

I wanted to understand the limitations of what you can do at compile time and make sure I don't miss a good way to archive it.

Yes, precisely. It is not a matter of OO/FP, just a limitation of the toolchain due to other priorities (like faster compilation). 

 

I just take this as not feasible and continue loving elixir :D

 Glad to hear!