[Proposal] Protected/Private modules

[Maciej Kaszubowski]

Hello, 

What?

I would like to propose introducing a possibility to make a module protected/private. Functions from such module would not be visible outside of the OTP application they are defined in. 

Why?

Currently, all modules included in the release are globally visible. This makes it harder to enforce correct architectural boundaries because we have no support from the compiler. We can only enforce the boundaries by being careful or by running external scripts, but both solutions fall short when the developers are under pressure or before deadlines. 

It would be nice if it was possible to create a module which can only be accessed from the inside of a library/application where it's defined. We have private functions, so it would be nice if we could do the same for modules which are one abstraction level higher. This would allow to clearly define a public interface for libraries/applications which would result in better design. Since one of the recent Elixir goals is to help creating maintainable software, I think this feature would be a really good step in this direction.

Issues

Proposed behaviour could be problematic due to the fact that all Elixir modules compile to Erlang modules which are all public. I came up with three possible ways to handle this:

1. Compile all modules modules as usual (resulting in public Erlang modules), but have Elixir compiler fail when the function from private/protected module is called. 

2. Don't create Erlang modules from private/protected Elixir modules and "copy" the functions to public modules that use them. 

3. Treat all modules are public and use mix xref task to validate this behaviour outside of the compilation step.

All of the solutions have advantages and disadvantages and maybe there are some others which I didn't think of. 

I'll be happy to know what you think about this. 

Cheers,

Maciej

[Paul Schoenfelder]

I think the only viable way to do something akin to "protected" modules, would be to have the compiler verify it, but generate public modules - this achieves the goal of protecting against improper use for the most part. That said, there are still ways to violate these boundaries, say by using `apply/3` or by assigning a module name to a binding and then calling a function via the binding. That's just the nature of dynamic languages though; the average case would still be protected anyway.

I'm not sure if this is really the right way to solve this particular problem though - I can't think of a situation I've encountered where I was like "this mistake wouldn't have been made if this module was protected". That said, I don't think it's a bad idea, but since it is not a guarantee and is rather easy to violate, it doesn't feel like the right solution.

Paul

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/ba637c6c-ea48-4bb2-be1f-3758dd901ced%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maciej Kaszubowski]

Sure, I agree with you that it would probably be possible to use such functions anyway, but I don't think it's much of an issue.

The problem I'm trying to solve here is not that calling such function would be "a mistake which could be prevented". My point is not to prevent mistakes, but to have a way to enforce boundaries and clear design of applications. 

I think the analogy to private functions is really helpful to explain this. Calling a function that's private wouldn't usually cause a bug, but doing so makes your module more maintainable and flexible because you can change the implementation details while being sure that nobody relies on. Private/protected modules would have the same effect but on the application, not module level. 

While it will really help to develop client-facing applications, I think it would help maintainers of libraries (including Elixir itself) even more. Just think about how many issues are opened by people using undocumented functions from modules which are used internally but aren't supposed to be called by external clients. This is quite a huge problem because there's no way to know this without looking at the documentation and can be easily missed while reviewing the code. With the proposed feature, we could have a clear distinction (enforced be a compiler) what is safe to call and what is not. If someone would like to call the function anyway (for example by using `apply/3`), this would at least be clear that it's not a correct solution to a problem. 

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[Christopher Keele]

Functions from such module would not be visible outside of the OTP application they are defined in. 

Currently, all modules included in the release are globally visible.

As module references are just atoms, I can't think of a good way to hide the modules, just the functions within them by making them private or undocumented.

For what its worth, when I want to document a module or function but hide it from exdoc, iex, and other tooling, I just do:

defmodule Secret do

  @moduledoc """

  Docs go here!

  """ && false

end

If such a module falsified all its docs as well as its moduledoc, I suspect you'd get many of the benefits of a 'private' module you're seeking without having Elixir have to add new compiler abstractions and diverge substantially from how erlang compiles today.

[Maciej Kaszubowski]

Yes, you can add `@moduledoc false`, but that doesn't really solve the problem. For me this is not enough, just like adding a comment saying "Please don't use this" wouldn't be enough if we didn't have private functions available. 

In my opinion, strong boundaries between components are key to creating maintainable software. Having a strong way to enforce them (by throwing a compile error) would really help. All the other methods (including `@moduledoc false`) rely on "good discipline" of developers - something that quickly disappears before the deadline. 

Now, I know this isn't an easy feature to add and I don't expect it to be added soon, but I strongly believe that it would have a great impact on maintainability of the software we create. For me, we should be really careful with arguments based on the way Erlang works. Erlang is a really mature solution and it's perfectly understandable that the rate of change is slower. In Elixir, we can move quicker and improve the language. What's more, the change wouldn't have to break backwards compatibility or compatibility with Erlang. The code could be still compiled to public Erlang modules, but Elixir could enforce the boundaries. It's a win-win situation in my opinion. 

[W-M Code]

I also commonly come across modules that are listed in ExDoc but turn out to only be part of the implementation of the 'main' module of the library, as well as the inverse, where I am myself creating a library and I'd want to split stuff up in multiple modules but end up not doing it because I don't want to confuse the user of the library with the extra modules.

More importantly than the second point though, is the fact that I actually do want to have documentation (including possibly doctests) for my 'private' modules' public functions (So functions that are used by other modules in my library). This to:

1. Inform my future self and other contributors to the open source project about design choices made in the implementation of certain functions.

2. Sometimes, test these functions. (There is some arguing about this; some people say that testing should only occur at the most external layer of your library, but I concur because often, libraries end up being nested (without it making sense to split stuff into multiple physical libraries)).

I really would like to use the markdown-syntax for this, and also to generate ExDoc pages that include this information (as well as showing it within IEx). Of course, with a disclaimer that this function is part of a 'private' module and should not be used by consumers of the library but only by the library itself.

 @Christopher Keele's trick to still use Markdown-syntax is interesting, but since it is not actually compiled to any documentation I don't think it is particularly useful.

Personally, I do think that having a way to annotate modules as protected or private (Possibly, protected is the more accurate name for this functionality?) that would:

- Generate disclaimers on top of documentation, possibly hiding the actual documentation behind a 'spoiler'-like tag to make sure that people don't glance over the disclaimer.
- Show references to the modules in a different color (i.e. 'faded out') in ExDoc (and possibly IEx) to make sure that people see that these modules are uninteresting for the outside world.

This would already protect against 95% of the 'bad use'. In the end, if a user wants to do something what is bad for them and we have informed them that it is bad, it is their own choice, after all.

And yes, maybe we could also have the compiler throw warnings when a user does decide to depend on an 'internal' function. But I think that is all that is needed.

~Wiebe-Marten Wijnja/Qqwy

[Maciej Kaszubowski]

These are really good examples, all the ideas about ExDoc are great. 

I also agree with the point about warnings. It might be actually even better then throwing a compilation error (at least for now). Having compiler throw warnings in this case would make it much easier to enforce boundaries (because we already have a way to treat warnings as errors) while keeping the compatibility with Erlang. I think it would be a reasonable step forward.

[Wojtek Mach]

(I tried sending below message to the list but is still doesn't show up so if it eventually does: sorry for dup!)

There is already kind of a notion of protected module in Elixir: a module with `@moduledoc false`. Such module is e.g. not autocompleted in IEx. You're right however that all modules are globally accessible.

I've recently encountered a SO answer [1] suggesting to use undocumented OTP :ram_file module and a prompt comment that since it's undocumented it *shouldn't* be depended upon. Thus, I think it's a good idea for xref to generate a warning in case of calling function from @moduledoc/@doc false outside of the OTP app. Especially if/when all BEAM languages store documentation chunks the same way. And of course, apply/3 is an escape hatch to silence the warning.

I started looking into adding the warning into xref [2] and it looks pretty promising. One thing missing is my early implementation still emits warning for undocumented function even if it's inside the same OTP app - something I want to fix in the final version.  

[1] https://stackoverflow.com/a/22809265

[2] https://github.com/wojtekmach/elixir/tree/wm-xref-undocumented

[Myron Marston]

I'm in favor of this proposal. In fact, I proposed something similar awhile back:

https://groups.google.com/d/msg/elixir-lang-core/X18SZnSDW7U/LZm8_8PYBQAJ

[Milad Rastian]

I'm not in favour making decision in IEx base on @moduledoc false or @doc false.

When someone uses them in a current project they won't be able to quickly check the module in IEx and has to enter the full module name or function name.

I understand your needs. However strongly believe that using doc attributes is not a proper way. The intention of doc attributes is to provide document not "hiding" it from others.

If it's decided to go on with this feature I recommend to use different macro(in compare to def and defmodule) or different module attribute.

[Kelvin Raffael Stinghen]

> I'm not in favour making decision in IEx base on @moduledoc false or @doc false.

Me too. I think the point Myron's proposal got is much more interesting. Read the thread for details: https://groups.google.com/d/msg/elixir-lang-core/X18SZnSDW7U/LZm8_8PYBQAJ