Help in understanding Julia's ways

[Julian Manzano]

Hi All,

I have been using C++ and Python for several years and I am very curious about Julia, I've got the REPL working on my workstation and I am really impressed so far with what I've seen.
However there are some design decisions in the language that I fail to understand and I would really appreciate if someone could explain the rationale:

The main point that I fail to understand is the decision not to allow member functions.
The typical explanation that I find everywhere is that Julia designers have chosen all the methods to be external because this is cleaner (specially for mathematical methods where there is no clear owner) and allows for multiple dispatch.
This explanation does not convince me for the following reasons:

1) We can have multiple dispatch a la Julia and still allow types to have methods. These two things seeem independent to me.

2) Dynamic multiple dispatch can also be done as a byproduct of single dispatch using the visitor pattern (C++, Java, etc.), so in that sense, multiple dispatch is not a new feature.

3) Lack of member functions forces all field to be public and therefore I cannot understand how Julia will avoid the object orgy anti-pattern (https://en.wikipedia.org/wiki/Object_orgy)

But hey, Julia still looks great, it is just that I would really like if someone could explain away my concerns, most likely I am missing something here.

Thanks!

Julian

[Jameson Nash]

Just to get it out of the way, I'll point out first that since all of the aforementioned languages are turing-complete, they can all solve all of the same programming problems. Therefore, there is never a question of whether one language can be used to emulate the features of another language. Instead, the question is whether one programming language makes the right paradigms sufficiently more useful or obvious than another language, for achieving the objectives of a particular programming problem.

1) OO-style dispatch is essentially single-dispatch on the first (implicit/hidden) argument. Multiple dispatch is a strict superset of that. So you could provide both, but there's no benefit. It complicates the user's mental model of the language to provide these as two independent features, rather than one unified system.

2) In C++ and Java, the type signature of the dispatch delegate is fully resolved at compile time. By contrast, in Julia, the dispatch occurs at runtime, when the actual type is known. The visitor design is exactly the sort of anti-pattern that Julia seeks to eliminate by removing the forced distinction between multiple dispatch functions (interfaces) and methods associated with types (ref. question #1).

3) As noted by the wikipedia article, this is a design problem, not a language problem. Some languages, like Python and Julia, therefore choose not to hide anything from the user, but simply provide recommendations against certain patterns. In Julia, it is generally discouraged to directly access the fields of an object outside some set of methods that are understood to be implementing the informal API for that type. Similarly, in Python, the convention is to prefix private data with `_`, since in general the dot-oriented access is the approved access API, but the general principle is the same.

[Scott Jones]

On Sunday, July 5, 2015 at 4:24:50 PM UTC-4, Jameson wrote:

Just to get it out of the way, I'll point out first that since all of the aforementioned languages are turing-complete, they can all solve all of the same programming problems. Therefore, there is never a question of whether one language can be used to emulate the features of another language. Instead, the question is whether one programming language makes the right paradigms sufficiently more useful or obvious than another language, for achieving the objectives of a particular programming problem.

1) OO-style dispatch is essentially single-dispatch on the first (implicit/hidden) argument. Multiple dispatch is a strict superset of that. So you could provide both, but there's no benefit. It complicates the user's mental model of the language to provide these as two independent features, rather than one unified system.

2) In C++ and Java, the type signature of the dispatch delegate is fully resolved at compile time. By contrast, in Julia, the dispatch occurs at runtime, when the actual type is known. The visitor design is exactly the sort of anti-pattern that Julia seeks to eliminate by removing the forced distinction between multiple dispatch functions (interfaces) and methods associated with types (ref. question #1).

3) As noted by the wikipedia article, this is a design problem, not a language problem. Some languages, like Python and Julia, therefore choose not to hide anything from the user, but simply provide recommendations against certain patterns. In Julia, it is generally discouraged to directly access the fields of an object outside some set of methods that are understood to be implementing the informal API for that type. Similarly, in Python, the convention is to prefix private data with `_`, since in general the dot-oriented access is the approved access API, but the general principle is the same.

I agree with your first two points, and find Julia to be much more powerful with multiple dispatch instead of single dispatch, however I think Julian's  third point is something that is a problem with Julia (although I believe it could be solved, without too much trouble).

If there were a way to have functions and types (or members of types) that are not accesible outside

the module they are defined in, then one could define an interface, and not have to worry about code breaking the encapsulation.  (For example, I've found hundreds of cases of accesses of .data in strings in

the registered packages, as well as others in different modules in Base)

Scott

[Julian Manzano]

Jameson, Scott

Thanks a lot for your comments.

@Jameson Regarding 1) I agree. Regarding point 2) Single dispatch in C++ and Java is polymorphic and the type is known only at runtime:  Base *p;   if( inputFromUser() ) p = new A; else p = new B;  p->foo(). For A and B subclasses of Base, p will created of type A or B at run time and therefore which implementation of foo() will be called will be decided at run time. Multiple dispatch in C++ and Java, same story but with a composition of several dynamic single dispatches. Still do not see a significant difference apart from the convenience of avoiding method composition.

Regarding point 3) I am not advocating for a solution that will prevent the user from accessing fields. For me it would be nice just to see something where you could at least annotate that some fields as 'internal'. In this way I would know immediately what I can use and what I can not (without breaking invariants or creating unnecessary coupling). As you point out in Python we annotate fields with '__' and Python will mangle those names so if you try to use them naively it will not work. May be in Julia we could annotate a group of fields with 'internal' such that only functions annotated 'impl' can access those fields without getting warnings.  This is even weaker than Scott's proposal because an an owner/implementer of a type you could have access to those fields even outside the module where the type was defined.


My two cents.

Julian

[Mauro]

> Thanks a lot for your comments.
>
> @Jameson Regarding 1) I agree. Regarding point 2) Single dispatch in C++
> and Java is polymorphic and the type is known only at runtime: Base *p;
> if( inputFromUser() ) p = new A; else p = new B; p->foo(). For A and B
> subclasses of Base, p will created of type A or B at run time and therefore
> which implementation of foo() will be called will be decided at run time.
> Multiple dispatch in C++ and Java, same story but with a composition of
> several dynamic single dispatches. Still do not see a significant
> difference apart from the convenience of avoiding method composition.
>
> Regarding point 3) I am not advocating for a solution that will prevent the
> user from accessing fields. For me it would be nice just to see something
> where you could at least annotate that some fields as 'internal'. In this
> way I would know immediately what I can use and what I can not (without
> breaking invariants or creating unnecessary coupling). As you point out in
> Python we annotate fields with '__' and Python will mangle those names so
> if you try to use them naively it will not work. May be in Julia we could
> annotate a group of fields with 'internal' such that only functions
> annotated 'impl' can access those fields without getting warnings. This is
> even weaker than Scott's proposal because an an owner/implementer of a type
> you could have access to those fields even outside the module where the
> type was defined.

My understanding is that fields of types are considered private in Julia
(public is the type itself and its parameters), so really no need to
annotate them further (but issue #1974 makes me worry). Although
sometimes a underscore is used to denote functions/fields as
extra-private. Of course, nothing stops you to access the fields. A
lax approach is often taken by Julia (by choice), so I suspect it will
be difficult to get support for adding language features which are
purely there to impose discipline.

However, there are many places in Base and elsewhere where the
fields-are-private is violated. Scott gave an example and here another
one: there is no accessor methods to get at Expr.head and Expr.args.
This should probably be refactored if someone has some time...

[Scott Jones]

@julian Unlike C++/Java, Julia usually doesn't do run-time dispatching (and when it has to, the performance can drop drastically).

Not only that, because of the JIT nature of Julia, it can take some generic code, that operates on the abstract type Unsigned, for example,

and generate different methods based on whether it is used with UInt8, UInt16, or UInt32.  (I noticed this when writing some Unicode handling code,

that a function that had a test something like `if ch <= 0xff ; return false ; elseif is_surrogate_char(ch) ; ... ; elseif ch > 0x10ffff`, in a loop, it actually generated

3 different methods, so that the UInt8 case, it simply returned false (which I think got inlined also into the calling function), and for the UInt16 case,

it eliminated the check for ch > 0x10ffff, because Julia knew that 0x10ffff is greater than typemax(UInt16).

That is what really brought home to me the power of Julia, because in other languages, I'd have to write the three versions myself.

@mauro, I've seen no indication that fields of types are considered private in Julia, they are frequently accessed far away from where they are implemented.

I also think, from a number of conversations at JuliaCon, that many people would welcome the ability to make things more disciplined.

Remember, my proposal is for *optionally* marking a type, field, function or method as private to a module, so that if somebody tried to access it with

Foo.privatemethod(blah) or str.privatedata, it would give a compiler (or run-time, if the type of str is not known at compile-time) error.

People who don't care that anybody can fiddle with their private parts would just keep working exactly as they do now.

Also, Julia already has the concept of inner and outer constructors, this essentially is no different, methods within a module are like

inner constructors, they would have access to all of the "private" fields, functions, methods, types of the module, just as only inner constructors can use new().

That doesn't prevent you from writing methods that just use the public (even if not exported) types, methods, etc. to extend the interface.

I think also that this would be a fairly simple change to make to Julia, that would break absolutely nothing.

@julian, mangling names etc. I think just makes life more difficult for debugging and a lot of other things, and doesn't even get you the guarantees that a software engineer

would want when trying to make show the correctness of a package.

Also, saying that these "private" methods etc. are only visible within the module isn't really restrictive at all for the author, I've noticed that large modules tend to have a file at the top level

which includes all of the files that make up the implementation, and exports all of the public interface, so this doesn't at all mean that one has to throw everything into one file.

Scot

[Julian Manzano]

On Monday, July 6, 2015 at 3:13:21 PM UTC+2, Julian Manzano wrote:

@ Scott, correct me if I am wrong: you say that fields of types should be accessible only to methods in the module where the type is defined, right?
For example in Julia 0.3.10 I can write:

module X
export myType,foo
type myType
   dontTouchMe
end
foo(x::myType) = x.dontTouchMe
end

using X
z = myType(666)
z.dontTouchMe    # this works in 0.3.10 but you say it shoudl stop working, right?
foo(z)                  # this works

Julian
 

[andrew cooke]

i feel like these conversations are often driven by people who want "new language" to be like whatever language they are most familiar with.  and those people, since they are naturally in an uncomfortable place (away from their "old" language) tend to be the most vocal.

so i just wanted to add a voice saying, no.  i don't want this.  i think the initial design of the language was correct. the added complexity here seems unnecessary to me.  things work just fine in more functional languages that do not have this kind of constraint.  they also work fine in python - if you want to use underscores in names as a sign to others, go ahead.

more than that, there are more important problems still to be addressed.  inheritance of fields, for example.

andrew

[Scott Jones]

On Monday, July 6, 2015 at 9:23:22 AM UTC-4, andrew cooke wrote:

i feel like these conversations are often driven by people who want "new language" to be like whatever language they are most familiar with.  and those people, since they are naturally in an uncomfortable place (away from their "old" language) tend to be the most vocal.

No, my proposal is just about adding an *extra* capability, which is necessary if you want to be able to make any sorts of guarantees about about separation of interfaces from implementations.

As I've already seen a lot of evidence that the "consenting adults" approach is not working in many cases, and there is frequent breakage when anybody changes something inside their module/package,

not expecting that somebody else has used some internal function (I just got bit by this last week, because the JSON.jl package was using two undocumented and unexported functions that were defined in utf16.jl.

Those were not really safe to be called externally, because they make certain assumptions about the type of the arguments, but since julia has no way of marking something as private, and people consistently ignore that something is not part of the interface, breakage occurred.

 

so i just wanted to add a voice saying, no.  i don't want this.  i think the initial design of the language was correct. the added complexity here seems unnecessary to me.  things work just fine in more functional languages that do not have this kind of constraint.  they also work fine in python - if you want to use underscores in names as a sign to others, go ahead.

underscores in names, and name mangling, is no solution, because people will just ignore that if they want to.

Remember, my idea would NOT change any current code, just allow programmers who *need* to have things more disciplined to be able to do so.

 

more than that, there are more important problems still to be addressed.  inheritance of fields, for example.

Being able to make secure, reliable programs in Julia I think is one of the most important issues, if julia is ever going to be used in real critical code.

Inheritance of fields sounds like something where you want "new language" to be like some other language.

[Tom Breloff]

I personally find underscores to be plenty good as "private" indicators.  Yes people may still reach in and access those fields directly, but I bet they'll hesitate a little bit, which is all I ask.  I certainly do NOT want to force people to jump through extra hoops (boilerplate code), but establishing good practice as a community is valuable.  If someone submits a pull request that accesses a field with an underscore directly, give them crap about it.  If there's a good reason: add comments, add tests, and make sure someone is ready to maintain it.

[Scott Jones]

I need a bit more reliability and security than that (remember, we want to be able to write applications for hospitals!).

The problem is, how are you going to police all of the _ accesses, to give somebody crap about it?

Are you going to have to grep all of the registered packages constantly?

I think that the tools need to be in place, so that people who need to control access, can do so.

It won't make a bit of difference to people who don't care about it, so why not have it?

[Julian Manzano]

Thank you all for making the matter clearer. I will use comments/notation to separate interface from implementation.
Julian

[Spencer Russell]

On Mon, Jul 6, 2015, at 10:29 AM, Scott Jones wrote:

The problem is, how are you going to police all of the _ accesses, to give somebody crap about it?

Are you going to have to grep all of the registered packages constantly?

 

This seems like something a Linter could check, and organizations that want to enforce that as a coding standard could certainly do it in an automated way. For that matter the Linter could disallow any directly field access and enforce the use of accessor functions.

 

-s

[Tobias Knopp]

Scott: Though it has never been verbalized in the documentation most people follow the rule that fields are private. Discussions can be found in #1974. Note that for immutable the fields are(!) the interface.

In practice this concept works very well and I have not seen people that are used to Julia worry about it. Its usually only an issue for people coming from C++/Java where one has to worry about visibility.

Note that visibility is not a security feature one should trust. By casting to pointers one can circumvent this in C++

Tobi

[Ismael VC]

Couldn't we just get a warning and let people shoot themselves in their foot if that's what they want?

Something like:

Warning: Using private method/type in module Foo at foo.jl:n.

I hate boilerplate redundancy, also I didn't knew about the "fields of types are private", I do understand that not exported objects are kinda "private".

[Pontus Stenetorp]

On 7 July 2015 at 04:12, Ismael VC <ismael...@gmail.com> wrote:
>
> Couldn't we just get a warning and let people shoot themselves in their foot
> if that's what they want?
>
> Something like:
>
> Warning: Using private method/type in module Foo at foo.jl:n.

I like that you are trying to find some middle ground. But if I am
doing this intentionally the warning will be a constant annoyance, so,
how should I silence it? Yet another command line option so that we
approach gcc/clang with different levels of warnings? Yuck...
Annotations (@silence?) like Java? Double yuck...

Giving users this level of power is something that I am happy with.
Last week it allowed me to add temporarily add a function looking deep
into a composite type from a library when debugging. Sure, it can be
abused, but it can equally well be used properly when necessary.
Sure, it may break, but ultimately it is always up to the caller not
to violate the API. Sure, sometimes base violates this idiom, but
this is most likely due to historic reasons and a lack of a consensus
more than anything else.

In the future, given enough experience and evidence to the contrary, I
would be happy to reconsider my position. But for now, using idioms
like the one below, is how I write my Julia code.

export Weights, W, b, fanin, fanout

immutable Weights{T<:FloatingPoint}
W::Matrix{T}
b::Vector{T}
end
Weights(fanin, fanout) = Weights(rand(fanout, fanin)./1024, zeros(fanout))
W(w) = w.w
b(w) = w.b
fanin(w) = size(W(w), 2)
fanout(w) = size(W(w), 1)

Apologies for the terribly short variable names, it is an example after all.

Pontus

[Tobias Knopp]

This is how most Julia code looks like. The methods are the interface and thus which is exposed in public.

It is pretty natural to program this way when doing generic programming and taking multiple dispatch into account.

I agree that this could be better documented somewhere but the solution is not to bring more complexity into the language. This has also been discussed several times on the mailing list, so I would be very surprised that introducing warnings for field access would gain any acceptance among the core Julia developers.

[Scott Jones]

I don't know, I think all of you have missed my point.

I am *not* saying to change anything at all in the behavior of current Julia programs, only ADDING the capability to keep methods, types, or fields which are not intended for public consumption hidden.

Being able to do this is very important to make any sorts of guarantees about a program.

I've already seen very many cases where things will be broken in the future, because people are accessing internal fields and methods instead of calling exported methods.

I had to do a PR specifically to fix a case where JSON.jl was using an internal function of utf16.jl just this week.

Many times, there are methods that are needed to implement some functionality, but which depend on certain things being set up correctly, which you do not want to expose publicly, that are not part of the

API, or which you plan on totally changing in the future, for example.

The only way to make sure that nobody is using those is to have the language support marking those as private in some fashion, not just some convention that on the whole, doesn't seem to be followed.

@tknopp You said that for immutables, that fields were the interface.  Please explain that to me, because although Julia conflates a number of different things into the idea of "immutable", I'd never heard that,

and don't really see that it could be true.  (Julia conflates the abstract idea of something not being settable, with the implementation detail of storing "bitstypes" directly in an "immutable" object, without boxing).

I can think easily think of cases where you want a read-only object, where none of the properties were visible (all access via methods).  That doesn't change just because Julia can dispatch on multiple arguments,

instead of just the (hidden) "self/this" argument.

@ninjin (Pontus) You said "Sure, it may break, but ultimately it is always up to the caller not to violate the API"

I totally disagree with that.   A language should *help* people be able to write correct code.  I'm not saying that Julia should be like CLU, where you simply couldn't access the implementation from outside the "cluster" (module), but simply that the level of access should be controllable by the author of the module/package.

String handling in julia is a great case for adding the ability to hide the implementation details, and only expose the methods you really want for the API.

Currently there are cases of .data throughout base and  packages, and the fact that UTF16String and UTF32String currently require a trailing 0 when being constructed is also exposed.

Many people have said that Julia's philosophy is one of "consenting adults", but I don't see that at all.  To have "consenting adults", you have to have consent on both sides.

Julia makes it such that it is impossible to prevent any stranger from walking up and fiddling with your "private parts" (and maybe giving you a virus in the form of corrupted data to boot!).

Julian is right about Julia suffering from https://en.wikipedia.org/wiki/Object_orgy  although I don't think that member functions are necessary at all to solve this (just a private keyword, that keeps things visible only within the module (and submodules)

[Tobias Knopp]

Ok I have to say that I am not that sure about the immutable thingy as for the regular type. For me an immutable is just like say an Int32 that also does not try to hide that it has 32 bits. Thus I do not see why an ARGB immutable that is bitwise represented by 4 packed bytes should hide its structure. But again, I am not 100% certain about this.

The issues you describe that packages use internals are real but shouldn't be the consequence be to simply fix them when found? Its absolutely normal that packages break when the standard library (base) moves on. I would say this is part of a regular development workflow and not any form of a serious issue. The Julia package landscape is healthy and IMHO in large parts of high quality. 

[Scott Jones]

I've seen what happens both in the internal code of a language, and the customer usage, over 3 decades, and know first-hand the issues that come up when you don't have the *option* of making things private.

(having the distinction between "public" and "private" in the language was something we had to add after years of problems)

Nothing about my proposal would *force* you to use it, it would not break *any* current julia code.

Also, I don't think it is at all good that people consider it "normal" that things break constantly, esp.

when there are simple ways that a lot of the pain could be avoided.

[Scott Jones]

On Tuesday, July 7, 2015 at 10:11:46 AM UTC-4, Tobias Knopp wrote:

Ok I have to say that I am not that sure about the immutable thingy as for the regular type. For me an immutable is just like say an Int32 that also does not try to hide that it has 32 bits. Thus I do not see why an ARGB immutable that is bitwise represented by 4 packed bytes should hide its structure. But again, I am not 100% certain about this.

So, in your case, it is fine to make everything public.

However, what happens when you realize that a byte isn't enough, and you want a 64-bit quantity packed a different way?

If you'd had accessor methods, getRed, getBlue, getGreen, etc. then you could merrily update your type, and everything would still work without breakage. 

[Stefan Karpinski]

On Sun, Jul 5, 2015 at 7:30 PM, Julian Manzano <julian....@gmail.com> wrote:

Hi All,

I have been using C++ and Python for several years

I'll comment on this later...

 

and I am very curious about Julia, I've got the REPL working on my workstation and I am really impressed so far with what I've seen.

Thanks :-)

 

However there are some design decisions in the language that I fail to understand and I would really appreciate if someone could explain the rationale:

The main point that I fail to understand is the decision not to allow member functions.
The typical explanation that I find everywhere is that Julia designers have chosen all the methods to be external because this is cleaner (specially for mathematical methods where there is no clear owner) and allows for multiple dispatch.
This explanation does not convince me for the following reasons:

1) We can have multiple dispatch a la Julia and still allow types to have methods. These two things seeem independent to me.

These things are distinct but not orthogonal. Multiple dispatch is a generalization of single dispatch. As such, it is strictly more expressive, and having two separate dispatch mechanisms – a weak one for "internal" methods and a more powerful one for external methods – would be non-orthogonal while not increasing the overall expressive power of the language at all. That's lose-lose from a language design perspective.

 

2) Dynamic multiple dispatch can also be done as a byproduct of single dispatch using the visitor pattern (C++, Java, etc.), so in that sense, multiple dispatch is not a new feature.

This isn't really true – or at least double dispatch requires an unacceptable amount of coupling between classes that should be unrelated to emulate proper multiple dispatch. This article has a pretty lucid explanation of the issues. In particular note the bit about the price of double dispatch:

• Adding new shapes will be cumbersome: we will have to change all existing visitors to add the new case.

• Dangerous for bugs too: when adding new subtypes of Shape, one might easily forget to override the accept method which will lead to unwanted behaviour: All visitors would treat instances of the newly created type like its super type.

So, single dispatch allows you to emulate multiple dispatch, but only awkwardly and in a way that makes any instance of the expression problem (and they are not uncommon at all) a maintenance nightmare.

Consider how to make `1 + obj` work just as well as `obj + 1` – i.e. how to make dispatch symmetric for a new type with a pre-existing type. In Python you can use __radd__ methods, but this is clearly a hack that just addresses this one particular instance of the expression problem in a non-general way. What you really need here is precisely multiple dispatch – with it, this is completely trivial to do. In fact, with Julia's promotion and conversion system (which is really just a clever application of multiple dispatch), you don't have to explicitly handle this case, you just provide the appropriate promotion rules and you can solve this entire class of promotion problems all at once in a completely general and extensible way. This is, in fact, how mixed-type arithmetic between built-in types are defined:

julia> @less 1 + 2.3

+(x::Number, y::Number) = +(promote(x,y)...)

If you define your own type, you hook into the same system and it just works. And it's efficient. Despite the fact that really basic mathematical operations like + are generic functions and mixed type addition conceptually goes through many layers of dispatch, it still boils down to the minimal two instructions:

julia> @code_llvm 1 + 2.5

define double @"julia_+_21457"(i64, double) {

top:

  %2 = sitofp i64 %0 to double

  %3 = fadd double %2, %1

  ret double %3

}

Compare this with the cost of dynamic dispatch through a vtable even in a fast language like C++ – if you had to do addition with two jumps through vtables, it would be a performance disaster. Perhaps you could backport some of the technology from Julia to a single dispatch design. In fact, modern C++ compilers rely heavily on devirtualization, which is essentially a mild form of how Julia's compiler manages this, but I'm really not sure if it would work in the large – there's a strangely nice synergy between dynamic type inference and multiple dispatch. Expressing complex polymorphic behaviors using dispatch is not only convenient, it also gives the compiler a huge amount of type information in a form that it can reason quite effectively about.

 

3) Lack of member functions forces all field to be public and therefore I cannot understand how Julia will avoid the object orgy anti-pattern (https://en.wikipedia.org/wiki/Object_orgy)

External dispatch does make it non-obvious how to enforce encapsulation. However, lots of languages don't enforce member encapsulation and get by just fine. Python, for example, doesn't have private members, and this doesn't seem to lead to any kind of orgy that I've ever noticed. They have a convention that members starting with _ are private and that works pretty well – if you access one of those and your code breaks in the future, you have only yourself to blame. In Julia, we take this attitude a bit further and consider all fields private unless otherwise indicated in the documentation of a type. So far this hasn't caused trouble.

You may find it strange for a language not to have class-based single-dispatch object-orientation because both of the languages you've used happen to have this model. While it's been fairly successful, this model hasn't been the panacea that it was touted to be in the 1990s and early 2000s. There are many programming languages that don't work this way at all and don't miss it. C is still the most popular language and doesn't do this. Almost all the functional languages don't do things this way. Give the multiple dispatch paradigm a try for a while. I think you'll find that it grows on you. Fair warning: after getting used to it, it's hard to go back.

[Julian Manzano]

@Stefan

First of all, thank you very much for your detailed comments. As I said I find Julia great and I would like to see if I can gradually use it in my workflow.
My intention with my questions was only to get some clarity from the experts on some points that I failed to understand just by reading docs and previous discussions.

Let me just make some brief friendly comments on your points :)

1,2) Regarding the multiple dispatch topic: I've got your point, multiple dispatch in Julia is easier and natural, not an afterthought as in C++/Java etc.

3) Encapsulation:

> External dispatch does make it non-obvious how to enforce encapsulation

True, but for example, there are some proposals (Scott's one) where you can explicitly indicate that some fields are private to the module where the type is defined. This is just a proposal that, as Scott pointed out, is opt in, will not break anybody's code and I honestly do not think adds any significant complexity to the language. But, hey, this is just 'a' proposal.

> Python, for example, doesn't have private members, and this doesn't seem to lead to any kind of orgy that I've ever noticed.

True, I am happy to have everything public like in Python. But at least like in Python you have a clearly defined policy that everybody follows and even more, the language helps with name mangling.
So even for newbies, is it almost impossible to breach the policy because they will get errors and if they take the time to understand what is going one and mangle the name to get access then they are painfully aware of what they are doing.
So may be, doing something similar in Julia could be just another proposal.

> You may find it strange for a language not to have class-based single-dispatch object-orientation because both of the languages you've used happen to have this model.

Actually I do not find this strange, I played with Haskell for some time, and even Mathematica and Fortran in ancient times so I more than happy to use a functional style or any other style. So more than happy with Julia here :)

Cheers,
Julian

[Milan Bouchet-Valat]

Then maybe this fact should be stressed more in the documentation. For
example, in the manual, nothing implies that fields should be
considered private at all:
http://docs.julialang.org/en/latest/manual/types/#composite-types

In the recent discussions, I had the feeling that many contributors
weren't clear on whether fields are considered private or not.

Also, to me the project of being able to overload the a.b field access
syntax [1] goes against the idea that fields are private. In that
issue, that feature was even considered as a way of implementing an
abstraction on top of a given structure. That's probably fine as long
as it's only used by packages interacting with other languages where
the convention is different, but beyond that it would really blur the
message about private fields.

So basically, I think we should be absolutely clear about whether
accessing fields from external code is subject to breaking at any point
or not. (Other than that, I agree with all of your points.)


Regards


1: https://github.com/JuliaLang/julia/issues/1974

[Julian Manzano]

Problem is, some fields for some cases should be public, because this is the most natural option...

[Tom Breloff]

So after lots of internal back and forth, I think I'm firmly in the camp that we DO NOT need additional syntax to restrict field access.  If you want something to be private, use an underscore, and be harsh on people that access it directly.  You can't protect other people from writing bad code... they will always find a way.  Making development a chore to protect others from their own stupidity is something I can't get behind.  I'm coming from 20 years of object oriented mindset, and so I completely understand Scott's feelings that internal changes are safer when internals are only accessed by getter/setter methods.  That may be true, but it sucks.  The absolute worst part about the getter/setter style is the horrible verbose codebase that does so little with so much code.  It's less maintainable because structural changes have far reaching consequences (except in the very simple/rare case where you swap out the field's type without changing the interface, which btw should be handled seamlessly by Julia in many cases).  

My solution: make sure there is a clear and public declaration of "Julia standards" so that people can understand intentions from other's code (underscores indicating private fields for example), and employ public shame when package developers break the social code. 

[Tony Kelman]

Although Python gets by fine without any enforced privacy, Python is also notorious for being poorly suited for writing large scale reliable systems - lack of privacy of internals is only one factor among many, but it contributes.

However this will be very difficult to convince enough of the core development group that it needs to be implemented (hint: step one is to create a proof-of-concept implementation) as a first-class language feature, when it could potentially be implemented via macros in user code. Especially on top of getfield overloading, #1974.

P.S: Scott, for everyone's benefit, please refrain from using the "private parts" metaphor.

[Stefan Karpinski]

On Wed, Jul 8, 2015 at 1:53 PM, Julian Manzano <julian....@gmail.com> wrote:

 

3) Encapsulation:

> External dispatch does make it non-obvious how to enforce encapsulation
True, but for example, there are some proposals (Scott's one) where you can explicitly indicate that some fields are private to the module where the type is defined. This is just a proposal that, as Scott pointed out, is opt in, will not break anybody's code and I honestly do not think adds any significant complexity to the language. But, hey, this is just 'a' proposal.

The real question with any notion of privacy is "private from whom?" Methods don't belong to any particular argument, so which of their arguments' private fields are they allowed to fiddle with? And what about more granular permissions like r/w? There are many ways to answer these issues, but whatever one chooses, it's more complicated than just marking some fields as private. It also doesn't enable anyone to do anything that they couldn't do before, which is what we tend to prioritize at this point.

[Scott Jones]

On Wednesday, July 8, 2015 at 9:23:28 AM UTC-4, Tony Kelman wrote:

Although Python gets by fine without any enforced privacy, Python is also notorious for being poorly suited for writing large scale reliable systems - lack of privacy of internals is only one factor among many, but it contributes.

What other factors do you see contributing to that in Python?  (I've learned Python, but once I learned about Julia, never looked back at it).

Are there any other impediments in Python to being able to write large scale reliable systems (which is precisely what my colleagues and I are trying to do), that are currently in Julia?

However this will be very difficult to convince enough of the core development group that it needs to be implemented (hint: step one is to create a proof-of-concept implementation) as a first-class language feature, when it could potentially be implemented via macros in user code. Especially on top of getfield overloading, #1974.

That would be fine - I don't know enough about Julia meta-programming  (that's next on my list of things I need to grok) to even know if it could be done as a macro.  

 

P.S: Scott, for everyone's benefit, please refrain from using the "private parts" metaphor.

I just repeated it (from: https://groups.google.com/d/msg/julia-users/sk8Gxq7ws3w/mvQbIb-FjRkJ), and if so, by the same token, people should stop using the "consenting adults" metaphor, which is *precisely* what leads to the "private parts" metaphor.  Note: https://groups.google.com/d/msg/julia-dev/Rj1xkrZkgcw/VE0K8Ji83QcJ, this has been around in Julia for years.

In order for the "consenting adults" metaphor to work, you have to have the means of *denying* consent, if you so wish.

[Tony Kelman]

Many factors, and many of the complaints apply equally to Javascript or any other dynamically typed language. Julia talks more about types than most other dynamic languages, but many things are still not statically checked and can lead to runtime-only errors. The problem boils down to whether or not a compiler is verifying invariants, interfaces, types, and safety in your code ahead of time. It's a common justification for using languages like Ada, Rust, Haskell, Scala, etc. Julia may be more amenable to offline static analysis than many other dynamic languages, but this really isn't the design space that's being targeted here, and tooling for this remains immature.

You've quoted that phrase multiple times since May, sorry I deliberately chose not to read that disaster of a many-pages-long mailing list thread so was unaware it was first written by someone else. I'd chastise him too and recommend a different choice of words. The "consenting adults" phrasing should probably be retired as well, given the growth and standards that we'd like to maintain as a community.

[Spencer Russell]

On Wed, Jul 8, 2015, at 01:28 PM, Stefan Karpinski wrote:

The real question with any notion of privacy is "private from whom?" Methods don't belong to any particular argument, so which of their arguments' private fields are they allowed to fiddle with? And what about more granular permissions like r/w? There are many ways to answer these issues, but whatever one chooses, it's more complicated than just marking some fields as private. It also doesn't enable anyone to do anything that they couldn't do before, which is what we tend to prioritize at this point.

 

This is an excellent point. I'm curious what the exact semantics are of the public/private keywords Scott is proposing.

 

I agree that defining interfaces is currently a problem but there's work on multiple fronts happening to make that better. I think though that the question should not be "How can I restrict what users can do with my package" and more "How do I communicate what I expect users to do".

 

It also seems that between getfield overloading and Traits/Interfaces we'll have mechanisms for package developers to get as granular as they want, and I wouldn't be surprised if someone ends up developing an `@private` macro that automatically overloads field access to raise an error or something.

 

Also FWIW I've always read "Consulting Adults" in a strictly platonic way until the addition of the private parts metaphor. Probably I've actually been reading it more as "Responsible" or "Trustworthy" in the sense that we don't need to go out of our way to protect again bad/stupid behavior in other code.

 

-s

[Scott Jones]

On Wednesday, July 8, 2015 at 2:02:24 PM UTC-4, Tony Kelman wrote:

Many factors, and many of the complaints apply equally to Javascript or any other dynamically typed language. Julia talks more about types than most other dynamic languages, but many things are still not statically checked and can lead to runtime-only errors. The problem boils down to whether or not a compiler is verifying invariants, interfaces, types, and safety in your code ahead of time. It's a common justification for using languages like Ada, Rust, Haskell, Scala, etc. Julia may be more amenable to offline static analysis than many other dynamic languages, but this really isn't the design space that's being targeted here, and tooling for this remains immature.

Well, I think with a few more tweaks, julia will fall right into a sweet spot of rapid application development while still completely capable of being used for large scale *reliable* and long-term application development, even for the "mission critical" sorts of applications that I'm used to, where reliablility, scalability, and performance are all necessary.

So far, I haven't seen another language with quite the "bones" that Julia has, even if everything is not totally fleshed out quite yet.

You've quoted that phrase multiple times since May, sorry I deliberately chose not to read that disaster of a many-pages-long mailing list thread so was unaware it was first written by someone else. I'd chastise him too and recommend a different choice of words. The "consenting adults" phrasing should probably be retired as well, given the growth and standards that we'd like to maintain as a community.

I think I only used it on-line about three times (and a few more times over beers during JuliaCon!) but I will desist, and I even found "private" parts comments going back to 2014 on GitHub, long before I even knew Julia existed.

[Scott Jones]

On Wednesday, July 8, 2015 at 2:30:40 PM UTC-4, Spencer Russell wrote:

On Wed, Jul 8, 2015, at 01:28 PM, Stefan Karpinski wrote:

The real question with any notion of privacy is "private from whom?" Methods don't belong to any particular argument, so which of their arguments' private fields are they allowed to fiddle with? And what about more granular permissions like r/w? There are many ways to answer these issues, but whatever one chooses, it's more complicated than just marking some fields as private. It also doesn't enable anyone to do anything that they couldn't do before, which is what we tend to prioritize at this point.

 

This is an excellent point. I'm curious what the exact semantics are of the public/private keywords Scott is proposing.

Encapsulation gives the implementor of something a lot more freedom, to improve things as time goes on, without worrying that somebody is depending on some internal part of the implemention, rather than the abstraction.  That to me is enabling something that can't be done currently.  It's something that in the short time I've been dealing with Julia, I've already been bitten by.

My idea of @public/@private (or whatever) is that public things, even if not exported, are OK to be used outside the module (i.e. as module.name).  That is good for things where you don't really want to pollute the enclosing namespace, and they are useful functions (or types, etc.).  Frequently utility functions for a type might fall into that category.

I'd probably say that unexported names are by default private (but then, I'd make warnings/errors

from the compiler based on a switch like deprecations use, i.e. Base.JLOptions.depwarn, which is

0 for no warnings, 1 for a single warning, 2 for getting an error.)

Since most people seem to like the everything visible model (at least while developing), that could be set to 0 by default.

You might also be able to make a type public, but then make certain fields private.

(you can't really do the opposite, because if the type is private, you won't see the fields anyway).

public and private to me, would just be at the module level.  If it is visible in the module, it would also be visible to all submodules.

I agree that defining interfaces is currently a problem but there's work on multiple fronts happening to make that better. I think though that the question should not be "How can I restrict what users can do with my package" and more "How do I communicate what I expect users to do".

 

It also seems that between getfield overloading and Traits/Interfaces we'll have mechanisms for package developers to get as granular as they want, and I wouldn't be surprised if someone ends up developing an `@private` macro that automatically overloads field access to raise an error or something.

 

Also FWIW I've always read "Consulting Adults" in a strictly platonic way until the addition of the private parts metaphor. Probably I've actually been reading it more as "Responsible" or "Trustworthy" in the sense that we don't need to go out of our way to protect again bad/stupid behavior in other code.

That surprises me, I've never heard that phrase used in any other way than the legal definition, or the way any dictionary I've ever seen defines it, which I won't quote here so as not to annoy people bothered by it.

(I think it's hard to square the frequency with which the phrase "consenting adults" has been thrown around for years in the julia community, given the definition of the phrase, with the "julia community standards" which I was pointed to, when I made the mistake of using the female pronoun for julia one time [you talk about boats using the female pronoun in English, and that's not "sexualizing" the boat, is it? "Genderizing", maybe])

-s