Why does python not have a mechanism for data hiding?
Sh4wn
first, python is one of my fav languages, and i'll definitely keep
developing with it. But, there's 1 one thing what I -really- miss:
data hiding. I know member vars are private when you prefix them with
2 underscores, but I hate prefixing my vars, I'd rather add a keyword
before it.
Python advertises himself as a full OOP language, but why does it miss
one of the basic principles of OOP? Will it ever be added to python?
Thanks in advance,
Lucas
Ben Finney
> first, python is one of my fav languages, and i'll definitely keep
> developing with it. But, there's 1 one thing what I -really- miss:
> data hiding. I know member vars are private when you prefix them with
> 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> before it.
From whom are you trying to hide your attributes?
In Python, the philosophy "we're all consenting adults here" applies.
You shouldn't pretend to know, at the time you write it, all the uses
to which your code will be put. Barriers such as enforced "private"
attributes will only cause resentment when people, despite your
anticipations, *need* to access them and are then forced to hack their
way around them.
If you want the users of your code to know that an attribute should
not be used as a public API for the code, use the convention of naming
the attribute with a single leading underscore. This is a string
signal that the attribute is part of the implementation, not the
interface. The reader is then on notice that they should not rely on
that attribute; but they are not *prohibited* from using it if
necessary to their ends.
> Python advertises himself as a full OOP language, but why does it
> miss one of the basic principles of OOP?
Who taught you that enforced restrictions on attribute access was a
"basic principle" of OO?
Perhaps you're confusing the "encapsulation" and "abstraction"
principles for enforced access restrictions; they're not.
> Will it ever be added to python?
I hope not.
--
\ "Why was I with her? She reminds me of you. In fact, she |
`\ reminds me more of you than you do!" -- Groucho Marx |
_o__) |
Ben Finney
Ben Finney
> If you want the users of your code to know that an attribute should
> not be used as a public API for the code, use the convention of
> naming the attribute with a single leading underscore. This is a
> string signal that the attribute is part of the implementation, not
> the interface.
Quite apart from whether it's a string signal, I meant to write that
it's a *strong* signal.
--
\ "Some people, when confronted with a problem, think 'I know, |
`\ I'll use regular expressions'. Now they have two problems." |
_o__) —Jamie Zawinski, in alt.religion.emacs |
Ben Finney
Fuzzyman
wrote:
> Sh4wn <luckyluk...@gmail.com> writes:
> > first, python is one of my fav languages, and i'll definitely keep
> > developing with it. But, there's 1 one thing what I -really- miss:
> > data hiding. I know member vars are private when you prefix them with
> > 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> > before it.
>
> From whom are you trying to hide your attributes?
Actually, 'data hiding', although vastly overused by the static crowd
can be a reasonable thing to want.
For example, at Resolver Systems we expose the spreadsheet object
model to our users. It hasa public, documented, API - plus a host of
undocumented internally used methods.
We would really *much* rather hide these, because anything our
customers start using (whether documented or not) we will probably
have to continue supporting and maintaining.
The 'we told you not to use that' approach, when applied to paying
customers doesn't really work... all they see is that you broke their
spreadsheet code by changing your API.
You can make members truly private by proxying, but it is a bit
ungainly.
Michael Foord
http://www.ironpythoninaction.com/
Roy Smith
<5ea78616-e955-4ee0...@79g2000hsk.googlegroups.com>,
Fuzzyman <fuzz...@gmail.com> wrote:
> The 'we told you not to use that' approach, when applied to paying
> customers doesn't really work... all they see is that you broke their
> spreadsheet code by changing your API.
I hear what you're saying, friend, and I feel your pain! Life gets so much
more complicated when you've got paying customers. It's especially
complicated when your customers have a market cap an order of magnitude
greater than your own :-)
Ville M. Vainio
> The 'we told you not to use that' approach, when applied to paying
> customers doesn't really work... all they see is that you broke
> their spreadsheet code by changing your API.
And the customer point of view is quite reasonable - they have a job
to do, and a limited timeframe; sometimes accessing privates directly
is much better than waiting for updates from vendor. Bad designs (as
far as choosing publics goes) happen.
Even if their softare breaks on upgrade, you can quite clearly point
out that they used an internal api - and they will keep on using the
old version until they solve the problem. Everybody wins.
Perhaps a lint-like validation tool would be optimal for this
problem...
Fuzzyman
> Fuzzyman <fuzzy...@gmail.com> writes:
> > The 'we told you not to use that' approach, when applied to paying
> > customers doesn't really work... all they see is that you broke
> > their spreadsheet code by changing your API.
>
> And the customer point of view is quite reasonable - they have a job
> to do, and a limited timeframe; sometimes accessing privates directly
> is much better than waiting for updates from vendor. Bad designs (as
> far as choosing publics goes) happen.
They will use whatever they find, whether it is the best way to
achieve a goal or not. Once they start using it they will expect us to
maintain it - and us telling them it wasn't intended to be used by
them in the first place won't cut it.
>
> Even if their softare breaks on upgrade, you can quite clearly point
> out that they used an internal api - and they will keep on using the
> old version until they solve the problem. Everybody wins.
>
Not if they are waiting for a fix or new feature in the upgrade - and
we can't refactor because they are relying on APIs that we want to
remove.
We very much lose.
> Perhaps a lint-like validation tool would be optimal for this
> problem...
So we can refuse to execute their code if they use private APIs?
Proxying so that we can really hide our internal APIs is a better
solution.
This hasn't happened to us yet (our application has only been in
commercial use since January), but it is one of the reasons that
Microsoft's COM APIs grew so wide and wild, and caused them very real
problems in trying to improve them. We are keen to avoid the same
situation.
Michael Foord
http://www.ironpythoninaction.com/
bearoph...@lycos.com
>In Python, the philosophy "we're all consenting adults here" applies.<
Michael Foord:
> They will use whatever they find, whether it is the best way to
> achieve a goal or not. Once they start using it they will expect us to
> maintain it - and us telling them it wasn't intended to be used by
> them in the first place won't cut it.
So they will use the methods with one or two underscores too. And
imply that you want to document them too.
They don't seem adult enough, then ;-)
Internal API hiding seems quite less useful if you don't need to let
customers manage your code.
Bye,
bearophile
Fuzzyman
> Ben Finney:
>
> >In Python, the philosophy "we're all consenting adults here" applies.<
>
> Michael Foord:
>
> > They will use whatever they find, whether it is the best way to
> > achieve a goal or not. Once they start using it they will expect us to
> > maintain it - and us telling them it wasn't intended to be used by
> > them in the first place won't cut it.
>
> So they will use the methods with one or two underscores too. And
> imply that you want to document them too.
> They don't seem adult enough, then ;-)
Great! We'll just tell them that...
Michael
Sh4wn
wrote:
Well for me, it the ideal way to make sure it contains so 'wrong'
data. You can create getter/setters, and in some cases only a getter
and validate the value given by the user. Then you'll not have to
worry about the data in the rest of your class, which makes life a lot
easier IMO.
Paddy
> On May 24, 2:58 pm, Ben Finney <bignose+hates-s...@benfinney.id.au>
> wrote:
>
> > Sh4wn <luckyluk...@gmail.com> writes:
> > > first, python is one of my fav languages, and i'll definitely keep
> > > developing with it. But, there's 1 one thing what I -really- miss:
> > > data hiding. I know member vars are private when you prefix them with
> > > 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> > > before it.
>
> > From whom are you trying to hide your attributes?
>
> Actually, 'data hiding', although vastly overused by the static crowd
> can be a reasonable thing to want.
>
> For example, at Resolver Systems we expose the spreadsheet object
> model to our users. It hasa public, documented, API - plus a host of
> undocumented internally used methods.
>
> We would really *much* rather hide these, because anything our
> customers start using (whether documented or not) we will probably
> have to continue supporting and maintaining.
>
> The 'we told you not to use that' approach, when applied to paying
> customers doesn't really work... all they see is that you broke their
> spreadsheet code by changing your API.
"We told you not to use it in the API Docs"
"Those names with leading undoerscores means _DO_NOT_USE_IT"
"THose methods whose docstrings say DO NOT USE EXTERNALLY"
And if they still use them, then they'd be problematic no matter what
language was used.
Customers ey?
Can't live without 'em...
... Actually that's customers Sir!
:-)
- Paddy.
Terry Reedy
"Benjamin Kaplan" <benjami...@case.edu> wrote in message
news:ec96e1390805241357i6e4...@mail.gmail.com...
| On Sat, May 24, 2008 at 10:14 AM, Fuzzyman <fuzz...@gmail.com> wrote:
|| > For example, at Resolver Systems we expose the spreadsheet object
| > model to our users. It hasa public, documented, API - plus a host of
| > undocumented internally used methods.
| >
| > We would really *much* rather hide these, because anything our
| > customers start using (whether documented or not) we will probably
| > have to continue supporting and maintaining.
| >
| > The 'we told you not to use that' approach, when applied to paying
| > customers doesn't really work... all they see is that you broke their
| > spreadsheet code by changing your API.
Python was not really written with 'difficult' customers in mind ;-)
One could largely hide private vars with a program that substituted random
names for single _ names, and removed the doc strings for functions,
classes, and methods with such names.
Such a program could even put such names in a separate module imported as
'_private_do_not_use_'.
Aahz
Not directly addressing your point except insofar as it appears that
your understanding of OOP may be limited:
"...some experts might say a C++ program is not object-oriented without
inheritance and virtual functions. As one of the early Smalltalk
implementors myself, I can say they are full of themselves." --zconcept
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
Need a book? Use your library!
Ville M. Vainio
>> Perhaps a lint-like validation tool would be optimal for this
>> problem...
>
> So we can refuse to execute their code if they use private APIs?
No, but it could complain and point out the exact offending lines,
pointing their development effort to right direction.
> Proxying so that we can really hide our internal APIs is a better
> solution.
How about pyprotocols and other interface packages?
Proxying is pretty workable too, and it easily makes sense that the
official API objects should be different from the logic executing
objects.
Duncan Booth
Or if you code in C++ and they *really* need to get at something you made
private they will still get at it. I've been there and done that: 'private'
in languages which have it is rarely an advantage and frequently a pain.
Also note that private in C++ while it does make it harder to access
(though not preventing it altogether) doesn't do any data hiding, so if you
are subclassing a vendor-provided function you still have to take note of
the private attributes to avoid naming collisions.
At least Python's convention which hides the implementation without making
it hard to access has some benefits.
Fuzzyman
> "Benjamin Kaplan" <benjamin.kap...@case.edu> wrote in message
>
> news:ec96e1390805241357i6e4...@mail.gmail.com...
> | On Sat, May 24, 2008 at 10:14 AM, Fuzzyman <fuzzy...@gmail.com> wrote:
> || > For example, at Resolver Systems we expose the spreadsheet object
> | > model to our users. It hasa public, documented, API - plus a host of
> | > undocumented internally used methods.
> | >
> | > We would really *much* rather hide these, because anything our
> | > customers start using (whether documented or not) we will probably
> | > have to continue supporting and maintaining.
> | >
> | > The 'we told you not to use that' approach, when applied to paying
> | > customers doesn't really work... all they see is that you broke their
> | > spreadsheet code by changing your API.
>
> Python was not really written with 'difficult' customers in mind ;-)
>
True. It's extremely suited to what we do though.Minor difficulties
like this are vastly outweighed by advantages. The difficulties are
real though.
> One could largely hide private vars with a program that substituted random
> names for single _ names, and removed the doc strings for functions,
> classes, and methods with such names.
>
We need to *use* those names to display the spreadsheet once the
calculation has finished (and their code has run).
> Such a program could even put such names in a separate module imported as
> '_private_do_not_use_'.
Splitting more of the functionality out is probably part of the best
solution.
Michael Foord
http://www.ironpythoninaction.com/
Ville M. Vainio
> Or if you code in C++ and they *really* need to get at something you
> made private they will still get at it. I've been there and done
> that: 'private' in languages which have it is rarely an advantage
> and frequently a pain.
Indeed. In C++, they recommend the "pimpl idiom" (private
implementation), which actually has real advantages ;-)
Fuzzyman
> Fuzzyman <fuzzy...@gmail.com> writes:
> >> Perhaps a lint-like validation tool would be optimal for this
> >> problem...
>
> > So we can refuse to execute their code if they use private APIs?
>
> No, but it could complain and point out the exact offending lines,
> pointing their development effort to right direction.
>
> > Proxying so that we can really hide our internal APIs is a better
> > solution.
>
> How about pyprotocols and other interface packages?
>
We're using IronPython. I haven't looked at pyprotocols but I know the
Zope interface package is at least partly written in C. Our
spreadsheet object model is very 'performance sensitive', so that's a
consideration. We should definitely explore the prior art in this area
before we implement anything ourselves.
Thanks
Michael Foord
http://www.ironpythoninaction.com/
Joe P. Cool
wrote:
> Sh4wn <luckyluk...@gmail.com> writes:
> > first, python is one of my fav languages, and i'll definitely keep
> > developing with it. But, there's 1 one thing what I -really- miss:
> > data hiding. I know member vars are private when you prefix them with
> > 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> > before it.
>
> From whom are you trying to hide your attributes?
I saw this "don't need it" pattern in discussions about the ternary
"if..else" expression and about "except/finally on the same block
level".
Now Python has both. Actually it is very useful to be able to
distinguish
between inside and outside. This is obvious for real world things e.g.
your
TV. Nobody likes to open the rear cover to switch the channel. Similar
arguments apply to software objects. "data hiding" is a harsh name, I
would
call it "telling what matters". The need for this becomes
indispensable in
really big software packages like the Eclipse framework with approx.
100000
classes. If you cannot tell the difference between inside and outside
you
are lost.
> In Python, the philosophy "we're all consenting adults here" applies.
Please don't sell a missing feature as a philosophy. Say you don't
need/want
it. But don't call it philosophy.
> You shouldn't pretend to know, at the time you write it, all the uses
> to which your code will be put.
It's *your* *decision* which uses will be available. Your explanation
appears
to me as a fear to decide.
> If you want the users of your code to know that an attribute should
> not be used as a public API for the code, use the convention of naming
> the attribute with a single leading underscore.
Littering your class definition with dozens of underscores is exactly
the
line noise we love to criticize in Perl.
> > Python advertises himself as a full OOP language, but why does it
> > miss one of the basic principles of OOP?
>
> Who taught you that enforced restrictions on attribute access was a
> "basic principle" of OO?
Nearly every introduction to OOP? Please don't tell me that
encapsulation
does not mean "enforced restriction". If the language has no syntactic
support for encapsulation then it does not have encapsulation. Similar
argument applies to Java where programmers sometimes claim that Java
"has"
properties because of setCrap/getCrap.
__
Joe
Terry Reedy
"Joe P. Cool" <joe.p...@googlemail.com> wrote in message
news:09847cfc-eaec-4adc...@m44g2000hsc.googlegroups.com...
| Actually it is very useful to be able to distinguish
| between inside and outside.
Which Python allows one to do.
| This is obvious for real world things e.g. your
| TV. Nobody likes to open the rear cover to switch the channel.
The original issue was that users might open the case to do something
unplanned and that they would come to depend on and clamor for the
maintenance of internal details that the manufacturer wants to change. But
in the real world, most customers know and accept that custom alterations
and extensions of a product my be model specific. Some things I buy even
have a disclaimer that user-visible details might be different from what
the instructions say, and only guarantee that the functionality will be as
good or better than specified.
[...]
| Please don't sell a missing feature as a philosophy.
I won't if you don't claim that a feature is missing because you don't like
its implementation. To most of us, replacing the nearly never used '__'
with a keyword would be a auful change.
| Please don't tell me that encapsulation does not mean "enforced
restriction".
There are obviously degrees of enforced restriction. Take your TV example.
Most users will respect a 'do not open' request by the manufacturer
(Python's '_'.). But some will unscrew anyway. So some makers use
fasteners that are resistent to opening without specialized tools from
specialized sources -- or perhaps specialized knowledge (Python's '__').
(This is very frustrating if one needs to open such a system.) Or some
mark the screws so they can detect and punish entry. At least one (Apple)
has intentionally altered internals to punish entry and alteration. Or
there is the 'embed in epoxy' method.
If you want something equivalent to epoxy embedding, design it, implement
it, and share it (if you want). Perhaps one could make an Opaque extension
class, perhaps even usable as a mixin class rather than as a single-base
class.
Terry Jan Reedy
miller...@gmail.com
> Python advertises himself as a full OOP language, but why does it miss
> one of the basic principles of OOP? Will it ever be added to python?
Others have already answered this directly, but I'd like to mention
that languages I know of which have this feature also have a feature
for getting around it. (e.g. C++ and friend classes) I don't know
about you, but I don't want features in the language that make me want
to circumvent them. Do you?
Russ P.
Dear Mr. Cool,
Excellent post. I agree with you entirely.
I've always considered the underscore conventions in Python an ugly
hack in an otherwise elegant language. I often avoid them even when
they technically belong just because I don't like the way they make my
code look.
Your analogy of the TV set is nice. The front control panel should be
clearely distinguished from the covered control panel and certainly
from the guts. It's more than just a matter of keeping the user from
doing something stupid -- it's a matter of design style.
I am also bothered a bit by the seeming inconsistency of the rules for
the single underscore. When used at file scope, they make the variable
or function invisible outside the module, but when used at class
scope, the "underscored" variables or functions are still fully
visible. For those who claim that the client should be left to decide
what to use, why is the client prohibited from using underscored
variables at file scope?
I may be full of beans here, but I would like to make a suggestion.
Why not add a keyword such as "private," and let it be the equivalent
of "protected" in C++? That would make member variables and functions
visible in inherited classes. The encapsulation wouldn't be as
airtight as "private" in C++, so clients could still get access if
they really need it in a pinch, but they would know they are bending
the rules slightly. I realize that single underscores do that already,
but they are just unsightly.
Paul Boddie
On May 25, 2:32 pm, "Joe P. Cool" <joe.p.c...@googlemail.com> wrote:
> > On 24 Mai, 15:58, Ben Finney <bignose+hates-s...@benfinney.id.au>
> > wrote:
> > > From whom are you trying to hide your attributes?
>
> > I saw this "don't need it" pattern in discussions about the ternary
> > "if..else" expression and about "except/finally on the same block
> > level". Now Python has both.
Well, there was no need to exclude try...except...finally in the first
place - Guido doubted the semantics of such a thing until he saw it in
Java, apparently. This is arguably one of the few features that has
incurred little cost with its addition, in contrast to many other
changes made to Python these days. Which brings us to the ternary
expression - something which will presumably see unprecedented usage
by those who jump on every new piece of syntax and "wear it out" in
short order.
Interestingly, Python did have an "access" keyword at some point
(before 1.4, I believe), so it isn't as if no-one had thought about
this before.
[...]
> > > Who taught you that enforced restrictions on attribute access was a
> > > "basic principle" of OO?
>
> > Nearly every introduction to OOP? Please don't tell me that
> > encapsulation
> > does not mean "enforced restriction". If the language has no syntactic
> > support for encapsulation then it does not have encapsulation. Similar
> > argument applies to Java where programmers sometimes claim that Java
> > "has" properties because of setCrap/getCrap.
But is "enforced restriction" essential in OOP? Note that languages
like C++ provide such features in order to satisfy constraints which
are not directly relevant to Python.
[...]
On May 26, 6:49 am, "Russ P." <Russ.Paie...@gmail.com> wrote:
>
> I've always considered the underscore conventions in Python an ugly
> hack in an otherwise elegant language. I often avoid them even when
> they technically belong just because I don't like the way they make my
> code look.
To be honest, I don't really like putting double underscores before
names just to make sure that instances of subclasses won't start
trashing attributes of a class I've written. (I actually don't care
about people accessing attributes directly from outside instances
because I've seen enough "if only they hadn't made that private/
protected, I could do my work" situations in languages like Java.)
Indeed, I experienced a situation only yesterday where I found that I
had been using an instance attribute defined in a superclass (which
was arguably designed for subclassing), but was the author really
expected to prefix everything with double underscores? Of course, by
not enforcing the private nature of attributes, Python does permit
things like mix-in classes very easily, however.
If anything, Python lacks convenient facilities (besides informal
documentation) for describing the instance attributes provided by the
code of a class and of superclasses. Of course, the dynamic
possibilities in Python makes trivial implementations of such
facilities difficult, but having access to such details would make
errors like those described above less likely. I imagine that pylint
probably helps out in this regard, too.
[...]
> I am also bothered a bit by the seeming inconsistency of the rules for
> the single underscore. When used at file scope, they make the variable
> or function invisible outside the module, but when used at class
> scope, the "underscored" variables or functions are still fully
> visible. For those who claim that the client should be left to decide
> what to use, why is the client prohibited from using underscored
> variables at file scope?
I don't remember why this is. I'll leave you to track down the
rationale for this particular behaviour. ;-)
> I may be full of beans here, but I would like to make a suggestion.
> Why not add a keyword such as "private," and let it be the equivalent
> of "protected" in C++? That would make member variables and functions
> visible in inherited classes. The encapsulation wouldn't be as
> airtight as "private" in C++, so clients could still get access if
> they really need it in a pinch, but they would know they are bending
> the rules slightly. I realize that single underscores do that already,
> but they are just unsightly.
If anything, I'd prefer having private attributes than some kind of
protected attributes which don't actually offer useful protection
(from accidental naming collisions).
Paul
Neil Hodgson
> The encapsulation wouldn't be as
C++ private is not airtight as you have access to pointer arithmetic
as well as
#define private public
Neil
Ville Vainio
> I may be full of beans here, but I would like to make a suggestion.
> Why not add a keyword such as "private," and let it be the equivalent
> of "protected" in C++? That would make member variables and functions
Better yet, have '@@foo' be translated to self.__foo. It would cut
some whining about 'self' as well, which would probably make it a more
worthwhile change (private attributes is a feature I don't really care
about much).
Obviously nobody should hold their breath and wait for this to happen.
This is a problem that needs to be solved on API doc generation / IDE
autocompleter layer. If everybody agreed on how to tag published vs.
private methods in docstrings or wherever, we would be mostly set.
David
and abstract base classes[2], so it is possible that it will get more
explicit support for private class attributes at some point, if there
is enough demand for it.
[1]http://www.python.org/dev/peps/pep-3107/
[2]http://www.python.org/dev/peps/pep-3119/
David.
Bruno Desthuilliers
> On 24 Mai, 15:58, Ben Finney <bignose+hates-s...@benfinney.id.au>
> wrote:
>> Sh4wn <luckyluk...@gmail.com> writes:
>>> first, python is one of my fav languages, and i'll definitely keep
>>> developing with it. But, there's 1 one thing what I -really- miss:
>>> data hiding. I know member vars are private when you prefix them with
>>> 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
>>> before it.
>> From whom are you trying to hide your attributes?
>
> I saw this "don't need it" pattern in discussions about the ternary
> "if..else" expression and about "except/finally on the same block
> level".
> Now Python has both. Actually it is very useful to be able to
> distinguish
> between inside and outside.
Doing it the Python way makes the distinction pretty obvious :
implementation stuff names all start with an underscore. You just can
not miss it.
> This is obvious for real world things e.g.
> your
> TV. Nobody likes to open the rear cover to switch the channel.
Nope, but when the switch is broken and the TV not under warranty no
more, I'm glad I can still open the rear cover and hack something by
myself, despite usual (and hard to miss) "no user serviceable parts
inside" and "warranty void if unsealed" warning stickers.
> Similar
> arguments apply to software objects. "data hiding" is a harsh name, I
> would
> call it "telling what matters".
Which is exactly what we do using naming convention.
>> In Python, the philosophy "we're all consenting adults here" applies.
>
> Please don't sell a missing feature as a philosophy. Say you don't
> need/want
> it. But don't call it philosophy.
Please understand that not having access restriction is a design choice,
not a technical inability (and FWIW the rationale behind that design
choice has been debated to hell and back). So yes, it's is actually a
'philosophic' problem.
>> You shouldn't pretend to know, at the time you write it, all the uses
>> to which your code will be put.
>
> It's *your* *decision* which uses will be available. Your explanation
> appears
> to me as a fear to decide.
Nope, just a matter of experience.
>> If you want the users of your code to know that an attribute should
>> not be used as a public API for the code, use the convention of naming
>> the attribute with a single leading underscore.
>
> Littering your class definition with dozens of underscores is exactly
> the
> line noise we love to criticize in Perl.
At least it makes what's 'inside' and what's 'outside' very obvious,
isn't it ?
>>> Python advertises himself as a full OOP language, but why does it
>>> miss one of the basic principles of OOP?
>> Who taught you that enforced restrictions on attribute access was a
>> "basic principle" of OO?
>
> Nearly every introduction to OOP?
Nearly every introduction to OOP is crap. Nearly every introduction to
OOP also introduces classes and inheritance as "basic principle" of OO.
Almost none introduction to OOP actually talk about *objects* - instead,
they introduce mostly what Stroustrup and Gostling understood of OOP.
> Please don't tell me that
> encapsulation
> does not mean "enforced restriction".
What about : "encapsulation does not mean *language enforced*
restriction" then ?
To me, encapsulation means that - as a client - you do not *need* to
care about implementation details, not that you can not get at it. And
while data hiding is indeed a possible mean to enforce some kind of
encapsulation, it's not quite the same thing.
Gabriel Genellina
>> I am also bothered a bit by the seeming inconsistency of the rules for
>> the single underscore. When used at file scope, they make the variable
>> or function invisible outside the module, but when used at class
>> scope, the "underscored" variables or functions are still fully
>> visible. For those who claim that the client should be left to decide
>> what to use, why is the client prohibited from using underscored
>> variables at file scope?
>
> I don't remember why this is. I'll leave you to track down the
> rationale for this particular behaviour. ;-)
There is no rationale because this is not how it works...
You can:
- import a module and access *all* of their names, even names prefixed with an underscore
- import any name from a module, even if it starts with an underscore
- import * from a module, and it will import all names listed in __all__, even if they start with an underscore
Only in that last case, and when the module doesn't define __all__, the list of names to be imported is built from all the global names excluding the ones starting with an underscore. And it seems the most convenient default. If one wants to access any "private" module name, any of the first two alternatives will do.
--
Gabriel Genellina
Russ P.
wrote:
Well, that's interesting, but it's not particularly relevant to the
original point. By default, underscored variables at file scope are
not made visible by importing the module in which they appear. But
underscored member variables of a class *are* made visible to the
client by default. That's seems at least slightly inconsistent to me.
The issue here, for me at least, is not whether the data or methods
should be absolutely hidden from the client. I'm perfectly willing to
say that the client should have a back door -- or even a side door --
to get access to "private" data or methods.
But I also believe that some standard way should be available in the
language to tell the client (and readers of the code) which methods
are *intended* for internal use only. And that method should be based
on more than a naming convention. Why? Because (1) I don't like
leading underscores in my identifiers, and (2) I think I should be
free to choose my identifiers independently from their properties.
Is this a major issue? No. Is it a significant issue. Yes, I think so.
Here's another suggestion. Why not use "priv" as shorthand for
"private"? Then,
priv height = 24
at file scope would make "height" invisible outside the module by
default. And the same line in a class definition would give "height"
the equivalent of "protected" status in C++.
I think "height" looks cleaner than "_height". And isn't clean code a
fundamental aspect of Python?
Arnaud Delobelle
Apart from the fact that modules and classes are very different ideas,
look at this code:
========================================
import random
class A(object):
def __init__(self, val):
self._val = val
class B(A):
def f(self, other):
return random.choice([self, other])._val
a = A(1)
b = B(2)
b.f(a)
========================================
So you want the last line to return 2 if b is chosen and raise a
ValueError if a is chosen. Good luck implementing that!
> The issue here, for me at least, is not whether the data or methods
> should be absolutely hidden from the client. I'm perfectly willing to
> say that the client should have a back door -- or even a side door --
> to get access to "private" data or methods.
>
> But I also believe that some standard way should be available in the
> language to tell the client (and readers of the code) which methods
> are *intended* for internal use only. And that method should be based
> on more than a naming convention. Why? Because (1) I don't like
> leading underscores in my identifiers, and (2) I think I should be
> free to choose my identifiers independently from their properties.
Python is a dynamic language, attributes can be added to objects at
any time in their life. Consider:
class A(object):
private x # Suspend belief for a minute...
def __init__(self, x):
self.x = x
a = A()
a.x = 2 # What behaviour do you propose here?
> Is this a major issue? No. Is it a significant issue. Yes, I think so.
>
> Here's another suggestion. Why not use "priv" as shorthand for
> "private"? Then,
>
> priv height = 24
>
> at file scope would make "height" invisible outside the module by
> default. And the same line in a class definition would give "height"
> the equivalent of "protected" status in C++.
>
> I think "height" looks cleaner than "_height". And isn't clean code a
> fundamental aspect of Python?
I didn't use to be very fond of it either but I have to admit that it
conveys very useful information.
--
Arnaud
Joe P. Cool
[...]
> | Please don't sell a missing feature as a philosophy.
>
> I won't if you don't claim that a feature is missing because you don't like
> its implementation. To most of us, replacing the nearly never used '__'
> with a keyword would be a auful change.
With "missing" I simply meant "not there".
> | Please don't tell me that encapsulation does not mean "enforced
> restriction".
>
> There are obviously degrees of enforced restriction. Take your TV example.
> Most users will respect a 'do not open' request by the manufacturer
> (Python's '_'.).
I program in Python quite often and I also use the '_' notation. But I
always found the inside/outside distinction very natural and
fundamental. The '_' notation is too informal and can arbitrarily be
overloaded with other meanings. I don't understand why it is so
important to avoid one keyword and paying for that by typing numerous
'_'. Underscore meaning is somewhat blurred - sometimes private,
sometimes protected. And it is line noise - like @$& in Perl.
> If you want something equivalent to epoxy embedding, design it, implement
> it, and share it (if you want). Perhaps one could make an Opaque extension
> class, perhaps even usable as a mixin class rather than as a single-base
> class.
Perhaps I'll give it a try - as soon as I have found out what you
mean :)
George Sakkis
> Here's another suggestion. Why not use "priv" as shorthand for
> "private"? Then,
>
> priv height = 24
>
> at file scope would make "height" invisible outside the module by
> default. And the same line in a class definition would give "height"
> the equivalent of "protected" status in C++.
On a slightly different question: can a data hiding mechanism be
implemented as an add-on by a third party library without a change in
the core language, just like zope.interface does for interfaces ? If
not, why not ?
> I think "height" looks cleaner than "_height". And isn't clean code a
> fundamental aspect of Python?
Note that even in languages that do implement data hiding, people
often use a naming convention to denote hidden members, e.g. an "m_"
prefix (though I find this uglier than plain underscores).
George
Gabriel Genellina
escribió:
> On May 26, 9:08 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>> En Mon, 26 May 2008 06:14:19 -0300, Paul Boddie <p...@boddie.org.uk>
>> escribió:
>>>> the single underscore. When used at file scope, they make the variable
>>>> or function invisible outside the module, but when used at class
>>>> scope, the "underscored" variables or functions are still fully
>>>> visible. For those who claim that the client should be left to decide
>>>> what to use, why is the client prohibited from using underscored
>>>> variables at file scope?
>>
>> explanation of how import works]
>
> Well, that's interesting, but it's not particularly relevant to the
> original point. By default, underscored variables at file scope are
> not made visible by importing the module in which they appear. But
> underscored member variables of a class *are* made visible to the
> client by default. That's seems at least slightly inconsistent to me.
To make things clear: _variables ARE visible when you import a module:
C:\TEMP>type module.py
_variable = 123
(invoke python)
py> import module
py> module._variable
123
py> dir(module)
['__builtins__', '__doc__', '__file__', '__name__', '_variable']
py>
py> from module import _variable
py> _variable
123
Only when you use "from module import *" _variable isn't imported.
(new python session):
py> from module import *
py> _variable
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name '_variable' is not defined
That last form should not be used normally, except when playing with the
interactive interpreter.
> The issue here, for me at least, is not whether the data or methods
> should be absolutely hidden from the client. I'm perfectly willing to
> say that the client should have a back door -- or even a side door --
> to get access to "private" data or methods.
>
> But I also believe that some standard way should be available in the
> language to tell the client (and readers of the code) which methods
> are *intended* for internal use only. And that method should be based
> on more than a naming convention. Why? Because (1) I don't like
> leading underscores in my identifiers, and (2) I think I should be
> free to choose my identifiers independently from their properties.
>
> Is this a major issue? No. Is it a significant issue. Yes, I think so.
Ok, that's what you'd like Python to be. Unfortunately your view appears
not to be shared by the language developers.
> Here's another suggestion. Why not use "priv" as shorthand for
> "private"? Then,
>
> priv height = 24
>
> at file scope would make "height" invisible outside the module by
> default. And the same line in a class definition would give "height"
> the equivalent of "protected" status in C++.
Python data model is centered on namespaces, and it's hard to tell whether
certain attribute is being accessed from "inside the class" or from
"outside the class" in order to allow or deny access.
It's not like static languages where the whole class definition has a
certain lexical scope and it can't be modified afterwards; the set of
allowed attributes is fixed at compile time. In Python you can
add/remove/alter attributes (including methods) dynamically. You can
create classes without using the class statement. The same method may be
shared by many unrelated classes. If you can devise a practical and
efficient mechanism to determine access right in all those varying
circumstances, please post it to the python-ideas list for further
discussion. (In the meantime we'll continue to use a naming convention for
us consenting adults.)
> I think "height" looks cleaner than "_height". And isn't clean code a
> fundamental aspect of Python?
I like the fact that the mere attribute name conveys useful information
abut its intended usage. And I don't care about the _, it doesn't look
ugly to me. But that's just my personal opinion.
--
Gabriel Genellina
alex23
> But I also believe that some standard way should be available in the
> language to tell the client (and readers of the code) which methods
> are *intended* for internal use only. And that method should be based
> on more than a naming convention. Why? Because (1) I don't like
> leading underscores in my identifiers, and (2) I think I should be
> free to choose my identifiers independently from their properties.
Have you considered using the Bridge pattern to separate your
interface from your implementation?
class Implementation:
def private_method(self):
raise NotImplementedError
class Interface:
def __init__(self):
self.imp = Implementation()
def public_method(self):
self.imp.private_method()
No offensive _methods on the interface, and the implementation is
still open for consenting adults to tinker with.
Russ P.
wrote:
> To make things clear: _variables ARE visible when you import a module:
>
> C:\TEMP>type module.py
> _variable = 123
>
> (invoke python)
> py> import module
> py> module._variable
> 123
> py> dir(module)
> ['__builtins__', '__doc__', '__file__', '__name__', '_variable']
> py>
> py> from module import _variable
> py> _variable
> 123
>
> Only when you use "from module import *" _variable isn't imported.
> (new python session):
>
> py> from module import *
> py> _variable
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> NameError: name '_variable' is not defined
Hmmm... that seems a bit strange. Why should "_variable" be visible
when you use "import module" but not when you use "from module import
*"?
> That last form should not be used normally, except when playing with the
> interactive interpreter.
OK, I have a confession to make. I use "from module import *" almost
exclusively. But then, I'm different from most folks. I can drink six
beers and still drive safely, for example. The key is to drive faster
so I can get past dangerous situations quicker.
Terry Reedy
"Russ P." <Russ.P...@gmail.com> wrote in message
news:03655ec7-2136-4cea...@p25g2000pri.googlegroups.com...
| Hmmm... that seems a bit strange. Why should "_variable" be visible
| when you use "import module"
Because you only add one name to the current namespace -- bound to a
module -- and there is no reason to exclude attributes of that object.
|but not when you use "from module import *"?
Because you are already adding m names and adding n more names may cause
problems -- which is what lead to the addition of the __all__ mechanism
some years back. Sorry, I forget the details because they don't concern
me.
Antoon Pardon
> Sh4wn <lucky...@gmail.com> writes:
>
>> first, python is one of my fav languages, and i'll definitely keep
>> developing with it. But, there's 1 one thing what I -really- miss:
>> data hiding. I know member vars are private when you prefix them with
>> 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
>> before it.
>
> From whom are you trying to hide your attributes?
>
> attributes will only cause resentment when people, despite your
> anticipations, *need* to access them and are then forced to hack their
> way around them.
I don't find this argument very compelling.
You can't anticipate all functionality people would like your function
to have. Acces to information in a (private) attribute is just one of
those possible functionallities. People will resent you if you don't
provide functionality they think fits logically in your package.
> If you want the users of your code to know that an attribute should
> not be used as a public API for the code, use the convention of naming
> the attribute with a single leading underscore. This is a string
> signal that the attribute is part of the implementation, not the
> interface. The reader is then on notice that they should not rely on
> that attribute; but they are not *prohibited* from using it if
> necessary to their ends.
But they will resent you just as much if you decide to rewrite
your module in such a way that the attribute is no longer present
or is used now in a slightly different way, so that it break code.
--
Antoon Pardon
Antoon Pardon
> Hi,
>
> first, python is one of my fav languages, and i'll definitely keep
> developing with it. But, there's 1 one thing what I -really- miss:
> data hiding. I know member vars are private when you prefix them with
> 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> before it.
>
>
> Lucas
If you really need it, you can do data hiding in python. It just
requires a bit more work.
----------------------------- Hide.py ---------------------------------
class Rec(object):
def __init__(__, **kwargs):
for key,value in kwargs.items():
setattr(__, key, value)
def __getitem__(self, key):
return getattr(self, key)
def __setitem__ (self, key, val):
setattr(self, key, val)
class Foo(object):
def __init__(self):
hidden = Rec(x=0, y=0)
def SetX(val):
hidden.x = val
def SetY(val):
hidden.y = val
def GetX():
return hidden.x
def GetY():
return hidden.y
self.SetX = SetX
self.SetY = SetY
self.GetX = GetX
self.GetY = GetY
--------------------------------------------------------------------------
$ python
Python 2.5.2 (r252:60911, Apr 17 2008, 13:15:05)
[GCC 4.2.3 (Debian 4.2.3-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> From Hide import Foo
>>> var = Foo()
>>> var.GetX()
0
>>> var.SetX(5)
>>> var.GetX()
5
>>> var.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'x'
>>> var.hidden.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'hidden'
--
Antoon Pardon
Duncan Booth
> If you really need it, you can do data hiding in python. It just
> requires a bit more work.
<snip>
> --- $ python
> Python 2.5.2 (r252:60911, Apr 17 2008, 13:15:05)
> [GCC 4.2.3 (Debian 4.2.3-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> From Hide import Foo
>>>> var = Foo()
>>>> var.GetX()
> 0
>>>> var.SetX(5)
>>>> var.GetX()
> 5
>>>> var.x
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: 'Foo' object has no attribute 'x'
>>>> var.hidden.x
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: 'Foo' object has no attribute 'hidden'
>
That sort of hiding isn't any more secure than the 'hiding' you get in C++.
>>> var.GetX.func_closure[0].cell_contents.x
5
All you've done is force the user who wants to bypass it to use a longer
expression, and if that's your criterion for 'hiding' then just use two
leading underscores.
--
Duncan Booth http://kupuguy.blogspot.com
Carl Banks
Red Herring.
1. This doesn't hide the variables; it just changes their spelling.
2. This also "hides" the variables from its own class.
In other words, it's a useless no-op.
In fact, I'd say this is even worse than useless. Creating accessor
functions is a sort of blessing for external use. Knowing that there
are accessor functions is likely to cause a user to show even less
restraint.
Carl Banks
Antoon Pardon
So?
>>>> var.GetX.func_closure[0].cell_contents.x
> 5
>
> All you've done is force the user who wants to bypass it to use a longer
> expression, and if that's your criterion for 'hiding' then just use two
> leading underscores.
That you can find a lock pick to get at an object doesn't contradict
that the object is locked away.
I think the intention of not having these variables accesable to
the application programmer is much stronger expressed than with
two leading underscores.
Even if the current implementation of the language makes it
relatively easy to get at the information if you really want
to.
--
Antoon Pardon
Antoon Pardon
I think you completed missed the point.
This is just a proof of concept thing. In a real example there would
of course no Set en Get methods but just methods that in the course
of their execution would access or update the hidden attributes
--
Antoon Pardon
Carl Banks
I'm not sure I missed the point so much as I failed to read your mind.
> This is just a proof of concept thing. In a real example there would
> of course no Set en Get methods but just methods that in the course
> of their execution would access or update the hidden attributes
Fair enough, but I don't see anything in your example that suggests a
way to discriminate between access from within the class and access
from outside the class, which is the crucial aspect of data hiding.
Carl Banks
Carl Banks
> Fair enough, but I don't see anything in your example that suggests a
> way to discriminate between access from within the class and access
> from outside the class, which is the crucial aspect of data hiding.
And, if you want an example of something that does that, how about
this metaclass. It creates a class that checks the stack frame to see
if the caller was defined in the same class.
Issues:
Classes are prevented from defining their own __setattr__ and
__getattribute__.
Classes and subclasses should not use the same names for their private
variables.
Private attribute access is pretty slow, but that's obvious.
Pretty easy to thwart.
#----------------------------------
import sys
import itertools
class PrivateAccessError(Exception):
pass
class PrivateDataMetaclass(type):
def __new__(metacls,name,bases,dct):
function = type(lambda x:x)
privates = set(dct.get('__private__',()))
codes = set()
for val in dct.itervalues():
if isinstance(val,function):
codes.add(val.func_code)
getframe = sys._getframe
count = itertools.count
def __getattribute__(self,attr):
if attr in privates:
for i in count(1):
code = getframe(i).f_code
if code in codes:
break
if code.co_name != '__getattribute__':
raise PrivateAccessError(
"attribute '%s' is private" % attr)
return super(cls,self).__getattribute__(attr)
def __setattr__(self,attr,val):
if attr in privates:
for i in count(1):
code = getframe(i).f_code
if code in codes:
break
if code.co_name != '__setattr__':
raise PrivateAccessError(
"attribute '%s' is private" % attr)
return super(cls,self).__setattr__(attr,val)
dct['__getattribute__'] = __getattribute__
dct['__setattr__'] = __setattr__
cls = type.__new__(metacls,name,bases,dct)
return cls
#----------------------------------
import traceback
class A(object):
__metaclass__ = PrivateDataMetaclass
__private__ = ['internal']
def __init__(self,n):
self.internal = n
def inc(self):
self.internal += 1
def res(self):
return self.internal
class B(A):
__private__ = ['internal2']
def __init__(self,n,m):
super(B,self).__init__(n)
self.internal2 = m
def inc(self):
super(B,self).inc()
self.internal2 += 2
def res(self):
return self.internal2 + super(B,self).res()
def bad(self):
return self.internal2 + self.internal
a = A(1)
a.inc()
print "Should print 2:"
print a.res()
print
print "Should raise PrivateAccessError:"
try:
print a.internal
except PrivateAccessError:
traceback.print_exc()
print
b = B(1,1)
b.inc()
print "Should print 5:"
print b.res()
print
print "Should raise PrivateAccessError:"
try:
print b.internal2
except PrivateAccessError:
traceback.print_exc()
print
print "Should raise PrivateAccessError:"
try:
print b.bad()
except PrivateAccessError:
traceback.print_exc()
print
#----------------------------------
Carl Banks
Antoon Pardon
>>
>> > In fact, I'd say this is even worse than useless. Creating accessor
>> > functions is a sort of blessing for external use. Knowing that there
>> > are accessor functions is likely to cause a user to show even less
>> > restraint.
>>
>> I think you completed missed the point.
>
> I'm not sure I missed the point so much as I failed to read your mind.
Fine with me, it is just the other side of the coin.
>> This is just a proof of concept thing. In a real example there would
>> of course no Set en Get methods but just methods that in the course
>> of their execution would access or update the hidden attributes
>
> Fair enough, but I don't see anything in your example that suggests a
> way to discriminate between access from within the class and access
> from outside the class, which is the crucial aspect of data hiding.
The fact is that hidden and its attributes are not accessible from
outside the instance. They are only accessible to the local functions
of __init__. By binding those local functions as atributes to the
instance, hidden can be modified by what for all practical purposes
looks like a method call, but really is a closure call.
--
Antoon Pardon
Carl Banks
You haven't hidden the data at all, all you've done is to change the
means of accessing it. What difference does it make whether I write
foo.getX() or foo.x? Everyone in the world still has full access to
the data.
You are not realizing that only useful(**) thing about data hiding is
that some code has access to the data, other code does not. If you
"hide" data equally from everyone it's just a useless spelling change.
** - Usefulness is questionable in most cases, but we assume it is
here for the sake of argument.
Carl Banks
Antoon Pardon
Can't you look beyond the specific example? The GetX is just an example.
Any local function of __init__ has access to hidden and its attributes
and could manipulate them, even if the class wouldn't define getters
and setters.
> You are not realizing that only useful(**) thing about data hiding is
> that some code has access to the data, other code does not. If you
> "hide" data equally from everyone it's just a useless spelling change.
I already explained this. The code is a proof of concept. I agree that
the example itself doesn't do anything usefull, that doesn't mean that
the concept it is trying to illustrate is useless too.
Your metaclass can be used to make attributes private. Your metaclass
doesn't stop doing that because someone makes a class with a getter
and a setter for those private attributes. The same goes for my example.
The closure makes a variable inaccessable to the outside. That doesn't
change because I defined a getter and a setter for that closure
variable.
--
Antoon Pardon
Russ P.
> You are not realizing that only useful(**) thing about data hiding is
> that some code has access to the data, other code does not. If you
> "hide" data equally from everyone it's just a useless spelling change.
I think you're missing the point.
As I see it, the primary value of data hiding is that it provides
useful information on which data and methods are intended for the
client and which are intended for internal use. It's like putting a
front panel on a TV set with the main controls intended for the
viewer.
People seem to be preoccupied with whether or not the back panel of
the TV is locked, but that is not the main issue. Sure, you probably
want to make the back panel removable, but you don't want the viewer
opening it up to change the channel, and you certainly don't want to
put all the internal adjustments for factory technicians together with
the controls for the end user.
As far as I am concerned, the current Python method of using
underscores to distinguish between internal and external methods and
data is an ugly hack that goes completely against the elegance of the
language in other areas. It is like a TV set with no back cover and
the volume and channel controls intermingled with the factory
controls. The underscores are just an afterthought like a red dot or
something used to tell the TV viewer what to fiddle with.
Python is a very nice language overall, but as far as I am concerned
the underscore convention is a blemish. I just wish people wouldn't
get so infatuated with the language that they cannot see the obvious
staring them in the face.
Carl Banks
> On Jun 2, 6:41 am, Carl Banks <pavlovevide...@gmail.com> wrote:
>
> > You are not realizing that only useful(**) thing about data hiding is
> > that some code has access to the data, other code does not. If you
> > "hide" data equally from everyone it's just a useless spelling change.
>
> I think you're missing the point.
Well that's nice: you're accusing me of missing the point after having
quoted something I wrote as if it represented by own views, even
though I footnoted it and said I was only doing it for the sake of
argument. Perhaps, outside this discussion, I am totally getting "the
point".
I can't tell, though, because I read your post twice and I have no
idea what you consider "the point" to be.
Best as I can tell you are claiming that data hiding isn't important,
but then you go on to imply Python is blemished because it doesn't
hide data. It really makes no sense: perhaps you can help us out by
giving us an example of something that illustrates what you're saying?
(FWIW, my actual view on the matter is I don't give a lick about data
hiding or marking internals.)
Carl Banks
Diez B. Roggisch
> Well that's nice: you're accusing me of missing the point after having
> quoted something I wrote as if it represented by own views, even
> though I footnoted it and said I was only doing it for the sake of
> argument. Perhaps, outside this discussion, I am totally getting "the
> point".
>
> I can't tell, though, because I read your post twice and I have no
> idea what you consider "the point" to be.
>
> Best as I can tell you are claiming that data hiding isn't important,
> but then you go on to imply Python is blemished because it doesn't
> hide data. It really makes no sense: perhaps you can help us out by
> giving us an example of something that illustrates what you're saying?
>
>
> (FWIW, my actual view on the matter is I don't give a lick about data
> hiding or marking internals.)
Do yourself a favor and google antoon's previous posts in this group. He
sure is a clever guy & I wouldn't call him a troll - but a bit trollish..
Diez
Russ P.
I think I made my point abundantly clear. I said that rigorously
denied access to encapsulated data is not important, but a clear
specification of what is intended for the client and what is intended
for internal use *is* important. And an ugly naming convention for
variable and function names is not the best way to do it.
I suggest you ask yourself why C++, Java, Ada, and probably most other
"object-oriented" languages, have encapsulation or "data hiding." And
no, I am not claiming that *every* application written in Python needs
it, but certainly *some* could benefit from it.
> (FWIW, my actual view on the matter is I don't give a lick about data
> hiding or marking internals.)
What kind of programming do you do?
Carl Banks
> On Jun 2, 3:04 pm, Carl Banks <pavlovevide...@gmail.com> wrote:
>
>
>
> > On Jun 2, 4:50 pm, "Russ P." <Russ.Paie...@gmail.com> wrote:
>
> > > On Jun 2, 6:41 am, Carl Banks <pavlovevide...@gmail.com> wrote:
>
> > > > You are not realizing that only useful(**) thing about data hiding is
> > > > that some code has access to the data, other code does not. If you
> > > > "hide" data equally from everyone it's just a useless spelling change.
>
> > > I think you're missing the point.
>
> > Well that's nice: you're accusing me of missing the point after having
> > quoted something I wrote as if it represented by own views, even
> > though I footnoted it and said I was only doing it for the sake of
> > argument. Perhaps, outside this discussion, I am totally getting "the
> > point".
>
> > I can't tell, though, because I read your post twice and I have no
> > idea what you consider "the point" to be.
> > Best as I can tell you are claiming that data hiding isn't important,
> > but then you go on to imply Python is blemished because it doesn't
> > hide data. It really makes no sense: perhaps you can help us out by
> > giving us an example of something that illustrates what you're saying?
>
> I think I made my point abundantly clear. I said that rigorously
> denied access to encapsulated data is not important, but a clear
> specification of what is intended for the client and what is intended
> for internal use *is* important. And an ugly naming convention for
> variable and function names is not the best way to do it.
Well let me tell you what's confusing me here: I can't figure out, if
this is your standpoint, what issue you could have had with what I
said. What specifically did you disagree with? What did I say that
was wrong? It seems like we are more in agreement than not.
> I suggest you ask yourself why C++, Java, Ada, and probably most other
> "object-oriented" languages, have encapsulation or "data hiding." And
> no, I am not claiming that *every* application written in Python needs
> it, but certainly *some* could benefit from it.
I know what it's there for, chief. That's exactly what I was saying
to Antoon, and you took issue with it and claimed I was missing the
point. What gives?
Carl Banks
Russ P.
I thought you were saying that encapsulation or so-called "data
hiding" is worthless. If I misunderstood you, then I apologize. I
don't have time to go back and sort it all out.
Here's what I think Python should have. I think it should have a
keyword, something like "priv," to identify data or functions as
"private." As I said earlier, "private" for class data or functions
("methods") could be implemented like "protected" in C++. That means
that derived classes would have access to it, but clients of the class
would not. If the client really needs or wants access, he could be
given a sort of "back door" access similar to the current Python rule
regarding double leading underscores. Thus, the client would have
access, but he would know very well that he is using something that
the original designer did not intend for him to use.
It's just a suggestion. I'm not a language expert, and I realize that
I could be missing something important.
I also realize, by the way, that Python allows a client of a class to
define a new class member from completely outside the class
definition. Obviously, that cannot be declared private. But if the
same identifier is already declared private within the class, than the
new definition should not be allowed (because it would defeat the
whole idea of "private" class members).
Paul Rubin
> I also realize, by the way, that Python allows a client of a class to
> define a new class member from completely outside the class
> definition. Obviously, that cannot be declared private.
This is bogus about 95% of the time though. For the cases where it is
really desired, I think it's best to require the target class to be
enable it specifically somehow, maybe by inheriting from a special
superclass. That could let the compiler statically resolve member
lookups the rest of the time.
Carl Banks
> I thought you were saying that encapsulation or so-called "data
> hiding" is worthless. If I misunderstood you, then I apologize. I
> don't have time to go back and sort it all out.
No, not at all. I was suggesting that Antoon's example of data hiding
wasn't useful because it didn't really hide data: it was merely a
spelling change.
(I'm not, mind you, saying that it's ordinarily a good trade-off to
encapsulate data, but I do get the point of it.)
> Here's what I think Python should have. I think it should have a
> keyword, something like "priv," to identify data or functions as
> "private." As I said earlier, "private" for class data or functions
> ("methods") could be implemented like "protected" in C++. That means
> that derived classes would have access to it, but clients of the class
> would not. If the client really needs or wants access, he could be
> given a sort of "back door" access similar to the current Python rule
> regarding double leading underscores. Thus, the client would have
> access, but he would know very well that he is using something that
> the original designer did not intend for him to use.
Reasonable enough. I've always thought C++ should have a
private_cast.
Carl Banks
Carl Banks
Ok. You could have made the proof-of-concept-ness of your example
more clear by not, you know, binding those local functions to globally
accessable names, thus thwarting your own concept.
But still, we all knew functions could do that. How could you make
data private to a group of functions (like, say, methods of a class,
or all functions in a module). It seems you'd have to define all
functions in the the same function scope, instead of where they'd more
naturally be defined (in the class or module scope). Unwieldy to say
the least, but doable, I suppose.
Carl Banks
alex23
> Here's what I think Python should have. I think it should have a
> keyword, something like "priv," to identify data or functions as
> "private."
As I stated earlier in this thread, if you want a public interface and
a private implementation, rather than adding another language feature
why not just separate them into two classes? This is exactly what the
Bridge pattern provides and would clearly denote your intention in the
code.
Russ P.
Yes, that looks interesting, but I think it has a couple of drawbacks.
First, it requires another completely separate class for the
"implementation" (although perhaps that could be a nested class).
Secondly, I think it essentially just adds a sort of inner namespace
through which the "private" data is accessed. That might be a good
idea, but I don't think it's quite the same as encapsulation.
Russ P.
It did seem a bit odd to me when I realized that you can add data
members (or even a "methods") to a class from completely outside the
class definition. That can be risky, of course, and as you suggest,
perhaps it shouldn't even be allowed by default.
I usually find that it's safer to initialize in the constructor all
(or nearly all) of the data members that will be needed in a class. If
I need a list that will be populated later, for example, I reserve the
name with an empty list in the constructor. Then, if for some reason
the list gets accessed before it is populated, I don't get an
exception.
alex23
> Yes, that looks interesting, but I think it has a couple of drawbacks.
> First, it requires another completely separate class for the
> "implementation" (although perhaps that could be a nested class).
That's hardly an enormous overhead, and it does clearly separate the
interface you want your "users" to have from the implementation. Even
better, since you seem so concerned with others meddling with your
implementation directly, they could provide their own quite easily if
they so choose.
> Secondly, I think it essentially just adds a sort of inner namespace
> through which the "private" data is accessed. That might be a good
> idea, but I don't think it's quite the same as encapsulation.
It's a clear separation of concerns, check. It removes the underscored
methods you find so aesthetically offensive, check. I have absolutely
no idea what _you_ mean by "encapsulation".
Then again, I have no issue with the current convention and personally
find the idea of adding a "private" keyword makes as much sense as
being able to syntactically define "model", "view" and "controller"
methods.
Russ P.
I'm not saying your bridge pattern is no good. It is a form of
encapsulation, I suppose, but it is cumbersome. With my suggested
"priv" keyword, you just use it once (per data member or function) and
you never need to use it again. With your approach, not only do you
need twice the number of classes, you need to use the longer names
every time you access the data.
Russ P.
> Then again, I have no issue with the current convention and personally
> find the idea of adding a "private" keyword makes as much sense as
> being able to syntactically define "model", "view" and "controller"
> methods.
Well, the designers of C++, Java, and Ada, to name just three very
popular languages (well, two) seem to think it makes sense. But maybe
you know more than they know.
alex23
> Well, the designers of C++, Java, and Ada, to name just three very
> popular languages (well, two) seem to think it makes sense. But maybe
> you know more than they know.
You do realise the same argument could be made about you and the
designers of Python, right?
But regardless, an argument from authority is a sure sign that this
isn't worth continued debate.
Good luck with the PEP.
Antoon Pardon
Well maybe you all knew that, yet while I read the thread, I saw
noone mentioning this possibility to get what the OP seemed to
want. I also did't see other posibilities. Your idea as an alternative
only came as a reaction to my idea.
I know my idea as presented was cumbersome. Maybe it can be worked out
more, maybe with the help of decorators to be less cumbersome.
I also know it can be circumvented in CPython.
Yet I think is was usefull to explore what is already possible in Python
instead of just argueing about whether people need it or not
I always find it odd when the "you don't need it" argument is used.
Of course that is correct, put python has acquired a lot of things
that were originnally labeled "you don't need it". So why people
still use it against introducing something, is something I don't
entirely understand.
--
Antoon Pardon
BJörn Lindqvist
> On Jun 2, 6:41 am, Carl Banks <pavlovevide...@gmail.com> wrote:
>
>> You are not realizing that only useful(**) thing about data hiding is
>> that some code has access to the data, other code does not. If you
>> "hide" data equally from everyone it's just a useless spelling change.
>
> I think you're missing the point.
>
> As I see it, the primary value of data hiding is that it provides
> useful information on which data and methods are intended for the
> client and which are intended for internal use. It's like putting a
> front panel on a TV set with the main controls intended for the
> viewer.
Here's my two cents. First of all, a TV is a bad analogy compared to
reusable software libraries. Really bad analogy. A TV is a horribly
complicated device which has to be dumbed down because otherwise it
would be to hard to use for ordinary people.
A software developers relation to a third party library is more
similar to a TV repair man trying to repair a TV than to a random
person watching TV. For a repair man, the front panel is just useless
and in the way.
Oh, and to continue on the TV analogy, one of the reason why a TV is
complicated is because its interface is totally different from its
implementation. Channels are just a bad abstraction for tuning the
receiver to different frequencies and for switching inputs. Merely
using a TV doesn't teach you anything about how it actually works.
KISS: Keep It Simple Stupid. And it is always simpler to not implement
the gunk needed for data hiding than to do it. By keeping things
simple you keep your code easy to implement, easy to understand and
easy to reuse.
Data hiding sacrifices implementation simplicity supposedly to make
the interface simpler and to keep backwards compatibility. It allows
you to change implementation details without affecting the
interface. But do you really want to do that? Consider this silly Java
example:
class Foo {
private int bar;
public int getBar() {
return bar;
}
};
Then for some reason you decide that hm, "bar" is not a good attribute
name so you change it to "babar". And you can do that without changing
the public interface! Woho! So now you have a public getter named
"getBar" that returns an attribute named "babar". That's in reality
just bad and whoever is maintaining the implementation is going to be
annoyed that the getters name doesn't match the attribute name.
What would have happened without data hiding? Renaming the public
attribute "bar" to "babar" probably cause some grief for someone
reusing your library, but you would keep your implementation pure.
What about semantic changes? Data hiding doesn't protect you against
that, so you'll have to change your interface anyway. The interface
for a car hasn't changed much in the last 100 years, but the
implementation has. How easy is it to repair a car nowadays compared
to 30 years ago?
And data hiding as a documentation aid is just a sham. "These methods
are public so you can call them, these aren't so hands off!" A reuser
of your library *will* want to know what happens on the inside, by
trying to make stuff impossible to reach you are just making that kind
of information much harder to come by.
The better method is to just write proper docstrings that tell the
user what the methods do and when they can be called.
Another good way to see how useless data hiding is, is to try and unit
test a very encapsulated library. You'll see that it is almost
impossible to write good unit tests unless you publicly export
almost everything in the code. At which point you come to realize that
all the data hiding was for naught.
--
mvh Björn
Lie
> On May 24, 2:58 pm, Ben Finney <bignose+hates-s...@benfinney.id.au>
> wrote:
>
> > Sh4wn <luckyluk...@gmail.com> writes:
> > > first, python is one of my fav languages, and i'll definitely keep
> > > developing with it. But, there's 1 one thing what I -really- miss:
> > > data hiding. I know member vars are private when you prefix them with
> > > 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> > > before it.
>
> > From whom are you trying to hide your attributes?
>
> Actually, 'data hiding', although vastly overused by the static crowd
> can be a reasonable thing to want.
>
> For example, at Resolver Systems we expose the spreadsheet object
> model to our users. It hasa public, documented, API - plus a host of
> undocumented internally used methods.
>
> We would really *much* rather hide these, because anything our
> customers start using (whether documented or not) we will probably
> have to continue supporting and maintaining.
>
> The 'we told you not to use that' approach, when applied to paying
> customers doesn't really work... all they see is that you broke their
> spreadsheet code by changing your API.
>
> You can make members truly private by proxying, but it is a bit
> ungainly.
Then don't document it, or separate internal documentation (which is
never to pass through the wall) and public documentation (which your
users use). Nobody would (apart from your dev team and anyone told by
your dev team, which means you may fire the person for "lack of
discipline") know that there is such a thing and in consequence
wouldn't use it.
Don't tell your user not to use something, just don't tell them that
it exists and they won't use it.
George Sakkis
And even more (well, almost all) languages use explicit delimiters for
defining blocks instead of indentation, so what's your point ?
Lie
> On Mon, Jun 2, 2008 at 10:50 PM, Russ P. <Russ.Paie...@gmail.com> wrote:
> > On Jun 2, 6:41 am, Carl Banks <pavlovevide...@gmail.com> wrote:
>
> >> You are not realizing that only useful(**) thing about data hiding is
> >> that some code has access to the data, other code does not. If you
> >> "hide" data equally from everyone it's just a useless spelling change.
>
> > I think you're missing the point.
>
> > As I see it, the primary value of data hiding is that it provides
> > useful information on which data and methods are intended for the
> > client and which are intended for internal use. It's like putting a
> > front panel on a TV set with the main controls intended for the
> > viewer.
>
> Here's my two cents. First of all, a TV is a bad analogy compared to
> reusable software libraries. Really bad analogy. A TV is a horribly
> complicated device which has to be dumbed down because otherwise it
> would be to hard to use for ordinary people.
I think it's actually quite a good analogy, a class may get quite
complicated and it may do completely different things that the public
interface seems to imply. Anyway, an analogy is an analogy, don't
expect it to be exactly the same as the case itself, expect it to
illustrate the point well enough and ignore the differences not being
illustrated.
TV is a good analogy since it illustrated the point quite well, that
there are some things user may freely interact, some that users should
not mess with, and things that is strictly not for you. Nevertheless,
with the proper knowledge and proper tools, any users could open the
case and get the special screwdriver to open the TV, if all else
fails, he could always get a hammer to break the casing and gone his
way through.
Python does not enforce data-hiding because it expect people that gone
his way to ignore the warning and get the special screwdriver to be
knowledgeable enough to mess with it. C/C++ expects people to use
hammer to break through their casings, and in the end, since the
casings has already been broken, the device may never look as
beautiful as before. In python, the device may appear to look just as
beautiful.
> A software developers relation to a third party library is more
> similar to a TV repair man trying to repair a TV than to a random
> person watching TV. For a repair man, the front panel is just useless
> and in the way.
No, for a knowledgable man (a TV repairman), he'd try first to fix the
TV without opening the case (such as seeing whether the power cable is
actually plugged), and if those attempts fails (or if he already know
where the damage is from the beginning), he'd then open the screws.
The public interface isn't "useless and in the way".
> Oh, and to continue on the TV analogy, one of the reason why a TV is
> complicated is because its interface is totally different from its
> implementation. Channels are just a bad abstraction for tuning the
> receiver to different frequencies and for switching inputs. Merely
> using a TV doesn't teach you anything about how it actually works.
Why couldn't a class have interface that's completely different thing
than the implementation.
And we expect those people is ready that the car may blow off right in
their face since they have violated the lines. If they broke the lines
and still think that we're guilty for his burnt face, that's their
problem.
Antoon Pardon
Hmm, difficult to react to this. On the one hand I have had people
argue that block delimiting in python is explicit too. So in that
case python doesn't differ from those other languages.
On the other hand if we accept that blocks are delimited implicitely
in python then it seems python doesn't follow its own zen:
Explicit is better than implicit
--
Antoon Pardon
Giuseppe Ottaviano
> Hmm, difficult to react to this. On the one hand I have had people
> argue that block delimiting in python is explicit too. So in that
> case python doesn't differ from those other languages.
>
> On the other hand if we accept that blocks are delimited implicitely
> in python then it seems python doesn't follow its own zen:
>
> Explicit is better than implicit
So also duck typing is against python's philosophy? :)
Lie
> On May 24, 2:58 pm, Ben Finney <bignose+hates-s...@benfinney.id.au>
> wrote:
>
> > Sh4wn <luckyluk...@gmail.com> writes:
> > > first, python is one of my fav languages, and i'll definitely keep
> > > developing with it. But, there's 1 one thing what I -really- miss:
> > > data hiding. I know member vars are private when you prefix them with
> > > 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> > > before it.
>
> > From whom are you trying to hide your attributes?
>
> Actually, 'data hiding', although vastly overused by the static crowd
> can be a reasonable thing to want.
>
> For example, at Resolver Systems we expose the spreadsheet object
> model to our users. It hasa public, documented, API - plus a host of
> undocumented internally used methods.
>
> We would really *much* rather hide these, because anything our
> customers start using (whether documented or not) we will probably
> have to continue supporting and maintaining.
>
> The 'we told you not to use that' approach, when applied to paying
> customers doesn't really work... all they see is that you broke their
> spreadsheet code by changing your API.
The problem is you're not hard enough, you let yourself to be enslaved
by your customer. If they have a problem because they used a private
interface, that's their problem, they have to fix it at their side or
go away and use a competing product[1]. Even if they're paying
customers they're not your master or your God, even if they're a
larger company than yours.
Python has an extremely good design because the BDFL doesn't just
listen to everyone and create a product that tries to please
everybody, no, he listens to those that have good ideas and tells the
stupid ideas to go away and he applies a subjective decision which
more often than not leads to a better python.
[1] In most cases, they would become silent at this point and fix
their code, because they know there is nothing they can do to change
your decision. It's often more expensive to move to competing products
so they'd either use old versions or fix those places where they've
used private interface, and avoid using private interface in the
future.
> You can make members truly private by proxying, but it is a bit
> ungainly.
>
> Michael Foordhttp://www.ironpythoninaction.com/
>
>
(snip)
greg
> On May 24, 9:41 am, Sh4wn <luckyluk...@gmail.com> wrote:
>
> > Python advertises himself as a full OOP language, but why does it miss
> > one of the basic principles of OOP? Will it ever be added to python?
>
> Others have already answered this directly, but I'd like to mention
> that languages I know of which have this feature also have a feature
> for getting around it. (e.g. C++ and friend classes) I don't know
> about you, but I don't want features in the language that make me want
> to circumvent them. Do you?
I curious as how 'private' fits with the Open Source philosophy?
Sure, I can (and do) hide stuff with the double underscore technique,
but anyone using my code can open it up and add an "accessor" method
anytime they want, so nothing is really hidden. I think the
"consenting adults" approach is the best one could hope for with FOSS.
Bruno Desthuilliers
> On Jun 2, 6:41 am, Carl Banks <pavlovevide...@gmail.com> wrote:
>
>> You are not realizing that only useful(**) thing about data hiding is
>> that some code has access to the data, other code does not. If you
>> "hide" data equally from everyone it's just a useless spelling change.
>
> I think you're missing the point.
>
> As I see it, the primary value of data hiding is that it provides
> useful information on which data and methods are intended for the
> client and which are intended for internal use. It's like putting a
> front panel on a TV set with the main controls intended for the
> viewer.
>
> People seem to be preoccupied with whether or not the back panel of
> the TV is locked, but that is not the main issue. Sure, you probably
> want to make the back panel removable, but you don't want the viewer
> opening it up to change the channel, and you certainly don't want to
> put all the internal adjustments for factory technicians together with
> the controls for the end user.
>
> As far as I am concerned, the current Python method of using
> underscores to distinguish between internal and external methods and
> data is an ugly hack that goes completely against the elegance of the
> language in other areas.
As far as I'm concerned, it's JustFine(tm). I don't have to ask myself
if an attribute is part of the API or not, I know it immediatly.
> It is like a TV set with no back cover and
> the volume and channel controls intermingled with the factory
> controls. The underscores are just an afterthought like a red dot or
> something used to tell the TV viewer what to fiddle with.
Your opinion. But beware of leaky TV-Set-metaphor abstractions
> Python is a very nice language overall, but as far as I am concerned
> the underscore convention is a blemish. I just wish people wouldn't
> get so infatuated with the language that they cannot see the obvious
> staring them in the face.
I definitively don't have problem with this naming convention, which I'd
find useful ever with a language having enforced access restrictions. If
that's the only - or worse - wart you find in Python, then it must
surely be a pretty good language !-)
sturlamolden
> first, python is one of my fav languages, and i'll definitely keep
> developing with it. But, there's 1 one thing what I -really- miss:
> data hiding. I know member vars are private when you prefix them with
> 2 underscores, but I hate prefixing my vars, I'd rather add a keyword
> before it.
Python has no data hiding because C++ has (void *).
Python underscores does some name mangling, but does not attempt any
data hiding.
Python and C has about the same approach to data hiding. It is well
tried, and works equally well in both languages:
# this is mine, keep your filthy paws off!!!
Irresponsible programmers should not be allowed near a computer
anyway. If you use data hiding to protect your code from yourself,
what you really need is some time off to reconsider your career.
Richard Levasseur
I really like this message and find it very true. Writing unit tests
for private data is nigh impossible. You end up either creating
accessors, or passing in parameters via the constructor (resulting in
a huge constructor). Personally, I'd rather have better test coverage
than data hiding.
Second, private vars with third party libs suck, and are nothing but
an infuriating frustration. I'm currently dealing with about 3 or 4
different libs, one of them uses private variables and its a huge
headache. I have to access some of those private vars occasionally to
make my thing work. The other libs i'm using don't have any private
vars (__) (only a couple protected ones, _), and its a breeze. The
docs say "this does x" or there's a comment that says "don't use this
unless you really know what you're doing," and I respect their
warnings.
When I was fooling around with sqlalchemy, it made heavy use of
protected vars but had a straight forward public api. Unfortunately,
writing plugins for it required access to some of those protected
vars. It wouldn't be possible if they were strictly controlled and
restricted by the language itself. Whenever I'd use those protected
vars, I expected an odd behavior or two. When using private vars, I
don't expect it to work at all, and really, refrain from using them
unless i've grokked the source.
My point is that I currently like the private/protected/public scheme
python has going on. It lets me fix or alter things if I have to, but
also provides a warning that I shouldn't be doing this.
As for customers using the internals and worrying about an upgrade
breaking them, it seems likes a silly issue, at least in python. If
there are internals that the customer would be playing with, then it
should be exposed publically, since they want it that way to begin
with. If they're using defunct variables or methods, you use
properties and __getattr__ to maintain backwards compatibility for a
version or two.
sturlamolden
> I think you completed missed the point.
>
> This is just a proof of concept thing. In a real example there would
> of course no Set en Get methods but just methods that in the course
> of their execution would access or update the hidden attributes
I have to agree with Banks here, you have not provided an example of
data hiding. It does not discriminate between attribute access from
within and from outside the class. You just assume that the attribute
named 'hidden' will be left alone. Also naming it hidden is stupid as
it is visible.
What you need is a mechanism that will thrown an exception whenever an
attribue is accessed from outside the class, but not from inside.
The mechanism must also be impossible to override with additional
code.
If Ada is what you want, Ada is what you should use.
Russ P.
You are comparing a syntactic convention with a more fundmaental
aspect of the language. But beyond that, I dislike braces as
delimiters for the same reason I dislike leading underscores: both are
unnecessary syntactic noise. And the whole idea of encoding properties
of an object in its name just seems tacky to me.
What is it about leading underscores that bothers me? To me, they are
like a small pebble in your shoe while you are on a hike. Yes, you can
live with it, and it does no harm, but you still want to get rid of it.
Russ P.
If you think that private data and methods should not be allowed
because they complicate unit testing, then I suggest you take a look
at how unit testing is done is C++, Java, and Ada. They seem to do
just fine. Also, I have stated several times now that "back door"
access should be allowed. That should satisfy any need for access to
"private" data in unit testing.
But I think there is a more fundamental issue here. You complain about
problems with software that uses data encapsulation. So two
possibilities exist here: either the designers of the code were not
smart enough to understand what data or methods the client would need,
or the client is not smart enough to understand what they need. Maybe
the solution is smarter programmers and clients rather than a dumber
language.
alex23
> If you think that private data and methods should not be allowed
> because they complicate unit testing, then I suggest you take a look
> at how unit testing is done is C++, Java, and Ada. They seem to do
> just fine.
Nice to put the burden of evidence back onto everyone else, but doing
a bit of searching I found the following "answers" to the question of
unit-testing private functions & methods:
> I suggest that tests should be written only for the public methods.
> You can use a debugger, probably Carbide. That way you can see
> all the variables. Otherwise, write the values to a log or EMCT.
> You can make the logging only happen for debug builds if you don't
> want the logging in the production code. If you really need to
> see the private variables from your code, declare them public in
> debug builds.
> Problem is testing private functions. Some can be fixed by
> promoting private to protected, inheriting the class adding
> testing in the class. Others get refactored out the classes
> they reside in and get put into their own functor classes[...]
So the basic answers I'm seeing that "do just fine" are:
1. Don't test private functions.
2. Add functionality _to_ the private functions for testing.
3. Change the interface for the purpose of testing.
All of which seem exceptionally inefficient and run counter to the
whole purpose of unit testing.
> But I think there is a more fundamental issue here. You complain about
> problems with software that uses data encapsulation. So two
> possibilities exist here: either the designers of the code were not
> smart enough to understand what data or methods the client would need,
> or the client is not smart enough to understand what they need. Maybe
> the solution is smarter programmers and clients rather than a dumber
> language.
This is the most ludicrous argument I've ever heard. Of _course_ we
can't predict every possible usage of our code that others might want
it for. If someone can easily extend code that I've written to improve
or increase its functionality, why would I want to prevent them from
doing so?
Then again, I tend to think of other programmers as "peers" rather
than clients. YMMV.
Ben Finney
> So the basic answers I'm seeing that "do just fine" are:
>
> 1. Don't test private functions.
> 2. Add functionality _to_ the private functions for testing.
> 3. Change the interface for the purpose of testing.
>
> All of which seem exceptionally inefficient and run counter to the
> whole purpose of unit testing.
It seems you have a different idea of what unit testing is for from
me.
Isn't the entire point of encapsulation to separate internal
components from the external interface?
Why would a unit test, the whole purpose of which is to assert some
aspect of the external behaviour of the unit of code, care about how
that code unit is implemented internally?
If changing the internal, encapsulated components of a unit causes its
external behaviour to change, that's a bug; either in the change made
(it shouldn't have altered the external behaviour), or in the unit
test asserting the wrong thing (it shouldn't be asserting anything
about internal state of the code).
--
\ “Try to become not a man of success, but try rather to become |
`\ a man of value.” —Albert Einstein |
_o__) |
Ben Finney
Russ P.
wrote:
Thank you. Let me just add that, as I said before, I think "private"
data (if it were added to Python) should be accessible through some
sort of "indirect" mechanism akin to the double-leading-underscore
rule. Then, even if it *is* needed for unit testing, it can be
accessed.
As for unit testing in C++, Java, and Ada, I confess I know nothing
about it, but I assume it gets done. Considering that Ada is used to
manage and control fighter jets, cruise missiles, and nuclear
arsenals, let's hope it gets done right.
Marc 'BlackJack' Rintsch
> alex23 <wuw...@gmail.com> writes:
>
>> So the basic answers I'm seeing that "do just fine" are:
>>
>> 1. Don't test private functions.
>> 2. Add functionality _to_ the private functions for testing.
>> 3. Change the interface for the purpose of testing.
>>
>> All of which seem exceptionally inefficient and run counter to the
>> whole purpose of unit testing.
>
> It seems you have a different idea of what unit testing is for from
> me.
For me it's about finding bugs where documentation and implementation
disagree. And if you document private functions it makes sense to me to
also test if they work as documented. Because the official API relies on
the correct implementation of the private parts it uses under the hood.
> Isn't the entire point of encapsulation to separate internal
> components from the external interface?
>
> Why would a unit test, the whole purpose of which is to assert some
> aspect of the external behaviour of the unit of code, care about how
> that code unit is implemented internally?
One part of writing unit tests is invoking functions with arguments that
you think are "corner cases". For example test if a function that takes a
list doesn't bomb out when you feed the empty list into it. Or if it
handles all errors correctly.
If a function `f()` calls internally `_g()` and that function might even
call other private functions, then you have to know how `f()` works
internally to create input that checks if error handling in `_g()` works
correctly. So it goes against your understanding of unit tests.
What do you do in such a situation? Build something from untested private
parts and just test the assembled piece? I prefer to test the private
functions too. After all the private functions are not private to the
everybody, there *are* functions that rely on them working correctly.
Ciao,
Marc 'BlackJack' Rintsch
Ben Finney
> On Wed, 04 Jun 2008 13:50:42 +1000, Ben Finney wrote:
>
> > It seems you [alex23] have a different idea of what unit testing
> > is for from me.
>
> For me it's about finding bugs where documentation and
> implementation disagree.
Where "documentation" is "specified externally-visible behaviour of
the unit", I agree with this.
> And if you document private functions
By definition, "private" functions are not part of the publicly
documented behaviour of the unit. Any behaviour exhibited by some
private component is seen externally as a behaviour of some public
component.
> it makes sense to me to also test if they work as documented.
If they affect the behaviour of some public component, that's where
the documentation should be.
If they *don't* affect the external behaviour, then they shouldn't be
there :-) Or, at least, their behaviour shouldn't be asserted as part
of the tests of external behaviour.
> Because the official API relies on the correct implementation of the
> private parts it uses under the hood.
Only to the extent that the documented behaviour of the API is
affected. The main benefit of marking something as "not public" is
that one *is* free to change its behaviour, so long as the public API
is preserved.
> One part of writing unit tests is invoking functions with arguments
> that you think are "corner cases". For example test if a function
> that takes a list doesn't bomb out when you feed the empty list into
> it. Or if it handles all errors correctly.
This sounds like part of the externally-visible behaviour of the code
unit; i.e. something that belongs in the public API. I agree that this
is the domain of a unit test.
> If a function `f()` calls internally `_g()` and that function might
> even call other private functions, then you have to know how `f()`
> works internally to create input that checks if error handling in
> `_g()` works correctly.
No, you don't need to know how it works internally; you need only know
what guarantees it must keep for its external behaviour.
If someone wants to alter the `_g()` function, or remove it entirely
while preserving the correct behaviour of `f()`, that should have no
effect on the external behaviour of `f()`.
That is to say, the knowledge of the "internals" of `f()` in your
example is actually knowledge of something that should be documented
as part of the external behaviour of `f()` — that, or it's not
relevant to the behaviour of `f()` and shouldn't be unit tested in
order that encapsulation is preserved.
> What do you do in such a situation? Build something from untested
> private parts and just test the assembled piece?
Assert the corner-case behaviour of `f()`, through unit tests that
operate on `f()` without caring about its internals.
> I prefer to test the private functions too. After all the private
> functions are not private to the everybody, there *are* functions
> that rely on them working correctly.
Then for *those* interfaces, unit tests can be devised that make
assertions about those interfaces.
--
\ "[T]he speed of response of the internet will re-introduce us |
`\ to that from which our political systems have separated us for |
_o__) so long, the consequences of our own actions." -- Douglas Adams |
Ben Finney
Antoon Pardon
>
> Python has an extremely good design because the BDFL doesn't just
> listen to everyone and create a product that tries to please
> everybody, no, he listens to those that have good ideas and tells the
> stupid ideas to go away and he applies a subjective decision which
> more often than not leads to a better python.
I agree that Guido van Rossum has done an excellent job. That doesn't
mean he has to be painted as unfailable in which the ideais he accepts
are good ideas and those he rejects are bad ideas almost by definition.
Guido has been known to change his mind, which is an admirabele quality,
but it does show that at some point he rejected a good idea or accepted
a bad idea.
--
Antoon Pardon
Antoon Pardon
> On Jun 2, 12:40 pm, Antoon Pardon <apar...@forel.vub.ac.be> wrote:
>
>> I think you completed missed the point.
>>
>> This is just a proof of concept thing. In a real example there would
>> of course no Set en Get methods but just methods that in the course
>> of their execution would access or update the hidden attributes
>
> I have to agree with Banks here, you have not provided an example of
> data hiding. It does not discriminate between attribute access from
> within and from outside the class. You just assume that the attribute
> named 'hidden' will be left alone. Also naming it hidden is stupid as
> it is visible.
No I don't assume that hidden wil be left alone. hidden is a free
variable in a closure and thus simply can't be accessed except by
local functions that were made accessible (and some mechanism
dependant on the CPython implementation).
> What you need is a mechanism that will thrown an exception whenever an
> attribue is accessed from outside the class, but not from inside.
And my example does this. It threw an AttributeError
> The mechanism must also be impossible to override with additional
> code.
Which as far as I know it is.
--
Antoon Pardon
Marc 'BlackJack' Rintsch
> Marc 'BlackJack' Rintsch <bj_...@gmx.net> writes:
>
>> On Wed, 04 Jun 2008 13:50:42 +1000, Ben Finney wrote:
>>
>> > It seems you [alex23] have a different idea of what unit testing
>> > is for from me.
>>
>> For me it's about finding bugs where documentation and
>> implementation disagree.
>
> Where "documentation" is "specified externally-visible behaviour of
> the unit", I agree with this.
>
>> And if you document private functions
>
> By definition, "private" functions are not part of the publicly
> documented behaviour of the unit. Any behaviour exhibited by some
> private component is seen externally as a behaviour of some public
> component.
But only indirectly, and it's often harder to predict the corner cases
that might trigger bugs or to test error testing in dependent private
functions. Private functions offer an API that's public to someone, so
they ought to be documented and tested.
>> it makes sense to me to also test if they work as documented.
>
> If they affect the behaviour of some public component, that's where
> the documentation should be.
As I said they are public themselves for someone.
>> Because the official API relies on the correct implementation of the
>> private parts it uses under the hood.
>
> Only to the extent that the documented behaviour of the API is
> affected. The main benefit of marking something as "not public" is
> that one *is* free to change its behaviour, so long as the public API
> is preserved.
One more reason to test the individual private functions because a change
of such a function shouldn't make it necessary to change the unit tests of
the public API.
>> One part of writing unit tests is invoking functions with arguments
>> that you think are "corner cases". For example test if a function
>> that takes a list doesn't bomb out when you feed the empty list into
>> it. Or if it handles all errors correctly.
>
> This sounds like part of the externally-visible behaviour of the code
> unit; i.e. something that belongs in the public API. I agree that this
> is the domain of a unit test.
>
>> If a function `f()` calls internally `_g()` and that function might
>> even call other private functions, then you have to know how `f()`
>> works internally to create input that checks if error handling in
>> `_g()` works correctly.
>
> No, you don't need to know how it works internally; you need only know
> what guarantees it must keep for its external behaviour.
How do you know the "corner cases" then? Often it is interesting how a
function that takes integers copes with zero, so that might be a test.
It's easy if you test a function directly but you need to know the
internals if you must find arguments that lead to a dependent function
called with zero. Contrived example:
def _g(i):
return (42 / i) if i else 0
def f(x):
return _g(x + 23)
Here ``f(-23)`` is a special corner case that should be tested somehow.
And I think it is better tested as explicit test of `_g()` than with a
test of `f()`. Testing for "corner cases" needs some knowledge about the
implementation, but that shouldn't be "transitive". The tests for `f()`
should assume that `_g()` itself has promised in its documentation was
already covered by a unit test.
> If someone wants to alter the `_g()` function, or remove it entirely
> while preserving the correct behaviour of `f()`, that should have no
> effect on the external behaviour of `f()`.
>
> That is to say, the knowledge of the "internals" of `f()` in your
> example is actually knowledge of something that should be documented
> as part of the external behaviour of `f()` — that, or it's not
> relevant to the behaviour of `f()` and shouldn't be unit tested in
> order that encapsulation is preserved.
`f()`'s documentation should mention that "it works for all integers
including -23" with -23 explicitly mentioned?
>> What do you do in such a situation? Build something from untested
>> private parts and just test the assembled piece?
>
> Assert the corner-case behaviour of `f()`, through unit tests that
> operate on `f()` without caring about its internals.
And this way missing many potential bugs?
>> I prefer to test the private functions too. After all the private
>> functions are not private to the everybody, there *are* functions
>> that rely on them working correctly.
>
> Then for *those* interfaces, unit tests can be devised that make
> assertions about those interfaces.
Now you lost me. So essentially you don't test private functions unless
they are used somewhere, then they should be tested too. As private
functions that are not used, shouldn't be there in the first place, every
function private or public should be tested, right!?
Ciao,
Marc 'BlackJack' Rintsch
Antoon Pardon
>>> it makes sense to me to also test if they work as documented.
>>
>> If they affect the behaviour of some public component, that's where
>> the documentation should be.
>
> As I said they are public themselves for someone.
Isn't that contradictory: "Public for someone" I always
thought "public" meant accessible to virtually anyone.
Not to only someone.
--
Antoon Pardon
NickC
> I saw this "don't need it" pattern in discussions about the ternary
> "if..else" expression and about "except/finally on the same block
> level".
> Now Python has both.
if/else was added solely because people kept coming up with ways of
embedding a pseudo conditional inside expressions and writing buggy
code in the process. All it really saves you in practice is a bit of
vertical whitespace, so, no, you still don't need it - but if you
insist on doing it, at least there's now an easy way to do it
correctly.
except/finally on the same block level was trivial to implement once
the reference interpreter switched to an AST based compiler for 2.5.
If you look at the AST, you'll find that it still only has TryExcept
and TryFinally, so again, you still don't need except/finally on the
same block level - all the syntax allows you to do is omit the second
try: line and its associated indentation.
> Actually it is very useful to be able to
> distinguish
> between inside and outside. This is obvious for real world things e.g.
> your
> TV. Nobody likes to open the rear cover to switch the channel. Similar
> arguments apply to software objects. "data hiding" is a harsh name, I
> would
> call it "telling what matters". The need for this becomes
> indispensable in
> really big software packages like the Eclipse framework with approx.
> 100000
> classes. If you cannot tell the difference between inside and outside
> you
> are lost.
>
> > In Python, the philosophy "we're all consenting adults here" applies.
>
> Please don't sell a missing feature as a philosophy. Say you don't
> need/want
> it. But don't call it philosophy.
Gosh, and here I thought treating programmers as non-idiots was
actually one of the guiding philosophies in the discussion on python-
dev. Good thing we have you here to tell us we're only imagining that.
> > You shouldn't pretend to know, at the time you write it, all the uses
> > to which your code will be put.
>
> It's *your* *decision* which uses will be available. Your explanation
> appears
> to me as a fear to decide.
Are you writing application code or library code? For application
code, you have a much greater idea of the uses for your code, so you
can be confident in your decision as to what should and should not be
visible. For library code, however, it's fairly common for a library
to provide something which is almost, but not quite, what the user
needs. Letting users poke around at their own risk is a nice courtesy
that can save them a lot of work in the long run.
So the decision to hide something is still made (by using an
underscore prefix), but an easy mechanism is provided for the library
user to override that decision.
> > If you want the users of your code to know that an attribute should
> > not be used as a public API for the code, use the convention of naming
> > the attribute with a single leading underscore.
>
> Littering your class definition with dozens of underscores is exactly
> the
> line noise we love to criticize in Perl.
Using underscores in names (leading or otherwise) separated by
plaintext keywords is a far cry from multiple different symbols that
mean different things in different contexts and can be chained
together fairly arbitrarily.
> > > Python advertises himself as a full OOP language, but why does it
> > > miss one of the basic principles of OOP?
>
> > Who taught you that enforced restrictions on attribute access was a
> > "basic principle" of OO?
>
> Nearly every introduction to OOP? Please don't tell me that
> encapsulation
> does not mean "enforced restriction". If the language has no syntactic
> support for encapsulation then it does not have encapsulation.
Module globals aren't visible outside the module without importing it.
Class attributes aren't visible outside the class without derefencing
it.
Instance attributes aren't visible outside an instance without
deferencing one.
*That* is the encapsulation/data hiding which OOP requires, and is the
kind which Python enforces. What you're asking for is encapsulation of
class and instance attributes based on the context in which the
dereferencing occurs (inside the class, inside a subclass of that
class, inside an instance of that class, inside an instance of a
subclass of that class, somewhere else entirely), and that has nothing
to do with the basics of OOP.
On the other hand, if you're so keen on this feature, perhaps you'd
like to make a concrete proposal regarding how you would like the
semantics to work in light of Python dynamic typing model. What will
it do when a method is invoked via the class dict rather than via
attribute retrieval? Can unbound methods access protected or private
attribute? How about descriptor get, set and delete methods? What
happens when a function is added to a class definition after creation
as a new method?
Cheers,
Nick.
NickC
> I am also bothered a bit by the seeming inconsistency of the rules for
> the single underscore. When used at file scope, they make the variable
> or function invisible outside the module, but when used at class
> scope, the "underscored" variables or functions are still fully
> visible. For those who claim that the client should be left to decide
> what to use, why is the client prohibited from using underscored
> variables at file scope?
They aren't - the only thing that won't see the underscore prefixed
names is "from x import *". If you do "import x" instead, all the
underscored names will be accessible as attributes of the module.
Paul Rubin
> if/else was added solely because people kept coming up with ways of
> embedding a pseudo conditional inside expressions and writing buggy
> code in the process. All it really saves you in practice is a bit of
> vertical whitespace, so, no, you still don't need it - but if you
> insist on doing it, at least there's now an easy way to do it
> correctly.
Come on, it's more than vertical whitespace, it's extraneous variables
and sometimes even extraneous functions and function call overhead.
And Python is supposed to be unbureaucratic. People kept looking for
ways to write conditional expressions instead of spewing the logic
across multiple statements for a reason: the code is often cleaner
that way.
NickC
> What is it about leading underscores that bothers me? To me, they are
> like a small pebble in your shoe while you are on a hike. Yes, you can
> live with it, and it does no harm, but you still want to get rid of it.
With leading underscores, you can see *at the point of dereference*
that the code is accessing private data. With a "this is private"
keyword you have no idea whether you're accessing private or public
data, because the two namespaces get conflated together.
I'll keep my pebble, thanks.
Cheers,
Nick.
NickC
> > Python was not really written with 'difficult' customers in mind ;-)
>
> True. It's extremely suited to what we do though.Minor difficulties
> like this are vastly outweighed by advantages. The difficulties are
> real though.
It's interesting to take a look at some of the work Brett Cannon has
done trying to come up with a sandbox for executing Python code that
actually manages to block access to dangerous functions like file() or
urllib.urlopen(). Powerful introspection capabilities and restricted
access to methods and attributes don't really play well together.
http://svn.python.org/view/python/branches/bcannon-objcap/securing_python.txt?rev=55685&view=markup
(I believe that work is on hiatus while he's been busy with other
projects, such as a more flexible Python-based reimplementation of the
import mechanism that would be make it possible to implement the
security restrictions needed to permit limited imports in a sandboxed
interpreter)
> > One could largely hide private vars with a program that substituted random
> > names for single _ names, and removed the doc strings for functions,
> > classes, and methods with such names.
>
> We need to *use* those names to display the spreadsheet once the
> calculation has finished (and their code has run).
>
> > Such a program could even put such names in a separate module imported as
> > '_private_do_not_use_'.
>
> Splitting more of the functionality out is probably part of the best
> solution.
Yeah, at this point your only hope is going to be making them go
through such wild contortions to get at the internal data they think
better of it. Actually blocking all access to something written in
Python is fairly tough (you generally need an extension class written
in non-Python code that hides access to certain attributes).
Cheers,
Nick.
NickC
> Guido has been known to change his mind, which is an admirabele quality,
> but it does show that at some point he rejected a good idea or accepted
> a bad idea.
And sometimes the person that talked him into accepting the bad idea
in the first place ends up agreeing with him when he eventually
rejects it ;)
Cheers,
Nick.
P.S. Read the list of references in PEP 343 if you want to know what
I'm talking about *cough*
Ben Finney
> On Wed, 04 Jun 2008 15:55:38 +1000, Ben Finney wrote:
>
> > By definition, "private" functions are not part of the publicly
> > documented behaviour of the unit. Any behaviour exhibited by some
> > private component is seen externally as a behaviour of some public
> > component.
>
> But only indirectly
No, that's the point: externally, such behaviour is exhibited by the
public API. Whatever internal moving parts actually lead to the
behaviour doesn't make any difference: it's the external behaviour
that's being discussed in the above.
> and it's often harder to predict the corner cases that might trigger
> bugs or to test error testing in dependent private functions.
Indeed. The person writing the unit tests should do so with full
knowledge of what the implementation looks like. This allows coverage
of those corner cases you rightly point out exist in many
implementations.
Those unit tests should *not*, though, exercise anything but the
public API, otherwise they're breaking encapsulation. Their assertion
should continue to be just as true after a refactoring of the internal
components as before.
> Private functions offer an API that's public to someone, so they
> ought to be documented and tested.
No, that's pretty much the point: private functions are intended for
use by nothing except other functions at the same scope. If they're to
be used in other contexts, they're public API, not private.
It's a feature of Python that such bad API design doesn't lead to
hideous workarounds: one can still easily get at the parts of an API
that the programmer mistakenly marked "private". That doesn't make it
any less a mistake to break encapsulation, but it does make it much
more easily fixed.
--
\ "During the Middle Ages, probably one of the biggest mistakes |
`\ was not putting on your armor because you were 'just going down |
_o__) to the corner.'" -- Jack Handey |
Ben Finney
NickC
True, but it really was the multitude of buggy workarounds for the
lack of a ternary expression that sealed the deal, rather than the
benefits of ternary expressions in their own right :)
Given that I personally use ternary expressions solely as the right
hand side of an assignment statement, the reduction in vertical
whitespace usage really is the only thing they gain me. I guess if you
embedded them as an argument to a function call or other more
complicated expression then there may be additional savings. I prefer
not to do that though, since such things can get quite difficult to
parse mentally when reading them later.
Cheers,
Nick.
NickC
wrote:
> Those unit tests should *not*, though, exercise anything but the
> public API, otherwise they're breaking encapsulation. Their assertion
> should continue to be just as true after a refactoring of the internal
> components as before.
Python must have bad unit tests then - the CPython test suite
explicitly tests private methods all the time.
There's actually an extremely good reason for doing it that way: when
the implementation of an internal method gets broken, the unit tests
flag it explicitly, rather than having to derive the breakage from the
breakage of 'higher level' unit tests (after all, you wouldn't factor
something out into its own method or function if you weren't using it
in at least a couple of different places).
Black box testing (testing only the public API) is certainly
important, but grey box and white box testing that either exploits
knowledge of the implementation when crafting interesting test cases,
or explicitly tests internal APIs can be highly beneficial in
localising faults quickly when something does break (and as any
experienced maintenance programmer will tell you, figuring out what
you actually broke is usually harder than fixing it after you find it).
Antoon Pardon
> On May 26, 7:32 am, "Joe P. Cool" <joe.p.c...@googlemail.com> wrote:
>> I saw this "don't need it" pattern in discussions about the ternary
>> "if..else" expression and about "except/finally on the same block
>> level".
>> Now Python has both.
>
> if/else was added solely because people kept coming up with ways of
> embedding a pseudo conditional inside expressions and writing buggy
> code in the process. All it really saves you in practice is a bit of
> vertical whitespace, so, no, you still don't need it - but if you
> insist on doing it, at least there's now an easy way to do it
> correctly.
If I remember correctly it was added because one of the python
developers was bitten by a bug in the standard library code
that was caused by the use of the and-or emulation, mentioned
in the FAQ.
And although one indeed doesn't need this. There are a lot
of things in Python one doesn't need. Python could be limited
to single operator expressions. You don't need:
x = a * b + c
You can write it just like this:
x = a * b
x = x + c
And if you want a list comprehension like the following:
ls = [ x * x + 4 for x in xrange(10)]
You can of course write it as follows:
def sqrplus4(a):
rs = a * a
return rs + 4
ls = [sqrplus4(x) for x in xrange(10)]
Now of course noone would defend such a limitation on the grounds
that one doesn't need the general case and that the general case
will only save you some vertical space.
But when it came to the ternary operator that was exactly the
argument used, to defend the lack of it.
>> > In Python, the philosophy "we're all consenting adults here" applies.
>>
>> Please don't sell a missing feature as a philosophy. Say you don't
>> need/want
>> it. But don't call it philosophy.
>
> Gosh, and here I thought treating programmers as non-idiots was
> actually one of the guiding philosophies in the discussion on python-
> dev.
I have heard the argument: "Such a feature will be abused too easily"
and similar too many times to find this credible.
--
Antoon Pardon