[Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi,

(I'd like to fork from a previous thread, "Pre-conditions and post-conditions", since it got long and we started discussing a couple of different things. Let's put the general discussion related to design-by-contract in this thread and I'll spawn another thread for the discussion about the concrete implementation of a design-by-contract library in Python.)

After the discussion we had on the list and after browsing the internet a bit, I'm still puzzled why design-by-contract was not more widely adopted and why so few languages support it. Please have a look at these articles and answers:

• https://www.leadingagile.com/2018/05/design-by-contract-part-one/
• https://ask.slashdot.org/story/07/03/10/009237/why-is-design-by-contract-not-more-popular -- this one is from 2007, but represents well IMO the way people discuss it
  
• https://stackoverflow.com/questions/481312/why-is-design-by-contract-not-so-popular-compared-to-test-driven-development and this answer in particular https://stackoverflow.com/a/28680756/1600678
  
• https://softwareengineering.stackexchange.com/questions/128717/why-is-there-such-limited-support-for-design-by-contract-in-most-modern-programm

I did see that there are a lot of misconceptions about it ("simple asserts", "developer overhead", "needs upfront design", "same as unit testing"). This is probably the case with any novel concept that people are not familiar with. However, what does puzzle me is that once the misconceptions are rectified ("it's not simple asserts", "the development is actually faster", "no need for upfront design", "not orthogonal, but dbc + unit testing is better than just unit testing"), the concept is still discarded.

After properly reading about design-by-contract and getting deeper into the topic, there is no rational argument against it and the benefits are obvious. And still, people just wave their hand and continue without formalizing the contracts in the code and keep on writing them in the descriptions.

Why is that so? I'm completely at loss about that -- especially about the historical reasons (some mentioned that design-by-contract did not take off since Bertrand Meyer holds the trademark on the term and because of his character. Is that the reason?).

One explanation that seems plausible to me is that many programmers are actually having a hard time with formalization and logic rules (e.g., implication, quantifiers), maybe due to missing education (e.g. many programmers are people who came to programming from other less-formal fields). It's hence easier for them to write in human text and takes substantial cognitive load to formalize these thoughts in code. Does that explains it?

What do you think? What is the missing part of the puzzle?

Cheers,

Marko

[David Mertz]

On Sun, Sep 23, 2018, 1:10 AM Marko Ristin-Kaufmann <marko....@gmail.com> wrote:

One explanation that seems plausible to me is that many programmers are actually having a hard time with formalization and logic rules (e.g., implication, quantifiers), maybe due to missing education (e.g. many programmers are people who came to programming from other less-formal fields). It's hence easier for them to write in human text and takes substantial cognitive load to formalize these thoughts in code. Does that explains it?

I've tried to explain my own reasons for not being that interested in DbC in other threads. I've been familiar with DbC libraries in Python for close to 20 years, and it never struck me as worth the effort of using.

I'm not alone in this. A large majority of folks formally educted in computer science and related fields have been aware of DbC for decades but deliberately decided not to use them in their own code. Maybe you and Bertram Meyer are simple better than that 99% of programmers... Or maybe the benefit is not so self-evidently and compelling as it feels to you.

To me, as I've said, DbC imposes a very large cost for both writers and readers of code. While it's possible to split hairs about the edge cases where assertions and unit tests cannot cover identical ground, the reality is that the benefits are extremely close between the different techniques. However, it's vastly easier to take a more incremental and as-needed approach using assertions and unit tests than it is with DbC. 

Moreover, unit tests have the giant advantage of living *elsewhere* than in the main code itself... This probably doesn't matter so much to writers, but it's a huge win for readers. Even with doctests—which I'm somewhat unusual in actually liking—even though the tests live in the same file and function/class as the operational code, it still feels relatively easy to separate the concerns visual when reading such code. I just cannot get that with DbC.

I know you can inside I'm wrong about all this, and my code would be better and faster if I would accept this niche orthodoxy. But I just do not see DbC becoming non-niche in any plausible future, neither in Python not in any other mainstream language. Opinions are free to differ, and I could be wrong.

[Angus Hollands]

Hi Marko,

I think there are several ways to approach this problem, though am not weighing in on whether DbC is a good thing in Python. I wrote a simple implementation of DbC which is currently a run-time checker. You could, with the appropriate tooling, validate statically too (as with all approaches). In my approach, I use a “proxy” object to allow the contract code to be defined at function definition time. It does mean that some things are not as pretty as one would like - anything that cannot be hooked into with magic methods i.e isinstance, but I think this is acceptable as it makes features like old easier. Also, one hopes that it encourages simpler contract checks as a side-effect. Feel free to take a look - https://github.com/agoose77/pyffel
It is by no means well written, but a fun PoC nonetheless.
Regards,
Angus

​

[Hugh Fisher]

> Date: Sun, 23 Sep 2018 07:09:37 +0200
> From: Marko Ristin-Kaufmann <marko....@gmail.com>
> To: Python-Ideas <python...@python.org>
> Subject: [Python-ideas] Why is design-by-contracts not widely adopted?

[ munch ]

> *. *After properly reading about design-by-contract and getting deeper into

> the topic, there is no rational argument against it and the benefits are
> obvious. And still, people just wave their hand and continue without
> formalizing the contracts in the code and keep on writing them in the
> descriptions.

Firstly, I see a difference between rational arguments against Design By
Contract (DbC) and against DbC in Python. Rejecting DbC for Python is
not the same as rejecting DbC entirely.

Programming languages are different, obviously. Python is not the same
as C is not the same as Lisp... To me this also means that different
languages are used for different problem domains, and in different styles
of development. I wouldn't use DbC in programming C or assembler
because it's not really helpful for the kind of low level close to the machine
stuff I use C or assembler for. And I wouldn't use DbC for Python because
I wouldn't find it helpful for the kind of dynamic, exploratory development
I do in Python. I don't write strict contracts for Python code because in a
dynamically typed, and duck typed, programming language they just don't
make sense to me. Which is not to say I think Design by Contract is bad,
just that it isn't good for Python.

Secondly, these "obvious" benefits. If they're obvious, I want to know why
aren't you using Eiffel? It's a programming language designed around DbC
concepts. It's been around for three decades, at least as long as Python or
longer. There's an existing base of compilers and support tools and libraries
and textbooks and experienced programmers to work with.

Could it be that Python has better libraries, is faster to develop for, attracts
more programmers? If so, I suggest it's worth considering that this might
be *because* Python doesn't have DbC.

Or is this an updated version of the old saying "real programmers write
FORTRAN in any language" ? If you are accustomed to Design by Contract,
think of your time in the Python world as a trip to another country. Relax
and try to program like the locals do. You might enjoy it.

--

cheers,
Hugh Fisher
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Marko Ristin-Kaufmann]

Hi,

Thank you for your replies, Hugh and David! Please let me address the points in serial.

Obvious benefits

You both seem to misconceive the contracts. The goal of the design-by-contract is not reduced to testing the correctness of the code, as I reiterated already a couple of times in the previous thread. The contracts document formally what the caller and the callee expect and need to satisfy when using a method, a function or a class. This is meant for a module that is used by multiple people which are not necessarily familiar with the code. They are not a niche. There are 150K projects on pypi.org. Each one of them would benefit if annotated with the contracts.

Please do re-read my previous messages on the topic a bit more attentively. These two pages I also found informative and they are quite fast to read (<15 min):
https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html
https://gleichmann.wordpress.com/2007/12/09/test-driven-development-and-design-by-contract-friend-or-foe/

Here is a quick summary of the argument.

When you are documenting a method you have the following options:

1) Write preconditions and postconditions formally and include them automatically in the documentation (e.g., by using icontract library).

2) Write precondtions and postconditions in docstring of the method as human text.

3) Write doctests in the docstring of the method.

4) Expect the user to read the actual implementation.

5) Expect the user to read the testing code.

Here is what seems obvious to me. Please do point me to what is not obvious to you because that is the piece of puzzle that I am missing (i.e. why this is not obvious and what are the intricacies). I enumerated the statements for easier reference:

a) Using 1) is the only option when you have to deal with inheritance. Other approaches can no do that without much repetition in practice.

b) If you write contracts in text, they will become stale over time (i.e. disconnected from the implementation and plain wrong and misleading). It is a common problem that the comments rot over time and I hope it does not need further argument (please let me know if you really think that the comments do not rot).

c) Using 3), doctests, means that you need mocking as soon as your method depends on non-trivial data structures. Moreover, if the output of the function is not trivial and/or long, you actually need to write the contract (as postcondition) just after the call in the doctest. Documenting preconditions includes writing down the error that will be thrown. Additionally, you need to write that what you are documenting actually also holds for all other examples, not just for this particular test case (e.g., in human text as a separate sentence before/after the doctest in the docstring).

d) Reading other people's code in 4) and 5) is not trivial in most cases and requires a lot of attention as soon as the method includes calls to submethods and functions. This is impractical in most situations since most code is non-trivial to read and is subject to frequent changes.

e) Most of the time, 5) is not even a viable option as the testing code is not even shipped with the package and includes going to github (if the package is open-sourced) and searching through the directory tree to find the test. This forces every user of a library to get familiar with the testing code of the library.

f) 4) and 5) are obviously a waste of time for the user -- please do explain why this might not be the case. Whenever I use the library, I do not expect to be forced to look into its test code and its implementation. I expect to read the documentation and just use the library if I'm confident about its quality. I have rarely read the implementation of the standard libraries (notable exceptions in my case are ast and subprocess module) or any well-established third-party library I frequently use (numpy, opencv, sklearn, nltk, zmq, lmdb, sqlalchemy). If the documentation is not clear about the contracts, I use trial-and-error to figure out the contracts myself. This again is obviously a waste of time of the user and it's far easier to read the contracts directly than use trial-and-error.

Contracts are difficult to read.

David wrote:

To me, as I've said, DbC imposes a very large cost for both writers and readers of code.

This is again something that eludes me and I would be really thankful if you could clarify. Please consider for an example, pypackagery (https://pypackagery.readthedocs.io/en/latest/packagery.html) and the documentation of its function resolve_initial_paths:

packagery.resolve_initial_paths(initial_paths)

Resolve the initial paths of the dependency graph by recursively adding *.py files beneath given directories.

Parameters:	initial_paths (List[Path]) – initial paths as absolute paths
Return type:	List[Path]
Returns:	list of initial files (i.e. no directories)
Requires:	all(pth.is_absolute() for pth in initial_paths)
Ensures:	len(result) >= len(initial_paths) if initial_paths else result == [] all(pth.is_absolute() for pth in result) all(pth.is_file() for pth in result)

How is this difficult to read, unless the reader is not familiar with formalism and has a hard time parsing the quantifiers and logic rules? Mind that all these bits are deemed important by the writer -- and need to be included in the function description  somehow -- you can choose between 1)-5). 1) seems obviously best to me. 1) will be tested at least at test time. If I have a bug in the implementation (e.g., I include a directory in the result), the testing framework will notify me again.

Here is what the reader would have needed to read without the formalism in the docstring as text (i.e., 2):

* All input paths must be absolute.

* If the initial paths are empty, the result is an empty list.

* All the paths in the result are also absolute.

* The resulting paths only include files.

and here is an example with doctest (3):

>>> result = packagery.resolve_initial_paths([])
[]

>>> with temppathlib.NamedTemporaryFile() as tmp1, \
...         temppathlib.NamedTemporaryFile() as tmp2:
...     tmp1.path.write_text("some text")
...     tmp2.path.write_text("another text")
...     result = packagery.resolve_initial_paths([tmp1, tmp2])
...     assert all(pth.is_absolute() for pth in result)
...     assert all(pth.is_file() for pth in result)

>>> with temppathlib.TemporaryDirectory() as tmp:
...     packagery.resolve_initial_paths([tmp.path])
Traceback (most recent call last):
        ...
ValueError("Unexpected directory in the paths")

>>> with temppathlib.TemporaryDirectory() as tmp:
...     pth = tmp.path / "some-file.py"
...     pth.write_text("some text")
...     packagery.initial_paths([pth.relative_to(tmp.path)])
Traceback (most recent call last):
        ...
ValueError("Unexpected relative path in the initial paths")

Now, how can reading the text (2, code rot) or reading the doctests (3, longer, includes contracts) be easier and more maintainable compared to reading the contracts? I would be really thankful for the explanation -- I feel really stupid as for me this is totally obvious and, evidently, for other people it is not.

I hope we all agree that the arguments about this example (resolve_initial_paths) selected here are not particular to pypackagery, but that they generalize to most of the functions and methods out there.

Writing contracts is difficult.

David wrote:

To me, as I've said, DbC imposes a very large cost for both writers and readers of code.

The effort of writing contracts include as of now:

* include icontract (or any other design-by-contract library) to setup.py (or requirements.txt), one line one-off

* include sphinx-icontract to docs/source/conf.py and docs/source/requirements.txt, two lines, one-off

* write your contracts (usually one line per contract).

The contracts (1) in the above-mentioned function look like this (omitting the contracts run only at test time):

@icontract.pre(lambda initial_paths: all(pth.is_absolute() for pth in initial_paths))
@icontract.post(lambda result: all(pth.is_file() for pth in result))
@icontract.post(lambda result: all(pth.is_absolute() for pth in result))
@icontract.post(lambda initial_paths, result: len(result) >= len(initial_paths) if initial_paths else result == [])
def resolve_initial_paths(initial_paths: List[pathlib.Path]) -> List[pathlib.Path]:
    ...

Putting aside how this code could be made more succinct (use "args" or "a" argument in the condition to represent the arguments, using from ... import ..., renaming "result" argument to "r", introducing a shortcut methods slowpre and slowpost to encapsulate the slow contracts not to be executed in the production), how is this difficult to write? It's 4 lines of code.

Writing text (2) is 4 lines. Writing doctests (3) is 23 lines and includes the contracts. Again, given that the writer is trained in writing formal expressions, the mental effort is the same for writing the text and writing the formal contract (in cases of non-native English speakers, I'd even argue that formal expressions are sometimes easier to write).

99% vs 1%

I'm not alone in this. A large majority of folks formally educated in computer science and related fields have been aware of DbC for decades but deliberately decided not to use them in their own code. Maybe you and Bertram Meyer are simple better than that 99% of programmers... Or maybe the benefit is not so self-evidently and compelling as it feels to you.

I think that ignorance plays a major role here. Many people have misconceptions about the design-by-contract. They just use 2) for more complex methods, or 3) for rather trivial methods. They are not aware that it's easy to use the contracts (1) and fear using them for non-rational reasons (e.g., habits).

This is also what Todd Plesel writes in https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html#IfDesignByContractIsSoGreat:

The vast majority of those developing software - even that intended to be reused - are simply ignorant of the concept. As a result they produce application programmer interfaces (APIs) that are under-specified thus passing the burden to the application programmer to discover by trial and error, the 'acceptable boundaries' of the software interface (undocumented contract's terms). But such ad-hoc operational definitions of software interface discovered through reverse-engineering are subject to change upon the next release and so offers no stable way to ensure software correctness.

The fact that many people involved in writing software lack pertinent education (e.g., CS/CE degrees) and training (professional courses, read software engineering journals, attend conferences etc.) is not a reason they don't know about DBC since the concept is not covered adequately in such mediums anyway. That is, ignorance of DBC extends not just throughout practitioners but also throughout educators and many industry-experts.

He lists some more factors and misconceptions that hinder the adoption. I would definitely recommend you to read at least that section if not the whole piece.

The conclusion paragraph "Culture Shift: Too Soon or Too Late" was also telling:

The simplicity and obvious benefits of Design By Contract lead one to wonder why it has not become 'standard practice' in the software development industry. When the concept has been explained to various technical people (all non-programmers), they invariably agree that it is a sensible approach and some even express dismay that software components are not developed this way.

It is just another indicator of the immaturity of the software development industry. The failure to produce high-quality products is also blatantly obvious from the non-warranty license agreement of commercial software. Yet consumers continue to buy software they suspect and even expect to be of poor quality. Both quality and lack-of-quality have a price tag, but the difference is in who pays and when. As long as companies can continue to experience rising profits while selling poor-quality products, what incentive is there to change? Perhaps the fall-out of the "Year 2000" problem will focus enough external pressure on the industry to jolt it towards improved software development methods. There is talk of certifying programmers like other professionals. If and when that occurs, the benefits of Design By Contract just might begin to be appreciated.

But it is doubtful. Considering the typical 20 year rule for adopting superior technology, DBC as exemplified by Eiffel, has another decade to go. But if Java succeeds in becoming a widely-used language and JavaBeans become a widespread form of reuse then it would already be too late for DBC to have an impact. iContract will be a hardly-noticed event much like ANNA for Ada and A++ for C++. This is because the philosophy/mindset/culture is established by the initial publication of the language and its standard library.

(Emphasis mine; iContract refers to a Java design-by-contract library)

Hence the salient argument is the lack of tools for DbC. So far, none of the existing DbC libraries in Python really have the capabilities to be included in the code base. The programmer had to duplicate the contract, the messages did not include the values involved in the condition, one could not inherit the contracts and the contracts were not included in the documentation. Some libraries supported some of these features, but none up to icontract library supported them all. icontract finally supports all these features.

I have never seen a rational argument how writing contracts (1) is inferior to approaches 2-5), except that it's hard for programmers untrained in writing formal expressions and for the lack of tools. I would be really thankful if you could address these points and show me where I am wrong given that formalism and tools are not a problem. We can train the untrained, and we can develop tools (and put them into standard library). This will push adoption to far above 1%.

Finally, it is obvious to me that the documentation is important. I see lacking documentation as one of the major hits in the productivity of a programmer. If there is a tool that could easily improve the documentation (i.e. formal contracts with one line of code per contract) and automatically keep it in sync with the code (by checking the contracts during the testing), I don't see any rational reason why you would dispense of such a tool. Again, please do correct me and contradict -- I don't want to sound condescending or arrogant -- I literally can't wrap my head around why anybody would dispense of a such an easy-to-use tool that gives you better documentation (i.e. superior to approaches 2-5) except for lack of formal skills and lack of supporting library. If you think that the documentation is not important, then please, do explain that since it goes counter to all my previous experience and intuition (which, of course, can be wrong).

Why not Eiffel?

Hugh wrote:

Secondly, these "obvious" benefits. If they're obvious, I want to know why
aren't you using Eiffel? It's a programming language designed around DbC
concepts. It's been around for three decades, at least as long as Python or
longer. There's an existing base of compilers and support tools and libraries
and textbooks and experienced programmers to work with.

Could it be that Python has better libraries, is faster to develop for, attracts
more programmers? If so, I suggest it's worth considering that this might
be *because* Python doesn't have DbC.

Python is easier to write and read, and there are no libraries which are close in quality in Eiffel space (notably, Numpy, OpenCV, nltk and sklearn). I really don't see how the quality of these libraries have anything to do with lack (or presence) of the contracts. OpenCV and Numpy have contracts all over their code (written as assertions and not documented), albeit with very non-informative violation messages. And they are great libraries. Their users would hugely benefit from a more mature and standardized contracts library with informative violation messages.

Duck Typing

Hugh wrote:

And I wouldn't use DbC for Python because
I wouldn't find it helpful for the kind of dynamic, exploratory development
I do in Python. I don't write strict contracts for Python code because in a
dynamically typed, and duck typed, programming language they just don't
make sense to me. Which is not to say I think Design by Contract is bad,
just that it isn't good for Python.

I really don't see how DbC has to do with duck typing (unless you reduce it to mere isinstance conditions, which would simply be a straw-man argument) -- could you please clarify? As soon as you need to document your code, and this is what most modules have to do in teams of more than one person (especially so if you are developing a library for a wider audience), you need to write down the contracts. Please see above where I tried to explained  that 2-5) are inferior approaches to documenting contracts compared to 1).

As I wrote above, I would be very, very thankful if you point me to other approaches (apart from 1-5) that are superior to contracts or state an argument why approaches 2-5) are superior to the contracts since that is what I miss to see.

Cheers,

Marko

[Barry Scott]

This is an interesting PoC, nice work! I like that its easy to read the tests.

Given a library like this the need to build DbC into python seems unnecessary.

What do other people think?

Barry

Regards,
Angus

​

[Barry Scott]

On 23 Sep 2018, at 11:33, Hugh Fisher <hugo....@gmail.com> wrote:

Could it be that Python has better libraries, is faster to develop for, attracts
more programmers? If so, I suggest it's worth considering that this might
be *because* Python doesn't have DbC.

I'm not sure how you get from the lack of DbC being a feature to python's success.

I use DbC in my python code via the asserts and its been very useful in my experience.

If there was a nice way to get better then the assert method I'd use it. Like Angus's PoC.

I assume that developers that are not interesting in DbC would simply not use any

library/syntax that supported it.

Barry

[Marko Ristin-Kaufmann]

Hi Barry,

I think the main issue with pyffel is that it can not support function calls in general. If I understood it right, and Angus please correct me, you would need to wrap every function that you would call from within the contract.

But the syntax is much nicer than icontract or dpcontracts (see these packages on pypi). What if we renamed "args" argument and "old" argument in those libraries to just "a" and "o", respectively? Maybe that gives readable code without too much noise:

@requires(lambda self, a, o: self.sum == o.sum - a.amount)

def withdraw(amount: int) -> None:

    ...

There is this lambda keyword in front, but it's not too bad?

I'll try to contact dpcontracts maintainers. Maybe it's possible to at least merge a couple of libraries into one and make it a de facto standard. @Agnus, would you also like to join the effort?

Cheers,

Marko

[Barry Scott]

On 24 Sep 2018, at 20:09, Marko Ristin-Kaufmann <marko....@gmail.com> wrote:

Hi Barry,

I think the main issue with pyffel is that it can not support function calls in general. If I understood it right, and Angus please correct me, you would need to wrap every function that you would call from within the contract.

But the syntax is much nicer than icontract or dpcontracts (see these packages on pypi). What if we renamed "args" argument and "old" argument in those libraries to just "a" and "o", respectively? Maybe that gives readable code without too much noise:

The args and old and not noise its easier to read the a and o.

a and o as aliases for more descriptive names maybe, but not as the only name.

@requires(lambda self, a, o: self.sum == o.sum - a.amount)

def withdraw(amount: int) -> None:

    ...

There is this lambda keyword in front, but it's not too bad?

The lambda smells of internals that I should not have to care about being exposed.

So -1 on lambda being required.

Also being able to supply a list of conditions was a +1.

[James Lu]

Perhaps it’s because fewer Python functions involve transitioning between states. Web development and statistics don’t involve many state transition. State transitions are where I think I would find it useful to write contracts out explicitly.

[Stephen J. Turnbull]

Barry Scott writes:

> > @requires(lambda self, a, o: self.sum == o.sum - a.amount)
> > def withdraw(amount: int) -> None:
> > ...
> >
> > There is this lambda keyword in front, but it's not too bad?
>
> The lambda smells of internals that I should not have to care about
> being exposed.
> So -1 on lambda being required.

If you want to get rid of the lambda you can use strings and then
'eval' them in the condition. Adds overhead.

If you want to avoid the extra runtime overhead of parsing
expressions, it might be nice to prototype with MacroPy. This should
also allow eliminating the lambda by folding it into the macro (I
haven't used MacroPy but it got really good reviews by fans of that
kind of thing). It would be possible to avoid decorator syntax if you
want to with this implementation.

I'm not sure that DbC is enough of a fit for Python that it's worth
changing syntax to enable nice syntax natively, but detailed reports
on a whole library (as long as it's not tiny) using DbC with a nice
syntax (MacroPy would be cleaner, but I think it would be easy to "see
through" the quoted conditions in an eval-based implementation) would
go a long way to making me sit up and take notice. (I'm not
influential enough to care about, but I suspect some committers would
be impressed too. YMMV)

Steve

[Marko Ristin-Kaufmann]

Hi Steve,

Thanks a lot for pointing us to macropy -- I was not aware of the library, it looks very interesting!

Do you have any experience how macropy fit with current IDEs and static linters (pylint, mypy)? I fired up pylint and mypy on the sample code from their web site, played a bit with it and it seems that they go along well.

I'm also a bit worried how macropy would work out in the libraries published to pypi -- imagine if many people start using contracts. Suddenly, all these libraries would not only depend on a contract library but on a macro library as well. Is that something we should care about? Potential dependency hell? (I already have a bad feeling about making icontract depend on asttokens and considerin-lining asttokens into icontract particularly for that reason).

I'm also worried about this one (from https://macropy3.readthedocs.io/en/latest/overview.html):

Note that this means you cannot use macros in a file that is run directly, as it will not be passed through the import hooks.

That would make contracts unusable in any stand-alone script, right?

Cheers,

Marko

[Marko Ristin-Kaufmann]

Hi Steve and others,

After some thinking, I'm coming to a conclusion that it might be wrong to focus too much about how the contracts are written -- as long as they are formal, easily  transformable to another representation and fairly maintainable.

Whether it's with a lambda, without, with "args" or "a", with "old" or "o" -- it does not matter that much as long as it is pragmatic and not something crazy complex. This would also mean that we should not add complexity (e.g., by adding macros) and limit the magic as much as possible.

It is actually much more important in which form they are presented to the end-user. I already made an example with sphinx-icontract in a message before -- an improved version might use mathematical symbols (e.g., replace all() with ∀, replace len() with |.|, nicely use subscripts for ranges, use case distinction with curly bracket "{" instead of if.. else ..., etc.). This would make them even shorter and easier to parse. Let me iterate the example I already pasted in the thread before to highlight what I have in mind:

packagery.resolve_initial_paths(initial_paths)

Resolve the initial paths of the dependency graph by recursively adding *.py files beneath given directories.

Parameters:	initial_paths (List[Path]) – initial paths as absolute paths
Return type:	List[Path]
Returns:	list of initial files (i.e. no directories)
Requires:	all(pth.is_absolute() for pth in initial_paths)

Ensures:	all(pth in result for pth in initial_paths if pth.is_file()) (Initial files also in result)

len(result) >= len(initial_paths) if initial_paths else result == [] all(pth.is_absolute() for pth in result) all(pth.is_file() for pth in result)

The contracts need to extend __doc__ of the function accordingly (and the contracts in __doc__ also need to reflect the inheritance of the contracts!), so that we can use help().

There should be also a plugin for Pycharm, Pydev, vim and emacs to show the contracts in an abbreviated and more readable form in the code and only show them in raw form when we want to edit them (i.e., when we move cursor over them). I suppose inheritance of contracts needs to be reflected in quick-inspection windows, but not in the code view.

Diffs and github/bitbucket/... code reviews might be a bit cumbersome since they enforce the raw form of the contracts, but as long as syntax is pragmatic, I don't expect this to be a blocker.

Is this a sane focus?

Cheers,

Marko

[Angus Hollands]

Hi Mario, 

yes I'd pass in some kind of 'old' object as a proxy to the old object state. 

My demo can handle function calls, unless they themselves ultimately call something which can't be proxies e.g is instance (which delegates to the test class, not the instance), or boolean evaluation of some expression (e.g an if block). I don't think that this is awful - contracts should probably be fairly concise while expressive - but definitely non-ideal. 

 

I haven't really time to work on this at the moment; I admit, it was a specific problem of interest, rather than a domain I have much experience with. In fact, it was probably an excuse to overload all of the operators on an object! 

Kind regards, 

Angus 

[Robert Collins]

On Mon, 24 Sep 2018 at 19:47, Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> Hi,
>
> Thank you for your replies, Hugh and David! Please let me address the points in serial.
>
> Obvious benefits
> You both seem to misconceive the contracts. The goal of the design-by-contract is not reduced to testing the correctness of the code, as I reiterated already a couple of times in the previous thread. The contracts document formally what the caller and the callee expect and need to satisfy when using a method, a function or a class. This is meant for a module that is used by multiple people which are not necessarily familiar with the code. They are not a niche. There are 150K projects on pypi.org. Each one of them would benefit if annotated with the contracts.

You'll lose folks attention very quickly when you try to tell folk
what they do and don't understand.

Claiming that DbC annotations will improve the documentation of every
single library on PyPI is an extraordinary claim, and such claims
require extraordinary proof.

I can think of many libraries where necessary pre and post conditions
(such as 'self is still locked') are going to be noisy, and at risk of
reducing comprehension if the DbC checks are used to enhance/extended
documentation.

Some of the examples you've been giving would be better expressed with
a more capable type system in my view (e.g. Rust's), but I have no
good idea about adding that into Python :/.

Anyhow, the thing I value most about python is its pithyness: its
extremely compact, allowing great developer efficiency, but the cost
of testing is indeed excessive if the tests are not structured well.
That said, its possible to run test suites with 10's of thousands of
tests in only a few seconds, so there's plenty of headroom for most
projects.

-Rob

[Stephen J. Turnbull]

Marko Ristin-Kaufmann writes:

> Thanks a lot for pointing us to macropy -- I was not aware of the library,
> it looks very interesting!
>
> Do you have any experience how macropy fit

Sorry, no. I was speaking as someone who is familiar with macros from
Lisp but doesn't miss them in Python, and who also has been watching
python-dev and python-ideas for about two decades now, so I've heard
of things like MacroPy and know how the core developers think to a
great extent.

> I'm also a bit worried how macropy would work out in the libraries
> published to pypi -- imagine if many people start using contracts.
> Suddenly, all these libraries would not only depend on a contract library
> but on a macro library as well.

That's right.

> Is that something we should care about?

Yes. Most Pythonistas (at least at present) don't much like macros.
They fear turning every program into its own domain-specific language.
I can't claim much experience with dependency hell, but I think that's
much less important from your point of view (see below).

My point is mainly that, as you probably are becoming painfully aware,
getting syntax changes into Python is a fairly drawnout process. For
an example of the kind of presentation that motivates people to change
their mind from the default state of "if it isn't in Python yet,
YAGNI" to "yes, let's do *this* one", see
https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings

Warning: Tim Peters is legendary, though still active occasionally.
All he has to do is post to get people to take notice. But this
Appendix is an example of why he gets that kind of R-E-S-P-E-C-T.[1]

So the whole thing is a secret plot ;-) to present the most beautiful
syntax possible in your PEP (which will *not* be about DbC, but rather
about a small set of enabling syntax changes, hopefully a singleton),
along with an extended example, or a representative sample, of usage.
Because you have a working implementation using MacroPy (or the less
pretty[2] but fewer dependencies version based on condition strings
and eval) people can actually try it on their own code and (you hope,
they don't :-) they find a nestful of bugs by using it.

> Potential dependency hell? (I already have a bad feeling about
> making icontract depend on asttokens and considerin-lining
> asttokens into icontract particularly for that reason).

I don't think so. First, inlining an existing library is almost
always a bad idea. As for the main point, if the user sticks to one
major revision, and only upgrades to compatible bugfixes in the
Python+stdlib distribution, I don't see why two or three libraries
would be a major issue for a feature that the developer/project uses
extremely frequently. I've rarely experienced dependency hell, and in
both cases it was web frameworks (Django and Zope, to be specific, and
the dependencies involved were more or less internal to those
frameworks). If you or people you trust have other experience, forget
what I just said. :-)

Of course it depends on the library, but as long as the library is
pretty strict about backward compatibility, you can upgrade it and get
new functionality for other callers in your code base (which are
likely to appear, you know -- human beings cannot stand to leave a
tool unused once they install it!)

> > Note that this means *you cannot use macros in a file that is run
> > directly*, as it will not be passed through the import hooks.

>
> That would make contracts unusable in any stand-alone script,
> right?

Yes, but really, no:

# The run.py described by the MacroPy docs assumes a script that
# runs by just importing it. I don't have time to work out
# whether that makes more sense. This idiom of importing just a
# couple of libraries, and then invoking a function with a
# conventional name such as "run" or "process" is quite common.
# If you have docutils install, check out rstpep2html.py.

import macropy.activate
from my_contractful_library import main
main()

and away you go. 5 years from now that script will be a badge of
honor among Pythonic DbCers, and you won't be willing to give it up!
Just kidding, of course -- the ideal outcome is that the use case is
sufficiently persuasive to justify a syntax change so you don't need
MacroPy, or, perhaps some genius will come along and provide some
obscure construct that is already legal syntax!

HTH

Footnotes:
[1] R.I.P. Aretha!

[2] Urk, I just realized there's another weakness to strings: you get
no help on checking their syntax from the compiler. For a proof-of-
concept that's OK, but if you end up using the DbC library in your
codebase for a couple years while the needed syntax change gathers
support, that would be really bad.

[Stephen J. Turnbull]

Angus Hollands writes:

> yes I'd pass in some kind of 'old' object as a proxy to the old object
> state.

Mostly you shouldn't need to do this, you can copy the state:

def method(self, args):
import copy
old = copy.deepcopy(self)

This is easy but verbose to do with a decorator, and I imagine a bunch
of issues about the 'old' object with multiple decorators, so I omit it
here. You might want a variety of such decorators. Ie, using
copy.copy vs copy.deepcopy vs a special-case copy for a particular
class because there are large objects that are actually constant that
you don't want to copy (an "is" test would be enough, so the copy
would actually implement part of the contract). Or the copy function
could be an argument to the decorator or a method on the object.

[Hugh Fisher]

> Date: Mon, 24 Sep 2018 09:46:16 +0200

> From: Marko Ristin-Kaufmann <marko....@gmail.com>
> To: Python-Ideas <python...@python.org>

> Subject: Re: [Python-ideas] Why is design-by-contracts not widely
> adopted?
> Message-ID:
> <CAGu4bVB4=Bou+DhoDzayx=i7eQ2Gr8KDDOFO...@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"

[munch]

> Their users would hugely benefit from a more mature
> and standardized contracts library with informative violation messages.

Will respond in another message, because it's a big topic.

> I really don't see how DbC has to do with duck typing (unless you reduce it
> to mere isinstance conditions, which would simply be a straw-man argument)
> -- could you please clarify?

I argue that Design by Contract doesn't make sense for Python and other
dynamically typed, duck typed languages because it's contrary to how the
language, and the programmer, expects to work.

In Python we can write something like:

def foo(x):
x.bar(y)

What's the type of x? What's the type of y? What is the contract of bar?
Don't know, don't care. x, or y, can be an instance, a class, a module, a
proxy for a remote web service. The only "contract" is that object x will
respond to message bar that takes one argument. Object x, do whatever
you want with it.

And that's a feature, not a bug, not bad design. It follows Postel's Law
for Internet protocols of being liberal in what you accept. It follows the
Agile principle of valuing working software over comprehensive doco.
It allows software components to be glued together quickly and easily.

It's a style of programming that has been successful for many years,
not just in Python but also in Lisp and Smalltalk and Perl and JavaScript.
It works.

Not for everything. If I were writing the avionics control routines for a
helicopter gas turbine, I'd use formal notation and static type checking
and preconditions and whatnot. But I wouldn't be using Python either.

> As soon as you need to document your code, and
> this is what most modules have to do in teams of more than one person
> (especially so if you are developing a library for a wider audience), you
> need to write down the contracts. Please see above where I tried to
> explained that 2-5) are inferior approaches to documenting contracts
> compared to 1).

You left off option 6), plain text. Comments. Docstrings. README files.
Web pages. Books. In my experience, this is what most people consider
documentation. A good book, a good blog post, can explain more about
how a library works and what the implementation requirements and
restrictions are than formal contract notation. In particular, contracts in
Eiffel don't explain *why* they're there.

As for 4) reading the code, why not? "Use the source, Luke" is now a
programming cliche because it works. It's particularly appropriate for
Python packages which are usually distributed in source form and, as
you yourself noted, easy to read.

--

cheers,
Hugh Fisher

[Marko Ristin-Kaufmann]

Hi Robert,

You'll lose folks attention very quickly when you try to tell folk
what they do and don't understand.

I apologize if I sounded offending, that was definitely not my intention. I appreciate that you addressed that.

I suppose it's cultural/language issue and the wording was probably inappropriate. Please let me clarify what I meant: there was a  misconception as DbC was reduced to a tool for testing, and, in a separate message, reduced to type-checks at runtime. These are clearly misconceptions, as DbC (as origianally proposed by Hoare and later popularized by Meyer) include other relevant aspects which are essential and hence can not be overseen or simply ignored. If we are arguing about DbC without these aspects then we are simply falling pray to a straw-man fallacy.

Claiming that DbC annotations will improve the documentation of every
single library on PyPI is an extraordinary claim, and such claims
require extraordinary proof.

I don't know what you mean by "extraordinary" claim and "extraordinary" proof, respectively. I tried to show that DbC is a great tool and far superior to any other tools currently used to document contracts in a library, please see my message https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. Let me re-use the enumeration I used in the message and give you a short summary.

The implicit or explicit contracts are there willy-nilly. When you use a module, either you need to figure them out using trial-and-error or looking at the implementation (4), looking at the test cases and hoping that they generalize (5), write them as doctests (3) or write them in docstrings as human text (2); or you write them formally as explicit contracts (1).

I could not identify any other methods that can help you with expectations when you call a function or use a class (apart from formal methods and proofs, which I omitted as they seem too esoteric for the current discussion).

Given that:

* There is no other method for representing contracts,

* people are trained and can read formal statements and

* there is tooling available to write, maintain and represent contracts in a nice way

I see formal contracts (1) as a superior tool. The deficiencies of other approaches are:

2) Comments and docstrings inevitably rot and get disconnected from the implementation in my and many other people's experience and studies.

3) Doctests are much longer and hence more tedious to read and maintain, they need extra text to signal the intent (is it a simple test or an example how boundary conditions are handled or ...). In any non-trivial case, they need to include even the contract itself.

4) Looking at other people's code to figure out the contracts is tedious and usually difficult for any non-trivial function.

5) Test cases can be difficult to read since they include much broader testing logic (mocking, set up). Most libraries do not ship with the test code. Identifying test cases which demonstrate the contracts can be difficult.

Any function that is used by multiple developers which operates on the restricted range of input values and gives out structured output values benefits from contracts (1) since the user of the function needs to figure them out to properly call the function and handle its results correctly. I assume that every package on pypi is published to be used by wider audience, and not the developer herself. Hence every package on pypi would benefit from formal contracts.

Some predicates are hard to formulate, and we will never be able to formally write down all the contracts. But that doesn't imply for me to not use contracts at all (analogously, some functionality is untestable, but that doesn't mean that we don't test what we can).

I would be very grateful if you could point me where this exposition is wrong (maybe referring to my original message, https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ, which I spent more thought on formulating).

So far, I was not confronted against nor read on the internet a plausible argument against formal contracts (the only two exceptions being lack of tools and less-skilled programmers have a hard time reading formal statements as soon as they include boolean logic and quantifiers). I'm actively working on the former, and hope that the latter would improve with time as education in computer sciences improves.

Another argument, which I did read often on internet, but don't really count is that quality software is not a priority and most projects hence dispense of documentation or testing. This should, hopefully, not apply to public pypi packages and is highly impractical for any medium-size project with multiple developers (and very costly in the long run).

I can think of many libraries where necessary pre and post conditions
(such as 'self is still locked') are going to be noisy, and at risk of
reducing comprehension if the DbC checks are used to enhance/extended
documentation.

It is up to the developer to decide which contracts are enforced during testing, production or displayed in the documentation (you can pick the subset of the three, it's not an exclusion). This feature ("enabled" argument to a contract) has been already implemented in the icontract library.

Some of the examples you've been giving would be better expressed with
a more capable type system in my view (e.g. Rust's), but I have no
good idea about adding that into Python  :/.

I don't see how type system would help regardless how strict it would be? Unless each input and each output represent a special type, which would be super confusing as soon as you would put them in the containers and have to struggle with invariance, contravariance and covariance. Please see https://github.com/rust-lang/rfcs/issues/1077 for a discussion about introducing DbC to Rust. Unfortunately, the discussion about contracts in Rust is also based on misconceptions (e.g., see https://github.com/rust-lang/rfcs/issues/1077#issuecomment-94582917) -- there seems to be something wrong in the way anybody proposing DbC exposes contracts to the wider audience and miss to address these issues in a good way. So most people just react instinctively with "80% already covered with type systems" / "mere runtime type checks, use assert" and "that's only an extension to testing, so why bother" :(.

I would now like to answer Hugh and withdraw from the discussion pro/contra formal contracts unless there is a rational, logical argument disputing the DbC in its entirety (not in one of its specific aspects or as a misconception/straw-man). A lot has been already said, many articles have been written (I linked some of the pages which I thought were short & good reads and I would gladly supply more reading material). I doubt I can find a better way to contribute to the discussion.

Cheers,

Marko

 

[Marko Ristin-Kaufmann]

Hi Steve,

I'll give it a shot and implement a proof-of-concept icontrac-macro library based on macropy and see if that works. I'll keep you posted.

Cheers,

Marko

[Marko Ristin-Kaufmann]

Hi Hugh,

> As soon as you need to document your code, and
> this is what most modules have to do in teams of more than one person
> (especially so if you are developing a library for a wider audience), you
> need to write down the contracts. Please see above where I tried to
> explained  that 2-5) are inferior approaches to documenting contracts
> compared to 1).

You left off option 6), plain text. Comments. Docstrings.

 That was actually the option 2):

2) Write precondtions and postconditions in docstring of the method as human text.

The problem with text is that it is not verifiable and hence starts to "rot". Noticing that text is wrong involves much more developer time & attention than automatically verifying the formal contracts.

In Python we can write something like:

def foo(x):
    x.bar(y)

What's the type of x? What's the type of y? What is the contract of bar?
Don't know, don't care. x, or y, can be an instance, a class, a module, a
proxy for a remote web service. The only "contract" is that object x will
respond to message bar that takes one argument. Object x, do whatever
you want with it.

I still don't see how this is connected to contracts or how contracts play a role there? If foo can accept any x and return any result then there is no contract. But hardly any function is like that. Most exercise a certain behavior on a subset of possible input values. The outputs also  satisfy certain contracts, i.e. they also live in a certain subset of possible outputs. (Please mind that I don't mean strictly numerical ranges here -- it can be any subset of structured data.) As I already mentioned, the contracts have nothing to do with typing. You can use them for runtime type checks -- but that's a reduction of the concept to a very particular use case. Usually contracts read like this (from the numpy example linked in another message, https://www.numpy.org/devdocs/reference/generated/numpy.ndarray.transpose.html#numpy.ndarray.transpose):

ndarray.transpose(*axes)

Returns a view of the array with axes transposed.

For a 1-D array, this has no effect. (To change between column and row vectors, first cast the 1-D array into a matrix object.) For a 2-D array, this is the usual matrix transpose. For an n-D array, if axes are given, their order indicates how the axes are permuted (see Examples). If axes are not provided and a.shape = (i[0], i[1], ... i[n-2], i[n-1]), then a.transpose().shape = (i[n-1], i[n-2], ... i[1], i[0]).

(emphasis mine)

Mind the three postconditions (case 1D array, case 2D array, case N-D array).

As for 4) reading the code, why not? "Use the source, Luke" is now a
programming cliche because it works. It's particularly appropriate for
Python packages which are usually distributed in source form and, as
you yourself noted, easy to read.

Because it is hard and costs a lot of time. The point of encapsulating a function is that I as a user don't have to know its details of implementation and its wider dependencies in the implementation. Looking at the code is the tool of last resort to figure out the contracts. Imagine if you had to look at the implementation of numpy.transpose() to figure out what happens when transposing a N-D array.

Cheers,

Marko

[Chris Angelico]

On Wed, Sep 26, 2018 at 3:19 AM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>> Claiming that DbC annotations will improve the documentation of every
>> single library on PyPI is an extraordinary claim, and such claims
>> require extraordinary proof.
>
>
> I don't know what you mean by "extraordinary" claim and "extraordinary" proof, respectively. I tried to show that DbC is a great tool and far superior to any other tools currently used to document contracts in a library, please see my message https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. Let me re-use the enumeration I used in the message and give you a short summary.
>

An ordinary claim is like "DbC can be used to improve code and/or
documentation", and requires about as much evidence as you can stuff
into a single email. Simple claim, low burden of proof.

An extraordinary claim is like "DbC can improve *every single project*
on PyPI". That requires a TON of proof. Obviously we won't quibble if
you can only demonstrate that 99.95% of them can be improved, but you
have to at least show that the bulk of them can.

> There are 150K projects on pypi.org. Each one of them would benefit if annotated with the contracts.

This is the extraordinary claim. To justify it, you have to show that
virtually ANY project would benefit from contracts. So far, I haven't
seen any such proof.

ChrisA

[Lee Braiden]

Eh. It's too easy to cry "show me the facts" in any argument.  To do that too often is to reduce all discussion to pendantry.

That verifying data against the contract a function makes code more reliable should be self evident to anyone with even the most rudimentary understanding of a function call, let alone a library or large application.  It's the reason why type checking exists, and why bounds checking exists, and why unit checking exists too.

[Chris Angelico]

On Wed, Sep 26, 2018 at 6:09 AM Lee Braiden <leeb...@gmail.com> wrote:
>
> Eh. It's too easy to cry "show me the facts" in any argument. To do that too often is to reduce all discussion to pendantry.
>
> That verifying data against the contract a function makes code more reliable should be self evident to anyone with even the most rudimentary understanding of a function call, let alone a library or large application. It's the reason why type checking exists, and why bounds checking exists, and why unit checking exists too.
>

It's easy, but it's also often correct.

From my reading of this thread, there HAS been evidence given that DbC
can be beneficial in some cases. I do not believe there has been
evidence enough to cite the number of projects on PyPI as "this is how
many projects would benefit".

Part of the trouble is finding a concise syntax for the contracts that
is still sufficiently expressive.

[Kyle Lahnakoski]

I use DbC occasionally to clarify my thoughts during a refactoring, and then only in the places that continue to make mistakes. In general, I am not in a domain that benefits from DbC.

Contracts are code: More code means more bugs. Declarative contracts are succinct, but difficult to debug when wrong; I believe this because the debugger support for contracts is poor; There is no way to step through the logic and see the intermediate reasoning in complex contracts.  A contract is an incomplete duplication of what the code already does: at some level of complexity I prefer to use a duplicate independent implementation and compare inputs/outputs.

Writing contracts cost time and money; and that cost should be weighed against the number and flexibility of the customers that use the code.  A one-time script, a webapp for you team, an Android app for your startup, fraud software, and Facebook make different accounting decisions.  I contend most code projects can not justify DbC.

On 2018-09-24 03:46, Marko Ristin-Kaufmann wrote:

When you are documenting a method you have the following options:

1) Write preconditions and postconditions formally and include them automatically in the documentation (e.g., by using icontract library).

2) Write precondtions and postconditions in docstring of the method as human text.

3) Write doctests in the docstring of the method.

4) Expect the user to read the actual implementation.

5) Expect the user to read the testing code.

There are other ways to communicate how a method works.

6) The name of the method
7) How the method is called throughout the codebase
8) observing input and output values during debugging
9) observing input and output values in production
10) relying on convention inside, and outside, the application
11) Don't communicate - Sometimes <complexity>/<num_customers> is too high; code is not repaired, only replaced.

This is again something that eludes me and I would be really thankful if you could clarify. Please consider for an example, pypackagery (https://pypackagery.readthedocs.io/en/latest/packagery.html) and the documentation of its function resolve_initial_paths:

packagery.resolve_initial_paths(initial_paths)

Resolve the initial paths of the dependency graph by recursively adding *.py files beneath given directories.

Parameters:	initial_paths (List[Path]) – initial paths as absolute paths
Return type:	List[Path]
Returns:	list of initial files (i.e. no directories)
Requires:	all(pth.is_absolute() for pth in initial_paths)
Ensures:	len(result) >= len(initial_paths) if initial_paths else result == [] all(pth.is_absolute() for pth in result) all(pth.is_file() for pth in result)

How is this difficult to read,[...]?

This contract does not help me: 

Does it work on Windows?
What is_absolute()?  is "file:///" absolute?
How does this code fail? 
What does a permission access problem look like? 
Can initial_paths can be None?
Can initial_paths be files? directories? 
What are the side effects?

resolve_initial_path() is a piece code is better understood by looking at the callers (#7), or not exposing it publicly (#11).  You can also use a different set of abstractions, to make the code easier to read: 
  
UNION(file for p in initial_paths for file in p.leaves() if file.extension=="py")

At a high level, I can see the allure of DbC:  Programming can be a craft, and a person can derive deep personal satisfaction from perfecting the code they work on. DbC provides you with more decoration, more elaboration, more ornamentation, more control.  This is not bad, but I see all your arguments as personal ascetic sense.  DbC is only appealing under certain accounting rules.  Please consider the possibility that "the best code" is: low $$$, buggy, full of tangles, and mostly gets the job done.   :)

[Chris Angelico]

On Wed, Sep 26, 2018 at 7:59 AM Kyle Lahnakoski <klahn...@mozilla.com> wrote:
> I use DbC occasionally to clarify my thoughts during a refactoring, and then only in the places that continue to make mistakes. In general, I am not in a domain that benefits from DbC.
>
> Contracts are code: More code means more bugs.

Contracts are executable documentation. If you can lift them directly
into user-readable documentation (and by "user" here I mean the user
of a library), they can save you the work of keeping your
documentation accurate.

> This contract does not help me:
>

> What is_absolute()? is "file:///" absolute?

I'd have to assume that is_absolute() is defined elsewhere. Which
means that the value of this contract depends entirely on having other
functions, probably ALSO contractually-defined, to explain it.

> How does this code fail?
> What does a permission access problem look like?

Probably an exception. This is Python code, and I would generally
assume that problems are reported as exceptions.

> Can initial_paths can be None?

This can be answered from the type declaration. It doesn't say
Optional, so no, it can't be None.

> Can initial_paths be files? directories?

Presumably not a question you'd get if you were actually using it; the
point of the function is to "[r]esolve the initial paths of the

dependency graph by recursively adding *.py files beneath given

directories", so you'd call it because you have directories and want
files back.

> What are the side effects?

Hopefully none, other than the normal implications of hitting the file system.

It's easy to show beautiful examples that may actually depend on other
things. Whether that's representative of all contracts is another
question.

ChrisA

[Marko Ristin-Kaufmann]

Hi Chris,

An extraordinary claim is like "DbC can improve *every single project*
on PyPI". That requires a TON of proof. Obviously we won't quibble if
you can only demonstrate that 99.95% of them can be improved, but you
have to at least show that the bulk of them can.

I tried to give the "proof" (not a formal one, though) in my previous message. The assumptions are that:

* There are always contracts, they can be either implicit or explicit. You need always to figure them out before you call a function or use its result.

* Figuring out contracts by trial-and-error and reading the code (the implementation or the test code) is time consuming and hard.

* The are tools for formal contracts.

* The contracts written in documentation as human text inevitably rot and they are much harder to maintain than automatically verified formal contracts.

* The reader is familiar with formal statements, and hence reading formal statements is faster than reading the code or trial-and-error.

I then went on to show why I think, under these assumptions, that formal contracts are superior as a documentation tool and hence beneficial. Do you think that any of these assumptions are wrong? Is there a hole in my logical reasoning presented in my previous message? I would be very grateful for any pointers!

If these assumptions hold and there is no mistake in my reasoning, wouldn't that qualify as a proof?

Cheers,

Marko

[Marko Ristin-Kaufmann]

Hi Kyle,

6) The name of the method
7) How the method is called throughout the codebase

10) relying on convention inside, and outside, the application

Sorry, by formulating 2) as "docstring" I excluded names of the methods as well as variables. Please assume that 2) actually entails those as well. They are human text and hence not automatically verifiable, hence qualify as 2).

8) observing input and output values during debugging
9) observing input and output values in production

Sorry, again I implicitly subsumed 8-9 under 4), reading the implementation code (including the trial-and-error). My assumption was that it is incomparably more costly to apply trial-and-error than read the contracts given that contracts can be formulated. Of course, not all contracts can be formulated all the time.

11) Don't communicate - Sometimes <complexity>/<num_customers> is too high; code is not repaired, only replaced.

I don't see this as an option for any publicly available, high-quality module on pypi or in any organization. As I already noted in my message to Hugh, the argument in favor of undocumented and/or untested code are not the arguments. I assume we want a maintainable and usable modules. I've never talked about undocumented throw-away exploratory code. Most of the Python features become futile in that case (type annotations and static type checking with mypy, to name only the few).

Does it work on Windows?

This is probably impossible to write as a contract, but needs to be tested (though maybe there is a way to check it and encapsulate the check in a separate function and put it into the contract).

What is_absolute()?  is "file:///" absolute?

Since the type is pathlib.Path (as written in the type annotation), it's pathlib.Path.is_absolute() method. Please see https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_absolute

At a high level, I can see the allure of DbC:  Programming can be a craft, and a person can derive deep personal satisfaction from perfecting the code they work on. DbC provides you with more decoration, more elaboration, more ornamentation, more control.  This is not bad, but I see all your arguments as personal ascetic sense.  DbC is only appealing under certain accounting rules.  Please consider the possibility that "the best code" is: low $$$, buggy, full of tangles, and mostly gets the job done.   :)

Actually, this goes totally contrary to most of my experience. Bad code is unmaintainable and ends up being much more costly down the line. It's also what we were taught in software engineering lectures in the university (some 10-15 years ago) and I always assumed that the studies presented there were correct. 

Saying that writing down contracts is costly is a straw-man. It is costly if you need to examine the function and write them down. If you are writing the function and just keep adding the contracts as-you-go, it's basically very little overhead cost. You make an assumption of the input, and instead of just coding on, you scroll up, write it down formally, and go back where you stopped and continue the implementation. Or you think for a minute what contracts your function needs to expect/satisfy before you start writing it (or during the design). I don't see how this can be less efficient than trial-and-error and making possibly wrong assumptions based on the output that you see without any documentation by running the code of the module.

Cheers,

Marko

[Marko Ristin-Kaufmann]

Hi Chris,

It's easy to show beautiful examples that may actually depend on other
things. Whether that's representative of all contracts is another
question.

I agree. There are also many contracts which are simply too hard to write down formally. But many are also easily captured in formal manner in my experience. The question is, of course, how many and you make a fair point there.

@Chris and others requesting data: my time is way too limited to provide a large-scale code analysis of many pypi packages (family obligations with a toddler, 9-6 job). I'm not doing research, and such a study would require substantial time resources. Is there an alternative request that you think that I (and other volunteers?)  could accomplish in a reasonable (free) time? Maybe you could compile a list of 100-200 (or even less) functions from representative modules and I try to annotate them with contracts and we see if that's convincing? It's up to you to pick representative functions and up to me to annotate them with contracts. That would diffuse the argument that I intentionally picked the functions whose contracts are easily and nice to annotate.

Cheers,

Marko

[Chris Angelico]

On Wed, Sep 26, 2018 at 2:47 PM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> Hi Chris,
>
>> An extraordinary claim is like "DbC can improve *every single project*
>> on PyPI". That requires a TON of proof. Obviously we won't quibble if
>> you can only demonstrate that 99.95% of them can be improved, but you
>> have to at least show that the bulk of them can.
>
>
> I tried to give the "proof" (not a formal one, though) in my previous message.

(Formal proof isn't necessary here; we say "extraordinary proof", but
it'd be more accurate to say "extraordinary evidence".)

> The assumptions are that:
> * There are always contracts, they can be either implicit or explicit. You need always to figure them out before you call a function or use its result.

Not all code has such contracts. You could argue that code which does
not is inferior to code which does, but not everything follows a
strictly-definable pattern.

> * Figuring out contracts by trial-and-error and reading the code (the implementation or the test code) is time consuming and hard.

Agreed.

> * The are tools for formal contracts.

That's the exact point you're trying to make, so it isn't evidence for
itself. Tools for formal contracts exist as third party in Python, and
if that were good enough for you, we wouldn't be discussing this.
There are no such tools in the standard library or language that make
formal contracts easy.

> * The contracts written in documentation as human text inevitably rot and they are much harder to maintain than automatically verified formal contracts.

Agreed.

> * The reader is familiar with formal statements, and hence reading formal statements is faster than reading the code or trial-and-error.

Disagreed. I would most certainly NOT assume that every reader knows
any particular syntax for such contracts. However, this is a weaker
point.

So I'll give you two and two halves for that. Good enough to make do.

> I then went on to show why I think, under these assumptions, that formal contracts are superior as a documentation tool and hence beneficial. Do you think that any of these assumptions are wrong? Is there a hole in my logical reasoning presented in my previous message? I would be very grateful for any pointers!
>
> If these assumptions hold and there is no mistake in my reasoning, wouldn't that qualify as a proof?
>

It certainly qualifies as proof that SOME code MAY benefit from
contracts. It does not reach the much higher bar to support the claim
that "there are X projects on PyPI and every single one of them would
benefit". For instance, would youtube-dl benefit from DbC? To most
people, it's an application, not a library. Even if you're invoking it
from within a Python script, it's usually easiest to use the main
entrypoint rather than delve into its internals. Case in point:

https://github.com/Rosuav/MegaClip/blob/master/megaclip.py#L71

It's actually easier to shell out to a subprocess than to call on
youtube-dl as a library. Would DbC benefit youtube-dl's internal
functions? Maybe, but that's for the youtube-dl people to decide; it
wouldn't in the slightest benefit my app.

You might argue that a large proportion of PyPI projects will be
"library-style" packages, where the main purpose is to export a bunch
of functions. But even then, I'm not certain that they'd all benefit
from DbC. Some would, and you've definitely made the case for that;
but I'm still -0.5 on adding anything of the sort to the stdlib, as I
don't yet see that *enough* projects would actually benefit.

People have said the same thing about type checking, too. Would
*every* project on PyPI benefit from MyPy's type checks? No. Syntax
for them was added, not because EVERYONE should use them, but because
SOME will use them, and it's worth having some language support. You
would probably do better to argue along those lines than to try to
claim that every single project ought to be using contracts.

[Robert Collins]

On Wed, 26 Sep 2018 at 05:19, Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> Hi Robert,
...

>> Claiming that DbC annotations will improve the documentation of every
>> single library on PyPI is an extraordinary claim, and such claims
>> require extraordinary proof.
>
>
> I don't know what you mean by "extraordinary" claim and "extraordinary" proof, respectively.

Chris already addressed this.

> I tried to show that DbC is a great tool and far superior to any other tools currently used to document contracts in a library, please see my message https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. Let me re-use the enumeration I used in the message and give you a short summary.

> When you are documenting a method you have the following options:
> 1) Write preconditions and postconditions formally and include them automatically in the documentation (e.g., by using icontract library).
> 2) Write precondtions and postconditions in docstring of the method as human text.
> 3) Write doctests in the docstring of the method.
> 4) Expect the user to read the actual implementation.
> 5) Expect the user to read the testing code.

So you can also:

0) Write clear documentation

e.g. just document the method without doctests. You can write doctests
in example documentation, or even in unit tests if desired: having use
cases be tested is often valuable.

> The implicit or explicit contracts are there willy-nilly. When you use a module, either you need to figure them out using trial-and-error or looking at the implementation (4), looking at the test cases and hoping that they generalize (5), write them as doctests (3) or write them in docstrings as human text (2); or you write them formally as explicit contracts (1).
>
> I could not identify any other methods that can help you with expectations when you call a function or use a class (apart from formal methods and proofs, which I omitted as they seem too esoteric for the current discussion).
>
> Given that:
> * There is no other method for representing contracts,
> * people are trained and can read formal statements and
> * there is tooling available to write, maintain and represent contracts in a nice way
>
> I see formal contracts (1) as a superior tool. The deficiencies of other approaches are:
> 2) Comments and docstrings inevitably rot and get disconnected from the implementation in my and many other people's experience and studies.
> 3) Doctests are much longer and hence more tedious to read and maintain, they need extra text to signal the intent (is it a simple test or an example how boundary conditions are handled or ...). In any non-trivial case, they need to include even the contract itself.
> 4) Looking at other people's code to figure out the contracts is tedious and usually difficult for any non-trivial function.
> 5) Test cases can be difficult to read since they include much broader testing logic (mocking, set up). Most libraries do not ship with the test code. Identifying test cases which demonstrate the contracts can be difficult.

I would say that contracts *are* a formal method. They are machine
interpretable rules about when the function may be called and about
what it may do and how it must leave things.
https://en.wikipedia.org/wiki/Formal_methods

The critique I offer of DbC in a Python context is the same as for
other formal methods: is the benefit worth the overhead. If you're
writing a rocket controller, Python might not be the best language for
it.

> Any function that is used by multiple developers which operates on the restricted range of input values and gives out structured output values benefits from contracts (1) since the user of the function needs to figure them out to properly call the function and handle its results correctly. I assume that every package on pypi is published to be used by wider audience, and not the developer herself. Hence every package on pypi would benefit from formal contracts.

In theory there is no difference between theory and practice, but in
practice there may be differences between practice and theory :).

Less opaquely, you're using a model to try and extrapolate human
behaviour. This is not an easy thing to do, and you are very likely to
be missing factors. For instance, perhaps training affects perceived
benefits. Perhaps a lack of experimental data affects uptake in more
data driven groups. Perhaps increased friction in changing systems is
felt to be a negative.

And crucially, perhaps some of these things are true. As previously
mentioned, Python has wonderful libraries; does it have more per
developer than languages with DbC built in? If so then that might
speak to developer productivity without these formal contracts: it may
be that where the risks of failure are below the threshold that formal
methods are needed, that we're better off with the tradeoff Python
makes today.

> Some predicates are hard to formulate, and we will never be able to formally write down all the contracts. But that doesn't imply for me to not use contracts at all (analogously, some functionality is untestable, but that doesn't mean that we don't test what we can).
>
> I would be very grateful if you could point me where this exposition is wrong (maybe referring to my original message, https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ, which I spent more thought on formulating).

I think the underlying problem is that you're treating this as a logic
problem (what does logic say applies here), rather than an engineering
problem (what can we measure and what does it tell us about whats
going on). At least, thats how it appears to me.

> So far, I was not confronted against nor read on the internet a plausible argument against formal contracts (the only two exceptions being lack of tools and less-skilled programmers have a hard time reading formal statements as soon as they include boolean logic and quantifiers). I'm actively working on the former, and hope that the latter would improve with time as education in computer sciences improves.
>
> Another argument, which I did read often on internet, but don't really count is that quality software is not a priority and most projects hence dispense of documentation or testing. This should, hopefully, not apply to public pypi packages and is highly impractical for any medium-size project with multiple developers (and very costly in the long run).

I have looked for but could not find any studies into the developer
productivity (and correctness) tradeoffs that DbC creates, other than
stuff from 20 years ago which by its age clearly cannot contrast with
modern Python/Ruby/Rust etc.

Consider this: the goal of software development is to deliver
features, at some level of correctness. One very useful measure of
productivity then is a measure of how long it takes a given team to
produce those features at that level of correctness.

If DbC reduces the time it takes to get those features, it is
increasing productivity.
If it increases the time it takes to get those features, it is
decreasing productivity, *even if it increases correctness*. Being
more correct than needed is not beneficial much of the time.

Does DbC deliver higher productivity @ a given correctness level? I
don't know - thats why I went looking for research, but I couldn't
find any (I may have missed it of course, I'd be happy to read some
citations). I'm specifically looking for empirical data here, not
extrapolation or rationalisations.

>> I can think of many libraries where necessary pre and post conditions
>> (such as 'self is still locked') are going to be noisy, and at risk of
>> reducing comprehension if the DbC checks are used to enhance/extended
>> documentation.
>
>
> It is up to the developer to decide which contracts are enforced during testing, production or displayed in the documentation (you can pick the subset of the three, it's not an exclusion). This feature ("enabled" argument to a contract) has been already implemented in the icontract library.
>
> Some of the examples you've been giving would be better expressed with
> a more capable type system in my view (e.g. Rust's), but I have no
> good idea about adding that into Python :/.

> I don't see how type system would help regardless how strict it would be? Unless each input and each output represent a special type, which would be super confusing as soon as you would put them in the containers and have to struggle with invariance, contravariance and covariance. Please see https://github.com/rust-lang/rfcs/issues/1077 for a discussion about introducing DbC to Rust. Unfortunately, the discussion about contracts in Rust is also based on misconceptions (e.g., see https://github.com/rust-lang/rfcs/issues/1077#issuecomment-94582917) -- there seems to be something wrong in the way anybody proposing DbC exposes contracts to the wider audience and miss to address these issues in a good way. So most people just react instinctively with "80% already covered with type systems" / "mere runtime type checks, use assert" and "that's only an extension to testing, so why bother" :(.
>
> I would now like to answer Hugh and withdraw from the discussion pro/contra formal contracts unless there is a rational, logical argument disputing the DbC in its entirety (not in one of its specific aspects or as a misconception/straw-man). A lot has been already said, many articles have been written (I linked some of the pages which I thought were short & good reads and I would gladly supply more reading material). I doubt I can find a better way to contribute to the discussion.

Sure; like I said, I think the fundamental question about DbC is
actually whether it helps:
a) all programs
b) all nontrivial programs
c) high assurance programs

My suspicion, for which I have only anecdata, is that its really in c)
today. Kindof where TDD was in the early 2000's (and as I understand
the research, its been shown to be a wash: you do get more tests than
test-last or test-during, and more tests is correlated with quality
and ease of evolution, but if you add that test coverage in
test-during or test-last, you end up with the same benefits).

[Marko Ristin-Kaufmann]

Hi Chris,

> * There are always contracts, they can be either implicit or explicit. You need always to figure them out before you call a function or use its result.

Not all code has such contracts. You could argue that code which does
not is inferior to code which does, but not everything follows a
strictly-definable pattern.

Well, you need to know what to supply to a function and what to expect, at least subconsciously. The question is just whether you can formulate these contracts or not; and whether you do or not.

> * The are tools for formal contracts.

That's the exact point you're trying to make, so it isn't evidence for
itself. Tools for formal contracts exist as third party in Python, and
if that were good enough for you, we wouldn't be discussing this.
There are no such tools in the standard library or language that make
formal contracts easy.

The original objection was that DbC in general is not beneficial; not that there are lacking tools for it (this probably got lost in the many messages on this thread). If you assume that there are no suitable tools for DbC, then yes, DbC is certainly not beneficial to any project since using it will be clumsy and difficult. It's a chicken-and-egg problem, so we need to assume that there are good tools for DbC in order for it to be beneficial.

Disagreed. I would most certainly NOT assume that every reader knows
any particular syntax for such contracts. However, this is a weaker
point.

 

The contracts are written as boolean expressions. While I agree that many programmers have a hard time with boolean expressions and quantifiers, I don't see this as a blocker to DbC. There is no other special syntax for DbC unless we decide to add it to the core language (which I don't see as probable). What I would like to have is a standard library so that inter-library interactions based on contracts are possible and an ecosystem could emerge around it.

Quite some people object that DbC should be rejected as their "average programmer Joe/Jane" can't deal with the formal expressions. Some readers have problems parsing "all(i > 0 for i in result) or len(result) < 3" and can only parse "All numbers in the result are positive and/or the result has fewer than 3 items". There are also programmers who have a hard time reading "all(key == value.some_property for key, value in result.items())" and could only read "The resulting values in the dictionary are keyed by their some_property attribute.".

I hope that education in computer science is improving and that soon programmers will be able to read these formal expressions. I'm also not sure if that is the case for non-English speakers. Hence I assumed that the readers of the contracts are familiar with formal boolean expressions. If you assume that readers have hard time parsing quantifiers and boolean logic then DbC is again certainly not beneficial.

It would make me sad if a feature is rejected on grounds that we have to accept that many programmers don't master the basics of computer science (which I consider the boolean expressions to be).

You might argue that a large proportion of PyPI projects will be
"library-style" packages, where the main purpose is to export a bunch
of functions. But even then, I'm not certain that they'd all benefit
from DbC.

Thanks for this clarification (and the download-yt example)! I actually only had packages-as-libraries in mind, not the executable scripts; my mistake. So, yes, "any Pypi package" should be reformulated to "any library on Pypi" (meant to be used by a wider audience than the developers themselves).

People have said the same thing about type checking, too. Would
*every* project on PyPI benefit from MyPy's type checks? No. Syntax
for them was added, not because EVERYONE should use them, but  because
SOME will use them, and it's worth having some language support. You
would probably do better to argue along those lines than to try to
claim that every single project ought to be using contracts.

I totally agree. The discussion related to DbC in my mind always revolved around these use cases where type annotations are beneficial as well. Thanks for pointing that out and I'd like to apologize for the confusion! For the future discussion, let's focus on these use cases and please do ignore the rest. I'd still say that there is a plethora of libraries published on Pypi (Is there a way to find out the stats?).

but I'm still -0.5 on adding anything of the sort to the stdlib, as I
don't yet see that *enough* projects would actually benefit.

Please see my previous message -- could you maybe say what would convince you that enough projects would actually benefit from formal contracts?

Cheers,

Marko

[Chris Angelico]

On Wed, Sep 26, 2018 at 5:10 PM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
> The original objection was that DbC in general is not beneficial; not that there are lacking tools for it (this probably got lost in the many messages on this thread). If you assume that there are no suitable tools for DbC, then yes, DbC is certainly not beneficial to any project since using it will be clumsy and difficult. It's a chicken-and-egg problem, so we need to assume that there are good tools for DbC in order for it to be beneficial.
>
>> Disagreed. I would most certainly NOT assume that every reader knows
>> any particular syntax for such contracts. However, this is a weaker
>> point.
>
>
> The contracts are written as boolean expressions. While I agree that many programmers have a hard time with boolean expressions and quantifiers, I don't see this as a blocker to DbC. There is no other special syntax for DbC unless we decide to add it to the core language (which I don't see as probable). What I would like to have is a standard library so that inter-library interactions based on contracts are possible and an ecosystem could emerge around it.
>

It's easy to say that they're boolean expressions. But that's like
saying that unit tests are just a bunch of boolean expressions too.
Why do we have lots of different forms of test, rather than just a big
fat "assert this and this and this and this and this and this"?
Because the key to unit testing is not "boolean expressions", it's a
language that can usefully describe what it is we're testing.
Contracts aren't just boolean expressions - they're a language (or a
mini-language) that lets you define WHAT the contract entails.

>> You might argue that a large proportion of PyPI projects will be
>> "library-style" packages, where the main purpose is to export a bunch
>> of functions. But even then, I'm not certain that they'd all benefit
>> from DbC.
>
>
> Thanks for this clarification (and the download-yt example)! I actually only had packages-as-libraries in mind, not the executable scripts; my mistake. So, yes, "any Pypi package" should be reformulated to "any library on Pypi" (meant to be used by a wider audience than the developers themselves).
>

Okay. Even with that qualification, though, I still think that not
every library will benefit from this. For example, matplotlib's
plt.show() method guarantees that... a plot will be shown, and the
user will have dismissed it, before it returns. Unless you're inside
Jupyter/iPython, in which case it's different. Or if you're in certain
other environments, in which case it's different again. How do you
define the contract for something that is fundamentally interactive?

You can define very weak contracts. For instance, input() guarantees
that its return value is a string. Great! DbC doing the job of type
annotations. Can you guarantee anything else about that string? Is
there anything else useful that can be spelled easily?

> I totally agree. The discussion related to DbC in my mind always revolved around these use cases where type annotations are beneficial as well. Thanks for pointing that out and I'd like to apologize for the confusion! For the future discussion, let's focus on these use cases and please do ignore the rest. I'd still say that there is a plethora of libraries published on Pypi (Is there a way to find out the stats?).
>

Ugh.... I would love to say "yes", but I can't. I guess maybe you
could look at a bunch of requirements.txt files and see which things
get dragged in that way? All you'll really get is heuristics at best,
and even that, I don't know how to provide. Sorry.

[Marko Ristin-Kaufmann]

Hi Chris,

It's easy to say that they're boolean expressions. But that's like
saying that unit tests are just a bunch of boolean expressions too.
Why do we have lots of different forms of test, rather than just a big
fat "assert this and this and this and this and this and this"?
Because the key to unit testing is not "boolean expressions", it's a
language that can usefully describe what it is we're testing.
Contracts aren't just boolean expressions - they're a language (or a
mini-language) that lets you define WHAT the contract entails.

Sorry, I misunderstood you. You are probably referring to knowing the terms like "preconditions, postconditions, invariants, strengthening/weakening", right? In that case, yes, I agree, I presuppose that readers are familiar with the concepts of DbC. Otherwise, of course, it makes no sense to use DbC if you assume nobody could actually figure out what it is :).

But that's not the objection that is made too often -- what I really read often is that DbC is not beneficial not because people are not familiar with or can learn DbC, but really because reading boolean expressions is hard. That is what I was referring to in my original message.

> Thanks for this clarification (and the download-yt example)! I actually only had packages-as-libraries in mind, not the executable scripts; my mistake. So, yes, "any Pypi package" should be reformulated to "any library on Pypi" (meant to be used by a wider audience than the developers themselves).
>

Okay. Even with that qualification, though, I still think that not
every library will benefit from this. For example, matplotlib's
plt.show() method guarantees that... a plot will be shown, and the
user will have dismissed it, before it returns. Unless you're inside
Jupyter/iPython, in which case it's different. Or if you're in certain
other environments, in which case it's different again. How do you
define the contract for something that is fundamentally interactive?

You can define very weak contracts. For instance, input() guarantees
that its return value is a string. Great! DbC doing the job of type
annotations. Can you guarantee anything else about that string? Is
there anything else useful that can be spelled easily?

In this case, no, I would not add any formal contracts to the function. Not all contracts can be formulated, and not all contracts are even meaningful. I suppose interactive functions are indeed a case that it is not possible. If any string can be returned, than the contract is empty.

However, contracts can be useful when testing the GUI -- often it is difficult to script the user behavior. What many people do is record a session and re-play it. If there is a bug, fix it. Then re-record. While writing unit tests for GUI is hard since GUI changes rapidly during development and scripting formally the user behavior is tedious, DbC might be an alternative where you specify as much as you can, and then just re-run through the session. This implies, of course, a human tester.

Let me take a look at matplotlib show:

matplotlib.pyplot.show(*args, **kw)[source]

Display a figure. When running in ipython with its pylab mode, display all figures and return to the ipython prompt.

In non-interactive mode, display all figures and block until the figures have been closed; in interactive mode it has no effect unless figures were created prior to a change from non-interactive to interactive mode (not recommended). In that case it displays the figures but does not block.

A single experimental keyword argument, block, may be set to True or False to override the blocking behavior described above.

Here are the contracts as a toy example; (I'm not familiar at all with the internals of matplotlib. I haven't spent much time really parsing and analyzing the docstring -- as I'll write below, it also confuses me since the docstring is not precise enough.)

* If in ipython with its pylab mode, all figures should be displayed.

* If in non-interactive mode, display all figures and block

I'm actually confused with what they mean with:

In non-interactive mode, display all figures and block until the figures have been closed; in interactive mode it has no effect unless figures were created prior to a change from non-interactive to interactive mode (not recommended). In that case it displays the figures but does not block.

A single experimental keyword argument, block, may be set to True or False to override the blocking behavior described above.

If only they spelled that out as a contract :)

"it has no effect": What does not have effect? The call to the function? Or the setting of some parameter? Or the order of the calls?

"unless ...to interactive mode": this would be actually really more readable as a formal contract. It would imply some refactoring to add a property when a figure was created (before or after the change to interactive mode from non-interactive mode). Right now, it's not very clear how you can test that even yourself as a caller. And what does "(not recommended)" mean? Prohibited?

The blocking behavior and the corresponding argument is hard to parse from the description for me. Writing it as a contract would, IMO, be much more readable.

Here is my try at the contracts. Assuming that there is a list of figures and that they have a property "displayed" and that "State.blocked" global variable refers to whether the interface is blocked or not::
@post(lambda: all(figure.displayed for figure in figures)

@post(lambda: not ipython.in_pylab_mode() or not State.blocked)

@post(lambda: not interactive() or State.blocked)

matplotlib.pyplot.show()
 

The function that actually displays the figure could ensure that the "displayed" property of the figure is set and that the window associated with the figure is visible.

Cheers,

Marko

[Paul Moore]

On Wed, 26 Sep 2018 at 06:41, Chris Angelico <ros...@gmail.com> wrote:
>
> On Wed, Sep 26, 2018 at 2:47 PM Marko Ristin-Kaufmann
> <marko....@gmail.com> wrote:
> >
> > Hi Chris,
> >
> >> An extraordinary claim is like "DbC can improve *every single project*
> >> on PyPI". That requires a TON of proof. Obviously we won't quibble if
> >> you can only demonstrate that 99.95% of them can be improved, but you
> >> have to at least show that the bulk of them can.
> >
> >
> > I tried to give the "proof" (not a formal one, though) in my previous message.
>
> (Formal proof isn't necessary here; we say "extraordinary proof", but
> it'd be more accurate to say "extraordinary evidence".)
>
> > The assumptions are that:
> > * There are always contracts, they can be either implicit or explicit. You need always to figure them out before you call a function or use its result.
>
> Not all code has such contracts. You could argue that code which does
> not is inferior to code which does, but not everything follows a
> strictly-definable pattern.

Also, the implicit contracts code currently has are typically pretty
loose. What you need to "figure out" is very general. Explicit
contracts are typically demonstrated as being relatively strict, and
figuring out and writing such contracts is more work than writing code
with loose implicit contracts. Whether the trade-off of defining tight
explicit contracts once vs inferring a loose implicit contract every
time you call the function is worth it, depends on how often the
function is called. For most single use (or infrequently used)
functions, I'd argue that the trade-off *isn't* worth it.

Here's a quick example from the pip codebase:

# Retry every half second for up to 3 seconds
@retry(stop_max_delay=3000, wait_fixed=500)
def rmtree(dir, ignore_errors=False):
shutil.rmtree(dir, ignore_errors=ignore_errors,
onerror=rmtree_errorhandler)

What contract would you put on this code? The things I can think of:

1. dir is a directory: obvious from the name, not worth the runtime
cost of checking as shutil.rmtree will do that and we don't want to
duplicate work.
2. dir is a string: covered by type declarations, if we used them. No
need for contracts
3. ignore_errors is a boolean: covered by type declarations.
4. dir should exist: Checked by shutil.rmtree, don't want to duplicate work.
5. After completion, dir won't exist. Obvious unless we have doubts
about what shutil.rmtree does (but that would have a contract too).
Also, we don't want the runtime overhead (again).

In addition, adding those contracts to the code would expand it
significantly, making readability suffer (as it is, rmtree is clearly
a thin wrapper around shutil.rmtree).

> > * Figuring out contracts by trial-and-error and reading the code (the implementation or the test code) is time consuming and hard.
>
> Agreed.

With provisos. Figuring out contracts in sufficient detail to use the
code is *in many cases* simple. For harder cases, agreed. But that's
why this is simply a proof that contracts *can* be useful, not that
100% of code would benefit from them.

> > * The are tools for formal contracts.
>
> That's the exact point you're trying to make, so it isn't evidence for
> itself. Tools for formal contracts exist as third party in Python, and
> if that were good enough for you, we wouldn't be discussing this.
> There are no such tools in the standard library or language that make
> formal contracts easy.
>
> > * The contracts written in documentation as human text inevitably rot and they are much harder to maintain than automatically verified formal contracts.
>
> Agreed.

Agreed, if contracts are automatically verified. But when runtime cost
comes up, people suggest that contracts can be disabled in production
code - which invalidates the "automatically verified" premise.

> > * The reader is familiar with formal statements, and hence reading formal statements is faster than reading the code or trial-and-error.
>
> Disagreed. I would most certainly NOT assume that every reader knows
> any particular syntax for such contracts. However, this is a weaker
> point.

Depends on what "formal statement" means. If it means "short snippet
of Python code", then yes, the reader will be familiar. But there's
only so much you can do in a short snippet of Python, without calling
out to other functions (which may or may not be "obvious" in their
behavour) so whether it's easier to read a contract is somewhat in
conflict with wanting strong contracts.

> So I'll give you two and two halves for that. Good enough to make do.
>
> > I then went on to show why I think, under these assumptions, that formal contracts are superior as a documentation tool and hence beneficial. Do you think that any of these assumptions are wrong? Is there a hole in my logical reasoning presented in my previous message? I would be very grateful for any pointers!
> >
> > If these assumptions hold and there is no mistake in my reasoning, wouldn't that qualify as a proof?
> >
>

[...]

> You might argue that a large proportion of PyPI projects will be
> "library-style" packages, where the main purpose is to export a bunch
> of functions. But even then, I'm not certain that they'd all benefit
> from DbC. Some would, and you've definitely made the case for that;
> but I'm still -0.5 on adding anything of the sort to the stdlib, as I
> don't yet see that *enough* projects would actually benefit.

The argument above, if it's a valid demonstration that all code would
benefit from contracts, would *also* imply that every function in the
stdlib should have contracts added. Are you proposing that, too, and
is your proposal not just for syntax for contracts, but *also* for
wholesale addition of contracts to the stdlib? If so, you should be
far more explicit that this is what you're proposing, because you'd
likely get even more pushback over that sort of churn in the stdlib
than over a syntax change to support contracts. Even Guido didn't push
that far with type annotations...

> People have said the same thing about type checking, too. Would
> *every* project on PyPI benefit from MyPy's type checks? No. Syntax
> for them was added, not because EVERYONE should use them, but because
> SOME will use them, and it's worth having some language support. You
> would probably do better to argue along those lines than to try to
> claim that every single project ought to be using contracts.

Precisely. To answer the original question, "Why is design by
contracts not widely adopted?" part of the answer is that I suspect
extreme claims like this have put many people off, seeing design by
contract as more of an evangelical stance than a practical tool. If it
were promoted more as a potentially useful addition to the
programmer's toolbox, and less of the solution to every problem, it
may have gained more traction. (Similar issues are why people are
skeptical of functional programming, and many other tools - even the
"strong typing vs weak typing" debate can have a flavour of this "my
proposal solves all the world's ills" attitude).

Personally, I'm open to the benefits of design by contract. But if I
need to buy into a whole philosophy to use it (or engage with its user
community) I'll pass.

Paul

[Chris Angelico]

On Wed, Sep 26, 2018 at 5:51 PM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> Hi Chris,
>
>> It's easy to say that they're boolean expressions. But that's like
>> saying that unit tests are just a bunch of boolean expressions too.
>> Why do we have lots of different forms of test, rather than just a big
>> fat "assert this and this and this and this and this and this"?
>> Because the key to unit testing is not "boolean expressions", it's a
>> language that can usefully describe what it is we're testing.
>> Contracts aren't just boolean expressions - they're a language (or a
>> mini-language) that lets you define WHAT the contract entails.
>
>
> Sorry, I misunderstood you. You are probably referring to knowing the terms like "preconditions, postconditions, invariants, strengthening/weakening", right? In that case, yes, I agree, I presuppose that readers are familiar with the concepts of DbC. Otherwise, of course, it makes no sense to use DbC if you assume nobody could actually figure out what it is :).
>

Let's say you want to define a precondition and postcondition for this function:

def fibber(n):
return n < 2 ? n : fibber(n-1) + fibber(n-2)

What would you specify? Can you say, as a postcondition, that the
return value must be a Fibonacci number? Can you say that, for any 'n'
greater than about 30, the CPU temperature will have risen? How do you
describe those as boolean expressions? The art of the contract depends
on being able to adequately define the conditions.

>
> However, contracts can be useful when testing the GUI -- often it is difficult to script the user behavior. What many people do is record a session and re-play it. If there is a bug, fix it. Then re-record. While writing unit tests for GUI is hard since GUI changes rapidly during development and scripting formally the user behavior is tedious, DbC might be an alternative where you specify as much as you can, and then just re-run through the session. This implies, of course, a human tester.
>

That doesn't sound like the function's contract. That sounds like a
test case - of which you would have multiple, using different
"scripted session" inputs and different outputs (some of success, some
of expected failure).

> Here is my try at the contracts. Assuming that there is a list of figures and that they have a property "displayed" and that "State.blocked" global variable refers to whether the interface is blocked or not::
> @post(lambda: all(figure.displayed for figure in figures)
> @post(lambda: not ipython.in_pylab_mode() or not State.blocked)
> @post(lambda: not interactive() or State.blocked)
> matplotlib.pyplot.show()
>

There is no such thing as "State.blocked". It blocks. The function
*does not return* until the figure has been displayed, and dismissed.
There's no way to recognize that inside the function's state.

Contracts are great when every function is 100% deterministic and can
maintain invariants and/or transform from one set of invariants to
another. Contracts are far less clear when the definitions are
muddier.

[Chris Angelico]

On Wed, Sep 26, 2018 at 6:36 PM Paul Moore <p.f....@gmail.com> wrote:
>
> On Wed, 26 Sep 2018 at 06:41, Chris Angelico <ros...@gmail.com> wrote:
> >
> > On Wed, Sep 26, 2018 at 2:47 PM Marko Ristin-Kaufmann
> > <marko....@gmail.com> wrote:
> > > * The contracts written in documentation as human text inevitably rot and they are much harder to maintain than automatically verified formal contracts.
> >
> > Agreed.
>
> Agreed, if contracts are automatically verified. But when runtime cost
> comes up, people suggest that contracts can be disabled in production
> code - which invalidates the "automatically verified" premise.

Even if they're only verified as a dedicated testing pass ("okay,
let's run the unit tests, let's run the contract verifier, let's run
the type checker, cool, we're good"), they're still better off than
unchecked comments/documentation in terms of code rot. That said,
though: the contract for a function and the documentation for the
function are inextricably linked *already*, and if you let your API
docs rot when you make changes that callers need to be aware of, you
have failed your callers. Wholesale use of contracts would not remove
the need for good documentation; what it might allow is easier version
compatibility testing. It gives you a somewhat automated (or at least
automatable) tool for checking if two similar libraries (eg version
X.Y and version X.Y-1) are compatible with your code. That would be of
some value, if it could be trusted; you could quickly run your code
through a checker and say "hey, tell me what the oldest version of
Python is that will run this", and get back a response without
actually running a gigantic test suite - since it could check based on
the *diffs* in the contracts rather than the contracts themselves. But
that would require a lot of support, all up and down the stack.

ChrisA

[Chris Angelico]

On Wed, Sep 26, 2018 at 6:37 PM Chris Angelico <ros...@gmail.com> wrote:
>
> On Wed, Sep 26, 2018 at 5:51 PM Marko Ristin-Kaufmann
> <marko....@gmail.com> wrote:
> >
> > Hi Chris,
> >
> >> It's easy to say that they're boolean expressions. But that's like
> >> saying that unit tests are just a bunch of boolean expressions too.
> >> Why do we have lots of different forms of test, rather than just a big
> >> fat "assert this and this and this and this and this and this"?
> >> Because the key to unit testing is not "boolean expressions", it's a
> >> language that can usefully describe what it is we're testing.
> >> Contracts aren't just boolean expressions - they're a language (or a
> >> mini-language) that lets you define WHAT the contract entails.
> >
> >
> > Sorry, I misunderstood you. You are probably referring to knowing the terms like "preconditions, postconditions, invariants, strengthening/weakening", right? In that case, yes, I agree, I presuppose that readers are familiar with the concepts of DbC. Otherwise, of course, it makes no sense to use DbC if you assume nobody could actually figure out what it is :).
> >
>
> Let's say you want to define a precondition and postcondition for this function:
>
> def fibber(n):
> return n < 2 ? n : fibber(n-1) + fibber(n-2)

Uhhhhhhhh....

def fibber(n):
return n if n < 2 else fibber(n-1) + fibber(n-2)

Let's, uhh, pretend that I didn't just mix languages there.

For the original, we can say:

@post(raises=ProgrammerFailedSanCheckError)

[Paul Moore]

On Wed, 26 Sep 2018 at 09:45, Chris Angelico <ros...@gmail.com> wrote:
>
> On Wed, Sep 26, 2018 at 6:36 PM Paul Moore <p.f....@gmail.com> wrote:
> >
> > On Wed, 26 Sep 2018 at 06:41, Chris Angelico <ros...@gmail.com> wrote:
> > >
> > > On Wed, Sep 26, 2018 at 2:47 PM Marko Ristin-Kaufmann
> > > <marko....@gmail.com> wrote:
> > > > * The contracts written in documentation as human text inevitably rot and they are much harder to maintain than automatically verified formal contracts.
> > >
> > > Agreed.
> >
> > Agreed, if contracts are automatically verified. But when runtime cost
> > comes up, people suggest that contracts can be disabled in production
> > code - which invalidates the "automatically verified" premise.
>
> Even if they're only verified as a dedicated testing pass ("okay,
> let's run the unit tests, let's run the contract verifier, let's run
> the type checker, cool, we're good"), they're still better off than
> unchecked comments/documentation in terms of code rot.

Absolutely. But if the contracts are checked at runtime, they are
precisely as good as tests - they will flag violations *in any
circumstances you check*. That's great, but nothing new. I understood
that one of the benefits of contracts was that it would handle cases
that you *forgot* to test - like assertions do, in essence - and would
need to be active in production (again, like assertions, if we assume
we've not explicitly chosen to run with assertions disabled) to get
that benefit.

There's certainly benefits for the "contracts as additional tests"
viewpoint. But whenever that's proposed as what people understand by
DbC, the response is "no, it's not like that at all". So going back to
the "why isn't DbC more popular" question - because no-one can get a
clear handle on whether they are "like tests" or "like assertions" or
"something else" :-)

Paul

[Marko Ristin-Kaufmann]

Hi Chris and Paul,

Let me please answer your messages in one go as they are related.

Paul wrote:

For most single use (or infrequently used) functions, I'd argue that the trade-off *isn't* worth it.

Here's a quick example from the pip codebase:

# Retry every half second for up to 3 seconds
@retry(stop_max_delay=3000, wait_fixed=500)
def rmtree(dir, ignore_errors=False):
    shutil.rmtree(dir, ignore_errors=ignore_errors,
                  onerror=rmtree_errorhandler)

Absolutely, I agree. If it's a single-use or infrequently used function that hardly anybody uses, it's not worth it. Moreover, if some contracts are harder to figure out than the implementation, then they are probably in most cases not worth the effort, too.

Please mind that I said: any library would benefit from it, as in the users of any library on pipy would benefit from better, formal and more precise documentation. That doesn't imply that all the contracts need to be specified or that you have to specify the contract for every function, or that you omit the documentation altogether. Some contracts are simply too hard to get explicitly. Some are not meaningful even if you could write them down. Some you'd like to add, but run only at test time since too slow in production. Some run in production, but are not included in the documentation (e.g., to prevent the system to enter a bad state though it is not meaningful for the user to actually read that contract).

Since contracts are formally written, they can be verified. Human text can not. Specifying all the contracts is in most cases not meaningful. In my day-to-day programming, I specify contracts on the fly and they help me express formally to the next girl/guy who will use my code what to expect or what not. That does not mean that s/he can just skip the documentation or that contracts describe fully what the function does. They merely help -- help him/her what arguments and results are expected. That does not mean that I fully specified all the predicates on the arguments and the result. It's merely a help à la
* "Ah, this argument needs to be bigger than that argument!" 
* "Ah, the resulting dictionary is shorter than the input list!"
* "Ah, the result does not include the input list"
* "Ah, this function accepts only files (no directories) and relative paths!"
* "Ah, I put the bounding box outside of the image -- something went wrong!"
* "Ah, this method allows me to put the bounding box outside of the image and will fill all the outside pixels with black!" etc.

For example, if I have an object detector operating on a region-of-interest and returning bounding boxes of the objects, the postconditions will not be: "all the bounding boxes are cars", that would impossible. But the postcondition might check that all the bounding boxes are within the region-of-interest or slightly outside, but not completely outside etc.

Let's be careful not to make a straw-man here, i.e. to push DbC ad absurdum and then discard it that way.

Also, the implicit contracts code currently has are typically pretty loose. What you need to "figure out" is very general. Explicit contracts are typically demonstrated as being relatively strict, and figuring out and writing such contracts is more work than writing code with loose implicit contracts. Whether the trade-off of defining tight explicit contracts once vs inferring a loose implicit contract every time you call the function is worth it, depends on how often the function is called. For most single use (or  infrequently used)  functions, I'd argue that the trade-off *isn't* worth it.

I don't believe such an approach would ever be pragmatical, i.e. automatic versioning based on the contracts. It might hint it (as a warning: you changed a contract, but did not bump the version), but relying solely on this mechanism to get the versioning would imply that you specified all the contracts. Though Bertrand might have always envisioned it as the best state of the world, even he himself repeatedly said that it's better to specify rather 10% than 0% contracts and 20% rather than 10% contracts.

Chris wrote:

def fibber(n):
    return n if n < 2 else fibber(n-1) + fibber(n-2)

 

It depends who you are writing this function for -- for example, instead of the contract, why not just include the code implementation as the documentation? The only meaningful contract I could imagine:

@pre n >= 0

Everything else would be just repetition of the function. If you implemented some optimized and complex fibber_optimized() function, then your contracts would probably look like:

@pre n >= 0

@post (n < 2 ) or result == fibber_optimized(n-1) + fibber_optimized(n-2)

@post not (n < 2) or result == n

> Here is my try at the contracts. Assuming that there is a list of figures and that they have a property "displayed" and that "State.blocked" global variable refers to whether the interface is blocked or not::
> @post(lambda: all(figure.displayed for figure in figures)
> @post(lambda: not ipython.in_pylab_mode() or not State.blocked)
> @post(lambda: not interactive() or State.blocked)
> matplotlib.pyplot.show()
>

There is no such thing as "State.blocked". It blocks. The function
*does not return* until the figure has been displayed, and dismissed.
There's no way to recognize that inside the function's state.

Contracts are great when every function is 100% deterministic and can
maintain invariants and/or transform from one set of invariants to
another. Contracts are far less clear when the definitions are
muddier.

Sorry, it has been ages that I used matplotlib. I thought it meant "blocking" as in "the drawing thread blocks". Since blocking basically means halting the execution-- then the contracts can't help. They are checked before and after the function executes. They can not solve the halting problem. For that you need formal methods (and I doubt that formal methods would ever work for matplotlib). The contracts do not check what happens during the execution of the function. They are not meant to do that. Even the invariants of the class are checked before and after the call to public methods (and the private methods are allowed to break them!).

Please mind that this is actually not the argument pro/against the contracts -- you are discussing in this particular case a tool (different to DbC) which should test the behavior during the execution of a function.

Chirs wrote:

> However, contracts can be useful when testing the GUI -- often it is difficult to script the user behavior. What many people do is record a session and re-play it. If there is a bug, fix it. Then re-record. While writing unit tests for GUI is hard since GUI changes rapidly during development and scripting formally the user behavior is tedious, DbC might be an alternative where you specify as much as you can, and then just re-run through the session. This implies, of course, a human tester.
>

That doesn't sound like the function's contract. That sounds like a
test case - of which you would have multiple, using different
"scripted session" inputs and different outputs (some of success, some
of expected failure).

Well, yes, you can view it as a testing technique; it assumes that scripting a session is often difficult for GUIs and sometimes harder (since combinatorial) than the implementation itself. What I saw people do is write the contracts, put the program in debug mode and let the human tester test it. Think of it as a random test where only checks are your pre/postconditions. The human annotator mimics a meaningful random walk and uses the application as s/he sees fit. I'm not championing this approach, just noting it here as a potential use case.

There's certainly benefits for the "contracts as additional tests"
viewpoint. But whenever that's proposed as what people understand by
DbC, the response is "no, it's not like that at all". So going back to
the "why isn't DbC more popular" question - because no-one can get a
clear handle on whether they are "like tests" or "like assertions" or
"something else" :-)

 

I think that it's a tool that you can use for many things:

* verifiable documentation

* deeper testing (every test case tests also other parts of the system, akin to asserts)

* automatic test generation

* hand-break akin to asserts

I find the first one, verifiable documentation, to be the most useful one in working with my team at the moment.

Cheers,

Marko

[Marko Ristin-Kaufmann]

P.S. My offer still stands: I would be very glad to annotate with contracts a set of functions you deem representative (e.g., from a standard library or from some widely used library). Then we can discuss how these contracts. It would be an inaccurate estimate of the benefits of DbC in Python, but it's at least better than no estimate. We can have as little as 10 functions for the start. Hopefully a couple of other people would join, so then we can even see what the variance of contracts would look like.

[Paul Moore]

On Wed, 26 Sep 2018 at 13:40, Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:

> Please mind that I said: any library would benefit from it, as in the users of any library on pipy would benefit from better, formal and more precise documentation. That doesn't imply that all the contracts need to be specified or that you have to specify the contract for every function, or that you omit the documentation altogether. Some contracts are simply too hard to get explicitly. Some are not meaningful even if you could write them down. Some you'd like to add, but run only at test time since too slow in production. Some run in production, but are not included in the documentation (e.g., to prevent the system to enter a bad state though it is not meaningful for the user to actually read that contract).
>
> Since contracts are formally written, they can be verified. Human text can not. Specifying all the contracts is in most cases not meaningful. In my day-to-day programming, I specify contracts on the fly and they help me express formally to the next girl/guy who will use my code what to expect or what not. That does not mean that s/he can just skip the documentation or that contracts describe fully what the function does. They merely help -- help him/her what arguments and results are expected. That does not mean that I fully specified all the predicates on the arguments and the result. It's merely a help à la
> * "Ah, this argument needs to be bigger than that argument!"
> * "Ah, the resulting dictionary is shorter than the input list!"
> * "Ah, the result does not include the input list"
> * "Ah, this function accepts only files (no directories) and relative paths!"
> * "Ah, I put the bounding box outside of the image -- something went wrong!"
> * "Ah, this method allows me to put the bounding box outside of the image and will fill all the outside pixels with black!" etc.

Whoops, I think the rules changed under me again :-(

Are we talking here about coding explicit executable contracts in the
source code of the library, or using (formally described in terms of
(pseudo-)code) contract-style descriptions in the documentation, or
simply using the ideas of contract-based thinking in designing and
writing code?

> For example, if I have an object detector operating on a region-of-interest and returning bounding boxes of the objects, the postconditions will not be: "all the bounding boxes are cars", that would impossible. But the postcondition might check that all the bounding boxes are within the region-of-interest or slightly outside, but not completely outside etc.

I understand that you have to pick an appropriate level of strictness
when writing contracts. That's not ever been in question (at least in
my mind).

> Let's be careful not to make a straw-man here, i.e. to push DbC ad absurdum and then discard it that way.

I'm not trying to push DbC to that point. What I *am* trying to do is
make it clear that your arguments (and in particular the fact that you
keep insisting that "everything" can benefit) are absurd. If you'd
tone back on the extreme claims (as Chris has also asked) then you'd
be more likely to get people interested. This is why (as you
originally asked) DbC is not more popular - its proponents don't seem
to be able to accept that it might not be the solution to every
problem. Python users are typically fairly practical, and think in
terms of "if it helps in this situation, I'll use it". Expecting them
to embrace an argument that demands they accept it applies to
*everything* is likely to meet with resistance.

You didn't address my question "does this apply to the stdlib"? If it
doesn't, your argument has a huge hole - how did you decide that the
solution you're describing as "beneficial to all libraries" doesn't
improve the stdlib? If it does, then why not demonstrate your case?
Give concrete examples - look at some module in the stdlib (for
example, pathlib) and show exactly what contracts you'd add to the
code, what the result would look like to the library user (who
normally doesn't read the source code) and to the core dev (who does).
Remember that pathlib (like all of the stdlib) doesn't use type
annotations, and that is a *deliberate* choice, mandated by Guido when
he first introduced type annotations. So you're not allowed to add
contracts like "arg1 is a string", nor are you allowed to say that the
lack of type annotations makes the exercise useless.

I think I've probably said all I can usefully say here. If you do
write up a DbC-enhanced pathlib, I'll be interested in seeing it and
may well have more to say as a result. If not, I think I'm just going
to file your arguments as "not proven".

[Paul Moore]

On Wed, 26 Sep 2018 at 13:43, Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> P.S. My offer still stands: I would be very glad to annotate with contracts a set of functions you deem representative (e.g., from a standard library or from some widely used library). Then we can discuss how these contracts. It would be an inaccurate estimate of the benefits of DbC in Python, but it's at least better than no estimate. We can have as little as 10 functions for the start. Hopefully a couple of other people would join, so then we can even see what the variance of contracts would look like.

Our mails crossed in the ether, sorry.

Pathlib.

[Marko Ristin-Kaufmann]

Hi Paul,

Are we talking here about coding explicit executable contracts in the
source code of the library, or using (formally described in terms of
(pseudo-)code) contract-style descriptions in the documentation, or
simply using the ideas of contract-based thinking in designing and
writing code?

The current implementation of icontract uses decorators to decorate the functions and classes (and metaclasses to support inheritance of contracts). You have an "enabled" flag which you can set on/off if you want to disable the contract in some situations.

We are talking about the explicit executable contracts :).

You didn't address my question "does this apply to the stdlib"? If it
doesn't, your argument has a huge hole - how did you decide that the
solution you're describing as "beneficial to all libraries" doesn't
improve the stdlib? If it does, then why not demonstrate your case?
Give concrete examples - look at some module in the stdlib (for
example, pathlib) and show exactly what contracts you'd add to the
code, what the result would look like to the library user (who
normally doesn't read the source code) and to the core dev (who does).
Remember that pathlib (like all of the stdlib) doesn't use type
annotations, and that is a *deliberate* choice, mandated by Guido when
he first introduced type annotations. So you're not allowed to add
contracts like "arg1 is a string", nor are you allowed to say that the
lack of type annotations makes the exercise useless.

Sorry, I missed that point; the messages are getting long :) Yes, the contracts would make sense in stdlib as well, I'd say.

@Chris Angelico  would annotating pathlib convince you that contracts are useful? Is it general enough to start with?

[Chris Angelico]

On Wed, Sep 26, 2018 at 11:04 PM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
> @Chris Angelico would annotating pathlib convince you that contracts are useful? Is it general enough to start with?
>

I won't promise it'll convince me, but it'll certainly be a
starting-point for real discussion. Also, it's a fairly coherent
"library-style" module - a nice warm-up. So, when you're ready for a
REAL challenge, annotate tkinter :)

Actually, annotating the builtins might be worth doing too. Either
instead of or after pathlib.

ChrisA

[Paul Moore]

On Wed, 26 Sep 2018 at 14:04, Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> @Chris Angelico would annotating pathlib convince you that contracts are useful? Is it general enough to start with?

Whoa - be careful here. No-one is saying that "contracts aren't
useful" (at least not that I'd heard). We're saying that contracts
*aren't a solution for every library in existence* (which was
essentially your claim). Annotating pathlib can't convince anyone of
that claim. At best (and this is all that I'm imagining) it might give
some indication of why you have such an apparently-unreasonable level
of confidence in contracts.

I do not expect *ever* to believe you when you say that all projects
would benefit from contracts. What I might get from such an exercise
is a better understanding of what you're imagining when you talk about
a real project "using contracts". Best case scenario - I might be
persuaded to use them occasionally. Worst case scenario - I find them
distracting and unhelpful, and we agree to differ on their value.

[Marko Ristin-Kaufmann]

Hi Paul,

Quite a few people replied on this thread and the previous one before the fork that dbc is either useless in Python or at best useful in avionics/niche applications.

I'm really only saying that contracts are a superior (complementary) tool to informal documentation, doctests, reading the implementation, reading the tests and trial-and-error.

For every library that have the contracts which can be written down formally in a pragmatic way, and when the users of the library are multiple -- then these libraries would benefit from dbc. 

That's all that I'm saying and it might have come over as arrogant due to limits of the medium. It was not my intention to sound so.

I'll have a look at pathlib then.

Cheers,

Marko

[Rhodri James]

On 25/09/18 21:09, Lee Braiden wrote:
> Eh. It's too easy to cry "show me the facts" in any argument. To do that
> too often is to reduce all discussion to pendantry.

I will resist pointing out the spelling mistake... oh damn :-)

The trouble with not crying "show me the facts" is that it is very easy
to make beautiful sounding assertions into a vacuum that fall apart the
moment you subject them to reality. I'm sure we can all think of
politicians of a variety of parties and nationalities with an
unfortunate habit of doing exactly that.

Marko is making some very big assertions about how much of a benefit
Design by Contract is. I flat-out don't believe him. It's up to him to
provide some evidence, since he's the one pressing for change.

> That verifying data against the contract a function makes code more
> reliable should be self evident to anyone with even the most rudimentary
> understanding of a function call, let alone a library or large
> application.

Let's assume that the contracts are meaningful and useful (which I'm
pretty sure won't be 100% true; some people are bound to assume that
writing contracts means they don't have to think). Assuming that you
aren't doing some kind of wide-ranging static analysis (which doesn't
seem to be what we're talking about), all that the contracts have bought
you is the assurance that *this* invocation of the function with *these*
parameters giving *this* result is what you expected. It does not say
anything about the reliability of the function in general.

It seems to me that a lot of the DbC philosophy seems to assume that
functions are complex black-boxes whose behaviours are difficult to
grasp. In my experience this is very rarely true. Most functions I
write are fairly short and easily grokked, even if they do complicated
things. That's part of the skill of breaking a problem down, IMHO; if
the function is long and horrible-looking, I've already got it wrong and
no amount of protective scaffolding like DbC is going to help.

> It's the reason why type checking exists,

Except Python doesn't type check so much as try operations and see if
they work.

> and why bounds checking exists,

Not in C, not most of the time :-)

> and why unit checking exists too.

Unit tests are good when you can do them. A fair bit of the embedded
code I write isn't very susceptible to automated testing, though, not
without spending twice as long writing (and testing!) the test
environment as the code.

--
Rhodri James *-* Kynesim Ltd

[Paul Moore]

On Wed, 26 Sep 2018 at 16:04, Rhodri James <rho...@kynesim.co.uk> wrote:
> Marko is making some very big assertions about how much of a benefit
> Design by Contract is. I flat-out don't believe him. It's up to him to
> provide some evidence, since he's the one pressing for change.

And to be fair, he's now offered to put some time into producing such
a demonstration, so asking for some facts has had a positive outcome.
(As well as demonstrating that Marko is happy to back up his position
and not just make unsubstantiated assertions - so that in itself is
good to know).

Paul

[James Lu]

> It's easy to say that they're boolean expressions. But that's like
> saying that unit tests are just a bunch of boolean expressions too.
> Why do we have lots of different forms of test, rather than just a big
> fat "assert this and this and this and this and this and this"?
> Because the key to unit testing is not "boolean expressions", it's a
> language that can usefully describe what it is we're testing.
> Contracts aren't just boolean expressions - they're a language (or a
> mini-language) that lets you define WHAT the contract entails.

Please read the earlier discussion from Marko. Contracts are like unit tests but acts as documentation that is right next to the function definition. It’s also much shorter in number of lines to define. You can write a simple unit smoke test to turn a contract into a unit test. Contracts serve unit testing and documentation at the same time.

Sent from my iPhone

> On Sep 26, 2018, at 3:18 AM, Chris Angelico <ros...@gmail.com> wrote:
>
> It's easy to say that they're boolean expressions. But that's like
> saying that unit tests are just a bunch of boolean expressions too.
> Why do we have lots of different forms of test, rather than just a big
> fat "assert this and this and this and this and this and this"?
> Because the key to unit testing is not "boolean expressions", it's a
> language that can usefully describe what it is we're testing.
> Contracts aren't just boolean expressions - they're a language (or a
> mini-language) that lets you define WHAT the contract entails.

[David Stanek]

On Wed, Sep 26, 2018 at 12:49 AM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
>> An extraordinary claim is like "DbC can improve *every single project*
>> on PyPI". That requires a TON of proof. Obviously we won't quibble if
>> you can only demonstrate that 99.95% of them can be improved, but you
>> have to at least show that the bulk of them can.
>
>
> I tried to give the "proof" (not a formal one, though) in my previous message.
>

I have to admit that I haven't kept up with the discussion today, but
I was also hoping to see some proof. I'm genuinely interested in
seeing if this is something that can help me and the teams I work
with. I was very interested in DbC a long time ago, but never found a
way to make it valuable to me.

I'd like to see a project from PyPI converted to use DbC. This would
make it easy to see the real world difference between an
implementation developed using DbC compared to one that is well
documented, tested and maybe even includes type hints. Toy or
cherry-picked examples don't help me get a feel for it.

--
david stanek
web: https://dstanek.com
twitter: https://twitter.com/dstanek
linkedin: https://www.linkedin.com/in/dstanek/

[Marko Ristin-Kaufmann]

Hi David,

I'm writing contracts for pathlib as a proof-of-concept.

See pypackagery for an example of a project on pypi that uses contracts:

https://pypi.org/project/pypackagery/

The docs with contracts are available at:

https://pypackagery.readthedocs.io/en/latest/packagery.html

Mind that the interface to icontract might change soon so I'd wait a month or two before it stabilizes.

Cheers, 

Marko

[Chris Barker via Python-ideas]

On Tue, Sep 25, 2018 at 10:10 PM Lee Braiden <leeb...@gmail.com> wrote:

It's the reason why type checking exists, and why bounds checking exists, and why unit checking exists too.

And yet Python has none of those. They all provide safety, but also place a burden on the developer.

So why use Python? I’m not arguing that all those features don’t have their advantages, but I am wondering why add them all to Python, rather than using a language designed for safety?

But Python has such a great ecosystem of packages! Yes, it does — but you might ask yourself why that is.

All that being said — do go and make a nice DbC package for Python — maybe all us naysayers will grow to love it!

But could we please stop cluttering this list with discussion of how great or not great it is?

These meta-conversations are getting really tiresome.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Chris Angelico]

On Thu, Sep 27, 2018 at 5:07 AM Chris Barker <chris....@noaa.gov> wrote:
>
> On Tue, Sep 25, 2018 at 10:10 PM Lee Braiden <leeb...@gmail.com> wrote:
>>
>> It's the reason why type checking exists, and why bounds checking exists, and why unit checking exists too.
>
> And yet Python has none of those. They all provide safety, but also place a burden on the developer.

Type checking? Depends on your definition. Since objects are typed,
Python is safe against the old "I passed an integer when it expected
an array" problem (which probably would result in a junk memory read,
in C), whether the function's arguments are type-checked or not. And
if you want actual type checking, that's available too, just not as
part of the core language (and it has language support for its
syntax).

Bounds checking? Most definitely exists. If you try to subscript a
string or list with a value greater than its length, you get an
exception.

Unit checking? If that's "unit testing", that's part of the standard
library, so yes, that definitely exists in Python.

ChrisA

[Robert Collins]

On Thu, 27 Sep 2018 at 00:44, Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> P.S. My offer still stands: I would be very glad to annotate with contracts a set of functions you deem representative (e.g., from a standard library or from some widely used library). Then we can discuss how these contracts. It would be an inaccurate estimate of the benefits of DbC in Python, but it's at least better than no estimate. We can have as little as 10 functions for the start. Hopefully a couple of other people would join, so then we can even see what the variance of contracts would look like.

i think requests would be a very interesting library to annotate. Just
had a confused developer wondering why calling an API with
session.post(...., data={...some object dict here}) didn't work
properly. (Solved by s/data/json), but perhaps illustrative of
something this might help with?

-Rob

[Chris Angelico]

On Thu, Sep 27, 2018 at 8:53 AM Robert Collins
<rob...@robertcollins.net> wrote:
>
> On Thu, 27 Sep 2018 at 00:44, Marko Ristin-Kaufmann
> <marko....@gmail.com> wrote:
> >
> > P.S. My offer still stands: I would be very glad to annotate with contracts a set of functions you deem representative (e.g., from a standard library or from some widely used library). Then we can discuss how these contracts. It would be an inaccurate estimate of the benefits of DbC in Python, but it's at least better than no estimate. We can have as little as 10 functions for the start. Hopefully a couple of other people would join, so then we can even see what the variance of contracts would look like.
>
> i think requests would be a very interesting library to annotate. Just
> had a confused developer wondering why calling an API with
> session.post(...., data={...some object dict here}) didn't work
> properly. (Solved by s/data/json), but perhaps illustrative of
> something this might help with?

Not sure what you mean by not working; my suspicion is that it DID
work, but didn't do what you thought it did (it would form-encode).
Contracts wouldn't help there, because it's fully legal and correct.

(Unless session.post() differs from requests.post(), but I doubt that
that'd be the case.)

ChrisA

[Robert Collins]

On Thu., 27 Sep. 2018, 11:00 Chris Angelico, <ros...@gmail.com> wrote:

On Thu, Sep 27, 2018 at 8:53 AM Robert Collins
<rob...@robertcollins.net> wrote:
>
> On Thu, 27 Sep 2018 at 00:44, Marko Ristin-Kaufmann
> <marko....@gmail.com> wrote:
> >
> > P.S. My offer still stands: I would be very glad to annotate with contracts a set of functions you deem representative (e.g., from a standard library or from some widely used library). Then we can discuss how these contracts. It would be an inaccurate estimate of the benefits of DbC in Python, but it's at least better than no estimate. We can have as little as 10 functions for the start. Hopefully a couple of other people would join, so then we can even see what the variance of contracts would look like.
>
> i think requests would be a very interesting library to annotate. Just
> had a confused developer wondering why calling an API with
> session.post(...., data={...some object dict here}) didn't work
> properly. (Solved by s/data/json), but perhaps illustrative of
> something this might help with?

Not sure what you mean by not working; my suspicion is that it DID
work, but didn't do what you thought it did (it would form-encode).
Contracts wouldn't help there, because it's fully legal and correct.

Giving post a data= results in form encoding, as you say.

Giving it a json= results in json encoding.

Works is a bit of a weasel word.

The python code did not crash. However it did not perform as desired, either. 

And since that is pretty much the entire value proposition of DbC... it seems like a good case to dissect.

(Unless session.post() differs from requests.post(), but I doubt that
that'd be the case.)

It doesn't.

Rob

[Chris Angelico]

On Thu, Sep 27, 2018 at 9:22 AM Robert Collins

<rob...@robertcollins.net> wrote:
>
> On Thu., 27 Sep. 2018, 11:00 Chris Angelico, <ros...@gmail.com> wrote:
>>
>> On Thu, Sep 27, 2018 at 8:53 AM Robert Collins
>> <rob...@robertcollins.net> wrote:
>> >
>> > On Thu, 27 Sep 2018 at 00:44, Marko Ristin-Kaufmann
>> > <marko....@gmail.com> wrote:
>> > >
>> > > P.S. My offer still stands: I would be very glad to annotate with contracts a set of functions you deem representative (e.g., from a standard library or from some widely used library). Then we can discuss how these contracts. It would be an inaccurate estimate of the benefits of DbC in Python, but it's at least better than no estimate. We can have as little as 10 functions for the start. Hopefully a couple of other people would join, so then we can even see what the variance of contracts would look like.
>> >
>> > i think requests would be a very interesting library to annotate. Just
>> > had a confused developer wondering why calling an API with
>> > session.post(...., data={...some object dict here}) didn't work
>> > properly. (Solved by s/data/json), but perhaps illustrative of
>> > something this might help with?
>>
>> Not sure what you mean by not working; my suspicion is that it DID
>> work, but didn't do what you thought it did (it would form-encode).
>> Contracts wouldn't help there, because it's fully legal and correct.
>
>
> Giving post a data= results in form encoding, as you say.
> Giving it a json= results in json encoding.
>
> Works is a bit of a weasel word.
>
> The python code did not crash. However it did not perform as desired, either.
>
> And since that is pretty much the entire value proposition of DbC... it seems like a good case to dissect.

Okay, but what is the contract that's being violated when you use
data= ? How would you define the contract that this would track?
That's what I'm asking.

[Robert Collins]

On 27 September 2018 at 11:50, Chris Angelico <ros...@gmail.com> wrote:

> Okay, but what is the contract that's being violated when you use
> data= ? How would you define the contract that this would track?
> That's what I'm asking.

I don't know :). From a modelling perspective the correctness of the
behaviour here depends on state in the server :/.

-Rob

[Chris Angelico]

On Thu, Sep 27, 2018 at 10:07 AM Robert Collins
<rob...@robertcollins.net> wrote:
>
> On 27 September 2018 at 11:50, Chris Angelico <ros...@gmail.com> wrote:
>
> > Okay, but what is the contract that's being violated when you use
> > data= ? How would you define the contract that this would track?
> > That's what I'm asking.
>
> I don't know :). From a modelling perspective the correctness of the
> behaviour here depends on state in the server :/.

Exactly. It's the same problem as with trying to write contracts for
matplotlib's plt.show() - its job is fundamentally external, so you
can't define preconditions and postconditions for it. This is one of
the limitations of contracts, or at least that's my understanding.
Maybe I can be proven wrong here?

ChrisA

[Greg Ewing]

Chris Angelico wrote:
> For example, matplotlib's
> plt.show() method guarantees that... a plot will be shown, and the
> user will have dismissed it, before it returns. Unless you're inside
> Jupyter/iPython, in which case it's different. Or if you're in certain
> other environments, in which case it's different again. How do you
> define the contract for something that is fundamentally interactive?

Indeed, this is what bothers me about DbC fanaticism. It seems to
have been conceived by people thinking about very computer-sciency
kinds of problems, e.g. you're implenenting a data structure which
has certain important invariants that can be expressed mathematically,
and code can be written that checks them reasonably efficiently.
All very neat and self-contained.

But a lot of real-world code isn't like that -- much of the time,
the correctness of a piece of code can't be tested without reference
to things outside that piece of code, or even outside of the program
altogether. How would you write DbC contracts for a GUI, for
example? Most of the postconditions consist of "an image looking
something like this appears on the screen", where "this" is a very
complicated function of the previous state of the system and the
user's input.

IMO, the reason DbC hasn't taken off is that it assumes an idealised
model of what programming consists of that doesn't match reality
well enough.

--
Greg

[David Mertz]

I'm not sure what to do with the repeated assurance that that various things "are obvious."  It really is NOT the case that me, or Paul Moore, or Hugh Fisher, or Greg Ewing are simply too simple minded to understand what DbC is.

The off-putting evangelical quality around DbC is something like the similar evangelical insistence that OOP solves all problems that one tended to hear in the late-1980s to mid-1990s especially.  The fact that no one can quite pin down just what this special quality of DbC is doesn't help... the reality is that they really ARE NOT much different from assertions, in either practice or theory.

Actually, the evangelical OOP thing connects with the evangelical DbC.  One of the "advantages" of DbC is often stated as support for inheritance.  But the truth is, I hardly ever use inheritance in my code, or at most very shallow inheritance in ways where invariants are mostly not preserved.  My use of OOP in Python is basically mixins plus magic methods... and no, it's not because I don't understand OOP (I wrote, for example, some of the most widely read papers on metaclass programming in Python, and designed and wrote about making a—toy, admittedly—method resolution order and OOP system for R; my frequent co-author wrote the canonical paper on C3 linearization, in which I'm acknowledged for my edits, but am not an author).

I also wrote an article or two about DbC in Python in the early 2000s.  None of this is very new.

On Mon, Sep 24, 2018 at 3:47 AM Marko Ristin-Kaufmann <marko....@gmail.com> wrote:

Obvious benefits

You both seem to misconceive the contracts. The goal of the design-by-contract is not reduced to testing the correctness of the code, as I reiterated already a couple of times in the previous thread. The contracts document formally what the caller and the callee expect and need to satisfy when using a method, a function or a class. This is meant for a module that is used by multiple people which are not necessarily familiar with the code. They are not a niche. There are 150K projects on pypi.org. Each one of them would benefit if annotated with the contracts.

Greg Ewing, Chris Angelica, and Paul Moore make the point very well that MOST code is simply not the sort of thing that is amenable to having contracts.  What one needs to state is either Turing complete or trivially reducible to type declarations.  Or the behavior is duck typed loosely enough that it can do the right thing without being able to specify the pre- and post-conditions more precisely than "do what this function does."  Very often, that looseness is simply NOT a problem, and is part of what makes Python flexible and successful.

Contracts are difficult to read.

David wrote:

To me, as I've said, DbC imposes a very large cost for both writers and readers of code.

This is again something that eludes me and I would be really thankful if you could clarify. Please consider for an example, pypackagery (https://pypackagery.readthedocs.io/en/latest/packagery.html) and the documentation of its function resolve_initial_paths:

The documentation you show below is definitely beautiful  I guess that's generated by your Sphinx enhancement, right?  There are similar systems for pulling things out of docstrings that follow conventions, but there's a small incremental value in making them live tests at the same time.

But reading the end-user documentation is not the reading I'm talking about.  The reading I mean is looking at the actual source code files.  Stating all the invariants you want code to follow makes the definitions of functions/methods longer and more cognitive effort to parse.  A ten line function body is likely to be accompanied by 15 lines of invariants that one can state about hopes for the function behavior.  That's not always bad... but it's definitely not always good.

This is why unit tests are often much better, organizationally.  The function definitions can remain relatively concise and clean because most of the time when you are reading or modifying them you don't WANT to think about those precise contracts.  Sure, maybe some tool support like folding of contracts could make the burden less; but not less than putting them in entirely separate files.

Most of the time, I'd like to look at the simplest expression of the code possible.  Then on a different day, or with a different set of eyes, I can look at the completely distinct file that has arbitrarily many unit tests to check invariants I'd like functions to maintain.  Yes, the issues are a little bit different in classes and nested hierarchies... but like I said, I never write those, and tend not to like code that does. 

packagery.resolve_initial_paths(initial_paths)

Resolve the initial paths of the dependency graph by recursively adding *.py files beneath given directories.

Parameters:	initial_paths (List[Path]) – initial paths as absolute paths
Return type:	List[Path]
Returns:	list of initial files (i.e. no directories)
Requires:	all(pth.is_absolute() for pth in initial_paths)
Ensures:	len(result) >= len(initial_paths) if initial_paths else result == [] all(pth.is_absolute() for pth in result) all(pth.is_file() for pth in result)

Again, this is a straw man.

 

Writing contracts is difficult.

David wrote:

To me, as I've said, DbC imposes a very large cost for both writers and readers of code.

The effort of writing contracts include as of now:

* include icontract (or any other design-by-contract library) to setup.py (or requirements.txt), one line one-off

* include sphinx-icontract to docs/source/conf.py and docs/source/requirements.txt, two lines, one-off

* write your contracts (usually one line per contract).

It's not much work to add `import icontract` of course.  But writing contracts is a lot of work.  Usually they work best only for pure functions, and only for ones that deal with rather complex data structures (flat is better than nested).  In Python, you usually simply do not have to think about the issues that contracts guard against.

 

I think that ignorance plays a major role here. Many people have misconceptions about the design-by-contract. They just use 2) for more complex methods, or 3) for rather trivial methods. They are not aware that it's easy to use the contracts (1) and fear using them for non-rational reasons (e.g., habits).

Again, this evangelical spirit to "burn the heretics" really isn't going to win folks over.  No one replying here is ignorant.  We all do not see any "silver bullet" in DbC that you advocate.  It's a moderately useful style that I wouldn't object to using if a codebase style guide demanded it.  But it's rarely the tool I would reach for on my own.  I just find it easier to understand and use a combination of assertions and unit tests.  Neither is exactly the same thing as DbC, but it's pretty darn close in practice.

And no, my hesitance isn't because I don't understand boolean logic.  I've also studied a bit of graduate logic and model theory in a long ago life.

Yours, David...

--

Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.

[Stephen J. Turnbull]

Robert Collins writes:

> I think the underlying problem is that you're treating this as a
> logic problem (what does logic say applies here), rather than an
> engineering problem (what can we measure and what does it tell us
> about whats going on).

Pure gold. I'm glad your name is "Robert Collins", or I would have
skipped the post and just muted the thread.

> My suspicion, for which I have only anecdata, is that its really in c)
> today. Kind of where TDD was in the early 2000's (and as I understand
> the research, its been shown to be a wash: you do get more tests than
> test-last or test-during,

That's a pretty big deal though, if it means the shop doesn't need The
Big Nurse to prod you to write more tests.

> and more tests is correlated with quality and ease of evolution,
> but if you add that test coverage in test-during or test-last, you
> end up with the same benefits).

"No Silver Bullet." QED

I think Watts Humphrey should have titled his classic, "Any Discipline
for Software Engineering", and subtitled it "'Whatever' Works, as Long
as You Actually Do It".[1] All of his books on the practice of
software engineering really do say that, by the way. He recommends
*starting* with *his* way because it worked for him and many students,
so you can just follow the cookbook until "actually doing" becomes
natural. Then change to doing what comes more naturally to you.


Footnotes:
[1] Fred Brooks would have done that, I think. Humphrey was way too
stuffy to do that. :-)

[Greg Ewing]

Chris Angelico wrote:
> if you let your API
> docs rot when you make changes that callers need to be aware of, you
> have failed your callers.

Yes, I find that documentation auto-generated from code is
usually a poor substitute for human-written documentation.
Dumping your formally-written contracts into the docs makes
the reader reverse-engineer them to figure out what the
programmer was really trying to say.

Which do you find easier to grok at a glance:

all(L[i] <= L[i+1] for i in range(len(L) - 1))

or

# The list is now sorted

--
Greg

[Chris Angelico]

On Thu, Sep 27, 2018 at 3:31 PM Greg Ewing <greg....@canterbury.ac.nz> wrote:
>
> Chris Angelico wrote:
> > if you let your API
> > docs rot when you make changes that callers need to be aware of, you
> > have failed your callers.
>
> Yes, I find that documentation auto-generated from code is
> usually a poor substitute for human-written documentation.
> Dumping your formally-written contracts into the docs makes
> the reader reverse-engineer them to figure out what the
> programmer was really trying to say.
>
> Which do you find easier to grok at a glance:
>
> all(L[i] <= L[i+1] for i in range(len(L) - 1))
>
> or
>
> # The list is now sorted
>

Well, with that particular example, you're capitalizing on the known
meaning of the English word "sorted". So to be fair, you should do the
same in Python:

postcondition: L == sorted(L)

This would be inappropriate for an actual sorting function, but let's
say you're altering the value of an item in a sorted list, and
shifting it within that list to get it to the new position, or
something like that.python-ideas <python...@python.org>

But yes, in general I do agree: it's frequently cleaner to use an
English word than to craft a Python equivalent.

ChrisA

[Stephen J. Turnbull]

Paul Moore writes:

> With provisos. Figuring out contracts in sufficient detail to use
> the code is *in many cases* simple. For harder cases, agreed. But
> that's why this is simply a proof that contracts *can* be useful,
> not that 100% of code would benefit from them.

Note that that is not what Marko wrote: he wrote that 100% of projects
on PyPI would benefit. Like others, I'd like to see at least a
before/after on *one* whole project (that's not tiny, so it has
several meaningful APIs and several calls for at least a few of them),
to see *how much* of that project actually benefits, and how much of
that benefit derives from "more sophisticated than assert" DbC tech.

For now, I think the whole thread is technically speaking off-topic,
though. Let's stipulate that DbC is a valuable technology that Python
programmers should have available.

(1) Why is it so valuable that it needs to be in the python.org
distribution? We already have asserts for the simplest cases.
Won't PyPI do for the full-blown Meyer-style implementation?
Development discussion for that implementation should take place
on GitHub[1] or a similar channel, IMO.

(2) We don't have an implementation to include, or if you like there
are many PoCs, and there's no good reason to suppose that there
will be consensus on the candidate for inclusion in this decade.
Again, development discussions for those implementations should
take place on GitHub before coming here to discuss (a) which is
best for the stdlib, (b) whether it's good enough, and (c) whether
it's needed in the stdlib at all. Is there a clear, leading
candidate that could be proposed "soon", icontracts maybe?

(3) The syntaxes proposed so far require inelegant constructs, like
lambdas or strs, so that decorator arguments can be evaluated
lazily. Lazy evaluation of arguments is something that newcomers
often want after being burned by "def foo(l=[]):". But there are
at least two plausible ways to handle this. One is like Lisp
macros: "defmacro mfoo(not, one, of, its, args, is, evaluated)".
Another would be a marker for args to be returned unevalled:
"def foo(eval_me, eval_me_not: unevalled, eval_me_too)".[2]

Thus, unevalled arguments *may* be a plausible syntax change that
would help support DbC as well as other possibly desirable use
cases, but we have no proposal to discuss. Do we?

I'm not yet suggesting that this thread *should* be terminated here
(and that's not to avoid charges of hypocrisy as I post to other
subthreads ;-). But I think we should be continuously aware of the
three questions I posed above.


Footnotes:
[1] Personally, I'd prefer it be GitLab. :-)

[2] Yes, I'm teasing the type annotations folks, I doubt this syntax
will fly.

[Stephen J. Turnbull]

Chris Angelico writes:

> Okay, but what is the contract that's being violated when you use
> data= ? How would you define the contract that this would track?
> That's what I'm asking.

The output is input to some other component. In production, that
component would not be under your control, perhaps, but in testing it
surely is. The contract would be associated with the other component,
and it would be "my input is JSON".

Alternatively, if "your" component is being used in a protocol that
specifies JSON, the contract could be "what I post is JSON". Presumably
that contract can't be checked in production, but in testing it could
be. If the inputs to your component satisfy their contracts, but JSON
isn't coming out, the problem is in your component.

Ie, I don't see how DbC can diagnose *how* a component is broken, in
general. It *can* localize the breakage, and provide some hints based
on the particular contract that is "violated".

I think this shows that in a broad class of cases, for existing code,
DbC doesn't do much that a developer with a debugger (such as print()
;-) can't already do, and the developer can do it much more flexibly.

However, getting meta,

> Just had a confused developer wondering why calling an API with
> session.post(...., data={...some object dict here}) didn't work
> properly. (Solved by s/data/json)

session.post seems to be a pretty horrible API. You'd think this
would either be

session.post_json(..., data={...})

or

session.post(..., data={...}, format='json')

with the former preferred (and session.post deprecated in favor of
session.form_encoded; I guess you could also use the latter and
require the 'format' argument).

How would this help in the DbC context? Your server (if you own it),
or your mock server in the test suite, will complain that its contract
"my input is JSON" is being violated, because it's explicitly an entry
condition, your programmer looks at the component that's supposed to
produce JSON, sees "form_encoded" either in the function name or the
format argument's value, and the likelihood of confusion is small.

The contribution of DbC here is not in the contract itself, but in the
discipline of thinking "how would I write this contract so that its
violation would point me to the coding error?", which leads to
refactoring the 'post' method.

Is DbC better than the server doing "assert is_valid_json(input)"?
That too works best with the suggested refactoring. Evidently it's
not better, but there are things that assert can't do. For
example, the server might incrementally parse the JSON, yielding
useful subobjects as you go along. Then you can't just assert
is_valid_json at the beginning of the response generator; you need to
do this at the server level. A DbC toolkit would presumably provide a
way to decorate the server with

try:
server.accept_json_object()
except JSONParseError as e:
report_contract_violation(e)

This is, of course, all imaginary, and I have no idea whether it would
work as suggested in practice. It will be interesting to me to see,
not only Marko's contracts in a DbC-ized module, but also places where
he demands refactoring so that an *informative* contract can be
written.

Steve

[Greg Ewing]

David Mertz wrote:
> the reality is that they really ARE NOT much different
> from assertions, in either practice or theory.

Seems to me that assertions is exactly what they are. Eiffel just
provides some syntactic sugar for dealing with inheritance, etc.
You can get the same effect in present-day Python if you're
willing to write the appropriate code.

--
Greg

[Paul Moore]

On Thu, 27 Sep 2018 at 08:03, Greg Ewing <greg....@canterbury.ac.nz> wrote:
>
> David Mertz wrote:
> > the reality is that they really ARE NOT much different
> > from assertions, in either practice or theory.
>
> Seems to me that assertions is exactly what they are. Eiffel just
> provides some syntactic sugar for dealing with inheritance, etc.
> You can get the same effect in present-day Python if you're
> willing to write the appropriate code.

Assertions, as far as I can see, are the underlying low level
*mechanism* that contracts would use. Just like they are the low level
mechanism behind unit tests (yeah, it's really exceptions, but close
enough). But like unit tests, contracts seem to me to be a philosophy
and a library / programming technique layered on top of that base. The
problem seems to me to be that DbC proponents tend to evangelise the
philosophy, and ignore requests to show the implementation (often
pointing to Eiffel as an "example" rather than offering something
appropriate to the language at hand). IMO, people don't tend to
emphasise the "D" in DbC enough - it's a *design* approach, and more
useful in that context than as a programming construct.

For me, the philosophy seems like a reasonable way of thinking, but
pretty old hat (I learned about invariants and pre-/post-conditions
and their advantages for design when coding in PL/1 in the 1980's,
about the same time as I was learning Jackson Structured Programming).
I don't think in terms of contracts as often as I should - but it's
unit tests that make me remember to do so. Would a dedicated
"contracts" library help? Probably not much, but maybe (if it were
lightweight enough) I could get used to the idea.

Like David, I find that having contracts inline is the biggest problem
with them. I try to keep my function definitions short, and contracts
can easily add 100% overhead in terms of lines of code. I'd much
prefer contracts to be in a separate file. (Which is basically what
unit tests written with DbC as a principle in mind would be). If I
have a function definition that's long enough to benefit from
contracts, I'd usually think "I should refactor this" rather than "I
should add contracts".

Paul

[Marko Ristin-Kaufmann]

Hi Paul,

I only had a contracts library in mind (standardized so that different modules with contracts can interact and that the ecosystem for automic testing could emerge). I was never thinking about the philosophy or design methodology (where you write _all_ the contracts first and then have the implementation fulfill them). I should have clarified that more. I personally also don't think that such a methodology is practical.

I really see contracts as verifiable docs that rot less fast than human text and are formally precise / less unambiguous than human text. Other aspects such as deeper tests and hand-braking (e.g., as postconditions which can't be practically implemented in python without exit stack context manager) are also nice to have.

I should be done with pathlib contracts by tonight if I manage to find some spare time in the evening.

Cheers,

Marko

[Marko Ristin-Kaufmann]

Hi,

I annotated pathlib with contracts:https://github.com/mristin/icontract-pathlib-poc. I zipped the HTML docs into https://github.com/mristin/icontract-pathlib-poc/blob/master/contracts-pathlib-poc.zip, you can just unpack and view the index.html.

One thing I did observe was that there were contracts written in text all over the documentation -- I tried to formulate most of them in code. Since I'm not the implementer nor very familiar with the module, please consider that many of these contracts can be definitely made more beautiful. There were some limitations to icontract-sphinx extension and icontracts which I noted at the top of the document.

(Note also that I had to rename the file to avoid import conflicts.)

Some of the contracts might seem trivial -- but mind that you, as a writer, want to tell the user what s/he is expected to fulfill before calling the function. For example, if you say:

rmdir()

Remove this directory. The directory must be empty.

Requires:	not list(self.iterdir()) (??? There must be a way to check this more optimally) self.is_dir()

self.is_dir() contract might seem trivial -- but it's actually not. You actually want to convey the message: dear user, if you are iterating through a list of paths, use this function to decide if you should call rmdir() or unlink(). Analogously with the first contract: dear user, please check if the directory is empty before calling rmdir() and this is what you need to call to actually check that.

I also finally assembled the puzzle. Most of you folks are all older and were exposed to DbC in the 80ies championed by DbC zealots who advertised it as the tool for software development. You were repulsed by their fanaticism -- the zealots also pushed for all the contracts to be defined, and none less. Either you have 100% DbC or no sane software development at all.

I, on the other side, were introduced to DbC in university much later -- Betrand held most of our software engineering lectures (including introduction to programming which was in Eiffel ;)). I started going to uni in 2004; by that time there was no fanaticism about DbC around -- not even by Bertrand himself. We were taught to use it just as yet another tool in our toolbox along unit testing and formal proofs. Not as a substitute for unit testing! Just as yet another instrument for correctness. There was no 100% DbC -- and we did quite some realistic school projects such as a backend for a flight-booking website in Eiffel (with contracts ;)). I remember that we got minus points if you wrote something in the documentation that could have been easily formalized. But nobody pushed for all the contracts; everybody was pragmatic. Nobody didn't even think about proposing to abolish unit testing and staff all the tests in the contracts to be smoke-tested.

While you read my proposals in the light of these 80ies style DbC proponents, I think always only about a much more minor thing: a simple tool to make some part of the documentation verifiable.

One lesson that I learned from all these courses was to what degree our understandings (especially among non-native speakers) differ. Even simple statements such as "x must be positive" can mean x > 0 and x >= 0 to different people. For me it became obvious that "x > 0" is clearer than "x must be positive" -- and this is that "obvious" which I always refer in my posts on this list. If a statement can not be formalized easily and introduces confusion, that's a different pair of shoes. But if it can -- why shouldn't you formalize it? I still can't wrap my head around the idea that it's not obvious that you should take the formal version over the informal version if both are equally readable, but one is far less unambiguous. It feels natural to me that if you want to kick out one, kick out the more ambiguous informal one. What's the point of all the maths if the informal languages just do as well?

And that's why I said that the libraries on pypi meant to be used by multiple people and which already have type annotations would obviously benefit from contracts -- while you were imagining that all of these libraries need to be DbC'ed 100%, I was imagining something much more humble. Thus the misunderstanding.

After annotating pathlib, I find that it actually needs contracts more thain if it had type annotations. For example:

stat()

Return the result of the stat() system call on this path, like os.stat() does.

Ensures:	result is not None ⇒ self.exists() result is not None ⇒ os.stat(str(self)).__dict__ == result.__dict__ (??? This is probably not what it was meant with ‘like os.stat() does’?)

But what does it mean "like os.stat() does"? I wrote equality over __dict__'s in the contract. That was my idea of what the implementer was trying to tell me. But is that the operator that should be applied? Sure, the contract merits a description. But without it, how am I supposed to know what "like" means?

Similarly with rmdir() -- "the directory must be empty" -- but how exactly am I supposed to check that?

Anyhow, please have a look at the contracts and let me know what you think. Please consider it an illustration. Try to question whether the contracts I wrote are so obvious to everybody even if they are obvious to you and keep in mind that the user does not look into the implementation. And please try to abstract away the aesthetics: neither icontract library that I wrote nor the sphinx extension are of sufficient quality. We use them for our company code base, but they still need a lot of polishing. So please try to focus only on the content. We are still talking about contracts in general, not about the concrete contract implementation.

Cheers,

Marko

 

[Paul Moore]

On Fri, 28 Sep 2018 at 02:24, Marko Ristin-Kaufmann <marko....@gmail.com> wrote:

Hi,

I annotated pathlib with contracts:https://github.com/mristin/icontract-pathlib-poc. I zipped the HTML docs into https://github.com/mristin/icontract-pathlib-poc/blob/master/contracts-pathlib-poc.zip, you can just unpack and view the index.html.

Thanks, for doing this! It's probably not going to result in the reaction you hoped for (see below) but I appreciate you taking the time to do it.

Some of the contracts might seem trivial -- but mind that you, as a writer, want to tell the user what s/he is expected to fulfill before calling the function. For example, if you say:

rmdir()

Remove this directory. The directory must be empty.

Requires:	not list(self.iterdir()) (??? There must be a way to check this more optimally) self.is_dir()

self.is_dir() contract might seem trivial -- but it's actually not. You actually want to convey the message: dear user, if you are iterating through a list of paths, use this function to decide if you should call rmdir() or unlink(). Analogously with the first contract: dear user, please check if the directory is empty before calling rmdir() and this is what you need to call to actually check that.

The problem (IMO) with both of these is precisely that they are written as Python expressions. Your prose descriptions of what they mean are fine, and *those* are what I would hope to see in documentation. This is far more obvious in later examples, where the code needed to check certain conditions is extremely unclear unless you spend time trying to interpret it.

 

I also finally assembled the puzzle. Most of you folks are all older and were exposed to DbC in the 80ies championed by DbC zealots who advertised it as the tool for software development. You were repulsed by their fanaticism -- the zealots also pushed for all the contracts to be defined, and none less. Either you have 100% DbC or no sane software development at all.

Well, yes, but your claims were *precisely* the same as those I saw back then. "All projects on PyPI would benefit", "the benefits are obvious", ...

As I said, DbC is a reasonable methodology, but the reason it's not more widely used is (in the first instance) because it has a *marketing* problem. Solving that with more of the same style of marketing won't help. Once you get beyond the marketing, there are *still* questions (see above and below), but if you can't even get people past the first step, you've lost.

 

And that's why I said that the libraries on pypi meant to be used by multiple people and which already have type annotations would obviously benefit from contracts -- while you were imagining that all of these libraries need to be DbC'ed 100%, I was imagining something much more humble. Thus the misunderstanding.

No, I was imagining that some libraries were small, some were used by small, specialised groups, and some were immensely successful without DbC. So claiming that they would "obviously" benefit is a very strong claim.

 

After annotating pathlib, I find that it actually needs contracts more thain if it had type annotations. For example:

stat()

Return the result of the stat() system call on this path, like os.stat() does.

Ensures:	result is not None ⇒ self.exists() result is not None ⇒ os.stat(str(self)).__dict__ == result.__dict__ (??? This is probably not what it was meant with ‘like os.stat() does’?)

But what does it mean "like os.stat() does"? I wrote equality over __dict__'s in the contract. That was my idea of what the implementer was trying to tell me. But is that the operator that should be applied? Sure, the contract merits a description. But without it, how am I supposed to know what "like" means?

Similarly with rmdir() -- "the directory must be empty" -- but how exactly am I supposed to check that?

Isn't that the whole point? The prose statement "the directory must be empty" is clear. But the exact code check isn't - and may be best handled by a series of unit tests, rather than a precondition.

Anyhow, please have a look at the contracts and let me know what you think. Please consider it an illustration. Try to question whether the contracts I wrote are so obvious to everybody even if they are obvious to you and keep in mind that the user does not look into the implementation. And please try to abstract away the aesthetics: neither icontract library that I wrote nor the sphinx extension are of sufficient quality. We use them for our company code base, but they still need a lot of polishing. So please try to focus only on the content. We are still talking about contracts in general, not about the concrete contract implementation

The thing that you didn't discuss in the above was the effect on the source code. Looking at your modified sources, I found it *significantly* harder to read your annotated version than the original. Basically every function and class was cluttered with irrelevant[1] conditions, which obscured the logic and the basic meaning of the code.

[1] Irrelevant in terms of the flow of the code - I appreciate that there's value in checking preconditions in the broader sense. It's like all error handling and similar - there's a balance to be had between "normal behaviour" and handling of exceptional cases. And I feel that the contracts tip that balance too far towards making exceptional cases the focus.

So ultimately this example has probably persuaded me that I *don't* want to add contract checking, except in very specific cases where the benefits outweigh the disadvantages. It's very subjective, though, so I'm fine if other people feel differently.

Paul

[Chris Angelico]

On Fri, Sep 28, 2018 at 11:25 AM Marko Ristin-Kaufmann
<marko....@gmail.com> wrote:
>
> Hi,
>
> I annotated pathlib with contracts:https://github.com/mristin/icontract-pathlib-poc. I zipped the HTML docs into https://github.com/mristin/icontract-pathlib-poc/blob/master/contracts-pathlib-poc.zip, you can just unpack and view the index.html.
>
> One thing I did observe was that there were contracts written in text all over the documentation -- I tried to formulate most of them in code. Since I'm not the implementer nor very familiar with the module, please consider that many of these contracts can be definitely made more beautiful. There were some limitations to icontract-sphinx extension and icontracts which I noted at the top of the document.
>

You do a lot of contracts that involve is_absolute and other checks.
But the postcondition on is_absolute merely says:

not result or self.root != ""

(Also: I'm not sure, but I think maybe that should be _root?? Leaving
that aside.)

So I'm not sure how much you can really ascertain about absolute
paths. You guarantee that is_absolute will return something plausible,
but unless you guarantee that it's returning the correct value,
depending on it for your preconditions seems dubious. A buggy
is_absolute could break basically everything, and your contracts
wouldn't notice.

It is still fundamentally difficult to make assertions about the file
system as pre/post contracts. Are you becoming aware of this?
Contracts, as has been stated multiple times, look great for
mathematically pure functions that have no state outside of their own
parameters and return values (and 'self', where applicable), but are
just poor versions of unit tests when there's anything external to
consider.

ChrisA

[Jonathan Fine]

I like this discussion. I'd like to add another theme, namely what
should happen when there is an error. (This is prompted by race
hazards when performing file system operations.)

Suppose fn_a() calls fn_b(), and fn_b() raises an exception. What then
should fn_a() do? It may be that this exception has left part or all
of the system in an inconsistent (invalid) state.

At this level of abstraction, it's not possible to sensibly answer
this question. Sometimes the whole system should be stopped. Other
times, an invalidation of an object is enough. And sometimes, a
rollback of the transaction is what's wanted.

Here's a well-known example (overflow exception in Ariane 5), which to
me shows that these problems can be very hard to get right.
https://en.wikipedia.org/wiki/Cluster_(spacecraft)

According to wikipedia (above) this failure resulted in "the first
example of large-scale static code analysis by abstract
interpretation".

I expect that in some situations design-by-contract will help here, by
encouraging a focus on providing a more complete specification of
behaviour. It would be good to have some real-life Python examples.
I'd afraid I don't have any (although I've looked either).

--
Jonathan

[Chris Angelico]

On Fri, Sep 28, 2018 at 7:29 PM Jonathan Fine <jfin...@gmail.com> wrote:
>
> I like this discussion. I'd like to add another theme, namely what
> should happen when there is an error. (This is prompted by race
> hazards when performing file system operations.)
>
> Suppose fn_a() calls fn_b(), and fn_b() raises an exception. What then
> should fn_a() do? It may be that this exception has left part or all
> of the system in an inconsistent (invalid) state.

That's why try/finally exists. You shouldn't have to worry about
contracts for that.

(Similarly, context managers, which are a way of wrapping up
try/finally into a convenient package.)

ChrisA

[Paul Moore]

On Fri, 28 Sep 2018 at 10:37, Chris Angelico <ros...@gmail.com> wrote:
>
> On Fri, Sep 28, 2018 at 7:29 PM Jonathan Fine <jfin...@gmail.com> wrote:
> >
> > I like this discussion. I'd like to add another theme, namely what
> > should happen when there is an error. (This is prompted by race
> > hazards when performing file system operations.)
> >
> > Suppose fn_a() calls fn_b(), and fn_b() raises an exception. What then
> > should fn_a() do? It may be that this exception has left part or all
> > of the system in an inconsistent (invalid) state.
>
> That's why try/finally exists. You shouldn't have to worry about
> contracts for that.
>
> (Similarly, context managers, which are a way of wrapping up
> try/finally into a convenient package.)

However, a contract would need to be able to express "Returns a fully
initialised widget or raises a BrokenWidget exception". Unit tests do
this sort of thing all the time.

Paul

[David Mertz]

On Fri, Sep 28, 2018, 4:38 AM Paul Moore <p.f....@gmail.com> wrote:

On Fri, 28 Sep 2018 at 02:24, Marko Ristin-Kaufmann <marko....@gmail.com> wrote:

I annotated pathlib with contracts:https://github.com/mristin/icontract-pathlib-poc. I zipped the HTML docs into https://github.com/mristin/icontract-pathlib-poc/blob/master/contracts-pathlib-poc.zip, you can just unpack and view the index.html.

The thing that you didn't discuss in the above was the effect on the source code. Looking at your modified sources, I found it *significantly* harder to read your annotated version than the original. Basically every function and class was cluttered with irrelevant[1] conditions, which obscured the logic and the basic meaning of the code.

My reaction was just the same as Paul's. I read the modified source, and found that the invariant declarations made it *dramatically* harder to read. The ratio was almost exactly as I characterized in a recent note: 15 lines of pre/post-conditions on a 10 like function.

Like Paul, I understand the documentation and testing value of these, but they were highly disruptive to the readability of the functions themselves.

As a result of reading the example, I'd be somewhat less likely to use a DbC library, and much more strongly opposed to having one in the standards library (and aghast at the idea of dedicated syntax)

[David Mertz]

On Thu, Sep 27, 2018, 9:25 PM Marko Ristin-Kaufmann <marko....@gmail.com> wrote:

Try to question whether the contracts I wrote are so obvious to everybody even if they are obvious to you and keep in mind that the user does not look into the implementation.

I had missed this comment, but this seems to be the biggest disconnect, or talking past each other.

I'm a user of many libraries. I USUALLY look at the implementation when in doubt about a function. If contracts are meant only for users who don't look at code, the detrimental effect on code readability is mitigated.

The other place I look, if not the actual implementation, is at the docstring. I don't remember if icontracts patches the docstring when it decorates a function. If not, that would be very helpful.

I agree that all the Sphinx documentation examples shown are very nice. Prose descriptions would often be nicer still, but the Boolean expressions are helpful for those unusual cases where I don't want to look at the code.

[Paul Moore]

On Fri, 28 Sep 2018 at 13:23, David Mertz <me...@gnosis.cx> wrote:
> I agree that all the Sphinx documentation examples shown are very nice. Prose descriptions would often be nicer still, but the Boolean expressions are helpful for those unusual cases where I don't want to look at the code.

I'm ambivalent about the Sphinx examples. I find the highly detailed
code needed to express a condition fairly unreadable (and I'm an
experienced Python programmer). For example

@pre(lambda args, result: not any(Path(arg).is_absolute() for arg in args) or
(result == [pth for arg in args for pth in [Path(arg)] if
pth.is_absolute()][-1]),
"When several absolute paths are given, the last is taken as an anchor
(mimicking os.path.join()’s behaviour)")

The only way I'd read that is by looking at the summary text - I'd
never get the sense of what was going on from the code alone. There's
clearly a number of trade-offs going on here:

* Conditions should be short, to avoid clutter
* Writing helper functions that are *only* used in conditions is more
code to test or get wrong
* Sometimes it's just plain hard to express a verbal constraint in code
* Marko isn't that familiar with the codebase, so there may be better
ways to express certain things

But given that *all* the examples I've seen of contracts have this
issue (difficult to read expressions) I suspect the problem is
inherent.

Another thing that I haven't yet seen clearly explained. How do these
contracts get *run*? Are they checked on every call to the function,
even in production code? Is there a means to turn them off? What's the
runtime overhead of a "turned off" contract (even if it's just an
if-test per condition, that can still add up)? And what happens if a
contract fails - is it an exception/traceback (which is often
unacceptable in production code such as services)? The lack of any
clear feel for the cost of adding contracts is part of what makes
people reluctant to use them (particularly in the face of the
unfortunately still common assertion that "Python is slow" :-()

[Marko Ristin-Kaufmann]

Hi,

(Posting from work, so sorry for the short response.)

@Paul Moore icontract.pre/post/inv have the enabled argument; if not enabled, the contract is ignored.

Similarly with rmdir() -- "the directory must be empty" -- but how exactly am I supposed to check that?

Isn't that the whole point? The prose statement "the directory must be empty" is clear. But the exact code check isn't - and may be best handled by a series of unit tests, rather than a precondition.

I meant "check" as a user, not as a developer. As in "What did the implementer think -- how am I supposed to check that the directory is empty?" A la: "Dear user, if you want to rmdir, here is what you need to check that it is indeed a dir, and here is what you need to check that it is empty. If both checks pass, run me." 

@David patching __doc__ automatically is on the short-term todo list. I suppose we'll just add sphinx directives (:requires:, :ensures: etc.)

* Marko isn't that familiar with the codebase, so there may be better
ways to express certain things

This is true :)

* Sometimes it's just plain hard to express a verbal constraint in code

 

In these cases you simply don't express it in code. Why would you? If it's complex code, possibility that you have an error is probably equal or higher than that your comment rots.

@pre(lambda args, result: not any(Path(arg).is_absolute() for arg in args) or
(result == [pth for arg in args for pth in [Path(arg)] if
pth.is_absolute()][-1]),
"When several absolute paths are given, the last is taken as an anchor
(mimicking os.path.join()’s behaviour)")

I'm really not familiar with the code base nor with how to write stuff nice and succinct in python. This particular contract was hard to define because there were no last() and no arg_is_absolute() functions.

Otherwise, it would have read:

@pre(lambda args, result: not any(arg_is_absolute(arg) for arg in args) or result == Path(last(arg for arg in args if arg_is_absolute(arg)))

When rendered, this wouldn't look too bad to read.

@Chris

It is still fundamentally difficult to make assertions about the file
system as pre/post contracts. Are you becoming aware of this?
Contracts, as has been stated multiple times, look great for
mathematically pure functions that have no state outside of their own
parameters and return values (and 'self', where applicable), but are
just poor versions of unit tests when there's anything external to
consider.

I never thought of these particular contracts as running in the production. I would set them to run only in testing and only on part of the tests where they are safe from race conditions (e.g., setting enabled=test_pathlib.SERIAL; toggling mechanism is something I haven't spent too much thought about either and would also need to be discussed/implemented.).

I really thought about them as documentation, not for correctness (or at best deeper tests during the testing in a "safe" local environment, for example when you want to check if all the contracts also hold on situations in my testsuite, not only during the test suite of pathlib).

In the end, I'm calling it the day. I really got tired in the last few days. Standardizing contracts for python is not worth the pain. We'll continue to develop icontract for our internal needs and keep it open source, so anybody who's interested can have a look. Thank you all for a very lively discussions!

Cheers,

Marko

[Brendan Barnwell]

On 2018-09-28 05:23, David Mertz wrote:
> On Thu, Sep 27, 2018, 9:25 PM Marko Ristin-Kaufmann
> <marko....@gmail.com <mailto:marko....@gmail.com>> wrote:
>
> Try to question whether the contracts I wrote are so obvious to
> everybody even if they are obvious to you and keep in mind that the
> user does not look into the implementation.
>
>
> I had missed this comment, but this seems to be the biggest disconnect,
> or talking past each other.
>
> I'm a user of many libraries. I USUALLY look at the implementation when
> in doubt about a function. If contracts are meant only for users who
> don't look at code, the detrimental effect on code readability is mitigated.

I agree with you that this seems to be a major disconnect in the
discussion here. However, on the issue itself, I quite agree with Marko
that it is *much* more important for the documentation to be readable
than for the function to be readable. I too read the source of
functions sometimes, and whenever I find myself having to do so, I
grouse at the authors of whatever libraries I'm using for not making the
documentation more clear. Ideally, no user should *ever* have to read
the function source code, because the documentation should make it
completely clear how to use the function without knowing *anything*
about how it is implemented.

Of course, this ideal will never be achieved, but I think it's
something to aim towards, and the idea that adding a few lines of DbC
decorators makes the source code too cluttered seems quite incorrect to
me. I glanced through the source code and didn't find it hard to read
at all. The contracts are all cleanly separated from the function
bodies because they're in decorators up front. I'm frankly quite
baffled that other people on this thread find that hard to read.

The problem with reading the source code is that you can't tell what
parts of the behavior are specified and which are implementation
details. The appeal of something like DbC is that it encourages (some
might say painfully forces) programmers to be very explicit about what
behavior they want to guarantee.

Whether writing these guarantees as Python expressions is better than
writing them as prose is another matter. Personally I do see some value
in the modifications that Marko made to pathlib. In a sense, writing
"documentation as Python code" is like annotating the source code to
mark which specific parts of the implementation are guarantees and which
may be implementation details. I think there is significant value in
knowing precisely what an API allows, in an explicit and verifiable form
such as that provided by DbC, rather than using human language, which is
much less precise, can leave room for misinterpretation, and, perhaps
most importantly, is harder to verify automatically.

Ultimately, I'm firmly of the opinion that, in publicly distributed
code, the function *IS* its documentation, not its source code. When a
function's actual behavior conflicts with its documented behavior, that
is a bug. When meaningful aspects of a functions behavior are not
specified in the documentation, that is also a bug. These may be bugs in
the documentation or in the behavior, but either way the point is that
reading the source code is not an acceptable substitute for making the
documentation a complete and self-sufficient vehicle for total
understanding of the function's behavior. It doesn't matter if the
function behaves as the author intended; it only matters if it behaves
as documented.

--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown

[Greg Ewing]

Chris Angelico wrote:
> It is still fundamentally difficult to make assertions about the file
> system as pre/post contracts.

When you consider race conditions, I'd say it's impossible.
A postcondition check involving the file system could fail
even if the function does its job perfectly.

--
Greg

[James Lu]

Many editors highlight decorators in a different color that makes it easier to ignore and can also fold decorators.

Contracts can also sometimes actively improve the flow of code.

I personally find a formal contract easier to read than informal documentation.

It also reduces the times where you need to spend time figuring out if documentation actually accurate and up to date

[Chris Angelico]

On Sat, Sep 29, 2018 at 7:19 AM Greg Ewing <greg....@canterbury.ac.nz> wrote:
>
> Chris Angelico wrote:
> > It is still fundamentally difficult to make assertions about the file
> > system as pre/post contracts.
>
> When you consider race conditions, I'd say it's impossible.
> A postcondition check involving the file system could fail
> even if the function does its job perfectly.

I guess that depends on the meaning of "contract". If it means "there
is a guarantee that this is true after this function returns", then
yes, the race condition means it's impossible. But as a part of the
function's declared intent, it's fine to have a postcondition of "the
directory will be empty" even though something could drop a file into
it.

But if it's the latter... contracts are just a different way of
defining unit tests.... and we're right back where we started.

ChrisA

[Greg Ewing]

Chris Angelico wrote:
> But as a part of the
> function's declared intent, it's fine to have a postcondition of "the
> directory will be empty" even though something could drop a file into
> it.

If you only intend the contract to serve as documentation,
I suppose that's okay, but it means you can't turn on runtime
checking of contracts, otherwise your program could suffer
spurious failures.

--
Greg

[Dan Sommers]

On 9/28/18 8:12 PM, Chris Angelico wrote:
> On Sat, Sep 29, 2018 at 7:19 AM Greg Ewing <greg....@canterbury.ac.nz> wrote:
>>
>> Chris Angelico wrote:
>>> It is still fundamentally difficult to make assertions about the file
>>> system as pre/post contracts.
>>
>> When you consider race conditions, I'd say it's impossible.
>> A postcondition check involving the file system could fail
>> even if the function does its job perfectly.
>
> I guess that depends on the meaning of "contract". If it means "there
> is a guarantee that this is true after this function returns", then
> yes, the race condition means it's impossible. But as a part of the
> function's declared intent, it's fine to have a postcondition of "the
> directory will be empty" even though something could drop a file into
> it.
>
> But if it's the latter... contracts are just a different way of
> defining unit tests.... and we're right back where we started.

At the risk of pedantry (on a Python list? I'm *shocked*):

I call BS on any contract that requires on entry or guarantees on exit
the state of the file system. At best, a function can guarantee that it
will make (or made) a request to the OS, and that the OS returned
"success" before the function continued.

Then again, a function that guarantees to work in a particular way based
on some condition of the file system would be okay. For example, a
function might claim to create a temporary file with some content *if*
some directory exists when the function tries to create the temporary
file. But as I think both of you are claiming, the best that function
can guarantee on exit is that it asked the OS to write to the file, and
that the OS agreed to do so. The function cannot guarantee that the
file or the content still exists when the function finally returns.

[Steven D'Aprano]

On Sun, Sep 23, 2018 at 07:09:37AM +0200, Marko Ristin-Kaufmann wrote:

> After the discussion we had on the list and after browsing the internet a
> bit, I'm still puzzled why design-by-contract was not more widely adopted
> and why so few languages support it.
[...]

> *. *After properly reading about design-by-contract and getting deeper into
> the topic, there is no rational argument against it and the benefits are
> obvious. And still, people just wave their hand and continue without
> formalizing the contracts in the code and keep on writing them in the
> descriptions.
>
> * Why is that so?
[...]

You are asking a question about human psychology but expecting logical,
technical answers. I think that's doomed to failure.

There is no nice way to say this, because it isn't nice.

Programmers and language designers don't use DbC because it is new and
different and therefore scary and wrong. Programmers, as a class, are
lazy (they even call laziness a virtue), deeply conservative,
superstitious, and don't like change. They do what they do because
they're used to it, not because it is the technically correct thing to
do, unless it is by accident.

(Before people's hackles raise too much, I'm speaking in generalities,
not about you personally. Of course you are one of the 1% awesomely
logical, technically correct programmers who know what you are doing and
do it for the right reasons after careful analysis. I'm talking about
the other 99%, you know the ones. You probably work with them. You've
certainly read their answers on Stackoverflow or The Daily WTF and a
million blogs.)

They won't use it until there is a critical mass of people using it, and
then it will suddenly flip from "that weird shit Eiffel does" to "oh
yeah, this is a standard technique that everyone uses, although we don't
make a big deal about it".

Every innovation in programming goes through this. Whether the
innovation goes mainstream or not depends on getting a critical mass,
and that is all about psychology and network effects and nothing to do
with the merit of the idea.

Remember the wars over structured programming? Probably not. In 2018,
the idea that people would *seriously argue against writing subroutines*
seems insane, but they did. As late as 1999, a former acquaintance of
mine was working on a BASIC project for a manager who insisted they use
GOTO and GOSUB in preference to subroutines.

Testing: the idea that we should have *repeatable automated tests* took
a long time to be accepted, and is still resisted by both developers and
their managers. What's wrong with just sitting a person down in front of
the program and checking for bugs by hand? We still sometimes have to
fight for an automated test suite, never mind something like test driven
development.

ML-based languages have had type inference for decades, and yet people
still think of type checking in terms of C and Java declarations. Or
they still think in terms of static VERSUS dynamic typing, instead of
static PLUS dynamic typing.

I could go on, but I think I've made my point.

I can give you some technical reasons why I don't use contracts in my
own Python code, even though I want to:

(1) Many of my programs are too small to bother. If I'm writing a quick
script, I don't even write tests. Sometimes "be lazy" is the right
answer, when the cost of bugs is small enough and the effort to avoid
them is greater. And that's fine. Nobody says that contracts must be
mandatory.

(2) Python doesn't make it easy to write contracts. None of the
solutions I've seen are nice. Ironically, the least worst I've seen is a
quick and dirty metaclass solution given by Guido in an essay way back
in Python 1.5 days:

https://www.python.org/doc/essays/metaclasses/

His solution relies only on a naming convention, no decorators, no
lambdas:

class C(Eiffel):
def method(self, arg):
return whatever
def method_pre(self, arg):
assert arg > 0
def method_post(self, Result, arg):
assert Result > arg


Still not pretty, but at least we get some block structure instead of a
wall of decorators.

Syntax matters. Without good syntax that makes it easy to write
contracts, it will never be anything but niche.


(3) In a sense, *of course I write contracts*. Or at least precondition
checks. I just don't call them that, and I embed them in the
implementation of the method, and have no way to turn them off. Nearly
all of us have done the same, we start with a method like this:

class C:
def method(self, alist, astring, afloat):
# do some work...

using nothing but naming conventions and an informal (and often vague)
specification in the docstring, and while that is sometimes enough in
small projects, the third time we get bitten we start adding defensive
checks so we get sensible exceptions:

def method(self, alist, astring, afloat):
if not isinstance(alist, list):
raise TypeError('expected a list')
if alist == []:
raise ValueError('list must not be empty')
# and so on...


These are pre-conditions! We just don't call them that. And we can't
disable them. They're not easy to extract from the source code and turn
into specifications. And so much boilerplate!

Let's invent syntax to make it more obvious what is going on:

def method(self, alist, astring, afloat):
requires:
isinstance(alist, list)
alist != []
isinstance(astring, str)
number_of_vowels(astring) > 0
isinstance(afloat, float)
not math.isnan(afloat)
implementation:
# code goes here


Its easy to distinguish the precondition checks from the implementation,
easy to ignore the checks if you don't care about them, and easy for an
external tool to analyse.

What's not to like about contracts? You're already using them, just in
an ad hoc, ugly, informal way.

And I think that is probably the crux of the matter. Most people are
lazy, and don't like having to do things in a systematic manner. For
years, programmers resisted writing functions, because unstructured code
was easier. We still resist writing documentation, because "its obvious
from the source code" is easier. We resist writing even loose
specifications, because NOT writing them is easier. We resist writing
tests unless the project demands it. We resist running a linter or
static checker over our code ("it runs, that means there are no
errors").

Until peer pressure and pain makes us do so, then we love them and could
not imagine going back to the bad old days before static analysis and
linters and unit tests.



--
Steve

[Steven D'Aprano]

On Sat, Sep 29, 2018 at 12:30:45PM +1200, Greg Ewing wrote:
> Chris Angelico wrote:
> >But as a part of the
> >function's declared intent, it's fine to have a postcondition of "the
> >directory will be empty" even though something could drop a file into
> >it.
>
> If you only intend the contract to serve as documentation,
> I suppose that's okay, but it means you can't turn on runtime
> checking of contracts, otherwise your program could suffer
> spurious failures.

If your code can cope with a particular file system state ("directory
isn't empty") then you don't need to specify it as a precondition. If it
can't cope, then it isn't a spurious failure, its a real failure. You
either get an error when the condition is checked, up front, or you get
an error in the middle of processing the directory.

Of course for many things (especially file system operations) you need
to be prepared to handle I/O errors *even if the precondition passed*.
So what? That hasn't changed, and nobody said that contracts were a cure
for Time Of Check To Time Of Use bugs.

Contracts are a tool to help developers write better code, not a magic
wand. You still need to write your code in a way which isn't vulnerable
to TOCTTOU failures, contracts or not.

Anticipating an objection: why bother with the precondition check if the
code has to handle an error anyway?

Because it is better to get an upfront error than an error halfway
through processing. In general you get a better, more informative error
message, closer to the place where it matters, if you fail fast.

Yes yes yes, in theory you might have a precondition requirement which
would fail up front but resolve itself before it matters:

directory not empty
start running your program
precondition check fails
... later
directory is emptied by another process
your program actually needs to use the empty directory

but do you really think it is good practice to design your code on the
basis of that? Better to write your code conservatively: if the
directory isn't empty up front, don't hope it will empty itself, fail
fast, but be prepared to handle the error case as well.


--
Steve

[Steven D'Aprano]

On Tue, Sep 25, 2018 at 08:01:28PM +1200, Robert Collins wrote:
> On Mon, 24 Sep 2018 at 19:47, Marko Ristin-Kaufmann
> <marko....@gmail.com> wrote:
[...]
> > There are 150K projects on pypi.org. Each one of them would
> > benefit if annotated with the contracts.
>
> You'll lose folks attention very quickly when you try to tell folk
> what they do and don't understand.
>
> Claiming that DbC annotations will improve the documentation of every
> single library on PyPI is an extraordinary claim, and such claims
> require extraordinary proof.

This is not a scientific paper, or an edited book. Its an email forum,
and communication is not always as precise as it could be. We should
read such statements charitably, not literally and pedantically.

But having said that... would it be an "extraordinary claim" to state
that all 150K projects on PyPI would benefit with better documentation,
tests or error checking? I don't think so.

I think it would be extraordinary to claim that there was even a single
project that *wouldn't* benefit from at least one such patch. I'd like
to see this glorious example of software perfection, and bow down in awe
to its author.

Contracts combine error checking, documentation and testing all in one.
Ergo, if a project would benefit from any of those things, it would
benefit from a contract.

Anticipating an objection: like tests and documentation and error
checking, contracts do not need to be "all or nothing". A contract can
be incomplete and still provide benefit. We can add contracts
incrementally. Even a single pre- or post-condition check is better than
no check at all.

So I'm with Marko here: every project on PyPI would, in principle,
benefit from some contracts. I say in principle only because in
practice, the available APIs for adding contracts to Python code are so
clunky that the cost of contracts are excessive. We're paying a syntax
tax on contracts which blows the cost all out of proportion.

Just as languages like Java pay a syntax tax on static typing (compared
to languages like ML and Haskell with type inference and few
declarations needed), and languages like COBOL have a syntax tax on,
well, everything.


> I can think of many libraries where necessary pre and post conditions
> (such as 'self is still locked') are going to be noisy, and at risk of
> reducing comprehension if the DbC checks are used to enhance/extended
> documentation.

So long as my editor lets me collapse the contract blocks, I don't need
to read them unless I want to. And automatically generated documentation
always tends towards the verbose. Just look at the output of help() in
the Python REPL. We learn to skim, and dig deeper only when needed.


> Some of the examples you've been giving would be better expressed with
> a more capable type system in my view (e.g. Rust's), but I have no
> good idea about adding that into Python :/.

Indeed.

A lot of things which could be preconditions in Python would be
type-checks in Eiffel. With gradual typing and type annotations, we
could move some pre-condition and post-condition checks into the type
checker.

(Static typing is another thing which doesn't need to be "all or
nothing".)


> Anyhow, the thing I value most about python is its pithyness: its
> extremely compact, allowing great developer efficiency,

None of that changes. The beauty of contracts is that you can have as
many or as few as make sense for each specific class or function, or for
that matter each project. If your project is sufficiently lightweight
and the cost of bugs is small enough, you might not bother with tests or
error checking, or contracts.

[Steven D'Aprano]

On Wed, Sep 26, 2018 at 05:40:45AM +1000, Chris Angelico wrote:

> > There are 150K projects on pypi.org. Each one of them would benefit
> > if annotated with the contracts.
>

> This is the extraordinary claim. To justify it, you have to show that
> virtually ANY project would benefit from contracts. So far, I haven't
> seen any such proof.

As per my previous email, I think the extraordinary claim is that there
exists even a single project which wouldn't benefit from at least one
contract.

Honestly, you sound almost like somebody saying

"Projects would benefit from getting an automated test
suite? Ridiculous!"

But to give you a charitable interpretation, I'll grant that given the
cost to benefit ratio of code churn, human effort, refactoring etc, it
is certainly possible that adding contracts to some especially mature
and high-quality projects, or quick-and-dirty low-quality projects where
nobody cares about bugs, would cost more than the benefit gained.
There's benefit, but not *nett* benefit.

That goes especially for Python code since the available interfaces for
contracts are so poor. But that's why we're talking about this on
Python-Ideas.

I just wish we didn't have to fight so hard to justify the very idea of
contracts themselves. That's like having to justify the idea of test
suites, documentation and error checking.

[Steven D'Aprano]

On Wed, Sep 26, 2018 at 04:03:16PM +0100, Rhodri James wrote:

> Let's assume that the contracts are meaningful and useful (which I'm
> pretty sure won't be 100% true; some people are bound to assume that
> writing contracts means they don't have to think).

Contracts are a tool. We shouldn't refuse effective tools because some
developers are too DailyWTF-worthy to use them. Why should the rest of
us miss out because of their incompetence? Contracts are not rocket-
science: there's nothing in them that most of us aren't already doing in
an ad-hoc, clumsy manner by embedding the contracts in the docstring,
inside the body of our methods, in external tests etc.


> Assuming that you
> aren't doing some kind of wide-ranging static analysis (which doesn't
> seem to be what we're talking about), all that the contracts have bought
> you is the assurance that *this* invocation of the function with *these*
> parameters giving *this* result is what you expected. It does not say
> anything about the reliability of the function in general.

This is virtually the complete opposite of what contracts give us. What
you are describing is the problem with *unit testing*, not contracts.

Unit tests only tell us that our function works with the specific input
the test uses. In contrast, contracts test the function with *every*
input the function is invoked with.

(Up to the point that you disable checking, of course. Which is under
your control: you decide when you are satisfied that the software is
sufficiently bug-free to risk turning off checking.)

Both are a form of testing, of course. As they say, tests can only
reveal the presence of bugs, they can't prove the absence of bugs. But
correctness checkers are out of scope for this discussion.


> It seems to me that a lot of the DbC philosophy seems to assume that
> functions are complex black-boxes whose behaviours are difficult to
> grasp.

I can't imagine how you draw that conclusion. That's like saying that
unit tests and documentation requires the assumption that functions are
complex and difficult to grasp.

This introduction to DbC shows that contracts work with simple methods:

https://www.eiffel.com/values/design-by-contract/introduction/

and here's an example:


put (x: ELEMENT; key: STRING) is
-- Insert x so that it will be retrievable through key.
require
count <= capacity
not key.empty
do
... Some insertion algorithm ...
ensure
has (x)
item (key) = x
count = old count + 1
end

Two pre-conditions, and three post-conditions. That's hardly complex.

[Aside: and there's actually a bug in this. What if the key already
exists? But this is from a tutorial, not production code. Cut them some
slack.]

If I were writing this in Python, I'd write something like this:

def put(self, x, key):
"""Insert x so that it will be retrievable through key."""
# Input checks are pre-conditions!
if self.count > capacity:
raise DatabaseFullError
if not key:
raise ValueError
# .. Some insertion algorithm ...


and then stick the post-conditions in a unit test, usually in a
completely different file:

class InsertTests(TestCase):
def test_return_result(self):
db = SomeDatabase()
db.put("value", "key")
self.AssertTrue("value" in db.values())
self.AssertEqual(db["key"], "value")
self.AssertEqual(db.count, 1)


Notice that my unit test is not actually checking at least one of the
post-conditions, but a weaker, more specific version of it. The
post-condition is that the count goes up by one on each insertion. My
test only checks that the count is 1 after inserting into an empty
database.


So what's wrong with the status quo?

- The pre-condition checks are embedded right there in the
method implementation, mixing up the core algorithm with the
associated error checking.

- Which in turn makes it hard to distinguish the checks from
the implementation, and impossible to do so automatically.

- Half of the checks are very far away, in a separate file, assuming
I even remembered or bothered to write the test.

- The post-conditions aren't checked unless I run my test suite, and
then they only check the canned input in the test suite.

- The pre-conditions can't be easily disabled in production.

- No class invariants.

- Inheritance is not handled correctly.


The status quo is all so very ad-hoc and messy. Design By Contract
syntax would allow (not force, allow!) us to add some structure to the
code:

- requirements of the function
- the implementation of the function
- the promise made by the function

Most of us already think about these as three separate things, and
document them as such. Our code should reflect the structure of how we
think about the code.


> In my experience this is very rarely true. Most functions I
> write are fairly short and easily grokked, even if they do complicated
> things. That's part of the skill of breaking a problem down, IMHO; if
> the function is long and horrible-looking, I've already got it wrong and
> no amount of protective scaffolding like DbC is going to help.

That's like saying that if a function is horrible-looking, then there's
no point in writing tests for it.

I'm not saying that contracts are only for horrible functions, but
horrible functions are the ones which probably benefit the most from
specifying exactly what they promise to do, and checking on every
invocation that they live up to that promise.


> >It's the reason why type checking exists,
>
> Except Python doesn't type check so much as try operations and see if
> they work.

Python (the interpreter) does type checking. Any time you get a
TypeError, that's a failed type check. And with type annotations, we can
run a static type checker on our code too, which will catch many of
these failures before we run the code.

Python code sometimes does type checking too, usually with isinstance.
That's following the principle of Fail Fast, rather than waiting for
some arbitrary exception deep inside the body of your function, you
should fail early on bad input.

[Stephen J. Turnbull]

Marko Ristin-Kaufmann writes:

> I annotated pathlib with contracts:
> https://github.com/mristin/icontract-pathlib-poc. I zipped the HTML docs

Thank your for completing this task so quickly.

I'm sorry, but I didn't find it convincing. I'll leave others to
discuss the docs, as they are clearly "proof of concept" and I expect
will be improved greatly.

Part of the problem is the style of contracts, which can probably be
improved with syntactic support. For example, the many double
negatives of the form "not (not (X)) or Y". I guess the idea is to
express all implications "X implies Y" in the equivalent form "not X
or Y". I tried a few in the form "Y if X else True" but didn't find
that persuasive. I also found the lambdas annoying.

I conclude that there's good reason to prefer the style where the
condition is expressed as a str, and eval'ed by the contract machinery.
This would get rid of the lambdas, and allow writing a contract parser
so you could write implications in a more natural form. I haven't
tried rewriting that way, and of course, I don't have a parser to
actually run stuff.

> - not list(self.iterdir()) (??? There must be a way to check
> this more optimally)

You might define a function like this:

def is_empty_dir(d):
for _ in os.scandir(d):
return False
return True

or the function could catch the StopIteration from next() directly
(but I think that's considered bad style). I don't think there's any
way to do this without a function call. Note that the implementation
of iterdir calls os.listdir, so this function isn't necessarily much
more efficient than "not list(self.iterdir())". (I'm pretty sure that
in this case the iterator keeps a copy of the list internally, and
listification probably involves object creation overhead, a memory
allocation and a machine-language block copy.) I'm not sure whether
scandir is any better. And I don't know how big a directory needs to
be before the overhead of function call and exception handling become
great enough to make it worthwhile.

> [You] want to convey the message: dear user, if you are iterating

> through a list of paths, use this function to decide if you should
> call rmdir() or unlink(). Analogously with the first contract: dear
> user, please check if the directory is empty before calling rmdir()
> and this is what you need to call to actually check that.

I don't understand this logic. I don't think I'd be looking at random
contracts for individual to learn how to handle filesystem objects.
I'd be more likely to do this kind of thing in most of my code:

def rmtree(path: Path) -> None:
try:
path.unlink()
except PermissionError:
for p in path.iterdir():
rmtree(p)
path.rmdir()

I wrote all of that without being a Path user and without checking
docs. (I cheated on PermissionError by testing in the interpreter,
I'd probably just use Exception if I didn't have an interpreter to
hand already.)

Note that this function is incorrect: PermissionError could occur
because I don't have write permission on the parent directory. I also
don't learn anything about PermissionError from your code, so your
contract is incomplete. *DbC is just not a guarantee of anything* --
if there's a guarantee, it derives from the quality of the development
organization that uses DbC.

> I also finally assembled the puzzle. Most of you folks are all
> older and were exposed to DbC in the 80ies championed by DbC

> zealots who advertised it as *the *tool for software
> development. You were repulsed [...].

Please don't make statements that require mind-reading. You do not
know that (I for example am 60, and have *heard of* design-by-contract
before but this is the first time I've seen it *seriously proposed for
use* or implementation in any project I participate in).

More important, several posters have explained why they don't see a
need for DbC in Python. Two common themes are proliferation of
conditions making both code and documentation hard to read, and
already using methods of similar power such as TDD or other systems
using unit tests. It's bad form to discount such explicit statements.

> And that's why I said that the libraries on pypi meant to be used by
> multiple people and which already have type annotations would obviously
> benefit from contracts -- while you were imagining that all of these
> libraries need to be DbC'ed 100%, I was imagining something much more
> humble. Thus the misunderstanding.

No, the misunderstanding is caused by your wording and by your lack of
attention to the costs of writing contracts. Criticizing yourself is
a much more effective strategy: you can control your words, but not
how others understand them. Your best bet is to choose words that are
hard to misunderstand.

> Similarly with rmdir() -- "the directory must be empty" -- but how
> exactly am I supposed to check that?

There's no reason to suppose the contract is a good place to look for
that. "not list(path.iterdir())" is the obvious and easy-to-read test
(but in your style, a nonempty condition would be "not not
list(path.iterdir()", which is not not not the way to do it in code
;-). But this may not be efficient for "large enough" directories.
However, if contracts are expected to be disabled in production code,
there's strong reason to write readable rather than efficient code for
contracts.

Please trim unneeded text. Fixed it for you this time.

[Stephen J. Turnbull]

Steven D'Aprano writes:

> I just wish we didn't have to fight so hard to justify the very
> idea of contracts themselves.

You don't. You need to justify putting them in the stdlib. That's a
pretty high bar. And as crappy as the contracts in Marko's pathlib
rewrite look to me, I suspect you'll need syntactic support (eg, for
"implies", see my post replying to Marko). If people want checking in
production where reasonable (seems to be one of the nice things about
contracts), you'll probably want a language change to support that
syntax efficiently, which is usually a prohibitively high bar.

[Stephen J. Turnbull]

Steven D'Aprano writes:

> put (x: ELEMENT; key: STRING) is
> -- Insert x so that it will be retrievable through key.
> require
> count <= capacity
> not key.empty
> do
> ... Some insertion algorithm ...
> ensure
> has (x)
> item (key) = x
> count = old count + 1
> end
>
> Two pre-conditions, and three post-conditions. That's hardly
> complex.

You can already do this:

def put(self, x: Element, key: str) -> None:

"""Insert x so that it will be retrievable through key."""

# CHECKING PRECONDITIONS
_old_count = self.count
assert self.count <= self.capacity,
assert key

# IMPLEMENTATION
... some assertion algorithm ...

# CHECKING POSTCONDITIONS
assert x in self
assert self[key] == x
assert self.count == _old_count

return

I don't see a big advantage to having syntax, unless the syntax allows
you to do things like turn off "expensive" contracts only. Granted,
you save a little bit of typing and eye movement (you can omit
"assert" and have syntax instead of an assignment for checking
postconditions dependent on initial state).

A document generator can look for the special comments (as with
encoding cookies), and suck in all the asserts following until a
non-assert line of code (or the next special comment). The
assignments will need special handling, an additional special comment
or something. With PEP 572, I think you could even do this:

assert ((_old_count := self.count),)

to get the benefit of python -O here.

> If I were writing this in Python, I'd write something like this:
>
> def put(self, x, key):
> """Insert x so that it will be retrievable through key."""
> # Input checks are pre-conditions!
> if self.count > capacity:
> raise DatabaseFullError
> if not key:
> raise ValueError
> # .. Some insertion algorithm ...

But this is quite different, as I understand it. Nothing I've seen in
the discussion so far suggests that a contract violation allows
raising differentiated exceptions, and it seems very unlikely from the
syntax in your example above. I could easily see both of these errors
being retryable:

for _ in range(3):
try:
db.put(x, key)
except DatabaseFullError:
db.resize(expansion_factor=1.5)
db.put(x, key)
except ValueError:
db.put(x, alternative_key)

> and then stick the post-conditions in a unit test, usually in a
> completely different file:

If you like the contract-writing style, why would you do either of
these instead of something like the code I wrote above?

> So what's wrong with the status quo?
>
> - The pre-condition checks are embedded right there in the
> method implementation, mixing up the core algorithm with the
> associated error checking.

You don't need syntax to separate them, you can use a convention, as I
did above.

> - Which in turn makes it hard to distinguish the checks from
> the implementation, and impossible to do so automatically.

sed can do it, why can't we?

> - Half of the checks are very far away, in a separate file,
> assuming I even remembered or bothered to write the test.

That was your choice. There's nothing about the assert statement that
says you're not allowed to use it at the end of a definition.

> - The post-conditions aren't checked unless I run my test suite, and
> then they only check the canned input in the test suite.

Ditto.

> - The pre-conditions can't be easily disabled in production.

What's so hard about python -O?

> - No class invariants.

Examples?

> - Inheritance is not handled correctly.

Examples? Mixins and classes with additional functionality should
work fine AFAICS. I guess you'd have to write the contracts in each
subclass of an abstract class, which is definitely a minus for some of
the contracts. But I don't see offhand why you would expect that the
full contract of a method of a parent class would typically make sense
without change for an overriding implementation, and might not make
sense for a class with restricted functionality.

> The status quo is all so very ad-hoc and messy. Design By Contract
> syntax would allow (not force, allow!) us to add some structure to the
> code:
>
> - requirements of the function
> - the implementation of the function
> - the promise made by the function

Possible already as far as I can see. OK, you could have the compiler
enforce the structure to some extent, but the real problem IMO is
going to be like documentation and testing: programmers just won't do
it regardless of syntax to make it nice and compiler checkable.

> Most of us already think about these as three separate things, and
> document them as such. Our code should reflect the structure of how we
> think about the code.

But what's the need for syntax? How about the common (in this thread)
complaint that even as decorators, the contract is annoying, verbose,
and distracts the reader from understanding the code? Note: I think
that, as with static typing, this could be mitigated by allowing
contracts to be optionally specified in a stub file. As somebody
pointed out, it shouldn't be hard to write contract strippers and
contract folding in many editors. (As always, we have to admit it's
very difficult to get people to change their editor!)

> > In my experience this is very rarely true. Most functions I
> > write are fairly short and easily grokked, even if they do complicated
> > things. That's part of the skill of breaking a problem down, IMHO; if
> > the function is long and horrible-looking, I've already got it wrong and
> > no amount of protective scaffolding like DbC is going to help.
>
> That's like saying that if a function is horrible-looking, then there's
> no point in writing tests for it.
>
> I'm not saying that contracts are only for horrible functions, but
> horrible functions are the ones which probably benefit the most from
> specifying exactly what they promise to do, and checking on every
> invocation that they live up to that promise.

I think you're missing the point then: ISTM that the implicit claim
here is that the time spent writing contracts for a horrible function
would be better spent refactoring it. As you mention in connection
with the Eiffel example, it's not easy to get all the relevant
contracts, and for a horrible function it's going to be hard to get
some of the ones you do write correct.

> Python (the interpreter) does type checking. Any time you get a
> TypeError, that's a failed type check. And with type annotations, we can
> run a static type checker on our code too, which will catch many of
> these failures before we run the code.

But an important strength of contracts is that they are *always* run,
on any input you actually give the function.

[Marko Ristin-Kaufmann]

Hi,

I compiled a couple of issues on github to provide a more structured ground for discussions on icontract features: https://github.com/Parquery/icontract/issues (@David Maertz: I also included the issue with automatically generated __doc__ in case you are still interested in it).

Cheers,

Marko

[Steven D'Aprano]

On Fri, Sep 28, 2018 at 01:49:01PM +0100, Paul Moore wrote:

> There's clearly a number of trade-offs going on here:
>
> * Conditions should be short, to avoid clutter
> * Writing helper functions that are *only* used in conditions is more
> code to test or get wrong

This is no different from any other refactoring. If you have a function
that checks its input:

def spam(arg):
if condition(arg) and other_condition(arg) or alt_condition(arg):
raise ValueError

and refactor it to a helper:

def invalid(arg):
return condition(arg) and other_condition(arg) or alt_condition(arg)

def spam(arg):
if invalid(arg):
raise ValueError

how is that a bad thing just because we call it a "precondition" instead
of calling it "error checking"?

Of course we don't necessarily want the proliferation of masses and
masses of tiny helper functions, but nor should we fear them. Helpers
should carry their own weight, and if they do, we should use them.
Whether they are used for contracts or not makes zero difference.

> * Sometimes it's just plain hard to express a verbal constraint in code

Indeed.

People seem to be arguing against some strawman of "Design By Contract
is a magic bullet that solves every imaginable problem". Of course it
doesn't. Some constraints are too hard to specify as code. Okay, then
don't do that.

DbC isn't "all or nothing". If you can't write a contract for something,
don't. You still get value from the contracts you do write.


[...]

> But given that *all* the examples I've seen of contracts have this
> issue (difficult to read expressions) I suspect the problem is
> inherent.

Are you specifically talking about *Python* examples? Or contracts in
general?

I don't know Eiffel very well, but I find this easy to read and
understand (almost as easy as Python). The trickiest thing is the
implicit "self".

put (x: ELEMENT; key: STRING) is
-- Insert x so that it will be retrievable through key.
require
count <= capacity
not key.empty
do
... Some insertion algorithm ...
ensure
has (x)
item (key) = x
count = old count + 1
end

https://www.eiffel.com/values/design-by-contract/introduction/


Here are a couple of examples from Cobra:

def fraction( numer as int, denom as int) as float
require
numer > 0
denom <> 0
body
...

def bumpState( incr as int) as int
require
incr > 0
ensure
result >= incr
.state = old.state + incr
body
.state += incr
return .state


http://cobra-language.com/trac/cobra/wiki/Contracts


If you find them difficult to read, I don't know what to say :-)

> Another thing that I haven't yet seen clearly explained. How do these
> contracts get *run*? Are they checked on every call to the function,

Yes, that's the point of them. In development they're always on. Every
time you run your dev code, it tests itself.


> even in production code?

That's up to you, but typical practice is to check pre-conditions (your
input) but not post-conditions (your output) in production.

> Is there a means to turn them off? What's the
> runtime overhead of a "turned off" contract (even if it's just an
> if-test per condition, that can still add up)?

Other languages may offer different options, but in Eiffel, contracts
checking can be set to:

no: assertions have no run-time effect.
require: monitor preconditions only, on routine entry.
ensure: preconditions on entry, postconditions on exit.
invariant: same as ensure, plus class invariant on both entry
and exit for qualified calls.
all: same as invariant, plus check instructions,
loop invariants and loop variants.

You can set the checking level globally, or class-by-class. The default
is to check only preconditions. That is, for methods to validate their
inputs. Quoting from the Eiffel docs:

When releasing the final version of a system, it is usually
appropriate to turn off assertion monitoring, or bring it down
to the ``require`` level. The exact policy depends on the
circumstances; it is a trade off between efficiency considerations,
the potential cost of mistakes, and how much the developers and
quality assurance team trust the product. When developing the
software, however, you should always assume -- to avoid loosening
your guard -- that in the end monitoring will be turned off.

https://www.eiffel.org/doc/eiffel/ET-_Design_by_Contract_%28tm%29%2C_Assertions_and_Exceptions


The intention for Python would be similar:

- we ought to be able disable contract checking globally;

- and preferrably on a case-by-case basis;

- a disabled contact ought to be like a disabled assertion,
that is, completely gone with no runtime effect at all;

- but due to the nature of Python's execution model, there will
probably be some (small) overhead at the time the function is
created, but not when the function is called.

Of course the overhead will depend on the implementation.

> And what happens if a
> contract fails - is it an exception/traceback (which is often
> unacceptable in production code such as services)?

What happens when any piece of error checking code fails and raises an
exception?

In the case of Python, a failed contract would be an exception. What you
do with the exception is up to you. The *intention* is that a failed
contract is a bug, so what you *ought to do* is fix the bug. But you
could catch it, retry the operation, restart the service, or whatever.

That's worth repeating: *contracts aren't for testing end-user input*
but for checking internal program state. A failed contract ought to be
considered a bug. Any *expected error state* (such as a missing file, or
bad user input, or a network outage) shouldn't be treated as a contract.


--
Steve

[David Mertz]

On Sat, Sep 29, 2018 at 7:20 AM Steven D'Aprano <st...@pearwood.info> wrote:

On Wed, Sep 26, 2018 at 04:03:16PM +0100, Rhodri James wrote:
> Assuming that you
> aren't doing some kind of wide-ranging static analysis (which doesn't
> seem to be what we're talking about), all that the contracts have bought
> you is the assurance that *this* invocation of the function with *these*
> parameters giving *this* result is what you expected.  It does not say
> anything about the reliability of the function in general.

This is virtually the complete opposite of what contracts give us. What
you are describing is the problem with *unit testing*, not contracts.

I think Steven's is backwards in its own way.

• Contracts test the space of arguments actually used during testing period (or during initial production if the performance hit is acceptable).
  
• Unit tests test the space of arguments thought of by the developers.

A priori, either one of those can cover cases not addressed by the other.  If unit tests use the hypothesis library or similar approaches, unit tests might very well examine arguments unlikely to be encountered in real-world (or test phase) use... these are nonetheless edge cases that are important to assure correct behavior on ("correct" can mean various things, of course: exceptions, recovery, default values whatever).

In contrast, contracts might well find arguments that the developers of unit tests had not thought of.  I tend to think someone sitting down trying to think of edge cases is going to be able to write more thorough tests than the accident of "what did we see during this run" ... but it could go either way.

Of course... my own approach to this concern would more likely be to use a logging decorator rather than a DbC one.  Then collect those logs that show all the arguments that were passed to a given function during a testing period, and roll those back into the unit tests.  My approach is a bit more manual work, but also more flexible and more powerful.

- Half of the checks are very far away, in a separate file, assuming
  I even remembered or bothered to write the test.

To me, this is the GREATEST VIRTUE of unit tests over DbC.  It puts the tests far away where they don't distract from reading and understanding the function itself.  I rarely want my checks proximate since I wear a very different hat when thinking about checks than when writing functionality (ideally, a lot of the time, I wear the unit test hat *before* I write the implementation; TDD is usually good practice).

 

- The post-conditions aren't checked unless I run my test suite, and
  then they only check the canned input in the test suite.

Yes, this is a great advantage of unit tests.  No cost until you explicitly run them.

 

- No class invariants.
- Inheritance is not handled correctly.

These are true.  Also they are things I care very little about.

 

--

Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.

[Steven D'Aprano]

On Sun, Sep 30, 2018 at 10:29:50AM -0400, David Mertz wrote:

> I think Steven's is backwards in its own way.
>

> - Contracts test the space of arguments *actually used during testing
> period* (or during initial production if the performance hit is
> acceptable).
> - Unit tests test the space of arguments *thought of by the developers*.
>
> *A priori,* either one of those can cover cases not addressed by the
> other.

Fair point.

But given that in general unit tests tend to only exercise a handful of
values (have a look at the tests in the Python stdlib) I think it is
fair to say that in practice unit tests typically do not have anywhere
near the coverage of live data used during alpha and beta testing.

> If unit tests use the hypothesis library or similar approaches,
> unit tests might very well examine arguments unlikely to be encountered in
> real-world (or test phase) use...

Indeed.

We can consider all of these things as complementary:

- doctests give us confidence that the documentation hasn't rotted;

- unit tests give us confidence that corner cases are tested;

- contracts give us confidence that regular and common cases are tested;

- regression tests give us confidence that bugs aren't re-introduced;

- smoke tests give us confidence that the software at least will run;

- static type checking allows us to drop type checks from our unit
tests and contracts;

but of course there can be overlap. And that's perfectly fine.


[...]

> - Half of the checks are very far away, in a separate file, assuming
> > I even remembered or bothered to write the test.
> >
>
> To me, this is the GREATEST VIRTUE of unit tests over DbC. It puts the
> tests far away where they don't distract from reading and understanding the
> function itself. I rarely want my checks proximate since I wear a very
> different hat when thinking about checks than when writing functionality
> (ideally, a lot of the time, I wear the unit test hat *before* I write the
> implementation; TDD is usually good practice).

I'm curious. When you write a function or method, do you include input
checks? Here's an example from the Python stdlib (docstring removed for
brevity):

# bisect.py
def insort_right(a, x, lo=0, hi=None):
if lo < 0:
raise ValueError('lo must be non-negative')
if hi is None:
hi = len(a)
while lo < hi:
mid = (lo+hi)//2
if x < a[mid]: hi = mid
else: lo = mid+1
a.insert(lo, x)

Do you consider that check for lo < 0 to be disruptive? How would you
put that in a unit test?

That check is effectively a pre-condition. Putting aside the question of
which exception should be raised (AssertionError or ValueError), we
could re-write that as a contract:

def insort_right(a, x, lo=0, hi=None):
require:
lo >= 0
# implementation follows, as above
if hi is None:
hi = len(a)
while lo < hi:
mid = (lo+hi)//2
if x < a[mid]: hi = mid
else: lo = mid+1
a.insert(lo, x)

Do you consider that precondition check for lo >= to be disruptive? More
or less disruptive than when it was in the body of the function
implementation?

> > - The post-conditions aren't checked unless I run my test suite, and
> > then they only check the canned input in the test suite.
> >
>
> Yes, this is a great advantage of unit tests. No cost until you explicitly
> run them.

If you're worried about the cost of verifying your program does the
right thing during testing and development, I think you're doing
something wrong :-)

If there are specific functions/classes where the tests are insanely
expensive, that's one thing. I have some code that wants to verify that
a number is prime as part of an informal post-condition check, but if it
is a *big* prime that check is too costly so I skip it.

But in general, if I'm testing or under active development, what do I
care if the program takes 3 seconds to run instead of 2.5 seconds?
Either way, its finished by the time I come back from making my coffee
:-)

But more seriously, fine, if a particular contract is too expensive to
run, disable it or remove it and add some unit tests. And then your
devs will complain that the unit tests are too slow, and stop running
them, and that's why we can't have nice things... *wink*

[David Mertz]

On Sun, Sep 30, 2018 at 11:34 AM Steven D'Aprano <st...@pearwood.info> wrote:

On Sun, Sep 30, 2018 at 10:29:50AM -0400, David Mertz wrote:
But given that in general unit tests tend to only exercise a handful of
values (have a look at the tests in the Python stdlib) I think it is
fair to say that in practice unit tests typically do not have anywhere
near the coverage of live data used during alpha and beta testing.

I still think it's a mixture.  I write tests to also address "this really shouldn't happen" cases as well (often in a loop, or using a Nose class for scaffolding).

There's some saying/joke about software testing along the lines of:

For an argument that should be in range 1-100: 

  try 50; try 1; try 100

  try 101; try 0; 

  try -1, 

  try 3.14

  try 1+1j;

  try the string "fish", 

  try a null pointer;

  etc.

Many of those oddball cases can easily be in a list of values to test in unit tests, but may be impossible or unlikely to make it to the function call in the normal/possible flow of the program.

 

I'm curious. When you write a function or method, do you include input
checks? Here's an example from the Python stdlib (docstring removed for
brevity):

# bisect.py
def insort_right(a, x, lo=0, hi=None):
    if lo < 0:
        raise ValueError('lo must be non-negative')
    if hi is None:
        hi = len(a)
    while lo < hi:
        mid = (lo+hi)//2
        if x < a[mid]: hi = mid
        else: lo = mid+1
    a.insert(lo, x)

Do you consider that check for lo < 0 to be disruptive? How would you
put that in a unit test?

I definitely put in checks like that.  However, I might well write a test like:

    assert lo >= 0, "lo must be non-negative"

That would allow disabling the check for production.  However, I presume the stdlib does this for a reason; it presumably wants to allow callers to catch a specific exception.  I haven't seen anything in the contract discussion that allows raising a particular exception when a contract is violated, only a general one for all contracts.

The case of 'if hi is None' is different.  It's remediation of a missing value where it's perfectly fine to impute an unspecified value.  So that would be a poor contract.

[Steven D'Aprano]

On Mon, Oct 01, 2018 at 12:36:18PM -0400, David Mertz wrote:

> There's some saying/joke about software testing along the lines of:
>
> For an argument that should be in range 1-100:
> try 50; try 1; try 100
> try 101; try 0;
> try -1,
> try 3.14
> try 1+1j;
> try the string "fish",
> try a null pointer;
> etc.
>
> Many of those oddball cases can easily be in a list of values to test in
> unit tests, but may be impossible or unlikely to make it to the function
> call in the normal/possible flow of the program.

Right. And for a *library*, you must handle out of range errors, because
the library cannot trust the input passed to it by third parties. If a
third party passes "fish" as an argument, that's not a bug in the
library and the library cannot fix it, it should handle the error
gracefully with a sensible error message.

But for an *application*, the internal routines are completely under
control of the one developer team. If an internal routine passes "fish",
that's a bug that the dev team can and ought to fix. Once that bug is
fixed, there's no need to check for "fish" in production code. Its a
waste of time.

(In principle at least -- it depends on how confident you or your QA
team are that all the bugs are ironed out. Do you ship your apps with
debugging turned off? Then you might ship with contract checking turned
off, or at least dialled back to a lower level.)

For an application, it doesn't matter if my function sets the computer
on fire when passed the string "fish", if there is no way for the
application to pass that string to the function. If it can't happen, it
can't happen and there's no need to defend against it beyond a
regression test.

(Modulo comments about how confident you are about the software
quality.)

For a library, anyone can pass "fish" to anything, so you better handle
it gracefully rather than setting the computer on fire.

For application developers, the point of contracts is to make error
conditions *impossible* rather than to just defend against them, so you
can turn off the defensive error checking in production because they
aren't needed.

> > I'm curious. When you write a function or method, do you include input
> > checks? Here's an example from the Python stdlib (docstring removed for
> > brevity):
> >
> > # bisect.py
> > def insort_right(a, x, lo=0, hi=None):
> > if lo < 0:
> > raise ValueError('lo must be non-negative')
> > if hi is None:
> > hi = len(a)
> > while lo < hi:
> > mid = (lo+hi)//2
> > if x < a[mid]: hi = mid
> > else: lo = mid+1
> > a.insert(lo, x)
> >
> > Do you consider that check for lo < 0 to be disruptive? How would you
> > put that in a unit test?
> >
>
> I definitely put in checks like that. However, I might well write a test
> like:
>
> assert lo >= 0, "lo must be non-negative"
>
> That would allow disabling the check for production.

For a library function, I would consider that an abuse of assert, since:

(1) It raises the wrong exception (AssertionError);

(2) It can be disabled;

(3) And it sends the wrong message, telling the reader that lo is an
internal implementation detail rather than part of the function's
external API.

> However, I presume
> the stdlib does this for a reason; it presumably wants to allow callers to
> catch a specific exception. I haven't seen anything in the contract
> discussion that allows raising a particular exception when a contract is
> violated, only a general one for all contracts.

I think that the Eiffel philosophy is that all contract violations are
assertion errors, but I'm not really confident in my understanding of
Eiffel's exception mechanism. It doesn't have a generalised try...except
statement like Python.

If Python had contracts, I'd want to see:

- by default, contract violations raise specific subclasses
of AssertionError, e.g. RequireError, EnsureError;

- but there ought to be a mechanism to allow specific errors
to raise more specific exceptions.

I haven't given any thought to that mechanism. It might simply be "the
status quo" -- there's no reason why we *must* move defensive input
checking into the (hypothetical) require block just because it is
available. You can still write your test the old fashioned way in the
body of the function and raise the exception of your choice.

> The case of 'if hi is None' is different. It's remediation of a missing
> value where it's perfectly fine to impute an unspecified value. So that
> would be a poor contract.

Since the check for None is not error checking at all, I didn't talk
about it or make it a precondition test.