adding a simulation mode

[andrea crotti]

I'm writing a program which has to interact with many external
resources, at least:
- mysql database
- perforce
- shared mounts
- files on disk

And the logic is quite complex, because there are many possible paths to
follow depending on some other parameters.
This program even needs to run on many virtual machines at the same time
so the interaction is another thing I need to check...

Now I successfully managed to mock the database with sqlalchemy and only
the fields I actually need, but I now would like to simulate also
everything else.

I would like for example that if I simulate I can pass a fake database,
a fake configuration and get the log of what exactly would happen. But
I'm not sure how to implement it now.. One possibility would be to have
a global variable (PRETEND_ONLY = False) that if set should be checked
before every potentially system-dependent command.

For example

copytree(src, dest) becomes:
if not PRETEND_ONLY:
copytree(src, dest)

But I don't like it too much because I would have to add a lot of
garbage around..

Another way is maybe to set the sys.excepthook to something that catchs
all the exceptions that would be thrown by these comands, but I'm not
sure is a good idea either..

Any suggestion?

[Steven D'Aprano]

On Wed, 04 Jul 2012 10:42:56 +0100, andrea crotti wrote:

> I'm writing a program which has to interact with many external
> resources, at least:
> - mysql database
> - perforce
> - shared mounts
> - files on disk
>
> And the logic is quite complex, because there are many possible paths to
> follow depending on some other parameters. This program even needs to
> run on many virtual machines at the same time so the interaction is
> another thing I need to check...
>
> Now I successfully managed to mock the database with sqlalchemy and only
> the fields I actually need, but I now would like to simulate also
> everything else.
>
> I would like for example that if I simulate I can pass a fake database,
> a fake configuration and get the log of what exactly would happen. But
> I'm not sure how to implement it now.. One possibility would be to have
> a global variable (PRETEND_ONLY = False) that if set should be checked
> before every potentially system-dependent command.

I think a better way would be to use a mock database, etc. For each thing
which you want to simulate, create a class that has the same interface
but a simulated implementation.

Then, have your code accept the thing as an argument.

E.g. instead of having a hard-coded database connection, allow the
database connection to be set (perhaps as an argument, perhaps as a
config option, etc.).

There are libraries to help with mocks, e.g.:

http://garybernhardt.github.com/python-mock-comparison/

> For example
>
> copytree(src, dest) becomes:
> if not PRETEND_ONLY:
> copytree(src, dest)

Ewww :(

Mocking the file system is probably the hardest part, because you
generally don't have a "FileSystem" object available to be replaced. In
effect, your program has one giant global variable, the file system.
Worse, it's not even a named variable, it's hard-coded everywhere you use
it.

I don't know of any good solution for that. I've often thought about it,
but don't have an answer.

I suppose you could monkey-patch a bunch of stuff:

if ONLY_PRETEND:
open = my_mock_open
copytree = my_mock_copytree
# etc.
main() # run your application



but that would also be painful.


--
Steven

[andrea crotti]

2012/7/4 Steven D'Aprano <steve+comp....@pearwood.info>:

>
> Then, have your code accept the thing as an argument.
>
> E.g. instead of having a hard-coded database connection, allow the
> database connection to be set (perhaps as an argument, perhaps as a
> config option, etc.).
>
> There are libraries to help with mocks, e.g.:
>
> http://garybernhardt.github.com/python-mock-comparison/

Ah yes this part is already done, I pass an object to the entry point
of the program which represents the database connection, which looks
like this:

class MockMysqlEngine(MySqlEngine):
# TODO: make the engine more generic would avoid this dirty hack
def __init__(self, *args, **kwargs):
# self.engine =
create_engine('sqlite:////home/andrea/testdb.sqlite', echo=True)
self.engine = create_engine('sqlite://', echo=True)
self.meta = MetaData(bind=self.engine)
self.session_maker = sessionmaker(bind=self.engine)


Now I populate statically the schema and populate with some test data
too, but I'm also implementing a weay to just pass some CSV files so
that other people can easily write some test cases with some other
possible database configurations.

(And I use mock for a few other things)

>
>
>> For example
>>
>> copytree(src, dest) becomes:
>> if not PRETEND_ONLY:
>> copytree(src, dest)
>
> Ewww :(
>
> Mocking the file system is probably the hardest part, because you
> generally don't have a "FileSystem" object available to be replaced. In
> effect, your program has one giant global variable, the file system.
> Worse, it's not even a named variable, it's hard-coded everywhere you use
> it.
>
> I don't know of any good solution for that. I've often thought about it,
> but don't have an answer.
>
> I suppose you could monkey-patch a bunch of stuff:
>
> if ONLY_PRETEND:
> open = my_mock_open
> copytree = my_mock_copytree
> # etc.
> main() # run your application
>
>
>
> but that would also be painful.
>

Yes there is no easy solution apparently.. But I'm also playing
around with vagrant and virtual machine generations, suppose I'm able
to really control what will be on the machine at time X, creating it
on demand with what I need, it might be a good way to solve my
problems (a bit overkill and slow maybe though).

I'll try the sys.excepthook trick first, any error should give me an
exception, so if I catch them all I think it might work already..

[andrea crotti]

> Yes there is no easy solution apparently.. But I'm also playing
> around with vagrant and virtual machine generations, suppose I'm able
> to really control what will be on the machine at time X, creating it
> on demand with what I need, it might be a good way to solve my
> problems (a bit overkill and slow maybe though).
>
> I'll try the sys.excepthook trick first, any error should give me an
> exception, so if I catch them all I think it might work already..

I actually thought that the sys.excepthook would be easy but it's not
so trivial apparently:
This simple sample never reaches the print("here"), because even if
the exception is catched it still quits with return code=1.

I also tried to catch the signal but same result, how do I make it
continue and just don't complain?

The other option if of course to do a big try/except, but I would
prefer the excepthook solution..


import sys
from shutil import copy

def my_except_hook(etype, value, tb):
print("got an exception of type", etype)


if __name__ == '__main__':
sys.excepthook = my_except_hook
copy('sdflsdk')
print("here")

[Mike C. Fletcher]

On 12-07-04 05:42 AM, andrea crotti wrote:
...

> copytree(src, dest) becomes:
> if not PRETEND_ONLY:
> copytree(src, dest)

import globalsub, unittest

class MyTest( unittest.TestCase ):
def setUp( self ):
globalsub.subs( shutil.copytree )
def tearDown( self ):
globalsub.restore( shutil.copytree )

You can also pass a function to subs like so:

def setUp( self ):
self.copied_trees = []
def fake_copytree( src, dest ):
assert os.path.exists( src )
self.copied_trees.append( (src, dest ))
return dest # or whatever the thing should return
globalsub.subs( shutil.copytree, fake_copytree )

$ pip install globalsub

HTH,
Mike

--
________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://www.vrplumber.com
http://blog.vrplumber.com

[Devin Jeanpierre]

For what it's worth, this is the reason that Allen Short wrote Exocet.

> This way, you can test your code without having to resort to sys.modules
> hackery, and you can better factor your applications by separating
> configuration and environment concerns from the rest of your code.

See:

- http://washort.twistedmatrix.com/2011/01/introducing-exocet.html
- http://washort.twistedmatrix.com/2011/03/exocet-second-look.html

-- Devin

[Paul Rubin]

andrea crotti <andrea....@gmail.com> writes:
> copytree(src, dest) becomes:
> if not PRETEND_ONLY:
> copytree(src, dest)
>
> But I don't like it too much because I would have to add a lot of
> garbage around..

I've had good results writing the module under test in the style of a
java applet, i.e. one of its args is a class instance representing the
"outside world", and ALL interaction that you might want to simulate is
done through this object:

def your_prog(parent):
conn = parent.db.make_connection(...)
blah = parent.copytree(...)

Then you make "real" and "mock" versions of the external interface, and
pass in an appropriate instance.

[Dieter Maurer]

andrea crotti <andrea....@gmail.com> writes:

> I'm writing a program which has to interact with many external
> resources, at least:
> - mysql database
> - perforce
> - shared mounts
> - files on disk
>
> And the logic is quite complex, because there are many possible paths to
> follow depending on some other parameters.
> This program even needs to run on many virtual machines at the same time
> so the interaction is another thing I need to check...
>
> Now I successfully managed to mock the database with sqlalchemy and only
> the fields I actually need, but I now would like to simulate also
> everything else.

There is a paradigm called "inversion of control" which can be used
to handle those requirements.

With "inversion of control", the components interact on the bases
of interfaces. The components themselves do not know each other, they
know only the interfaces they want to interact with. For the interaction
to really take place, a component asks a registry "give me a component
satisfying this interface", gets it and uses the interface.

If you follow this paradigm, it is easy to switch components: just
register different alternatives for the interface at hand.


"zope.interface" and "zope.component" are python packages that
support this paradigm. Despite the "zope" in their name, they can be
used outside of "Zope".

"zope.interface" models interfaces, while "zope.component" provides
so called "utilities" (e.g. "database utility", "filesystem utility", ...)
and "adapters" and the corresponding registries.


Of course, they contain only the infrastructure for the "inversion of control"
paradigm. Up to you to provide the implementation for the various
mocks.

[andrea crotti]

2012/7/5 Dieter Maurer <die...@handshake.de>:

>
> There is a paradigm called "inversion of control" which can be used
> to handle those requirements.
>
> With "inversion of control", the components interact on the bases
> of interfaces. The components themselves do not know each other, they
> know only the interfaces they want to interact with. For the interaction
> to really take place, a component asks a registry "give me a component
> satisfying this interface", gets it and uses the interface.
>
> If you follow this paradigm, it is easy to switch components: just
> register different alternatives for the interface at hand.
>
>
> "zope.interface" and "zope.component" are python packages that
> support this paradigm. Despite the "zope" in their name, they can be
> used outside of "Zope".
>
> "zope.interface" models interfaces, while "zope.component" provides
> so called "utilities" (e.g. "database utility", "filesystem utility", ...)
> and "adapters" and the corresponding registries.
>
>
> Of course, they contain only the infrastructure for the "inversion of control"
> paradigm. Up to you to provide the implementation for the various
> mocks.
>

Thanks that's a good point, in short I could do something like:


class FSActions:
@classmethod
def copy_directories(cls, src, dest)
raise NotImplementedError

@classmethod
....

And then have different implementations of this interface. This would
work, but I don't really like the idea of constructing interfaces that
provide only the few things I need.

Instead of being good APIs they might become just random
functionalities put together to make my life easier, and at that point
maybe just some clear mocking might be even better..

[andrea crotti]

One thing that I don't quite understand is why some calls even if I
catch the exception still makes the whole program quit.
For example this

try:
copytree('sjkdf', 'dsflkj')
Popen(['notfouhd'], shell=True)
except Exception as e:
print("here")


behaves differently from:

try:
Popen(['notfouhd'], shell=True)
copytree('sjkdf', 'dsflkj')
except Exception as e:
print("here")

because if copytree fails it quits anyway.
I also looked at the code but can't quite get why.. any idea?

[andrea crotti]

One way instead that might actually work is this

def default_mock_action(func_name):
def _default_mock_action(*args, **kwargs):
print("running {} with args {} and {}".format(func_name, args, kwargs))

return _default_mock_action


def mock_fs_actions(to_run):
"""Take a function to run, and run it in an environment which
mocks all the possibly dangerous operations
"""
side_effect = [
'copytree',
'copy',
]

acts = dict((s, default_mock_action(s)) for s in side_effect)

with patch('pytest.runner.commands.ShellCommand.run',
default_mock_action('run')):
with patch.multiple('shutil', **acts):
to_run()


So I can just pass the main function inside the mock like
mock_fs_actions(main)

and it seems to do the job, but I have to list manually all the things
to mock and I'm not sure is the best idea anyway..

[John Gordon]

In <mailman.2039.1342099...@python.org> andrea crotti <andrea....@gmail.com> writes:

> try:
> copytree('sjkdf', 'dsflkj')
> Popen(['notfouhd'], shell=True)
> except Exception as e:
> print("here")

> behaves differently from:

> try:
> Popen(['notfouhd'], shell=True)
> copytree('sjkdf', 'dsflkj')
> except Exception as e:
> print("here")

> because if copytree fails it quits anyway.
> I also looked at the code but can't quite get why.. any idea?

copytree() could contain a call to sys.exit(), although that seems like
a rude thing to do.

--
John Gordon A is for Amy, who fell down the stairs
gor...@panix.com B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

[andrea crotti]

Well that's what I thought, but I can't find any explicit exit
anywhere in shutil, so what's going on there?

[John Gordon]

In <mailman.2043.1342102...@python.org> andrea crotti <andrea....@gmail.com> writes:

> Well that's what I thought, but I can't find any explicit exit
> anywhere in shutil, so what's going on there?

Try catching SystemExit specifically (it doesn't inherit from Exception,
so "except Exception" won't catch it.)

[andrea crotti]

2012/7/12 John Gordon <gor...@panix.com>:

> In <mailman.2043.1342102...@python.org> andrea crotti <andrea....@gmail.com> writes:
>
>> Well that's what I thought, but I can't find any explicit exit
>> anywhere in shutil, so what's going on there?
>
> Try catching SystemExit specifically (it doesn't inherit from Exception,
> so "except Exception" won't catch it.)
>
> --

Ah yes that actually works, but I think is quite dodgy, why was it
done like this?
In shutil there is still no mention of SystemExit, and trying to raise
the single exceptions by and
doens't still make it exit, so I would still like to know how it is
possible just for curiosity..

[Dieter Maurer]

andrea crotti wrote at 2012-7-12 14:20 +0100:
>One thing that I don't quite understand is why some calls even if I
>catch the exception still makes the whole program quit.
>For example this
>

> try:
> copytree('sjkdf', 'dsflkj')
> Popen(['notfouhd'], shell=True)
> except Exception as e:
> print("here")
>
>
>behaves differently from:
>
> try:
> Popen(['notfouhd'], shell=True)
> copytree('sjkdf', 'dsflkj')
> except Exception as e:
> print("here")
>
>because if copytree fails it quits anyway.
>I also looked at the code but can't quite get why.. any idea?

There are ways to quit a program immediately without giving
exception handlers a chance to intervene -- though Python does
not make this easy.

Your code above should not do this. If it does, there is likely a bug.


You told us, that the two alternatives above behaved differently --
I expect 'behaved differently with respect to the printing of "here"'.
If you tell us, which alternative printed "here" and which did not,
we would be able to deduce which of the "Popen" or "copytree"
caused the immediate exit.

"Popen" might contain a call to "os._exit" (one of the ways to immediately
quit) -- though it should only call it in the forked child not in the
calling process. "coyptree" might under exceptional circumstances (extremely
deeply nested structures -- surely not for non-existent source and target)
cause a stack overflow (which, too, can lead to immediate death).


In addition, "Popen" and maybe even "copytree" may call platform
dependent functions. Thus, platform information could be relevant.

Under "*nix", you should be able to get some information from
the exit code of a suddenly quiting process. It tells whether
the process died from a fatal signal (a stack overflow would
result in the fatal SIGSEGV) or whether it existed willingly with
an exit code.

--
Dieter

[Steven D'Aprano]

On Thu, 12 Jul 2012 15:17:03 +0100, andrea crotti wrote:

> Well that's what I thought, but I can't find any explicit exit anywhere
> in shutil, so what's going on there?

Hard to say, since you don't give any context to your question.

When replying to posts, please leave enough quoted to establish context.
Neither email nor usenet are guaranteed delivery services, and they
certainly don't guarantee to deliver messages in order. Assume that your
readers may not have seen the message you are replying to, and you will
probably get more and better responses.



--
Steven

[Steven D'Aprano]

On Thu, 12 Jul 2012 16:37:42 +0100, andrea crotti wrote:

> 2012/7/12 John Gordon <gor...@panix.com>:
>> In <mailman.2043.1342102...@python.org> andrea crotti
>> <andrea....@gmail.com> writes:
>>
>>> Well that's what I thought, but I can't find any explicit exit
>>> anywhere in shutil, so what's going on there?
>>
>> Try catching SystemExit specifically (it doesn't inherit from
>> Exception, so "except Exception" won't catch it.)
>>
>

> Ah yes that actually works, but I think is quite dodgy, why was it done
> like this?

Built-in exceptions SystemExit, KeyboardInterrupt and GeneratorExit
deliberately do not inherit from Exception since they are not meant to be
caught by "catch-all" try...except Exception clauses.

You can see the exception hierarchy here:

http://docs.python.org/library/exceptions.html#exception-hierarchy

Please do NOT catch BaseException, since that is the wrong thing to do.
If you must catch SystemExit, KeyboardInterrupt, etc. they you should do
so as separate catch clauses:

try:
main()
except SystemExit as e:
print(e) # see if we can find out who is raising this
except KeyboardInterrupt:
print("Mwahahaha my pretty, you cannot cancel this!!!")
print("...er, now what do I do?")
except Exception:
print("why am I catching exceptions I can't recover from?")


--
Steven

[Steven D'Aprano]

On Thu, 12 Jul 2012 14:20:18 +0100, andrea crotti wrote:

> One thing that I don't quite understand is why some calls even if I
> catch the exception still makes the whole program quit.

Without seeing your whole program, we can't possibly answer this. But by
consulting my crystal ball, I bet you have something like this:

try:
do_stuff() # run your program
except Exception as e:
# pointlessly catch exceptions I can't handle, which has the
# bonus of making debugging MUCH MUCH harder
print("here")
# end of file, nothing further to do


When do_stuff() fails, "here" gets printed, and then the program exits
because there's nothing else to do.

Catching exceptions doesn't magically cause the code to continue from the
point of the error. It doesn't work like that. Execution skips from where
the error occurred to the except clause. Once the except clause has run,
anything following the except clause runs, and then the program ends as
normal.

If you haven't already done so, I recommend you go through the tutorial:

http://docs.python.org/py3k/tutorial/index.html

in particular the part about exception handling:

http://docs.python.org/py3k/tutorial/errors.html

> For example this
>
> try:
> copytree('sjkdf', 'dsflkj')
> Popen(['notfouhd'], shell=True)
> except Exception as e:
> print("here")

What is "Popen" and where is it from?

My first guess was os.popen, but that doesn't take a shell argument:

py> os.popen(['ls', '-l'], shell=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: popen() got an unexpected keyword argument 'shell'

> behaves differently from:
>
> try:
> Popen(['notfouhd'], shell=True)
> copytree('sjkdf', 'dsflkj')
> except Exception as e:
> print("here")
>
> because if copytree fails it quits anyway.

Well of course it does. If copytree fails, the try block ends and
execution skips straight to the except block, which runs, and then the
program halts because there's nothing else to be done.

That at least is my guess, based on the described symptoms.




--
Steven

[andrea crotti]

2012/7/13 Steven D'Aprano <steve+comp....@pearwood.info>:

> Well of course it does. If copytree fails, the try block ends and
> execution skips straight to the except block, which runs, and then the
> program halts because there's nothing else to be done.
>
> That at least is my guess, based on the described symptoms.
>

Well I think that's what I was stupidly missing, I always had only one
possibly failing thing in a try/except block,
and I always gave for granted that it doesn't jump to the except block
on first error, but of course it makes
more sense if it does...

Thanks a lot

[Hans Mulder]

On 13/07/12 04:16:53, Steven D'Aprano wrote:
> On Thu, 12 Jul 2012 16:37:42 +0100, andrea crotti wrote:
>
>> 2012/7/12 John Gordon <gor...@panix.com>:
>>> In <mailman.2043.1342102...@python.org> andrea crotti
>>> <andrea....@gmail.com> writes:
>>>
>>>> Well that's what I thought, but I can't find any explicit exit
>>>> anywhere in shutil, so what's going on there?
>>>
>>> Try catching SystemExit specifically (it doesn't inherit from
>>> Exception, so "except Exception" won't catch it.)
>>>
>>
>> Ah yes that actually works, but I think is quite dodgy, why was it done
>> like this?

It may be that the function you're calling found a problem that the
author thinks is so grave that they shouldn't give you an opportunity
to deal with it.

If that's the case, I would be inclined to think that they are wrong.

> Built-in exceptions SystemExit, KeyboardInterrupt and GeneratorExit
> deliberately do not inherit from Exception since they are not meant to be
> caught by "catch-all" try...except Exception clauses.
>
> You can see the exception hierarchy here:
>
> http://docs.python.org/library/exceptions.html#exception-hierarchy
>
> Please do NOT catch BaseException, since that is the wrong thing to do.

I would agree if you had said "in production code".

If you are investigating why a third-party function is stopping your
interpreter, then catching BaseException may tell you that the code
is raising the wrong kind of Exception. Once you know what kind the
function is raising, you should catch only that particular excpetion
subclass.

> If you must catch SystemExit, KeyboardInterrupt, etc. they you should do
> so as separate catch clauses:
>
> try:
> main()
> except SystemExit as e:
> print(e) # see if we can find out who is raising this

If you want to find out who is raising the exception, you could
try this:

except SystemExit:
import traceback
traceback.print_exc()

That will print a complete stack trace.

If you only need to know what kind of exception you have,
you can do:

print(repr(e))

A simple print(e) will print str(e), which in the case of
SystemExit, is an empty string. That's not very informative.

> except KeyboardInterrupt:
> print("Mwahahaha my pretty, you cannot cancel this!!!")
> print("...er, now what do I do?")
> except Exception:
> print("why am I catching exceptions I can't recover from?")

Hope this helps,

-- HansM

[Prasad, Ramit]

> > Please do NOT catch BaseException, since that is the wrong thing to do.
>
> I would agree if you had said "in production code".
>
> If you are investigating why a third-party function is stopping your
> interpreter, then catching BaseException may tell you that the code
> is raising the wrong kind of Exception. Once you know what kind the
> function is raising, you should catch only that particular excpetion
> subclass.

I would say the opposite. In production code usually I want it
to recover, log as much information as I need (including sending
any notifications), and NOT just die.

In development, not catching the exception will give me a full
trace back automatically. Why bother with trying to catch and
print something when the interpreter will do it for me? Not
to mention that removes any hassle of trying to catch the
"right" exception or figuring out the best way to print it.
I suppose if there are arguments on the exception that were not
printed then I might want to catch it, but has been rare
in my experience.

Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

[Chris Angelico]

On Sat, Jul 14, 2012 at 3:08 AM, Prasad, Ramit
<ramit....@jpmorgan.com> wrote:
> I would say the opposite. In production code usually I want it
> to recover, log as much information as I need (including sending
> any notifications), and NOT just die.
>
> In development, not catching the exception will give me a full
> trace back automatically.

Here's another take on the matter. In development, your script is your
execution unit, so you let the interpreter print out your tracebacks.
In production, there will usually be one, maybe two subunits (for
instance, a TCP-based server might have the socket connection as an
execution unit, and possibly a command parser inside that), and at the
top of that subunit, you have a broad exception handler that resets
that one unit (goes back and accepts another client, or waits for
another command). Otherwise, wide-scope exception handling is usually
a bad thing.

ChrisA