And the logic is quite complex, because there are many possible paths to
follow depending on some other parameters.
This program even needs to run on many virtual machines at the same time
so the interaction is another thing I need to check...

Now I successfully managed to mock the database with sqlalchemy and only
the fields I actually need, but I now would like to simulate also
everything else.

I would like for example that if I simulate I can pass a fake database,
a fake configuration and get the log of what exactly would happen.  But
I'm not sure how to implement it now..  One possibility would be to have
a global variable (PRETEND_ONLY = False) that if set should be checked
before every potentially system-dependent command.

For example

copytree(src, dest) becomes:
if not PRETEND_ONLY:
    copytree(src, dest)

But I don't like it too much because I would have to add a lot of
garbage around..

Another way is maybe to set the sys.excepthook to something that catchs
all the exceptions that would be thrown by these comands, but I'm not
sure is a good idea either..

Then, have your code accept the thing as an argument.

E.g. instead of having a hard-coded database connection, allow the
database connection to be set (perhaps as an argument, perhaps as a
config option, etc.).

There are libraries to help with mocks, e.g.:

http://garybernhardt.github.com/python-mock-comparison/

Mocking the file system is probably the hardest part, because you
generally don't have a "FileSystem" object available to be replaced. In
effect, your program has one giant global variable, the file system.
Worse, it's not even a named variable, it's hard-coded everywhere you use
it.

I don't know of any good solution for that. I've often thought about it,
but don't have an answer.

I suppose you could monkey-patch a bunch of stuff:

if ONLY_PRETEND:
     open = my_mock_open
     copytree = my_mock_copytree
     # etc.
main()  # run your application

but that would also be painful.

class MockMysqlEngine(MySqlEngine):
    # TODO: make the engine more generic would avoid this dirty hack
    def __init__(self, *args, **kwargs):
        # self.engine =
create_engine('sqlite:////home/andrea/testdb.sqlite', echo=True)
        self.engine = create_engine('sqlite://', echo=True)
        self.meta = MetaData(bind=self.engine)
        self.session_maker = sessionmaker(bind=self.engine)

Now I populate statically the schema and populate with some test data
too, but I'm also implementing a weay to just pass some CSV files so
that other people can easily write some test cases with some other
possible database configurations.

(And I use mock for a few other things)

I also tried to catch the signal but same result, how do I make it
continue and just don't complain?

The other option if of course to do a big try/except, but I would
prefer the excepthook solution..

import sys
from shutil import copy

def my_except_hook(etype, value, tb):
    print("got an exception of type", etype)

class MyTest( unittest.TestCase ):
     def setUp( self ):
         globalsub.subs( shutil.copytree )
     def tearDown( self ):
         globalsub.restore( shutil.copytree )

You can also pass a function to subs like so:

     def setUp( self ):
         self.copied_trees = []
         def fake_copytree( src, dest ):
             assert os.path.exists( src )
             self.copied_trees.append( (src, dest ))
             return dest # or whatever the thing should return
         globalsub.subs( shutil.copytree, fake_copytree )

$ pip install globalsub

HTH,
Mike

- http://washort.twistedmatrix.com/2011/01/introducing-exocet.html
- http://washort.twistedmatrix.com/2011/03/exocet-second-look.html

   def your_prog(parent):
      conn = parent.db.make_connection(...)
      blah = parent.copytree(...)

With "inversion of control", the components interact on the bases
of interfaces. The components themselves do not know each other, they
know only the interfaces they want to interact with. For the interaction
to really take place, a component asks a registry "give me a component
satisfying this interface", gets it and uses the interface.

If you follow this paradigm, it is easy to switch components: just
register different alternatives for the interface at hand.

"zope.interface" and "zope.component" are python packages that
support this paradigm. Despite the "zope" in their name, they can be
used outside of "Zope".

"zope.interface" models interfaces, while "zope.component" provides
so called "utilities" (e.g. "database utility", "filesystem utility", ...)
and "adapters" and the corresponding registries.

class FSActions:
      @classmethod
      def copy_directories(cls, src, dest)
          raise NotImplementedError

      @classmethod
      ....

And then have different implementations of this interface.  This would
work, but I don't really like the idea of constructing interfaces that
provide only the few things I need.

    try:
        copytree('sjkdf', 'dsflkj')
        Popen(['notfouhd'], shell=True)
    except Exception as e:
        print("here")

behaves differently from:

    try:
        Popen(['notfouhd'], shell=True)
        copytree('sjkdf', 'dsflkj')
    except Exception as e:
        print("here")

def default_mock_action(func_name):
    def _default_mock_action(*args, **kwargs):
        print("running {} with args {} and {}".format(func_name, args, kwargs))

    return _default_mock_action

def mock_fs_actions(to_run):
    """Take a function to run, and run it in an environment which
    mocks all the possibly dangerous operations
    """
    side_effect = [
        'copytree',
        'copy',
    ]

    acts = dict((s, default_mock_action(s)) for s in side_effect)

    with patch('pytest.runner.commands.ShellCommand.run',
default_mock_action('run')):
        with patch.multiple('shutil', **acts):
            to_run()

So I can just pass the main function inside the mock like
        mock_fs_actions(main)

Your code above should not do this. If it does, there is likely a bug.

You told us, that the two alternatives above behaved differently --
I expect 'behaved differently with respect to the printing of "here"'.
If you tell us, which alternative printed "here" and which did not,
we would be able to deduce which of the "Popen" or "copytree"
caused the immediate exit.

"Popen" might contain a call to "os._exit" (one of the ways to immediately
quit) -- though it should only call it in the forked child not in the
calling process. "coyptree" might under exceptional circumstances (extremely
deeply nested structures -- surely not for non-existent source and target)
cause a stack overflow (which, too, can lead to immediate death).

In addition, "Popen" and maybe even "copytree" may call platform
dependent functions. Thus, platform information could be relevant.

Under "*nix", you should be able to get some information from
the exit code of a suddenly quiting process. It tells whether
the process died from a fatal signal (a stack overflow would
result in the fatal SIGSEGV) or whether it existed willingly with
an exit code.

When replying to posts, please leave enough quoted to establish context.
Neither email nor usenet are guaranteed delivery services, and they
certainly don't guarantee to deliver messages in order. Assume that your
readers may not have seen the message you are replying to, and you will
probably get more and better responses.

You can see the exception hierarchy here:

http://docs.python.org/library/exceptions.html#exception-hierarchy

Please do NOT catch BaseException, since that is the wrong thing to do.
If you must catch SystemExit, KeyboardInterrupt, etc. they you should do
so as separate catch clauses:

try:
    main()
except SystemExit as e:
    print(e)  # see if we can find out who is raising this
except KeyboardInterrupt:
    print("Mwahahaha my pretty, you cannot cancel this!!!")
    print("...er, now what do I do?")
except Exception:
    print("why am I catching exceptions I can't recover from?")

try:
    do_stuff()  # run your program
except Exception as e:
    # pointlessly catch exceptions I can't handle, which has the
    # bonus of making debugging MUCH MUCH harder
    print("here")
# end of file, nothing further to do

When do_stuff() fails, "here" gets printed, and then the program exits
because there's nothing else to do.

Catching exceptions doesn't magically cause the code to continue from the
point of the error. It doesn't work like that. Execution skips from where
the error occurred to the except clause. Once the except clause has run,
anything following the except clause runs, and then the program ends as
normal.

If you haven't already done so, I recommend you go through the tutorial:

http://docs.python.org/py3k/tutorial/index.html

in particular the part about exception handling:

http://docs.python.org/py3k/tutorial/errors.html

My first guess was os.popen, but that doesn't take a shell argument:

py> os.popen(['ls', '-l'], shell=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: popen() got an unexpected keyword argument 'shell'

That at least is my guess, based on the described symptoms.

If that's the case, I would be inclined to think that they are wrong.

If you are investigating why a third-party function is stopping your
interpreter, then catching BaseException may tell you that the code
is raising the wrong kind of Exception.  Once you know what kind the
function is raising, you should catch only that particular excpetion
subclass.

except SystemExit:
    import traceback
    traceback.print_exc()

That will print a complete stack trace.

If you only need to know what kind of exception you have,
you can do:

    print(repr(e))

A simple print(e) will print str(e), which in the case of
SystemExit, is an empty string.  That's not very informative.

In development, not catching the exception will give me a full
trace back automatically. Why bother with trying to catch and
print something when the interpreter will do it for me? Not
to mention that removes any hassle of trying to catch the
"right" exception or figuring out the best way to print it.
I suppose if there are arguments on the exception that were not
printed then I might want to catch it, but has been rare
in my experience.

Ramit