I'm relatively new to Python and have googled and googled but haven't found a reasonable answer to this question, so I thought I'd ask it here.
I'm beginning a large Python project which contains many packages, modules and classes. The organisation of those is clear to me.
Now, the classes can contain many methods (100s of data analysis methods) which operate on instances of the class they belong to. These methods can be long and complex. So if I put these methods all in the module file inside the class, the file will get insanely long. Reading on google, the answer is usually "refactor", but that really doesn't make sense here. It's just that the methods are many, and each method can be a long piece of code. So, is there a way to put these methods in their own files and have them 'included' in the class somehow? I read a little about mixins but all the solutions looked very hacky. Is there an official python way to do this? I don't like having source files with 100's of lines of code in, let alone 1000's.
Martin Hewitson <martinhewit...@mac.com> writes:
> So, is there a way to put these methods in their own files and have
> them 'included' in the class somehow? ... Is there an official python
> way to do this? I don't like having source files with 100's of lines
> of code in, let alone 1000's.
That code sounds kind of smelly... why are there so many methods per
class?
Python lets you inject new methods into existing classes (this is
sometimes called duck punching) but I don't recommend doing this.
A few hundred lines of code in a file is completely reasonable. Even a
few thousand is ok. IME it starts getting unwieldy at 3000 or so.
> I'm relatively new to Python and have googled and googled but haven't
> found a reasonable answer to this question, so I thought I'd ask it here.
> I'm beginning a large Python project which contains many packages, modules
> and classes. The organisation of those is clear to me.
> Now, the classes can contain many methods (100s of data analysis methods)
> which operate on instances of the class they belong to. These methods can
> be long and complex. So if I put these methods all in the module file
> inside the class, the file will get insanely long. Reading on google, the
> answer is usually "refactor", but that really doesn't make sense here.
> It's just that the methods are many, and each method can be a long piece
> of code. So, is there a way to put these methods in their own files and
> have them 'included' in the class somehow? I read a little about mixins
> but all the solutions looked very hacky. Is there an official python way
> to do this? I don't like having source files with 100's of lines of code
> in, let alone 1000's.
You googled, found the right answer ("refactor"), didn't like it and are now looking to cure the symptoms of the original problem?
Seriously, a good editor can deal with a long source file, but a class with hundreds of methods will bring trouble to any old brain.
On 2, Nov, 2012, at 08:38 AM, Paul Rubin <no.em...@nospam.invalid> wrote:
> Martin Hewitson <martinhewit...@mac.com> writes:
>> So, is there a way to put these methods in their own files and have
>> them 'included' in the class somehow? ... Is there an official python
>> way to do this? I don't like having source files with 100's of lines
>> of code in, let alone 1000's.
> That code sounds kind of smelly... why are there so many methods per
> class?
Simply because there are many different ways to process the data. The class encapsulates the data, and the user can process the data in many ways. Of course, one could have classes which encapsulate the algorithms, as well as the data, but it also seems natural to me to have algorithms as methods which are part of the data class, so the user operates on the data using methods of that class.
> Python lets you inject new methods into existing classes (this is
> sometimes called duck punching) but I don't recommend doing this.
Is there not a way just to declare the method in the class and put the actual implementation in another file on the python path so that it's picked up a run time?
> A few hundred lines of code in a file is completely reasonable. Even a
> few thousand is ok. IME it starts getting unwieldy at 3000 or so.
In all other projects (in other languages) I've always tried to keep individual files short to ease maintainability and search-ability (if there is such a word).
Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
>> I'm relatively new to Python and have googled and googled but haven't
>> found a reasonable answer to this question, so I thought I'd ask it here.
>> I'm beginning a large Python project which contains many packages, modules
>> and classes. The organisation of those is clear to me.
>> Now, the classes can contain many methods (100s of data analysis methods)
>> which operate on instances of the class they belong to. These methods can
>> be long and complex. So if I put these methods all in the module file
>> inside the class, the file will get insanely long. Reading on google, the
>> answer is usually "refactor", but that really doesn't make sense here.
>> It's just that the methods are many, and each method can be a long piece
>> of code. So, is there a way to put these methods in their own files and
>> have them 'included' in the class somehow? I read a little about mixins
>> but all the solutions looked very hacky. Is there an official python way
>> to do this? I don't like having source files with 100's of lines of code
>> in, let alone 1000's.
> You googled, found the right answer ("refactor"), didn't like it and are now > looking to cure the symptoms of the original problem?
> Seriously, a good editor can deal with a long source file, but a class with > hundreds of methods will bring trouble to any old brain.
Well, here we disagree. Suppose I have a class which encapsulates time-series data. Below is a list of the absolute minimum methods one would have to process that data. That's close to 100 already before even having any specialised methods for dealing with the data. Each of these methods will have maybe 20 lines of documentation. That's 2000 lines already. And what if someone wants to extend that class to add their own processing methods? It would a maintenance nightmare for them to edit the actual class file, which they would then have to repeat each time a new version of the 'official' class file is released.
So maybe some rethinking of this design is needed to handle this 'limitation' of python. Perhaps grouping the processing algorithms into methods of processing classes, then pass the data objects to these methods. But somehow I don't like that. I have the feeling these methods would end up peppered with things like:
if this data type, do this
else if this data type, do this
else ....
normally this would be solved by overloading methods in different data subclasses.
> Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
> Thanks for your thoughts,
> Martin
20 lines of documentation per method? As far as I'm concerned that's not a smell, that's a stink.
> On 2, Nov, 2012, at 09:00 AM, Peter Otten <__pete...@web.de> wrote:
>> Martin Hewitson wrote:
>>> Dear list,
>>> I'm relatively new to Python and have googled and googled but haven't
>>> found a reasonable answer to this question, so I thought I'd ask it here.
>>> I'm beginning a large Python project which contains many packages, modules
>>> and classes. The organisation of those is clear to me.
>>> Now, the classes can contain many methods (100s of data analysis methods)
>>> which operate on instances of the class they belong to. These methods can
>>> be long and complex. So if I put these methods all in the module file
>>> inside the class, the file will get insanely long. Reading on google, the
>>> answer is usually "refactor", but that really doesn't make sense here.
>>> It's just that the methods are many, and each method can be a long piece
>>> of code. So, is there a way to put these methods in their own files and
>>> have them 'included' in the class somehow? I read a little about mixins
>>> but all the solutions looked very hacky. Is there an official python way
>>> to do this? I don't like having source files with 100's of lines of code
>>> in, let alone 1000's.
>> You googled, found the right answer ("refactor"), didn't like it and are now
>> looking to cure the symptoms of the original problem?
>> Seriously, a good editor can deal with a long source file, but a class with
>> hundreds of methods will bring trouble to any old brain.
> Well, here we disagree. Suppose I have a class which encapsulates time-series data. Below is a list of the absolute minimum methods one would have to process that data. That's close to 100 already before even having any specialised methods for dealing with the data. Each of these methods will have maybe 20 lines of documentation. That's 2000 lines already. And what if someone wants to extend that class to add their own processing methods? It would a maintenance nightmare for them to edit the actual class file, which they would then have to repeat each time a new version of the 'official' class file is released.
Do not make them methods of the time-series class. Make functions that take a time-series object. Then you can organize the functions in separate modules to your heart's content and import them all into a single convenience namespace.
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
On Fri, Nov 2, 2012 at 7:08 PM, Martin Hewitson <martinhewit...@mac.com> wrote:
> On 2, Nov, 2012, at 08:38 AM, Paul Rubin <no.em...@nospam.invalid> wrote:
>> Martin Hewitson <martinhewit...@mac.com> writes:
>>> So, is there a way to put these methods in their own files and have
>>> them 'included' in the class somehow? ... Is there an official python
>>> way to do this? I don't like having source files with 100's of lines
>>> of code in, let alone 1000's.
>> That code sounds kind of smelly... why are there so many methods per
>> class?
> Simply because there are many different ways to process the data. The class encapsulates the data, and the user can process the data in many ways. Of course, one could have classes which encapsulate the algorithms, as well as the data, but it also seems natural to me to have algorithms as methods which are part of the data class, so the user operates on the data using methods of that class.
Are these really needing to be methods, or ought they to be
module-level functions? Remember, Python code doesn't have to be
organized into classes the way Java code is.
> On 2, Nov, 2012, at 09:00 AM, Peter Otten <__pete...@web.de> wrote:
>> Martin Hewitson wrote:
>>> Dear list,
>>> I'm relatively new to Python and have googled and googled but haven't
>>> found a reasonable answer to this question, so I thought I'd ask it
>>> here.
>>> I'm beginning a large Python project which contains many packages,
>>> modules and classes. The organisation of those is clear to me.
>>> Now, the classes can contain many methods (100s of data analysis
>>> methods) which operate on instances of the class they belong to. These
>>> methods can be long and complex. So if I put these methods all in the
>>> module file inside the class, the file will get insanely long. Reading
>>> on google, the answer is usually "refactor", but that really doesn't
>>> make sense here. It's just that the methods are many, and each method
>>> can be a long piece of code. So, is there a way to put these methods in
>>> their own files and have them 'included' in the class somehow? I read a
>>> little about mixins but all the solutions looked very hacky. Is there an
>>> official python way to do this? I don't like having source files with
>>> 100's of lines of code in, let alone 1000's.
>> You googled, found the right answer ("refactor"), didn't like it and are
>> now looking to cure the symptoms of the original problem?
>> Seriously, a good editor can deal with a long source file, but a class
>> with hundreds of methods will bring trouble to any old brain.
> Well, here we disagree. Suppose I have a class which encapsulates
> time-series data. Below is a list of the absolute minimum methods one
> would have to process that data. That's close to 100 already before even
> having any specialised methods for dealing with the data. Each of these
> methods will have maybe 20 lines of documentation. That's 2000 lines
> already. And what if someone wants to extend that class to add their own
> processing methods?
from timeseries import TimeSeries
class MyTimeSeries(TimeSeries):
def average():
# specialised implementation
> It would a maintenance nightmare for them to edit the
> actual class file, which they would then have to repeat each time a new
> version of the 'official' class file is released.
Patient: Doctor, it hurts when I ... Doctor: Then don't do that.
> So maybe some rethinking of this design is needed to handle this
> 'limitation' of python. Perhaps grouping the processing algorithms into
> methods of processing classes, then pass the data objects to these
> methods. But somehow I don't like that. I have the feeling these methods
> would end up peppered with things like:
> if this data type, do this
> else if this data type, do this
> else ....
> normally this would be solved by overloading methods in different data
> subclasses.
You could ask your TimeSeries for the appropriate Statistics subclass
stats = ts.get_stats()
print stats.mean()
where get_stats() is a classmethod that returns an object that provides min(), max(), average() etc.
Another approach are mix-in classes:
class Stats:
def min(): ...
def average(): ...
class SpecialStats(Stats):
def min(): return 42
class TimeSeries(BaseTimeSeries, Stats):
pass
class SpecialTimeSeries(BaseTimeSeries, SpecialStats):
pass
On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamore...@yahoo.co.uk> wrote:
> On 02/11/2012 08:08, Martin Hewitson wrote:
>> Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
>> Thanks for your thoughts,
>> Martin
> 20 lines of documentation per method? As far as I'm concerned that's not a smell, that's a stink.
Wow, I don't think I've ever been criticised before for writing too much documentation :)
I guess we have different end users. This is not a set of classes for other developers to use: it's a set of classes which creates a data analysis environment for scientists to use. They are not programmers, and expect the algorithms to be documented in detail.
> I'm relatively new to Python and have googled and googled but haven't found a reasonable answer to this question, so I thought I'd ask it here.
> I'm beginning a large Python project which contains many packages, modules and classes. The organisation of those is clear to me.
> Now, the classes can contain many methods (100s of data analysis methods) which operate on instances of the class they belong to. These methods can be long and complex. So if I put these methods all in the module file inside the class, the file will get insanely long. Reading on google, the answer is usually "refactor", but that really doesn't make sense here. It's just that the methods are many, and each method can be a long piece of code. So, is there a way to put these methods in their own files and have them 'included' in the class somehow? I read a little about mixins but all the solutions looked very hacky. Is there an official python way to do this? I don't like having source files with 100's of lines of code in, let alone 1000's.
> Many thanks,
> Martin
I have read the other responses, so I may get some flak for encouraging bad habits. Nevertheless, I did have a similar requirement, and I found a solution that worked for me.
My situation was not as extreme as yours. I had a class with a number of methods. Some of them were of an 'operational' nature - they represented the main functionality of the class, and could be called often. Some of them were of a 'setup' nature - they were optionally called when the object was instantiated, but would only be called once.
I found that when I wanted to focus on one set of methods, the other set 'got in my way', and vice-versa. My solution was to take the 'setup' methods and put them in another file. This is how I did it.
On Fri, 02 Nov 2012 08:40:06 +0000, Mark Lawrence wrote:
> On 02/11/2012 08:08, Martin Hewitson wrote:
>> Even if one takes reasonable numbers: 20 methods, each method has 20
>> lines of documentation, then we immediately have 400 lines in the file
>> before writing a line of code. It would seem much more natural to me to
>> have these methods in their own file, grouped nicely in
>> sub-directories. But it seems this is not the python way. Sigh.
>> Thanks for your thoughts,
>> Martin
> 20 lines of documentation per method? As far as I'm concerned that's
> not a smell, that's a stink.
Depends on the method. For some, 20 lines is 18 lines too many. For others, that's 80 lines too few.
Martin Hewitson wrote:
> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamore...@yahoo.co.uk>
> wrote:
>> 20 lines of documentation per method? As far as I'm concerned that's not
>> a smell, that's a stink.
> Wow, I don't think I've ever been criticised before for writing too much
> documentation :)
> I guess we have different end users. This is not a set of classes for
> other developers to use: it's a set of classes which creates a data
> analysis environment for scientists to use. They are not programmers, and
> expect the algorithms to be documented in detail.
While I would never discourage thorough documentation you may be better off with smaller docstrings and the details in an external document. Python projects typically use rst-files processed by sphinx.
> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamore...@yahoo.co.uk> wrote:
>> On 02/11/2012 08:08, Martin Hewitson wrote:
>>> Even if one takes reasonable numbers: 20 methods, each method has 20 lines of documentation, then we immediately have 400 lines in the file before writing a line of code. It would seem much more natural to me to have these methods in their own file, grouped nicely in sub-directories. But it seems this is not the python way. Sigh.
>>> Thanks for your thoughts,
>>> Martin
>> 20 lines of documentation per method? As far as I'm concerned that's not a smell, that's a stink.
> Wow, I don't think I've ever been criticised before for writing too much documentation :)
> I guess we have different end users. This is not a set of classes for other developers to use: it's a set of classes which creates a data analysis environment for scientists to use. They are not programmers, and expect the algorithms to be documented in detail.
> Martin
You've completely missed the point. 99% of the time if you can't write down what a method does in at most half a dozen lines, the method is screaming out to be refactored. Rightly or wrongly you've already rejected that option, although I suspect that rightly is nearer the mark in this case on the grounds that practicality beats purity.
>> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamore...@yahoo.co.uk>
>> wrote:
>>> 20 lines of documentation per method? As far as I'm concerned that's not
>>> a smell, that's a stink.
>> Wow, I don't think I've ever been criticised before for writing too much
>> documentation :)
>> I guess we have different end users. This is not a set of classes for
>> other developers to use: it's a set of classes which creates a data
>> analysis environment for scientists to use. They are not programmers, and
>> expect the algorithms to be documented in detail.
> While I would never discourage thorough documentation you may be better off
> with smaller docstrings and the details in an external document. Python
> projects typically use rst-files processed by sphinx.
In the science/math community, we tend to build the Sphinx API reference from the thorough, authoritative docstrings. We like having complete docstrings because we are frequently at the interactive prompt. We tend to have broad APIs, so having a single source of documentation and not repeating ourselves is important.
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
>> On 2, Nov, 2012, at 09:40 AM, Mark Lawrence <breamore...@yahoo.co.uk> wrote:
>>> On 02/11/2012 08:08, Martin Hewitson wrote:
>>>> Even if one takes reasonable numbers: 20 methods, each method has 20 lines
>>>> of documentation, then we immediately have 400 lines in the file before
>>>> writing a line of code. It would seem much more natural to me to have these
>>>> methods in their own file, grouped nicely in sub-directories. But it seems
>>>> this is not the python way. Sigh.
>>>> Thanks for your thoughts,
>>>> Martin
>>> 20 lines of documentation per method? As far as I'm concerned that's not a
>>> smell, that's a stink.
>> Wow, I don't think I've ever been criticised before for writing too much
>> documentation :)
>> I guess we have different end users. This is not a set of classes for other
>> developers to use: it's a set of classes which creates a data analysis
>> environment for scientists to use. They are not programmers, and expect the
>> algorithms to be documented in detail.
>> Martin
> You've completely missed the point. 99% of the time if you can't write down
> what a method does in at most half a dozen lines, the method is screaming out to
> be refactored. Rightly or wrongly you've already rejected that option, although
> I suspect that rightly is nearer the mark in this case on the grounds that
> practicality beats purity.
You've completely missed the context. These are not really complicated methods doing lots of things all at once such that can be refactored to simpler methods. The docstrings are not just glorified comments for other developers reading the source code. They are the online documentation for non-programmer end-users who are using the interactive prompt as an interactive data analysis environment. Frequently, they not only have to describe what it's doing, but also introduce the whole concept of what it's doing, why you would want to do such a thing, and provide examples of its use. That's why they are so long. For example:
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
> Well, here we disagree. Suppose I have a class which encapsulates
> time-series data. Below is a list of the absolute minimum methods one
> would have to process that data.
Just as a suggestion, you can separate these into categories:
1. Things that modify the data, yielding a different (although derived) data set, e.g. import/load, split, join, plus, minus, zeropad.
2. Things that operate on the data without modifying it, e.g. export/save, average, find, plot, integrate.
The latter can easily be removed from the class. Since they don't touch the content, they can't invalidate internals and can't break encapsulation.
For the former, providing general means to construct or modify the data (like e.g. adding records or joining sequences) is also all that needs to remain inside the class to ensure internal consistency, everything else can be built on top of these using external functions.
> On 2, Nov, 2012, at 08:38 AM, Paul Rubin <no.em...@nospam.invalid>
> wrote:
>> Martin Hewitson <martinhewit...@mac.com> writes:
>>> So, is there a way to put these methods in their own files and
>>> have them 'included' in the class somehow? ... Is there an
>>> official python way to do this? I don't like having source files
>>> with 100's of lines of code in, let alone 1000's.
>> That code sounds kind of smelly... why are there so many methods
>> per class?
> Simply because there are many different ways to process the data. The
> class encapsulates the data, and the user can process the data in
> many ways. Of course, one could have classes which encapsulate the
> algorithms, as well as the data, but it also seems natural to me to
> have algorithms as methods which are part of the data class, so the
> user operates on the data using methods of that class.
This is largely a matter of taste and a question of circumstances, but I'd like to point out here that your "natural" is not universal. If you take a different approach, namely that a class should encapsulate in order to maintain its internal consistency but otherwise be as small as possible, then algorithms operating on some data are definitely not part of that data. The advantage is that the data class gets smaller, and in the algorithms you don't risk ruining the internal integrity of the used data.
Further, encapsulating algorithms into classes is also not natural. Algorithms are often expressed much better as functions. Shoe-horning things into a class in the name of OOP is IMHO misguided.
Concerning mixins, you can put them into separate modules[1]. If it is clearly documented that class FooColourMixin handles the colour-related stuff for class Foo, and reversely that class Foo inherits FooShapeMixin and FooColourMixin that provide handling of shape and colour, then that is fine. It allows you to not only encapsulate things inside class Foo but to partition things inside Foo. Note that mixins are easier to write than in C++. If the mixin needs access to the derived class' function bar(), it just calls self.bar(). There is no type-casting or other magic involved. The same applies to data attributes (non-function attributes), basically all attributes are "virtual". The compile-time, static type checking of e.g. C++ doesn't exist.
>> Python lets you inject new methods into existing classes (this is
>> sometimes called duck punching) but I don't recommend doing this.
> Is there not a way just to declare the method in the class and put
> the actual implementation in another file on the python path so that
> it's picked up a run time?
To answer your question, no, not directly. Neither is there a separation like in C++ between interface and implementation, nor is there something like in C# with partial classes. C++ interface/implementation separation is roughly provided by abstract base classes. C# partial classes are most closely emulated with mixins.
That said, modifying classes is neither magic nor is it uncommon:
class foo:
pass
import algo_x
foo.algo = algo_x.function
Classes are not immutable, you can add and remove things just like you can do with objects.
BTW: If you told us which language(s) you have a background in, it could be easier to help you with identifying the idioms in that language that turn into misconceptions when applied to Python.
Greetings!
Uli
[1] Actually, modules themselves provide the kind of separation that I think you are after. Don't always think "class" if it comes to encapsulation and modularization!
> Am 02.11.2012 09:20, schrieb Martin Hewitson:
>> Well, here we disagree. Suppose I have a class which encapsulates
>> time-series data. Below is a list of the absolute minimum methods one
>> would have to process that data.
> [...]
> > 'abs' 'acos' 'asin' 'atan' 'atan2' 'average' 'cohere' 'conv' 'corr'
> > 'cos' 'cov' 'cpsd' 'detrend' 'dft' 'diff' 'downsample' 'exp'
> > 'export' 'fft' 'fftfilt' 'filter' 'filtfilt' 'find' 'heterodyne'
> > 'hist' 'imag' 'integrate' 'interp' 'join' 'le' 'lincom' 'ln' 'load'
> > 'log' 'log10' 'lscov' 'max' 'mean' 'median' 'min' 'minus' 'mode'
> > 'mpower' 'mrdivide' 'mtimes' 'ne' 'norm' 'or' 'plot' 'plus'
> > 'polyfit' 'power' 'psd' 'rdivide' 'real' 'resample' 'rms' 'round'
> > 'save' 'scale' 'search' 'select' 'sin' 'smoother' 'sort'
> > 'spectrogram' 'split' 'sqrt' 'std' 'sum' 'sumjoin' 'svd' 'tan' 'tfe'
> > 'timeaverage' 'times' 'timeshift' 'transpose' 'uminus' 'upsample'
> > 'zeropad'
> Just as a suggestion, you can separate these into categories:
> 1. Things that modify the data, yielding a different (although derived) data set, e.g. import/load, split, join, plus, minus, zeropad.
> 2. Things that operate on the data without modifying it, e.g. export/save, average, find, plot, integrate.
> The latter can easily be removed from the class. Since they don't touch the content, they can't invalidate internals and can't break encapsulation.
> For the former, providing general means to construct or modify the data (like e.g. adding records or joining sequences) is also all that needs to remain inside the class to ensure internal consistency, everything else can be built on top of these using external functions.
Thank you all so much for your thoughts and suggestions. I need to absorb all of this and decide on the best approach in this case.
> BTW: If you told us which language(s) you have a background in, it could be easier to help you with identifying the idioms in that language that turn into misconceptions when applied to Python.
I'm considering porting some MATLAB code to python to move away from commercial software. Python seemed like the right choice simply because of the wonderful numpy, scipy and matplotlib.
So my project will build on these packages to provide some additional state and functionality.
> [1] Actually, modules themselves provide the kind of separation that I think you are after. Don't always think "class" if it comes to encapsulation and modularization!
> -- > http://mail.python.org/mailman/listinfo/python-list
>> BTW: If you told us which language(s) you have a background in, it could be easier to help you with identifying the idioms in that language that turn into misconceptions when applied to Python.
>> Greetings!
>> Uli
>> [1] Actually, modules themselves provide the kind of separation that I think you are after. Don't always think "class" if it comes to encapsulation and modularization!
>> --
>> http://mail.python.org/mailman/listinfo/python-list
> I'm considering porting some MATLAB code to python to move away from commercial software. Python seemed like the right choice simply because of the wonderful numpy, scipy and matplotlib.
> So my project will build on these packages to provide some additional state and functionality.
> Cheers,
> Martin
Just in case you're not aware there are separate mailing lists for numpy and matplotlib, presumably scipy as well, should you have specific questions. Further matplotlib is now at version 1.2rc3 with Python 3 support, yippee. Combine that with the recently released Python 3.3 and it should make one hell of a tool kit :)
Martin Hewitson <martinhewit...@mac.com> writes:
> Well, here we disagree. Suppose I have a class which encapsulates
> time-series data. Below is a list of the absolute minimum methods one
> would have to process that data. ... > 'abs'
> 'acos'
> 'asin'
> ...
Ok, THERE is your problem. Why do you have separate implementations of
all those functions? Does the abs of a time series simply mean the abs
of each element of the series? In that case you want just ONE method,
something like "map", which applies an arbitrary function to all
elements of the series. Then for time series ts, instead of saying
ts.abs(), you'd say ts.map(abs) where abs is the existing, built-in
absolute value function. You could similarly say ts.map(acos) etc.
That gets rid of almost all of those methods.
> Martin Hewitson <martinhewit...@mac.com> writes:
>> Well, here we disagree. Suppose I have a class which encapsulates
>> time-series data. Below is a list of the absolute minimum methods one
>> would have to process that data. ... >> 'abs'
>> 'acos'
>> 'asin'
>> ...
> Ok, THERE is your problem. Why do you have separate implementations of
> all those functions? Does the abs of a time series simply mean the abs
> of each element of the series? In that case you want just ONE method,
> something like "map", which applies an arbitrary function to all
> elements of the series. Then for time series ts, instead of saying
> ts.abs(), you'd say ts.map(abs) where abs is the existing, built-in
> absolute value function. You could similarly say ts.map(acos) etc.
> That gets rid of almost all of those methods.
Well, because one of the features that the framework will have is to capture history steps (in a tree structure) so that each processing step the user does is tracked. So while methods such as abs(), cos(), etc will eventually just call a built-in method, there will be some house-keeping around them. All that said, as I've been trying to implement this structure, it turns out that in Python, this is more naturally achieved (I think) if each algorithm is implemented as a class, so that each algorithm can return its set of supported parameters for validation against the user inputs and, ultimately, for inclusion in a step in the history tree. Since most of that infrastructure will turn out to be boiler-plate code, it would make sense to have an algorithm base class, which all other algorithms (abs, cos, etc) will inherit from. Then I just need to get my head around the interplay between these algorithm classes and the data classes. Some more prototyping needed.
Thanks for the info about map(); this will likely turn out to be very useful, if not at the user level, at least within the framework. Again, a main requirement is that the users should be able to work without knowing much about Python or programming in general; they just get this toolkit and, after minimal training should be able to do most of what they want in an intuitive way.
Martin Hewitson <martinhewit...@mac.com> writes:
>> you want just ONE method, something like "map"...
> Well, because one of the features that the framework will have is to
> capture history steps (in a tree structure) so that each processing
> step the user does is tracked. So while methods such as abs(), cos(),
> etc will eventually just call a built-in method, there will be some
> house-keeping around them.
Make the "map" wrapper do the house-keeping.
> turns out that in Python, this is more naturally achieved (I think) if
> each algorithm is implemented as a class, so that each algorithm can
> return its set of supported parameters for validation against the user
> inputs and, ultimately, for inclusion in a step in the history
> tree. Since most of that infrastructure will turn out to be
> boiler-plate code, it would make sense to have an algorithm base
> class, which all other algorithms (abs, cos, etc) will inherit
> from.
That sounds like over-use of classes and inheritance. It's probably
easiest to just use a dictionary with functions in it (untested):
On Fri, 02 Nov 2012 07:16:09 +0100, Martin Hewitson wrote:
> I'm beginning a large Python project which contains many packages,
> modules and classes. The organisation of those is clear to me.
[...]
> I don't like having source files with
> 100's of lines of code in, let alone 1000's.
Why? Do you do your software development on an iPhone?
Hundreds of lines is nothing, especially if you are including docstrings and comments in that count. My Python startup file is over 100 lines, and it is trivial.
100 lines is approximately and a half pages using a 10pt font size (depending on the font and the platform, of course). In any case, it's not a lot. If you average two lines of code every comment, blank line or docstring, 100 lines is only 66 lines of actual code.
To give you an example of what I consider pushing the limit of how much code should go into a single file, look at the source code to decimal.py. In Python 3.2, that file is 6249 lines. There are:
- 2880 lines of actual code
- 2070 lines in docstrings, including blanks
- 606 commented lines
- 693 blank lines outside of docstrings
The module includes 20 top-level functions, 18 classes, and 213 methods. Most of those methods are in just two classes, Context and Decimal, with 76 and 117 methods respectively. So there is an average of 12 lines of code per function or method.
I wouldn't like to have to deal with a single file twice that size, but decimal.py is relatively easy to deal with. It's not a trivial module by any means, but nor is it especially over-complex. The complexity is all in the computations, not the code layout.
Please understand me -- I'm not saying that you should stuff all your code into a single module. Code should only be placed into a single module when the code is related. But in my opinion, you should not split a logical single unit of code, a module of related code, into separate files just because it is "too big" until it is at least as large as decimal.py.
