import and package confusion
Dale Amon
admit I do not understand what is going on with
import of hierarchical packages/modules. Perhaps someone
can get me on the road again. Here is a subset of what I
am trying to accomplish:
The package directory set up:
VLMLegacy/
__init__.py
Reader.py
Conditions.py
VLM4997/
__init__.py
Conditions.py
WINGTL/
__init__.py
Conditions.py
The inheritance:
object
Reader
Conditions
VLM4997.Conditions
WINGTL.Conditions
Now how do I use import or from to be able to
use these modules? The following is not 'real'
code and is only intended to give some idea of
what I am trying to accomplish:
import sys
sys.path.extend (['../lib', '../bin'])
import VLMLegacy.VLM4997.Conditions
import VLMLegacy.WINGTL.Conditions
b = VLM4997.Conditions(2)
b.test()
c = WINGTL.Conditions(2)
c.test()
And of course note that both of those must inherit
VLMLegacy.Conditions().
Scott David Daniels
>
> b = VLM4997.Conditions(2)
I see some confusion here.
(1) This should give a NameError('VLM4997')
(2) Even if changed to b = VLMLegacy.VLM4997.Conditions(2),
You are calling an imported module, rather than a function or class.
So, using my crystal ball (which came back from the shop covered
in some sticky fluid), I guess you might mean:
import VLMLegacy.VLM4997.Conditions.Conditions as VLM4997_Conditions
import VLMLegacy.WINGTL.Conditions.Conditions as WINGTL_Conditions
...
b = VLM4997_Conditions(2)
b.test()
...
--Scott David Daniels
Scott....@Acm.Org
Dale Amon
missing. Is it the case that if you have a module C in a
package A:
A.C
that there is no way to load it such that you can use:
x = A.C()
in your code? This is just a simpler case of what I'm
trying to do now, which has a module C in a sub-package
to be imported:
A.B.C
ie with files:
mydir/A/B/C.py
mydir/mymain.py
and executed in mymain.py as:
x = A.B.C()
I may still chose to do it the way you suggested, but I
would still like to understand why this does not work.
Scott David Daniels
> I am trying to get to the heart of what it is I am
> missing. Is it the case that if you have a module C in a
> package A:
> A.C
> that there is no way to load it such that you can use:
> x = A.C()
> in your code?
import sys
sys()
sys is a module, and as such, it is not callable.
Just because you put a class inside a module, does not mean
that class magically does something by virtue of having the
same name as the module.
A module is a namespace to hold classes, functions, etc....
A package is a namespace to hold modules (possibly more).
I don't understand why you don't use files like:
VLMLegacy/
__init__.py
Reader.py
VLM4997.py
WINGTL.py
But, presuming some kind of rationale,
put the code you want in
VLMLegacy/VLM4997/__init__.py
--Scott David Daniels
Scott....@Acm.Org
--Scott David Daniels
Scott....@Acm.Org
Dale Amon
> Dale Amon wrote:
>> missing. Is it the case that if you have a module C in a package A:
>> A.C
>> that there is no way to load it such that you can use:
>> x = A.C()
>> in your code?
> OK, here's a simple question. What do you expect from:
> import sys
> sys()
> sys is a module, and as such, it is not callable.
> Just because you put a class inside a module, does not mean
> that class magically does something by virtue of having the
> same name as the module.
>
> A module is a namespace to hold classes, functions, etc....
> A package is a namespace to hold modules (possibly more).
>
> I don't understand why you don't use files like:
>
> VLMLegacy/
> __init__.py
> Reader.py
> VLM4997.py
> WINGTL.py
Well, it is far more complex than that: I just cut it down to
the most minimal case I could.
> But, presuming some kind of rationale,
> put the code you want in VLMLegacy/VLM4997/__init__.py
That doesn't really do it. Perhaps I should try to describe
the situation better.
There are n different similar systems, each with multiple classes.
They could either be implimented as a class at the first level:
VLMLegacy
Condition.py
Plan.py
|
|
etc
but in that case each class will be filled with conditionals
that try to do the correct thing depending on which system's
data they are reading. That approach has already gotten *insane*
and I need to objectify things: put all the common code into
abstract superclasses, and then create a subclass for each
different system (of which there will be an unknown number added
over time), ie:
VLMLegacy/
Conditions.py Abstract classes
Plan.py
|
|
etc
TYPE1/ Subclasses of above specific to Type 1
Conditions.py
Plan.py
|
|
etc
TYPE2/ Subclasses for Type 2
Conditions.py
Plan.py
|
|
etc
|
TYPEn/ Subclasses for Type n
Conditions.py
Plan.py
|
|
etc
Every VLMLegacy.TYPEn.Conditions (or other class) has exactly
the same set of methods; each of those methods inherits much
of its basic behavior from VLMLegacy.Conditions.
If I make every subclass a unique name, things will
rapidly get out of hand, especially when I start
adding TYPEn+1,2... etc.
So yes, the approach isn't arbitrary, it is a solution
to real design problems which even the above does not
fully do justice to.
What I would really like to do when executing is more
like:
type = "VLM4997"
type.Header(args)
type.Plan(args)
type.Conditions(args)
Where the type might change from execution to execution
or even on different iterations.
Dale Amon
> type = "VLM4997"
> type.Header(args)
> type.Plan(args)
> type.Conditions(args)
> Where the type might change from execution to execution
> or even on different iterations.
Actually let me make that reflect more accurately what
is going on:
obj = Deck(rdr)
obj.header = type.Header(rdr)
obj.plan[0] = type.Plan(rdr)
obj.plan[1] = type.Plan(rdr)
obj.cond = type.Conditions(rdr)
obj.cond.calcsomething(args)
and so forth through many pages of code...
Scott David Daniels
> On Wed, Apr 29, 2009 at 01:12:33PM -0700, Scott David Daniels wrote:
>
>>
>>> I am trying to get to the heart of what it is I am
>>> missing. Is it the case that if you have a module C in a package A:
>>> A.C
>>> that there is no way to load it such that you can use:
>>> x = A.C()
>>> in your code?
>>>
>> OK, here's a simple question. What do you expect from:
>> import sys
>> sys()
>> sys is a module, and as such, it is not callable.
>>
of your misunderstanding. A class and a module are _not_the_same_thing_.
sys is not a package, it is a module.
>> Just because you put a class inside a module, does not mean
>> that class magically does something by virtue of having the
>> same name as the module.
>>
>> A module is a namespace to hold classes, functions, etc....
>> A package is a namespace to hold modules (possibly more).
>>
>> I don't understand why you don't use files like:
>>
>> VLMLegacy/
>> __init__.py
>> Reader.py
>> VLM4997.py
>> WINGTL.py
several classes, several functions, several constants....
> Every VLMLegacy.TYPEn.Conditions (or other class) has exactly...
>
But VLMLegacy.TYPEn.Conditions is a _module_, not a _class_.
I suggest VLM4497.py look a bit like the following:
from VLMLegacy import abstract_classes
class Condition(abstract_classes.BaseCondition):
...
class Plan(abstract_classes.BasePlan):
...
Header = abstract_classes.BaseHeader # Note we needed no customization
...
> the same set of methods; each of those methods inherits much
> of its basic behavior from VLMLegacy.Conditions.
>
> If I make every subclass a unique name, things will
> rapidly get out of hand, especially when I start
> adding TYPEn+1,2... etc.
>
> So yes, the approach isn't arbitrary, it is a solution
> to real design problems which even the above does not
> fully do justice to.
>
> What I would really like to do when executing is more
> like:
>
> type = "VLM4997"
> type.Header(args)
> type.Plan(args)
> type.Conditions(args)
>
> Where the type might change from execution to execution
> or even on different iterations.
>
Well, "VLM4997" is a _string_, and it has no attributes (nor methods)
named "Header", "Plan", or "Conditions." And "type" is a perfectly awful
name for a variable, since it hides the builtin named type. You seem to
confuse names, files, and classes defined in files (at least in your
writing).
--Scott David Daniels
Scott....@Acm.Org
Dale Amon
> You did not answer the question above, and I think the answer is the root
> of your misunderstanding. A class and a module are _not_the_same_thing_.
> sys is not a package, it is a module.
>>> Just because you put a class inside a module, does not mean
>>> that class magically does something by virtue of having the
>>> same name as the module.
>>>
>>> A module is a namespace to hold classes, functions, etc....
>>> A package is a namespace to hold modules (possibly more).
>>>
>>> I don't understand why you don't use files like:
>>>
>>> VLMLegacy/
>>> __init__.py
>>> Reader.py
>>> VLM4997.py
>>> WINGTL.py
> Unlike Java, we are free to have several things in a module:
> several classes, several functions, several constants....
These modules would grow to be hundreds of pages long and a
difficult to deal with to debug a problem related to one obscure
system without looking at (or potentially screwing up) any of the
others. I prefer one class per module.
This gets more into philosophy, but I figure any function or method
that does not fit on one page is too big; and any source file that
is more than 20 pages long should be broken in half. I like my modules
in the 5-10 page size range, including the embedded Unix ManPages and
the cvs history. But that's just my house style.
> Well, "VLM4997" is a _string_, and it has no attributes (nor methods)
> named "Header", "Plan", or "Conditions." And "type" is a perfectly awful
> name for a variable, since it hides the builtin named type. You seem to
> confuse names, files, and classes defined in files (at least in your
> writing).
Actually I'm not. I am simply trying to use a pseudo code
to explain roughly what is going on. There will be a string
that selects what the set of classes are to be used on any
given iteration and it will be used to generate the name
of the class and/or name of the module where it is to be found.
I'm an old ObjC hacker. I often put the class or method in a
variable and do the bindings at runtime. I am already doing some
of that sort of thing in this system with the method names and
it works nicely.
The point I take away from this is that packages and
modules have dotted names, but Classes do not and there
is no way to do exactly what I wanted to do.
The dot syntax would have been quite nice (I quite like the
"::" syntax in Perl) and would have made the code much
clearer. The way you suggested with a 'typename_classname'
generated using a from/import statement will just have to
suffice.
Scott David Daniels
> ....
> The point I take away from this is that packages and
> modules have dotted names, but Classes do not and there
> is no way to do exactly what I wanted to do.
of the lack of clarity is your imprecision about names.
If you insist on having a class per module, you will
always have redundant-looking class names somewhere.
You will help yourself out a lot by not sharing the class
name and the base class name (not the least in error
messages), but it is possible to have them the same.
If your module name, base class name, specialized name,
and derived class name are all the same, there are _way_
too many chances to confuse yourself about exactly what
you are talking about.
Assuming:
VLMLegacy/
__init__.py
Reader.py [defines class Reader]
Conditions.py [defines class Conditions]
...
VLM4997/
__init__.py
Reader.py [defines class Reader]
Conditions.py [defines class Conditions]
WINGTL/
__init__.py
Conditions.py
Reader.py [defines class Reader]
Conditions.py [defines class Conditions]
After
import VLMLegacy.Reader
import VLM4997.Reader
There are four things named Reader:
VLMLegacy.Reader # A module
VLMLegacy.Reader.Reader # A class
VLM4997.Reader # a module
VLM4997.Reader.Reader # a class
How I'd do it to "sort of" fit your style:
VLMLegacy/
__init__.py
base_reader.py [defines class BaseReader]
base_conditions.py [defines class BaseConditions]
...
VLM4997/
__init__.py
reader.py [defines class Reader]
conditions.py [defines class Conditions]
WINGTL/
__init__.py
reader.py [defines class Reader]
conditions.py [defines class Conditions]
I'd make the VLMLegacy/__init__.py file read:
from VLMLegacy.base_reader import BaseReader
from VLMLegacy.base_conditions import BaseConditions
...
I'd make VLM4997/reader.py read:
import VLMLegacy
class Reader(VLMLegacy.BaseReader):
...
I'd make VLM4997/__init__.py read:
from VLMLegacy.VLM997.reader import Reader
from VLMLegacy.VLM997.conditions import Conditions
...
The the consuming python code can do:
import VLMLegacy.VLM997
something = VLMLegacy.VLM997.Reader()
Dale Amon
and just so you can see:
#!/usr/bin/python
import sys
sys.path.extend (['../lib', '../bin'])
from VLMLegacy.CardReader import CardReader
rdr = CardReader ("../example/B767.dat","PRINTABLE")
iotypes = ["WINGTL","VLMPC","VLM4997"]
for iotype in iotypes:
packagename = "VLMLegacy." + iotype + ".Conditions"
classname = iotype + "_Conditions"
code = "from %s import Conditions as %s" \
% (packagename, classname)
x = compile (code,"foo","exec")
exec x
cls = globals()[classname]
a = cls(rdr,2)
a.test()
Dale Amon
> Dale Amon wrote:
>> ....
>> The point I take away from this is that packages and
>> modules have dotted names, but Classes do not and there
>> is no way to do exactly what I wanted to do.
> Nope. You have not been clear with what you want, and part
> of the lack of clarity is your imprecision about names.
>
> If you insist on having a class per module, you will
> always have redundant-looking class names somewhere.
> You will help yourself out a lot by not sharing the class
> name and the base class name (not the least in error
> messages), but it is possible to have them the same.
That in particular may happen. This has all been a matter
of running tests to see how close I could get to the
desired concept using Python. With some working test code
I have now, the answer is 'close enough'.
Your assistance has been useful, regardless of whether
it sounded that way or not ;-)
Dave Angel
> The point I take away from this is that packages and
> modules have dotted names, but Classes do not and there
> is no way to do exactly what I wanted to do.
>
> "::" syntax in Perl) and would have made the code much
> clearer. The way you suggested with a 'typename_classname'
> generated using a from/import statement will just have to
> suffice.
>
>
>
predictably, and quite flexibly. The problem was that by using the same
name for module and class, you didn't realize you needed to include both.
If a particular module is called mymodule, and you want to use a class
within it called myclass, you can do it readily with:
import mymodule
obj = mymodule.myclass(arg1, arg2)
If you have the same class name in more than one module, you can make
either one simply by doing:
obj = mymodule1.myclass(arg1, arg2)
obj = .mymlodule2.myclass(arg1, arg2)
And in particular if you simply do the following, you can choose between
those modules:
if test:
mod = mymodule1
else:
mod = mymodule2
obj = mod.myclass(arg1, arg2)
Now the same logic would work if you added packages to the mix, and if
you happened to call the class the same thing as the module. But it
would be very confusing in a message like this, so I choose not to.
The compiler doesn't care if the names are the same, but when you get
scoping wrong, the error messages will be confusing.
Please don't sink to exec or eval to solve what is really a
straightforward problem.
Dale Amon
> The dot syntax works very
> predictably, and quite flexibly. The problem was that by using the same
> name for module and class, you didn't realize you needed to include both.
It is one of the hazards of working in many very different
languages. But I see where that confusion lies and that is
a useful thing to know.
> And in particular if you simply do the following, you can choose between
> those modules:
>
> if test:
> mod = mymodule1
> else:
> mod = mymodule2
> obj = mod.myclass(arg1, arg2)
Not really applicable to the case I have. There can be lots of
different ones and the input selection comes from a command line
string so...
> Please don't sink to exec or eval to solve what is really a
> straightforward problem.
I do not really see any other way to do what I want. If
there is a way to get rid of the exec in the sample code
I have used, I would love to know... but I can't see how
to import something where part of the name comes from user
command line input without interpreting the code via exec.
[See the test module I posted.] I'm dealing with something
like this:
myprogram --type WINGTL file.dat
the set of types will have new members added as they are
discovered and I intend to minimize code changes to doing
nothing but create a subpackage directory with the new
modules, drop it in place. New functionality with no
mods to the existing code...
Scott David Daniels
> On Wed, Apr 29, 2009 at 10:02:46PM -0400, Dave Angel wrote:
>> Please don't sink to exec or eval to solve what is really a
>> straightforward problem.
>
> I do not really see any other way to do what I want. If
> there is a way to get rid of the exec in the sample code
> I have used, I would love to know... but I can't see how
> to import something where part of the name comes from user
> command line input without interpreting the code via exec.
at this point it would help to know what version of Python
you are using. If importlib is available, use that. Otherwise
you should investigate __import__.
--Scott David Daniels
Scott....@Acm.Org
alex23
> I do not really see any other way to do what I want. If
> there is a way to get rid of the exec in the sample code
> I have used, I would love to know... but I can't see how
> to import something where part of the name comes from user
> command line input without interpreting the code via exec.
Are you familiar with __import__?
iotypes = ["WINGTL","VLMPC","VLM4997"]
for iotype in iotypes:
packagename = "VLMLegacy." + iotype + ".Conditions"
classname = iotype + "_Conditions"
module = __import__(packagename)
cls = getattr(module, classname)
# etc
Much cleaner :) And far more secure: any time you think "user input"
AND "exec", you're probably going in the wrong direction...
Arnaud Delobelle
> I do not really see any other way to do what I want. If
> there is a way to get rid of the exec in the sample code
> I have used, I would love to know... but I can't see how
> to import something where part of the name comes from user
> command line input without interpreting the code via exec.
__import__
--
Arnaud
Jeroen Ruigrok van der Werven
>import sys
>sys.path.extend (['../lib', '../bin'])
>
>rdr = CardReader ("../example/B767.dat","PRINTABLE")
>
>for iotype in iotypes:
> packagename = "VLMLegacy." + iotype + ".Conditions"
> classname = iotype + "_Conditions"
> % (packagename, classname)
> x = compile (code,"foo","exec")
> exec x
> cls = globals()[classname]
> a = cls(rdr,2)
> a.test()
>
Although I can applaud your creativity, this really is very hackish for
Python and definitely has a bad 'smell'.
Like you I work with many different languages, but I am not going to force
the natural flow of Python into something resembling Java or whatever
other language.
Simply doing:
Right now your code boils down to:
from VLMLegacy.VLM4997.Conditions import Conditions as VLM4997_Conditions
from VLMLegacy.VLMPC.Conditions import Conditions as VLMPC_Conditions
from VLMLegacy.WINGTL.Conditions import Conditions as WINGTL_Conditions
And while you are, of course, allowed to do so, it's not the way you would
want to approach it in Python.
For each subpackage/module you could add an import and __all__ to
__init__.py to expose Conditions and then shorten it all to:
import VLMLegacy.VLM4997 as VLM4997
import VLMLegacy.VLMPC as VLMPC
import VLMLegacy.WINGTL as WINGTL
So that you can do:
a = VLM4997.Conditions(rdr, 2)
a.test()
--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
イェルーン ラウフロック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
May you get - not what you deserve - but your heart's desire...
Dave Angel
> On Wed, Apr 29, 2009 at 10:02:46PM -0400, Dave Angel wrote:
>
>> predictably, and quite flexibly. The problem was that by using the same
>> name for module and class, you didn't realize you needed to include both.
>>
>
> It is one of the hazards of working in many very different
> languages. But I see where that confusion lies and that is
> a useful thing to know.
>
>
>> And in particular if you simply do the following, you can choose between
>> those modules:
>>
>> if test:
>> mod = mymodule1
>> else:
>> mod = mymodule2
>> obj = mod.myclass(arg1, arg2)
>>
>
> Not really applicable to the case I have. There can be lots of
> different ones and the input selection comes from a command line
> string so...
>
>
>> straightforward problem.
>>
>
> there is a way to get rid of the exec in the sample code
> I have used, I would love to know... but I can't see how
> to import something where part of the name comes from user
> command line input without interpreting the code via exec.
> like this:
>
> myprogram --type WINGTL file.dat
>
> the set of types will have new members added as they are
> discovered and I intend to minimize code changes to doing
> nothing but create a subpackage directory with the new
> modules, drop it in place. New functionality with no
> mods to the existing code...
>
>
>
Python version. If you can use __import__(), then realize that
mod = __import__("WINGTL")
will do an import, using a string as the import name. I don' t have the
experience to know how it deals with packages, but I believe I've heard
it does it the same as well.
One more possibility, if you're only trying to get a single package
hierarchy at a time, it might be possible to arrange them in such a way
that the sys.path search order gets you the package you want. Rather
than the top level being a package, it'd be an ordinary directory, and
you'd add it to the sys.path variable so that when you import a
subpackage (which would now really just be a package), you'd get the
right one.
DaveA
Gabriel Genellina
> On Apr 30, 1:10�pm, Dale Amon <a...@vnl.com> wrote:
>> I do not really see any other way to do what I want. If
>> there is a way to get rid of the exec in the sample code
>> I have used, I would love to know... but I can't see how
>> to import something where part of the name comes from user
>> command line input without interpreting the code via exec.
>
> Are you familiar with __import__?
>
> iotypes = ["WINGTL","VLMPC","VLM4997"]
> for iotype in iotypes:
> packagename = "VLMLegacy." + iotype + ".Conditions"
> classname = iotype + "_Conditions"
> module = __import__(packagename)
> cls = getattr(module, classname)
> # etc
(doesn't work as written, because __import__ returns the top package when
given a dotted name)
Replace the last three lines with:
__import__(packagename)
module = sys.modules[packagename]
cls = getattr(module, "Conditions")
--
Gabriel Genellina
Dale Amon
-On [20090430 02:21], Dale Amon (am...@vnl.com) wrote:
>>sys.path.extend (['../lib', '../bin'])
>>
>>rdr = CardReader ("../example/B767.dat","PRINTABLE")
>>
>>for iotype in iotypes:
>> packagename = "VLMLegacy." + iotype + ".Conditions"
>> classname = iotype + "_Conditions"
>> % (packagename, classname)
>> x = compile (code,"foo","exec")
>> exec x
>> cls = globals()[classname]
>> a = cls(rdr,2)
>> a.test()
>
>
> from VLMLegacy.VLM4997.Conditions import Conditions as VLM4997_Conditions
> from VLMLegacy.VLMPC.Conditions import Conditions as VLMPC_Conditions
> from VLMLegacy.WINGTL.Conditions import Conditions as WINGTL_Conditions
>
> And while you are, of course, allowed to do so, it's not the way you would
> want to approach it in Python.
>
> For each subpackage/module you could add an import and __all__ to
> __init__.py to expose Conditions and then shorten it all to:
>
> import VLMLegacy.VLM4997 as VLM4997
> import VLMLegacy.VLMPC as VLMPC
> import VLMLegacy.WINGTL as WINGTL
>
> So that you can do:
>
> a = VLM4997.Conditions(rdr, 2)
> a.test()
If my proof of concept test code were actually all there was
you would be correct. But what I wish to accomplish is that
a string supplied from the command line does a run time load
of code that is not even explicitely mentioned in the main
body of the system.
myprogram --type NEWTYPE old.dat
Where NEWTYPE did not exist when the above code was written and
distributed. Think of the following scenario.
* Customer tells me, we have old data decks which are not
quite in any of the supported formats.
* I supply a new subpackage NEWTYPE with the varient code.
* They copy it into the package directory, VLMLegacy/NEWTYPE.
* They type the name of that package as a command line arg as
above.
* The code I wrote takes their string and dynamically binds
and uses the new code without changing anything else in
the code base.
Now I agree it is hackish. I don't really want to eval,
I just want to import a module whose name is contained
in a variable. I'm not unhappy with the second part where
I use globals()[thestring] to get from a string to a
class object; I am indeed bothered that I have to eval
to do the import as I have been unable so far to find
a way to make it work dynamically. I'd be perfectly happy
if something like this worked:
from VLMLegacy.CardReader import *
opts, args = p.parse_args()
iotype = opts.type
targetname = "Deck"
packagename = "VLMLegacy." + iotype + "." targetname
classname = iotype + "_" + targetname
# I only wish something like this worked...
from packagename import targetname as classname
cls = globals()[classname]
file = args[0]
rdr = CardReader(file,opts.punchtype)
a = cls(rdr,2)
a.read()
<do minutes or hours of calculations>
but it doesn't. Oh, and it gets worse later. The data I'm
reading is potentially straight off ancient Fortran decks
with no spaces between numbers. ;-)
Dale Amon
> As Scott David Daniels says, you have two built-in choices, depending on
> Python version. If you can use __import__(), then realize that
> mod = __import__("WINGTL")
>
> will do an import, using a string as the import name. I don' t have the
> experience to know how it deals with packages, but I believe I've heard
> it does it the same as well.
> One more possibility, if you're only trying to get a single package
> hierarchy at a time, it might be possible to arrange them in such a way
> that the sys.path search order gets you the package you want. Rather
> than the top level being a package, it'd be an ordinary directory, and
> you'd add it to the sys.path variable so that when you import a
> subpackage (which would now really just be a package), you'd get the
> right one.
That would be too unpredictable. But I find the first option
very interesting. I was looking at the __import__ in the Python
book and thought it *might* be able to do it, but I was not sure
if it was a solution or an enticing event horizon.
I'm using 2.5 btw.
Dale Amon
> En Thu, 30 Apr 2009 03:04:40 -0300, alex23 <wuw...@gmail.com> escribió:
>> Are you familiar with __import__?
>>
>> iotypes = ["WINGTL","VLMPC","VLM4997"]
>> for iotype in iotypes:
>> packagename = "VLMLegacy." + iotype + ".Conditions"
>> classname = iotype + "_Conditions"
>> module = __import__(packagename)
>> cls = getattr(module, classname)
>> # etc
>
> given a dotted name)
> Replace the last three lines with:
>
> __import__(packagename)
> module = sys.modules[packagename]
> cls = getattr(module, "Conditions")
Thanks. That works marvelously. I just knew there
had to be a better way.
MRAB
> On Thu, Apr 30, 2009 at 08:32:31AM +0200, Jeroen Ruigrok van der Werven wrote:
> -On [20090430 02:21], Dale Amon (am...@vnl.com) wrote:
>>> sys.path.extend (['../lib', '../bin'])
>>>
>>> rdr = CardReader ("../example/B767.dat","PRINTABLE")
>>>
>>> for iotype in iotypes:
>>> packagename = "VLMLegacy." + iotype + ".Conditions"
>>> classname = iotype + "_Conditions"
cls = getattr(__import__(packagename), targetname)
Dale Amon
just to put a bow tie on this thread:
#!/usr/bin/python
import sys
sys.path.extend (['../lib', '../bin'])
from VLMLegacy.CardReader import CardReader
rdr = CardReader ("../example/B767.dat","PRINTABLE")
iotypes = ["WINGTL","VLMPC","VLM4997"]
for iotype in iotypes:
classname = "Conditions"
__import__("VLMLegacy." + iotype + "." + classname)
module = sys.modules[packagename]
cls = getattr(module, classname)
a = cls(rdr,2)
a.test()
Works like a champ!
It would have taken days for me to find that by trial and
error and rtfm and google. So thank you all. Even if at times
I was rather unclear about what I was trying to accomplish.
Now I can move on to parsing those pesky Fortran card
images... There wouldn't happen to be a way to take n
continguous slices from a string (card image) where each
slice may be a different length would there? Fortran you
know. No spaces between input fields. :-)
I know a way to do it, iterating over a list of slice sizes,
perhaps in a list comprehension, but some of the august python
personages here no doubt know better ways.
Terry Reedy
> Now I can move on to parsing those pesky Fortran card
> images... There wouldn't happen to be a way to take n
> continguous slices from a string (card image) where each
> slice may be a different length would there? Fortran you
> know. No spaces between input fields. :-)
>
> I know a way to do it, iterating over a list of slice sizes,
Yes.
> perhaps in a list comprehension, but some of the august python
> personages here no doubt know better ways.
No. Off the top of my head, here is what I would do something like
(untested)
def card_slice(card, sizes):
"Card is data input string. Sizes is an iterable of int field sizes,
where negatives are skipped fields. Return list of strings."
pos, ret = 0, []
for i in sizes:
if i > 0:
ret.append(card[pos:pos+i])
else:
i = -i
pos += i
return ret
To elaborate this, make sizes an iterable of (size, class) pairs, where
class is str, int, or float (for Fortran) or other for more generel use.
Then
...
for i,c in sizes:
if i > 0:
ret.append(c(card[pos:pos+i]))
Terry Jan Reedy
MRAB
> Dale Amon wrote:
>
>> Now I can move on to parsing those pesky Fortran card
>> images... There wouldn't happen to be a way to take n
>> continguous slices from a string (card image) where each slice may be
>> a different length would there? Fortran you know. No spaces between
>> input fields. :-)
>>
>> I know a way to do it, iterating over a list of slice sizes,
>
> Yes.
>
>> perhaps in a list comprehension, but some of the august python
>> personages here no doubt know better ways.
>
> No. Off the top of my head, here is what I would do something like
> (untested)
>
> def card_slice(card, sizes):
> "Card is data input string. Sizes is an iterable of int field sizes,
> where negatives are skipped fields. Return list of strings."
> pos, ret = 0, []
> for i in sizes:
> if i > 0:
> ret.append(card[pos:pos+i])
> else:
> i = -i
> pos += i
I would shorten that a little to:
if i > 0:
ret.append(card[pos:pos+i])
pos += abs(i)
> return ret
>
> To elaborate this, make sizes an iterable of (size, class) pairs, where
> class is str, int, or float (for Fortran) or other for more generel use.
> Then
> ...
> for i,c in sizes:
> if i > 0:
> ret.append(c(card[pos:pos+i]))
>
> Terry Jan Reedy
>
norseman
> Dale Amon wrote:
>
>> Now I can move on to parsing those pesky Fortran card
>> images... There wouldn't happen to be a way to take n
>> continguous slices from a string (card image) where each slice may be
>> a different length would there? Fortran you know. No spaces between
>> input fields. :-)
>>
>> I know a way to do it, iterating over a list of slice sizes,
>
> Yes.
>
>> perhaps in a list comprehension, but some of the august python
>> personages here no doubt know better ways.
>
> No. Off the top of my head, here is what I would do something like
> (untested)
>
> def card_slice(card, sizes):
> "Card is data input string. Sizes is an iterable of int field sizes,
> where negatives are skipped fields. Return list of strings."
> pos, ret = 0, []
> for i in sizes:
> if i > 0:
> ret.append(card[pos:pos+i])
> else:
> i = -i
> pos += i
>
> To elaborate this, make sizes an iterable of (size, class) pairs, where
> class is str, int, or float (for Fortran) or other for more generel use.
> Then
> ...
> for i,c in sizes:
> if i > 0:
> ret.append(c(card[pos:pos+i]))
>
> Terry Jan Reedy
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
Terry is on right track.
I use this:
* 1#
comment="""
Structure for TYPE III file named <tmp_att.dbf>
Number of bytes per record : 129
Number of fields in record : 25
Number of records in file : 0
Date file was last updated : 11/20/ 8
Field Label Type Size/Dec. Offset
1 AREA N 14 3 1
2 PERIMITER N 14 3 15
3 DUMMY1 N 11 0 29
4 DUMMY2 N 11 0 40
5 BL_X N 12 0 51
6 BL_Y N 12 0 63
7 ACRES C 13 0 75
.
.
"""
Then:
open file
skip any header bytes
while not eof:
read full record (129 bytes in this case) into buffer (tstln)
#parse:
PERIMITER= tstln[15:29]
DUMMY1= tstln[29:40]
DUMMY2= tstln[40:51]
BL_X= tstln[51:63]
BL_Y= tstln[63:75]
ACRES= tstln[75:88]
.
.
do stuff
#loop
The other method he mentions works too. I create an Ashton-Tate dBASE
III+ from a template file, add the structure I want and populate it
from, well... a wide variety of sources. The 'header' has all the field
names and sizes and such. I can then use the above manual method or let
the program do the parsing. The second method is much more flexible.
import StringIO
source1= StringIO.StringIO("""(This file must be converted with
BinHex 4.0)
:#h0[GA*MC6%ZC'*Q!$q3#!#3"!4L!*!%'H!$#!B$!*!%B36M!*!98e4"9%P26Pp
133"$#3$'44i!!!!"!*!,48a&63#3"d-R!-C&"!!!!!%!N!YC48&568m!N!9$+`$
'43B!!!!"!*!,4%&C-$%!-3#3"%-a!-C&"J!!!!%!N!Y%39N`-J!b!*!%3cF!aN8
'!!!!!3#3#d4"@6!c!$-!N!4$23$'43B!!!!"!*!,4%&C-$3!0!#3"%0$!-C&"J!
!!!%!N!Y%39N`03!e!*!%3dN!aN8'!!!!!3#3#d4"@6!f!$B!N!4$6`$'43B!!!!
"!*!,4%&C-$F!0`#3"%09!-C&"J!!!!%!N!Y%39N`1!!i!*!%3eX!aN8'!!!!!3#
3#d4"@6!j!$N!N!4$B3$'43B!!!!"!*!,4%&C-6!!-!#3"%0R!-C&"J!!!!%!N!Y
%39Na-3#3"N0Y!-C&"J!!!!%!N!Y%39Na-J!b!*!%3h-!aN8'!!!!!3#3#d4"@6%
c!$-!N!4$H3$'43B!!!!"!*!,4%&C-63!0!#3"%0r!-C&"J!!!!%!N!Y%39Na03!
e!*!%3i8!aN8'!!!!!3#3#d4"@6%f!$B!N!4$L`$'43B!!!!"!*!,4%&C-6F!0`#
3"%14!-C&"J!!!!%!N!Y%39Na1!!i!*!%3jF!aN8'!!!!!3#3#d4"@6%j!$N!N!4
$R3$'43B!!!!"!*!,4%&C-M!!-!#3"%1M!-C&"J!!!!%!N!Y%39Nb-3!a!*!%3kN
!aN8'!!!!!3#3#d4"@6)b!$)!N!4$V`$'43B!!!!"!*!,4%&C-M-!-`#3"%1e!-C
&"J!!!!%!N!Y%39Nb0!!d!*!%3lX!aN8'!!!!!3#3#d4"@6)e!$8!N!4$`3$'43B
!!!!"!*!,4%&C-MB!0J#3"%2(!-C&"J!!!!%!N!Y%39Nb0`!h!*!%3md!aN8'!!!
!!3#3#d4"@6)i!$J!N!4$d`$'43B!!!!"!*!,4%&C-MN!13#3"%2C!-C&"J!!!!%
!N!Y%39Nc-!!`!*!%3pm!aN8'!!!!!3#3#d4"@6-a!$%!N!4$j3$'43B!!!!"!*!
,$4TU9`!!:
""")
hexbin(source1, 'source1.dbf')
source1.close()
del source1
######### above makes a structure
def rdhdr(adbf):
adbf.seek(4) #ver,yr,mo,day
hdr= struct.unpack('L',adbf.read(4)) #number of records
hdr= hdr + struct.unpack('H',adbf.read(2)) #length of header
hdr= hdr + struct.unpack('H',adbf.read(2)) #length of records
adbf.seek(32)
fld= 1
while adbf.tell() < (hdr[1] - 32): #each field def is 32bytes
adbf.seek(fld*32)
hdrn= struct.unpack('11s',adbf.read(11))[0].strip('x\00')
adbf.seek(5,1)
hdrs= (struct.unpack('B',adbf.read(1))[0])
hdr= hdr + ((hdrn,hdrs,fld),) #name,size,seq.number
fld= fld + 1
return(hdr)
################ above sets up for parsing (reading or writing or both)
If the Fortran is in fact from a punch card then your record will be 80
columns (IBM) or 90 (UniVac). The green bar paper is 132. In each
case, offset zero is page control and last 4 columns of card are for
sequencing the cards in case you (or someone) dropped the deck. Last 6
columns for sequence on green bar as I recall. (decks numbers additive)
The advantage of using the .dbf is it creates a user friendly file.
Excel, well - almost any spread sheet or database program.
Plus it is a dbf and database operations are 'right there'. Microsoft
office, OpenOffice and the list goes on, all read/write .dbf
By the way - CSV (comma seperated values) were in use in the past, but
due to memory (or lack of) were only for sequential use. You have to
count the commas and compare to number expected at EOL.
SDF (standard data format) the card was for random access.
good old lseek(record_number * record_size, basepoint)
snipets are from actual code dated Feb 2008.
using Python 2.5.2 on Linux Slackware 10.2
today: 20090430
Steve
alex23
wrote:
> (doesn't work as written, because [...]
Man, I don't get Python... I can write a program that runs properly on
the first try but every time I post untested code to c.l.p I regret
it...
Thanks, Gabriel :)
Gabriel Genellina
> On Apr 30, 5:33�pm, "Gabriel Genellina" wrote:
>> (doesn't work as written, because [...]
>
> Man, I don't get Python... I can write a program that runs properly on
> the first try but every time I post untested code to c.l.p I regret
> it...
Most of Python features are rather natural and/or intuitive so it's easy
to express your idea of a program in Python code. BUT in this particular
case, __import__ behaviour is NOT intuitive nor natural nor practical
nor... so it's quite reasonable you fall in the trap :)
(it happens to me too: every time I post some untested code, it has a bug!)
--
Gabriel Genellina
Terry Reedy
Which is why I either cut&paste working code or say 'untested', as I did
in this thread ;-)
Arnaud Delobelle
> Now I can move on to parsing those pesky Fortran card
> images... There wouldn't happen to be a way to take n
> continguous slices from a string (card image) where each
> slice may be a different length would there? Fortran you
> know. No spaces between input fields. :-)
>
> I know a way to do it, iterating over a list of slice sizes,
> perhaps in a list comprehension, but some of the august python
> personages here no doubt know better ways.
It's a challenge to do it in a list comprehension, but here it is!
Warning: unpythonic code follows(*). Sensitive readers may want to look
away now.
>>> data
'AAABBBBBCCCCCCCCCDD'
>>> field_sizes
[3, 5, 9, 2]
>>> [data[i:j] for j in [0] for s in field_sizes for i, j in [(j, j+s)]]
['AAA', 'BBBBB', 'CCCCCCCCC', 'DD']
>>>
Now you can look again.
(*) Pythonicity is often much debated as it lies for a good part in the
eye of the snake charmer. I don't think this snippet would generate
much debate!
--
Arnaud
alex23
> It's a challenge to do it in a list comprehension, but here it is!
> 'AAABBBBBCCCCCCCCCDD'
> >>> field_sizes
> [3, 5, 9, 2]
> >>> [data[i:j] for j in [0] for s in field_sizes for i, j in [(j, j+s)]]
> ['AAA', 'BBBBB', 'CCCCCCCCC', 'DD']
Ugh. I know using a listcomp was the point but I find using struct so
much more elegant.
>>> import struct
>>> data = 'AAABBBBBCCCCCCCCCDD'
>>> format = '3s5s9s2s'
>>> struct.unpack(format, data)
('AAA', 'BBBBB', 'CCCCCCCCC', 'DD')
Friends don't let friends abuse list comprehensions.
I made sure the code works this time :)