Python Generators
mpc
I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?
#!/usr/bin/env python
import os,struct,time
def generate_headers(cachedir):
for name, blocksize in cachefiles:
pathname = os.path.join(cachedir,name)
f = open(pathname,"rb")
f.seek(4096)
while True:
header = f.read(36)
if not header: break
fields = struct.unpack(">9I",header)
if fields[0] == 0x00010008: yield f, fields
fp = f.tell()
offset = fp % blocksize
if offset:
f.seek(blocksize - offset,1)
f.close()
def concatenate(sequences):
for seq in sequences:
for item in seq:
yield item
all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
cachefiles = [('_CACHE_001_',256),('_CACHE_002_',1024),('_CACHE_003_',
4096)]
n = 0
while True:
n += 1
time.sleep(0.5)
headers = concatenate(generate_headers(path) for path in
all_caches)
for h in headers:
print h,n
# this doesn't work either
while True:
n += 1
time.sleep(0.5)
for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n
# however if I hard code the path
path = "FFCache"
while True:
n += 1
headers = generate_headers(path)
for h in headers:
print h,n
#but of course i do not wish to hard code the path.
sturlamolden
> generator embedded in the argument only once. Can anyone explain while
> the generator will not re-initiate, and suggest a simple fix?
I am not sure what you are trying to do, but it seems a bit confused.
>>> def concat(seq):
for s in seq: yield s
>>> seq = xrange(3)
>>> for n in xrange(5):
h = concat(s for s in seq)
for i in h: print i,n
0 0
1 0
2 0
0 1
1 1
2 1
0 2
1 2
2 2
0 3
1 3
2 3
0 4
1 4
2 4
Generators work they way they should. Even when one is used as
argument for another.
Peter Otten
> I am trying to write a while loop that will iterate over generators to
> capture all the headers of FFCache directories. However, the
> generators embedded within the argument of another generator do not
> seem to re-initiate. the example below loops through and initiates the
> generator embedded in the argument only once. Can anyone explain while
> the generator will not re-initiate, and suggest a simple fix?
A generator or generator expression does indeed only run once.
>>> for i in range(3):
... print "---", i, "---"
... for k in gen: print k,
... print
...
--- 0 ---
1 3
--- 1 ---
--- 2 ---
The fix is to use a list or list comprehension, or make a new generator
every time you need one:
>>> for i in range(3):
... print "---", i, "---"
... gen = (i for i in range(5) if i%2)
... for k in gen: print k,
... print
...
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3
At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.
> while True:
> n += 1
> time.sleep(0.5)
all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
> for path in all_caches:
> headers = generate_headers(path)
> for h in headers:
> print h,n
(Untested, because you didn't bother to provide a self-contained example)
Peter
Peter Otten
> I am trying to write a while loop that will iterate over generators to
> capture all the headers of FFCache directories. However, the
> generators embedded within the argument of another generator do not
> seem to re-initiate. the example below loops through and initiates the
> generator embedded in the argument only once. Can anyone explain while
> the generator will not re-initiate, and suggest a simple fix?
A generator or generator expression does indeed only run once.
>>> gen = (i for i in range(5) if i%2)
--- 2 ---
> while True:
> n += 1
> time.sleep(0.5)
all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
> for path in all_caches:
> headers = generate_headers(path)
> for h in headers:
> print h,n
(Untested, because you didn't bother to provide a self-contained example)
Peter
Matt Nordhoff
<snip>
> def concatenate(sequences):
> for seq in sequences:
> for item in seq:
> yield item
You should check out itertools.chain(). It does this. You call it like
"chain(seq1, seq2, ...)" instead of "chain(sequences)" though, which may
be a problem for you.
The rest of itertools might be interesting too:
<http://docs.python.org/lib/module-itertools.html>
<snip>
--
Marius Gedminas
> mpc wrote:
> > def concatenate(sequences):
> > for seq in sequences:
> > for item in seq:
> > yield item
>
> You should check out itertools.chain(). It does this. You call it like
> "chain(seq1, seq2, ...)" instead of "chain(sequences)" though, which may
> be a problem for you.
Solved rather easily by chain(*sequences):
>>> from itertools import chain
>>> def concat(sequences):
... return chain(*sequences)
...
>>> concat([[1,2], [3, 4], [5], [6, 7, 8]])
<itertools.chain object at 0xb7cffc0c>
>>> list(concat([[1,2], [3, 4], [5], [6, 7, 8]]))
[1, 2, 3, 4, 5, 6, 7, 8]
wondering if google groups will add a .sig or not-ly,
Marius Gedminas