is_iterable function.

7 views
Skip to first unread message

Neil Cerutti

unread,
Jul 25, 2007, 2:24:24 PM7/25/07
to
def is_iterable(obj):
try:
iter(obj)
return True
except TypeError:
return False

Is there a better way?

--
Neil Cerutti

danmc...@yahoo.com

unread,
Jul 25, 2007, 2:58:40 PM7/25/07
to
You can use the built-in dir() function to determine whether or not
the __iter__ method exists:

class Iterable(object):
def __iter__(self):
pass

class NotIterable(object):
pass

def is_iterable(thing):
return '__iter__' in dir(thing)

print 'list is iterable = ', is_iterable(list())
print 'int is iterable = ', is_iterable(10)
print 'float is iterable = ', is_iterable(1.2)
print 'dict is iterable = ', is_iterable(dict())
print 'Iterable is iterable = ', is_iterable(Iterable())
print 'NotIterable is iterable = ', is_iterable(NotIterable())

Results:
list is iterable = True
int is iterable = False
float is iterable = False
dict is iterable = True
Iterable is iterable = True
NotIterable is iterable = False

Marc 'BlackJack' Rintsch

unread,
Jul 25, 2007, 3:11:34 PM7/25/07
to
On Wed, 25 Jul 2007 11:58:40 -0700, danmc...@yahoo.com wrote:

> You can use the built-in dir() function to determine whether or not
> the __iter__ method exists:

Doesn't work:

In [58]: is_iterable('hello')
Out[58]: False

But strings *are* iterable.

And just calling `iter()` doesn't work either:

In [72]: class A:
....: def __getitem__(self, key):
....: if key == 42:
....: return 'answer'
....: raise KeyError
....:

In [73]: iter(A())
Out[73]: <iterator object at 0xb7829b2c>

In [74]: a = iter(A())

In [75]: a.next()
---------------------------------------------------------------------------
<type 'exceptions.KeyError'> Traceback (most recent call last)

/home/bj/<ipython console> in <module>()

/home/bj/<ipython console> in __getitem__(self, key)

<type 'exceptions.KeyError'>:


So there's no reliable way to test for "iterables" other than actually
iterate over the object.

Ciao,
Marc 'BlackJack' Rintsch

Carsten Haese

unread,
Jul 25, 2007, 3:15:26 PM7/25/07
to pytho...@python.org
On Wed, 2007-07-25 at 11:58 -0700, danmc...@yahoo.com wrote:
> You can use the built-in dir() function to determine whether or not
> the __iter__ method exists:
>
> class Iterable(object):
> def __iter__(self):
> pass
>
> class NotIterable(object):
> pass
>
> def is_iterable(thing):
> return '__iter__' in dir(thing)
>
> print 'list is iterable = ', is_iterable(list())
> print 'int is iterable = ', is_iterable(10)
> print 'float is iterable = ', is_iterable(1.2)
> print 'dict is iterable = ', is_iterable(dict())
> print 'Iterable is iterable = ', is_iterable(Iterable())
> print 'NotIterable is iterable = ', is_iterable(NotIterable())
>
> Results:
> list is iterable = True
> int is iterable = False
> float is iterable = False
> dict is iterable = True
> Iterable is iterable = True
> NotIterable is iterable = False

Testing for __iter__ alone is not enough:

>>> class X(object):
... def __getitem__(self,i):
... if i<10: return i
... else: raise IndexError, i
...
>>> x = X()
>>> is_iterable(x)
False
>>> iter(x)
<iterator object at 0xb7f0182c>
>>> for i in x: print i
...
0
1
2
3
4
5
6
7
8
9

No __iter__ in sight, but the object is iterable.

--
Carsten Haese
http://informixdb.sourceforge.net


Neil Cerutti

unread,
Jul 25, 2007, 3:26:57 PM7/25/07
to
On 2007-07-25, danmc...@yahoo.com <danmc...@yahoo.com> wrote:
> You can use the built-in dir() function to determine whether or not
> the __iter__ method exists:

I think you might have to check for __getitem__, also, which may
support the sequence protocol.

> def is_iterable(thing):
> return '__iter__' in dir(thing)

So then:

def is_iterable(thing):
return '__iter__' in dir(thing) or '__getitem__' in dir(thing)

Speaking of the iter builtin function, is there an example of the
use of the optional sentinel object somewhere I could see?

--
Neil Cerutti

Jeff McNeil

unread,
Jul 25, 2007, 3:37:45 PM7/25/07
to danmc...@yahoo.com, pytho...@python.org
That's not going to hold true for generator functions or iterators
that are implemented on sequences without '__iter__.' You might also
want to check for __getitem__. I'm not sure if there's a way to tell
if a function is a generator without actually calling it.

-Jeff

> --
> http://mail.python.org/mailman/listinfo/python-list
>

Duncan Booth

unread,
Jul 25, 2007, 3:41:12 PM7/25/07
to
Neil Cerutti <hor...@yahoo.com> wrote:

> Speaking of the iter builtin function, is there an example of the
> use of the optional sentinel object somewhere I could see?

for line in iter(open('somefile.txt', 'r').readline, ''):
print line

Neil Cerutti

unread,
Jul 25, 2007, 3:42:53 PM7/25/07
to
On 2007-07-25, Marc 'BlackJack' Rintsch <bj_...@gmx.net> wrote:
> So there's no reliable way to test for "iterables" other than
> actually iterate over the object.

A TypeError exception is perhaps too generic for comfort in this
use case:

def deeply_mapped(func, iterable):
for item in iterable:
try:
for it in flattened(item):
yield func(it)
except TypeError:
yield func(item)

I'd be more confortable excepting some sort of IterationError (or
using an is_iterable function, of course). I guess there's always
itertools. ;)

--
Neil Cerutti

Carsten Haese

unread,
Jul 25, 2007, 3:46:14 PM7/25/07
to pytho...@python.org
On Wed, 2007-07-25 at 19:11 +0000, Marc 'BlackJack' Rintsch wrote:
> And just calling `iter()` doesn't work either:
>
> In [72]: class A:
> ....: def __getitem__(self, key):
> ....: if key == 42:
> ....: return 'answer'
> ....: raise KeyError
> ....:
>
> In [73]: iter(A())
> Out[73]: <iterator object at 0xb7829b2c>
>
> In [74]: a = iter(A())
>
> In [75]: a.next()
> ---------------------------------------------------------------------------
> <type 'exceptions.KeyError'> Traceback (most recent call last)
>
> /home/bj/<ipython console> in <module>()
>
> /home/bj/<ipython console> in __getitem__(self, key)
>
> <type 'exceptions.KeyError'>:
>
>
> So there's no reliable way to test for "iterables" other than actually
> iterate over the object.

You seem to say that your 'a' object is not iterable. I disagree. While
it's true that it raises an exception upon retrieval of the zeroth
iteration, that situation is quite different from attempting to iterate
over the number 10, where you can't even ask for a zeroth iteration.

To illustrate this point further, imagine you write an object that
iterates over the lines of text read from a socket. If the connection is
faulty and closes the socket before you read the first line, the zeroth
iteration raises an exception. Does that mean the object is iterable or
not depending on the reliability of the socket connection? I find that
notion hard to swallow.

Carsten Haese

unread,
Jul 25, 2007, 3:47:50 PM7/25/07
to pytho...@python.org
On Wed, 2007-07-25 at 19:26 +0000, Neil Cerutti wrote:
> Speaking of the iter builtin function, is there an example of the
> use of the optional sentinel object somewhere I could see?

Example 1: If you use a DB-API module that doesn't support direct cursor
iteration with "for row in cursor", you can simulate it this way:

for row in iter(cursor.fetchone, None):
# do something

Example 2: Reading a web page in chunks of 8kB:

f = urllib.urlopen(url)
for chunk in iter(lambda:f.read(8192), ""):
# do something

HTH,

Neil Cerutti

unread,
Jul 25, 2007, 4:01:29 PM7/25/07
to
On 2007-07-25, Carsten Haese <car...@uniqsys.com> wrote:
> On Wed, 2007-07-25 at 19:26 +0000, Neil Cerutti wrote:
>> Speaking of the iter builtin function, is there an example of the
>> use of the optional sentinel object somewhere I could see?
>
> Example 1: If you use a DB-API module that doesn't support direct cursor
> iteration with "for row in cursor", you can simulate it this way:
>
> for row in iter(cursor.fetchone, None):
> # do something
>
> Example 2: Reading a web page in chunks of 8kB:
>
> f = urllib.urlopen(url)
> for chunk in iter(lambda:f.read(8192), ""):
> # do something

Ah! Thanks for the examples. That's much simpler than I was
imagining. It's also somewhat evil, but I suppose it conserves a
global name to do it that way.

--
Neil Cerutti

George Sakkis

unread,
Jul 25, 2007, 4:02:14 PM7/25/07
to
On Jul 25, 3:26 pm, Neil Cerutti <horp...@yahoo.com> wrote:

> Speaking of the iter builtin function, is there an example of the
> use of the optional sentinel object somewhere I could see?

# iterate over random numbers from 1 to 10; use 0 as a sentinel to
stop the iteration
for n in iter(lambda:random.randrange(10), 0):
print n

More generally, iter(callable, sentinel) is just a convenience
function for the following generator:

def iter(callable, sentinel):
while True:
c = callable()
if c == sentinel: break
yield c


George

Message has been deleted

Steve Holden

unread,
Jul 25, 2007, 6:04:03 PM7/25/07
to pytho...@python.org
Carsten Haese wrote:
> On Wed, 2007-07-25 at 19:26 +0000, Neil Cerutti wrote:
>> Speaking of the iter builtin function, is there an example of the
>> use of the optional sentinel object somewhere I could see?
>
> Example 1: If you use a DB-API module that doesn't support direct cursor
> iteration with "for row in cursor", you can simulate it this way:
>
> for row in iter(cursor.fetchone, None):
> # do something
>
[...]
This would, of course, be a horribly inefficient way to handle a
database result with 1,500,000 rows. Calling fetchall() might also have
its issues. The happy medium is to use a series of calls to fetchmany(N)
with an appropriate value of N.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Bruno Desthuilliers

unread,
Jul 22, 2007, 12:38:30 AM7/22/07
to
Neil Cerutti a écrit :

> def is_iterable(obj):
> try:
> iter(obj)
> return True
> except TypeError:
> return False
>
> Is there a better way?

The only other alternative I see is worse:

def iterable(obj):
# strings are iterable and don't have an __iter__ method...
for name in ('__iter__', '__getitem__'):
try:
getattr(obj, name)
return True
except AttributeError:
pass
else:
return False


Neil Cerutti

unread,
Jul 26, 2007, 11:02:39 AM7/26/07
to
Based on the discussions in this thread (thanks all for your
thoughts), I'm settling for:

def is_iterable(obj):
try:
iter(obj).next()


return True
except TypeError:
return False

except KeyError:
return False

The call to iter will fail for objects that don't support the
iterator protocol, and the call to next will fail for a
(hopefully large) subset of the objects that don't support the
sequence protocol.

This seems preferable to cluttering code with exception handling
and inspecting tracebacks. But it's still basically wrong, I
guess.

To repost my use case:

def deeply_mapped(func, iterable):
""" Recursively apply a function to every item in a iterable object,
recursively descending into items that are iterable. The result is an
iterator over the mapped values. Similar to the builtin map function, func
may be None, causing the items to returned unchanged.

>>> import functools
>>> flattened = functools.partial(deeply_mapped, None)
>>> list(flattened([[1], [2, 3, []], 4]))
[1, 2, 3, 4]

>>> list(flattened(((1), (2, 3, ()), 4)))
[1, 2, 3, 4]

>>> list(flattened([[[[]]], 1, 2, 3, 4]))
[1, 2, 3, 4]

>>> list(flattened([1, [[[2, 3]], 4]]))
[1, 2, 3, 4]

"""
for item in iterable:
if is_iterable(item):
for it in deeply_mapped(func, item):
if func is None:
yield it
else:
yield func(it)
else:
if func is None:
yield item
else:
yield func(item)
--
Neil Cerutti

Marc 'BlackJack' Rintsch

unread,
Jul 26, 2007, 1:15:30 PM7/26/07
to
On Thu, 26 Jul 2007 15:02:39 +0000, Neil Cerutti wrote:

> Based on the discussions in this thread (thanks all for your
> thoughts), I'm settling for:
>
> def is_iterable(obj):
> try:
> iter(obj).next()
> return True
> except TypeError:
> return False
> except KeyError:
> return False
>
> The call to iter will fail for objects that don't support the
> iterator protocol, and the call to next will fail for a
> (hopefully large) subset of the objects that don't support the
> sequence protocol.

And the `next()` consumes an element if `obj` is not "re-iterable".

Ciao,
Marc 'BlackJack' Rintsch

Neil Cerutti

unread,
Jul 26, 2007, 2:56:46 PM7/26/07
to

Crap.

So how best to imlement deeply_mapped?

The following still breaks for objects that don't provide
__iter__, do provide __getitem__, but don't support the sequence
protocol.

def deeply_mapped(func, iterable):
""" Recursively apply a function to every item in a nested container,


recursively descending into items that are iterable. The result is an
iterator over the mapped values. Similar to the builtin map function, func
may be None, causing the items to returned unchanged.

>>> import functools
>>> flattened = functools.partial(deeply_mapped, None)
>>> list(flattened([[1], [2, 3, []], 4]))
[1, 2, 3, 4]

>>> list(flattened(((1), (2, 3, ()), 4)))
[1, 2, 3, 4]

>>> list(flattened([[[[]]], 1, 2, 3, 4]))
[1, 2, 3, 4]

>>> list(flattened([1, [[[2, 3]], 4]]))
[1, 2, 3, 4]

>>> def magic(o):
... if o == 3:
... raise TypeError('Three is a magic number')
... return o

>>> list(deeply_mapped(magic, [1, [[[2, 3]], 4]]))
Traceback (most recent call last):
...
TypeError: Three is a magic number

"""
frame = inspect.currentframe()
info = inspect.getframeinfo(frame)
filename = info[0]
funcname = info[2]


for item in iterable:
try:

for it in deeply_mapped(func, item):
if func is None:
yield it
else:
yield func(it)

except TypeError:
eframe = inspect.trace()[-1]
efilename = eframe[1]
efuncname = eframe[3]
if efilename != filename or efuncname != funcname:
raise

George Sakkis

unread,
Jul 26, 2007, 5:20:18 PM7/26/07
to
On Jul 26, 2:56 pm, Neil Cerutti <horp...@yahoo.com> wrote:
> On 2007-07-26, Marc 'BlackJack' Rintsch <bj_...@gmx.net> wrote:
>
>
>
> > On Thu, 26 Jul 2007 15:02:39 +0000, Neil Cerutti wrote:
>
> >> Based on the discussions in this thread (thanks all for your
> >> thoughts), I'm settling for:
>
> >> def is_iterable(obj):
> >> try:
> >> iter(obj).next()
> >> return True
> >> except TypeError:
> >> return False
> >> except KeyError:
> >> return False
>
> >> The call to iter will fail for objects that don't support the
> >> iterator protocol, and the call to next will fail for a
> >> (hopefully large) subset of the objects that don't support the
> >> sequence protocol.
>
> > And the `next()` consumes an element if `obj` is not
> > "re-iterable".
>
> Crap.
>
> So how best to imlement deeply_mapped?
>
> The following still breaks for objects that don't provide
> __iter__, do provide __getitem__, but don't support the sequence
> protocol.

That's not the only problem; try a string element to see it break too.
More importantly, do you *always* want to handle strings as
iterables ?

The best general way to do what you're trying to is pass is_iterable()
as an optional argument with a
sensible default, but allow the user to pass a different one that is
more appropriate for the task at hand:

def is_iterable(obj):
try: iter(obj)
except: return False
else: return True

def flatten(obj, is_iterable=is_iterable):
if is_iterable(obj):
for item in obj:
for flattened in flatten(item, is_iterable):
yield flattened
else:
yield obj

from functools import partial
flatten_nostr = partial(flatten, is_iterable=lambda obj: not
isinstance(obj,basestring)
and
is_iterable(obj))

print list(flatten_nostr([1, [[[2, 'hello']], (4, u'world')]]))


By the way, it's bad design to couple two distinct tasks: flattening a
(possibly nested) iterable and applying a function to its elements.
Once you have a flatten() function, deeply_mapped is reduced down to
itertools.imap.

HTH,
George

Neil Cerutti

unread,
Jul 26, 2007, 6:10:55 PM7/26/07
to
On 2007-07-26, George Sakkis <george...@gmail.com> wrote:
> That's not the only problem; try a string element to see it
> break too. More importantly, do you *always* want to handle
> strings as iterables ?
>
> The best general way to do what you're trying to is pass
> is_iterable() as an optional argument with a sensible default,
> but allow the user to pass a different one that is more
> appropriate for the task at hand:
>
> def is_iterable(obj):
> try: iter(obj)
> except: return False
> else: return True

> def flatten(obj, is_iterable=is_iterable):

That makes good sense.

Plus the subtly different way you composed is_iterable is clearer
than what I originally wrote. I haven't ever used a try with an
else.

> if is_iterable(obj):
> for item in obj:
> for flattened in flatten(item, is_iterable):
> yield flattened
> else:
> yield obj
>

> By the way, it's bad design to couple two distinct tasks:
> flattening a (possibly nested) iterable and applying a function
> to its elements. Once you have a flatten() function,
> deeply_mapped is reduced down to itertools.imap.

I chose to implement deeply_mapped because it illustrated the
problem of trying to catch a TypeError exception when one might
be thrown by some other code. I agree with your opinion that it's
a design flaw, and most of my problems with the code were caused
by that flaw.

--
Neil Cerutti

Reply all
Reply to author
Forward
0 new messages