dictConfig: logging.StreamHandler object is not iterable.

[Tim Williams]

(Apologies for using Google Groups to post)

I'm trying to use dictConfig to configure logging. I keep running into the error that the logging.StreamHandler object is not iterable.

I'm using Python 3.4.3 on a Windows 7 box.

C:\Python34\python.exe 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)]


I want to use the configobj module to create the dictionary from an INI file, but that's not the problem I'm having.

Here is my test code:
#############
import logging, logging.config, logging.handlers
import configobj

# config = configobj.ConfigObj('loggingtest.ini')
# config['version']=eval(config['version'])
config = {
'version': 1,
'level': 'INFO',
'formatters': {'fmt1': {'format': '%(asctime)s: (%(levelname)s) %(message)s',
'datefmt': ''}
},
'loggers': {'root': {'level': 'INFO',
'handlers': 'cfg://handlers.console'},
'file': {'level': 'WARN',
'handlers': 'cfg://handlers.file'}
},
'handlers': {'console': {'class': 'logging.StreamHandler',
'level': 'INFO',
'stream': 'ext://sys.stdout'},
'file': {'class': 'logging.FileHandler',
'level': 'WARN',
'filename': 'test.log'}
},
}


logging.config.dictConfig(config)
################

When I run it, I get this traceback:

Traceback (most recent call last):
File "C:\Python34\lib\logging\config.py", line 611, in configure
self.configure_logger(name, loggers[name])
File "C:\Python34\lib\logging\config.py", line 775, in configure_logger
self.common_logger_config(logger, config, incremental)
File "C:\Python34\lib\logging\config.py", line 767, in common_logger_config
self.add_handlers(logger, handlers)
File "C:\Python34\lib\logging\config.py", line 748, in add_handlers
for h in handlers:
TypeError: 'StreamHandler' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "L:\workspace\MyPython\src\testlog.py", line 27, in <module>
logging.config.dictConfig(config)
File "C:\Python34\lib\logging\config.py", line 789, in dictConfig
dictConfigClass(config).configure()
File "C:\Python34\lib\logging\config.py", line 614, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure logger 'root': 'StreamHandler' object is not iterable


I even tried creating a JSON file to create the dictionary, and got the same error.

Thanks for any help
--
Tim

[Peter Otten]

Tim Williams wrote:

> (Apologies for using Google Groups to post)
>
> I'm trying to use dictConfig to configure logging. I keep running into the
> error that the logging.StreamHandler object is not iterable.
>
> I'm using Python 3.4.3 on a Windows 7 box.
>
> C:\Python34\python.exe 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40)
> [MSC v.1600 64 bit (AMD64)]
>
>
> I want to use the configobj module to create the dictionary from an INI
> file, but that's not the problem I'm having.
>
> Here is my test code:
> #############
> import logging, logging.config, logging.handlers
> import configobj
>
> # config = configobj.ConfigObj('loggingtest.ini')
> # config['version']=eval(config['version'])
> config = {
> 'version': 1,
> 'level': 'INFO',
> 'formatters': {'fmt1': {'format': '%(asctime)s: (%(levelname)s)
> %(message)s',
> 'datefmt': ''}
> },
> 'loggers': {'root': {'level': 'INFO',
> 'handlers': 'cfg://handlers.console'},

If I'm reading

<https://docs.python.org/dev/library/logging.config.html#dictionary-schema-details>

"""
loggers - the corresponding value will be a dict in which each key is a
logger name and each value is a dict describing how to configure the
corresponding Logger instance.

The configuring dict is searched for the following keys:

[...]

handlers (optional). A list of ids of the handlers for this logger.
"""

correctly this should be

'loggers': {'root': {'level': 'INFO',

'handlers': ['console']},

> 'file': {'level': 'WARN',
> 'handlers': 'cfg://handlers.file'}

Same here:

... 'handlers': ['file'] ...

[Tim Williams]

Peter,

Thanks. That did it for my dictionary.
What I'm really trying to do is use configobj to do this from an INI file. The one problem is that all the values are strings, hence the

config['version']=eval(config['version'])

It looks like if I do something similar with the handlers in the loggers, like:

config['loggers']['root']['handlers']=eval(config['loggers']['root']['handlers'])
config['loggers']['file']['handlers']=eval(config['loggers']['file']['handlers'])

logging.config.dictConfig(dict(config))

seems to work.

I'm using configobj instead of configparser because configobj can nest sections. The INI file I'm trying to build has more sections for other things.

version = 1
level = INFO
[formatters]
[[fmt1]]
format = "%(asctime)s: (%(levelname)s) %(message)s"
datefmt =
[handlers]
[[console]]
class = logging.StreamHandler
level = INFO
stream = ext://sys.stdout
[[file]]
class = logging.FileHandler
level = WARN
filename = test.log
[loggers]
[[root]]
level = INFO
handlers = [console]
[[file]]
level = WARN
handlers = ['file']
propagate = 1

[Tim Williams]

On Wednesday, May 24, 2017 at 2:46:54 PM UTC-4, Tim Williams wrote:

Just as a followup, if I use 'unrepr=True' in my ConfigObj, I don't have to convert the strings.

[Peter Otten]

Tim Williams wrote:

> Just as a followup, if I use 'unrepr=True' in my ConfigObj, I don't have
> to convert the strings.

I'd keep it simple and would use JSON...

[Tim Williams]

I looked at JSON at first, but went with configobj because I didn't see where it did string interpolation, which I needed for other parts of my INI file, and I'm trying to use it to specify my log file in my handler.

Which brings me to ...

I have this stripped down INI file:

[loggng]
version = 1
level = 'INFO'
RootDir = 'TestData'
CaptureDrive = 'C:/'
LogFile = '%(CaptureDrive)s%(RootDir)s/test.log'
[[formatters]]

[[[fmt1]]]
format = '%(asctime)s: (%(levelname)s) %(message)s'
datefmt =
[[handlers]]
[[[console]]]
class = 'logging.StreamHandler'
level = 'INFO'
stream = 'ext://sys.stdout'
[[[file]]]
class = 'logging.FileHandler'
level = 'WARN'

filename = '%(LogFile)s'
#filename = 'cfg://loggng.LogFile'
[[loggers]]
[[[root]]]
level = 'INFO'
handlers = ['file','console']

When I parse the INI file with configobj:
config = configobj.ConfigObj('loggingtest.ini', unrepr=True, raise_errors=True)

config['loggng']['handlers']['file']['filename'] evaluates correctly to C:/TestData/test.log

However, when I try to call logging.config.dictConfig() on it, the stream that is opened on creating the logging.FileHandler object is "%(LogFile)s", not "C:/TestData/test.log".

Stepping into the debugger, I see that the dictionary is changed in BaseConfigurator.convert()

def convert(self, value):
"""
Convert values to an appropriate type. dicts, lists and tuples are
replaced by their converting alternatives. Strings are checked to
see if they have a conversion format and are converted if they do.
"""
if not isinstance(value, ConvertingDict) and isinstance(value, dict):
value = ConvertingDict(value)
value.configurator = self

BEFORE calling ConvertingDict(value), the value dict has the correct value for key filename:

>>> value
{'class': 'logging.FileHandler', 'level': 'WARN', 'filename': 'C:/TestData/test.log'}

AFTER calling ConvertingDict(value):

>>> value
{'filename': '%(LogFile)s', 'class': 'logging.FileHandler', 'level': 'WARN'}

If I try to use
filename = 'cfg://loggng.LogFile'
in my INI file, I get a ValueError calling logging.config.dictConfig(config):


pydev debugger: starting (pid: 70744)

Traceback (most recent call last):

File "C:\Python34\lib\logging\config.py", line 557, in configure
handler = self.configure_handler(handlers[name])
File "C:\Python34\lib\logging\config.py", line 723, in configure_handler
kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
File "C:\Python34\lib\logging\config.py", line 723, in <listcomp>
kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
File "C:\Python34\lib\logging\config.py", line 318, in __getitem__
return self.convert_with_key(key, value)
File "C:\Python34\lib\logging\config.py", line 284, in convert_with_key
result = self.configurator.convert(value)
File "C:\Python34\lib\logging\config.py", line 455, in convert
value = converter(suffix)
File "C:\Python34\lib\logging\config.py", line 404, in cfg_convert
d = self.config[m.groups()[0]]
File "C:\Python34\lib\logging\config.py", line 317, in __getitem__
value = dict.__getitem__(self, key)
KeyError: 'loggng'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "L:\timothy.j.williams1\Documents\My Programs\eclipse_neon\eclipse\plugins\org.python.pydev_5.3.0.201610121612\pysrc\pydevd.py", line 1531, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "L:\timothy.j.williams1\Documents\My Programs\eclipse_neon\eclipse\plugins\org.python.pydev_5.3.0.201610121612\pysrc\pydevd.py", line 938, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "L:\timothy.j.williams1\Documents\My Programs\eclipse_neon\eclipse\plugins\org.python.pydev_5.3.0.201610121612\pysrc\_pydev_imps\_pydev_execfile.py", line 25, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "L:\workspace\MyPython\src\testlog.py", line 8, in <module>

logging.config.dictConfig(config)
File "C:\Python34\lib\logging\config.py", line 789, in dictConfig
dictConfigClass(config).configure()

File "C:\Python34\lib\logging\config.py", line 565, in configure

'%r: %s' % (name, e))

ValueError: Unable to configure handler 'file': 'loggng'

testlog.py:

import logging, logging.config, logging.handlers
import configobj

logconfig = configobj.ConfigObj('loggingtest.ini', unrepr=True, raise_errors=True)

config=dict(logconfig['loggng'])
logging.config.dictConfig(config)
logger=logging.getLogger('root')
print(config['handlers']['file'])
print(logger.handlers[0].stream)

loggingtest.ini:
[loggng]
version = 1
level = 'INFO'
RootDir = 'TestData'
CaptureDrive = 'C:/'
LogFile = '%(CaptureDrive)s%(RootDir)s/test.log'
[[formatters]]

[[[fmt1]]]
format = '%(asctime)s: (%(levelname)s) %(message)s'
datefmt =
[[handlers]]
[[[console]]]
class = 'logging.StreamHandler'
level = 'INFO'
stream = 'ext://sys.stdout'
[[[file]]]
class = 'logging.FileHandler'
level = 'WARN'

filename = '%(LogFile)s'
#filename = 'cfg://loggng.LogFile'
[[loggers]]
[[[root]]]
level = 'INFO'
handlers = ['file','console']

Thanks for any help.

[Peter Otten]

Tim Williams wrote:

> On Wednesday, May 24, 2017 at 5:47:37 PM UTC-4, Peter Otten wrote:
>> Tim Williams wrote:
>>
>> > Just as a followup, if I use 'unrepr=True' in my ConfigObj, I don't
>> > have to convert the strings.
>>
>> I'd keep it simple and would use JSON...
>
> I looked at JSON at first, but went with configobj because I didn't see
> where it did string interpolation, which I needed for other parts of my
> INI file, and I'm trying to use it to specify my log file in my handler.
>
> Which brings me to ...

> I have this stripped down INI file:
>

...

How do you get

> LogFile = '%(CaptureDrive)s%(RootDir)s/test.log'

to be interpolated while leaving

> format = '%(asctime)s: (%(levelname)s) %(message)s'

as is?

> However, when I try to call logging.config.dictConfig() on it, the stream
> that is opened on creating the logging.FileHandler object is
> "%(LogFile)s", not "C:/TestData/test.log".

I don't even get this far:

>>> c = configobj.ConfigObj("second.ini", unrepr=True)
>>> c.dict()

Traceback (most recent call last):

...
configobj.MissingInterpolationOption: missing option "asctime" in
interpolation.

I tried to escape % as %%, but that doesn't seem to work. When I provide
bogus replacements

>>> c = configobj.ConfigObj("third.ini", unrepr=True)
>>> pprint.pprint(c.dict()["loggng"])
{'CaptureDrive': 'C:/',
'LogFile': 'C:/TestData/test.log',
'RootDir': 'TestData',
'asctime': 'ASCTIME',
'formatters': {'fmt1': {'datefmt': '',
'format': 'ASCTIME: (LEVELNAME) MESSAGE'}},

'handlers': {'console': {'class': 'logging.StreamHandler',
'level': 'INFO',
'stream': 'ext://sys.stdout'},
'file': {'class': 'logging.FileHandler',

'filename': 'C:/TestData/test.log',
'level': 'WARN'}},
'level': 'INFO',
'levelname': 'LEVELNAME',
'loggers': {'root': {'handlers': ['file', 'console'], 'level': 'INFO'}},
'message': 'MESSAGE',
'version': 1}

I get the expected output.

[Tim Williams]

I'm at home now, so I don't have my environment, but if I do a c.dict() I get the error about asctime also. If I just pass in the dict object or do a 'dict(config['loggng'])', I don't get that.

[Tim Williams]

On Thursday, May 25, 2017 at 9:43:40 PM UTC-4, Tim Williams wrote:
> On Thursday, May 25, 2017 at 5:16:13 PM UTC-4, Peter Otten wrote:

(snip)

(Back at work.)
Looking at the non-interpolation of '%(asctime)s', etc again, I'm wondering that myself. Maybe this is a bug in configobj? That doesn't make sense though. I'm wondering if the keyword 'format' has something to do with it.

[Tim Williams]

Changed key 'foramt' to 'formt'. No change on non-interpolation of
'%(asctime)s'

[Tim Williams]

Peter,

I'm starting to think that maybe my problem is somewhere with configobj. Calling ConfigObj.dict() (really configobj.Section.dict()) tries to interpolate the values in the 'format' key, but not when creating the ConfigObj object, but it does interpolate the LogFile key value.

I've spent too much time trying to track this down. I'll just hard-code my filename in my INI file. Maybe I'll get back to it, but I need to move on.

Thanks for your help.
Tim

[Peter Otten]

Tim Williams wrote:

> I've spent too much time trying to track this down. I'll just hard-code my
> filename in my INI file. Maybe I'll get back to it, but I need to move on.

The only alternative I see would be to build your own InterpolationEngine
which understands some kind of escaping so that e. g.

format = '%%(asctime)s: (%%(levelname)s) %%(message)s'

would become

format = '%(asctime)s: (%(levelname)s) %(message)s'

when you invoke the dict() method.

[Tim Williams]

It helps to sometimes just let a problem rest for awhile. I found a workaround to my problem. In my INI file, don't create a 'filename' key in the config['handlers']['file'] section. Set it in my python code after loading the INI file. The following works just fine for me. It doesn't address the issue of config.dict() string interpolating the format key, but it does what I want it to do and that's my bottom line.

Test script:
############

import logging, logging.config, logging.handlers
import configobj

logconfig = configobj.ConfigObj('loggingtest.ini', unrepr=True, raise_errors=True)

config=logconfig['loggng']
config['handlers']['file']['filename'] = logconfig['loggng']['LogFile']
logging.config.dictConfig(config)
formatters = config['formatters']['fmt1']

logger=logging.getLogger('root')
print(config['handlers']['file'])

for h in logger.handlers:
print('handler {}, stream {}'.format(h.name, h.stream))
############

loggingtest.ini:
###########

[loggng]
version = 1
level = 'INFO'
RootDir = 'TestData'
CaptureDrive = 'C:/'
LogFile = '%(CaptureDrive)s%(RootDir)s/test.log'
[[formatters]]
[[[fmt1]]]

format = '%(asctime)s (%(levelname)s) %(message)s'

datefmt =
[[handlers]]
[[[console]]]
class = 'logging.StreamHandler'
level = 'INFO'

#stream = 'ext://sys.stdout'
formatter = 'fmt1'

[[[file]]]
class = 'logging.FileHandler'
level = 'WARN'

formatter = 'fmt1'

[[loggers]]
[[[root]]]
level = 'INFO'
handlers = ['file','console']

###########