ConfigParser: use newline in INI file

[Thorsten Kampe]

Hi,

ConfigParser escapes `\n` in ini values as `\\n`. Is there a way to
signal to ConfigParser that there is a line break?

Thorsten

[Terry Reedy]

On 10/1/2016 10:56 AM, Thorsten Kampe wrote:

> ConfigParser escapes `\n` in ini values as `\\n`. Is there a way to
> signal to ConfigParser that there is a line break?

Without an example or two, I don't really understand the question enough
to answer.

--
Terry Jan Reedy

[Ben Finney]

Thorsten Kampe <thor...@thorstenkampe.de> writes:

> ConfigParser escapes `\n` in ini values as `\\n`.

How do you demonstrate that?

Here is an example text of a config file::

>>> import io
>>> import textwrap

>>> config_text = textwrap.dedent(r"""
... [foo]
... wibble = Lorem\nipsum.
... """)
>>> print(config_text)

[foo]
wibble = Lorem\nipsum.

So that text has the characters “\n” in it. (Note that I had to use the
‘r’ prefix on the string literal, in order to turn off the special
meaning of “\” and get that character literally.)

When I use a ConfigParser it reads “\n” and reproduces exactly those
characters::

>>> import configparser

>>> config_file = io.StringIO(config_text)
>>> config = configparser.ConfigParser()
>>> config.read_file(config_file)
>>> print(config['foo']['wibble'])
Lorem\nipsum.

So you see that the “\n” characters are preserved exactly by the
ConfigParser.

> Is there a way to signal to ConfigParser that there is a line break?

Yes, you use a line break. See the ‘configparser’ module documentation:

Values can also span multiple lines, as long as they are indented
deeper than the first line of the value. Depending on the parser’s
mode, blank lines may be treated as parts of multiline values or
ignored.

<URL:https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>

Thus::

>>> config_text = textwrap.dedent(r"""
... [foo]
... wibble = Lorem
... ipsum.
... """)
>>> print(config_text)

[foo]
wibble = Lorem
ipsum.

>>> config_file = io.StringIO(config_text)
>>> config = configparser.ConfigParser()
>>> config.read_file(config_file)
>>> print(config['foo']['wibble'])
Lorem
ipsum.

--
\ “Only the shallow know themselves.” —Oscar Wilde, _Phrases and |
`\ Philosophies for the Use of the Young_, 1894 |
_o__) |
Ben Finney

[Thorsten Kampe]

* Terry Reedy (Sat, 1 Oct 2016 15:44:39 -0400)

>
> On 10/1/2016 10:56 AM, Thorsten Kampe wrote:
>

> > ConfigParser escapes `\n` in ini values as `\\n`. Is there a way to

> > signal to ConfigParser that there is a line break?
>

> Without an example or two, I don't really understand the question enough
> to answer.

>>> !cat INI.ini
[Asciidoc]
_test_stdout = <div class="paragraph">\n<p>Hello, World!</p>\n</div>

>>> import configparser

>>> config = configparser.ConfigParser()

>>> config.read('INI.ini')
['INI.ini']

>>> config['Asciidoc']['_test_stdout']
'<div class="paragraph">\\n<p>Hello, World!</p>\\n</div>'

...as you can see, ConfigParser escaped the backslash by doubling it.
Which is fine in most cases - except when I want to have something
indicating an newline.

Thorsten

[Thorsten Kampe]

* Ben Finney (Sun, 02 Oct 2016 07:12:46 +1100)

>
> Thorsten Kampe <thor...@thorstenkampe.de> writes:
>
> > ConfigParser escapes `\n` in ini values as `\\n`.

Indenting solves the problem. I'd rather keep it one line per value
but it solves the problem.

Thorsten

[Ned Batchelder]

If you want to have \n mean a newline in your config file, you can
do the conversion after you read the value:

>>> "a\\nb".decode("string-escape")
'a\nb'

--Ned.

[Thorsten Kampe]

* Ned Batchelder (Sat, 1 Oct 2016 17:41:28 -0700 (PDT))

Interesting approach (although it does not work with Python 3: decode
is only for byte-strings and the string-escape encoding is not
defined).

If I understand this correctly, this is equivalent to
`.replace('\\n', '\n')` ?

Thorsten

[Chris Angelico]

On Sun, Oct 2, 2016 at 9:34 PM, Thorsten Kampe
<thor...@thorstenkampe.de> wrote:
>> If you want to have \n mean a newline in your config file, you can
>> do the conversion after you read the value:
>>
>> >>> "a\\nb".decode("string-escape")
>> 'a\nb'
>

> Interesting approach (although it does not work with Python 3: decode
> is only for byte-strings and the string-escape encoding is not
> defined).
>
> If I understand this correctly, this is equivalent to
> `.replace('\\n', '\n')` ?

Yes, and a bunch of other conversions too. Notably, it will convert
'\\\\' into '\\' (I'd explain that with raw string literals but they
get wonky with trailing backslashes!), without breaking the
replacement of other constructs. But if you're okay with the string
r"\n" being impossible to enter, then go ahead and just use replace().
And you don't have to use \n specifically - if you're using replace(),
you can use any other token you like to separate lines. VX-REXX had a
nifty way to handle this: you may use any one-character separator you
like, simply by starting the string with it.

;foo;bar;quux;spam
!foo!bar!quux!spam
=foo=bar=quux=spam

If only a few of your tokens are multi-line, you could use this
technique - no escaping needed.

ChrisA

[Peter Otten]

Thorsten Kampe wrote:

> * Ned Batchelder (Sat, 1 Oct 2016 17:41:28 -0700 (PDT))

>> If you want to have \n mean a newline in your config file, you can
>> do the conversion after you read the value:
>>
>> >>> "a\\nb".decode("string-escape")
>> 'a\nb'
>

> Interesting approach (although it does not work with Python 3: decode
> is only for byte-strings and the string-escape encoding is not
> defined).

The equivalent for Python 3 is

>>> import codecs
>>> codecs.decode("a\\nb", "unicode-escape")
'a\nb'

You can teach ConfigParser to do this automatically with a custom
Interpolation:

$ cat cpdemo.py
#!/usr/bin/env python3
import configparser as cp
import codecs

class MyInterpolation(cp.BasicInterpolation):
def before_get(self, parser, section, option, value, defaults):
return super().before_get(
parser, section, option,
codecs.decode(value, "unicode-escape"),
defaults)

p = cp.ConfigParser(interpolation=MyInterpolation())
p.read_string("""\
[foo]
bar=ham\\nspam
""")
print(p["foo"]["bar"])
$ ./cpdemo.py
ham
spam

[jim.wo...@gmail.com]

Wow! Thanks so much Ned. I've been looking for the solution to this issue for several days and had nearly given up.
Jim

[tony]

How does that translate to Python3?

[jim.wo...@gmail.com]

I have no idea. I'm on 2.7.3
I'd be interested in knowing if someone would try it on 3.
I do very little Python programming. I've written a messaging system for Linuxcnc and could not get it to work using bash, so I tried Python and now, thanks to you, I have it working.
I can provide you with the code if you want to play with it. When it is run with an integer as a parameter it displays a message from an INI file.
Jim

[Peter Otten]

tony wrote:

>> On Saturday, October 1, 2016 at 7:41:40 PM UTC-5, Ned Batchelder wrote:

>>> If you want to have \n mean a newline in your config file, you can
>>> do the conversion after you read the value:
>>>
>>> >>> "a\\nb".decode("string-escape")
>>> 'a\nb'

> How does that translate to Python3?

[tony]

Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>>
>>> "a\\nb".decode("string-escape")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'
>>>

[jim.wo...@gmail.com]

And now we know!
I think they should have named Python 3 something else

[jim.wo...@gmail.com]

I just tried "unicode_decode" in place of "string-decode" and it works. My impression is that it will also work in Python 3

[Tim Chase]

On 2019-03-07 17:19, tony wrote:
> Python 3.5.3 (default, Sep 27 2018, 17:25:39)

> >>> "a\\nb".decode("string-escape")

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: 'str' object has no attribute 'decode'

Looks like bytestring.decode('unicode_escape') does what you're
describing:

>>> b"hello\nworld".decode('unicode_escape')
'hello\nworld'
>>> print(b"hello\nworld".decode('unicode_escape'))
hello
world

-tkc