Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser
Arnaud Lacombe
On Tue, Feb 1, 2011 at 6:27 PM, Ulf Magnusson <ulfaliz...@gmail.com> wrote:
> Hi,
>
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems. �It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
>
Does that mean that when a kconfig-language change will happen, one
will not only must have to be Lex/YaCC and C fluent, but also, perl
and python ? That seem to be a lot of duplication to me.
I'd rather see the backend be changed so that it could be used with
SWIG to generate language bindings. I had WIP in this area, but
changes needed are rather intrusive. That said, the testsuite might be
interesting, I've been thinking about that for quite some time.
- Arnaud
> For a much longer introduction including multiple examples, see
> arch/kconfig/kconfiglib.py.
>
> Have fun!
>
> Signed-off-by: Ulf Magnusson <ulfaliz...@gmail.com>
> ---
> Convenience links:
>
> Documentation, generated from kconfiglib.py with pydoc -w:
> http://dl.dropbox.com/u/10406197/kconfiglib.html
>
> Examples as separate files:
> http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
>
>
> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> (2.6.38-rc3 at the time of writing). �Due to recent Kconfig changes, the
> kconfigtest.py test suite - which compares output character-for-character -
> will indicate failure on older (a few months old) kernels versions even though
> the outputs are functionally equivalent.
>
> �Documentation/kbuild/kconfig-language.txt | � �5 +
> �Documentation/kbuild/kconfig.txt � � � � �| � �8 +
> �README � � � � � � � � � � � � � � � � � �| � 13 +
> �scripts/kconfig/Makefile � � � � � � � � �| � 26 +-
> �scripts/kconfig/kconfiglib.py � � � � � � | 3918 +++++++++++++++++++++++++++++
> �scripts/kconfig/kconfigtest.py � � � � � �| �396 +++
> �6 files changed, 4365 insertions(+), 1 deletions(-)
> �create mode 100644 scripts/kconfig/kconfiglib.py
> �create mode 100644 scripts/kconfig/kconfigtest.py
> [...]
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Ulf Magnusson
> Hi,
>
> On Tue, Feb 1, 2011 at 6:27 PM, Ulf Magnusson <ulfaliz...@gmail.com> wrote:
> > Hi,
> >
> > This is the initial release of Kconfiglib: a Python library for
> > scripting, debugging, and extracting information from Kconfig-based
> > configuration systems. It can be used to programmatically generate a
> > .config when the '*conf' tools are too inflexible, to quickly find out
> > interesting information about a Kconfig configuration such as dependency
> > relations between symbols and where undefined symbols are referenced,
> > and in applications that need to parse and extract information from
> > Kconfig files.
> >
> Does that mean that when a kconfig-language change will happen, one
> will not only must have to be Lex/YaCC and C fluent, but also, perl
> and python ? That seem to be a lot of duplication to me.
>
> I'd rather see the backend be changed so that it could be used with
> SWIG to generate language bindings. I had WIP in this area, but
> changes needed are rather intrusive. That said, the testsuite might be
> interesting, I've been thinking about that for quite some time.
>
> - Arnaud
>
changes would probably have to be very invasive/obfuscating in order to
extract all the information you can get out of Kconfiglib. So I instead
went the minimally invasive route with a completely stand-alone library
together with a test suite.
I'm willing to update Kconfiglib when future additions/modifications are
made to the Kconfig language so that all tests in the compatibility test
suite keep passing. I've added lots of comments and documentation to
internal methods and tried to keep the code clean and straightforward,
so it should be pretty easy for others to get into the code as well.
If by Perl you're referring to streamline_config.pl, I suspect it could
be rewritten as a smallish Kconfiglib script, though I haven't looked at
it closely. That version might also be much better at respecting symbol
dependencies.
Ulf Magnusson
> Hi,
>
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems. It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
>
> arch/kconfig/kconfiglib.py.
>
> Have fun!
>
> Signed-off-by: Ulf Magnusson <ulfaliz...@gmail.com>
> ---
> Convenience links:
>
> Documentation, generated from kconfiglib.py with pydoc -w:
> http://dl.dropbox.com/u/10406197/kconfiglib.html
>
> Examples as separate files:
> http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
>
>
> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the
> kconfigtest.py test suite - which compares output character-for-character -
> will indicate failure on older (a few months old) kernels versions even though
> the outputs are functionally equivalent.
>
Seems linux-kbuild and linux-doc won't accept the patch (too large?), so
here's a link to the message on linux-kernel:
https://lkml.org/lkml/2011/2/1/439
/Ulf Magnusson
Ulf Magnusson
Randy Dunlap
> Documentation/kbuild/kconfig-language.txt | 5 +
> Documentation/kbuild/kconfig.txt | 8 +
> README | 13 +
> scripts/kconfig/Makefile | 26 +-
> scripts/kconfig/kconfiglib.py | 3918 +++++++++++++++++++++++++++++
> scripts/kconfig/kconfigtest.py | 396 +++
> 6 files changed, 4365 insertions(+), 1 deletions(-)
> create mode 100644 scripts/kconfig/kconfiglib.py
> create mode 100644 scripts/kconfig/kconfigtest.py
>
> diff --git a/README b/README
> index 1b81d28..bb5e68f 100644
> --- a/README
> +++ b/README
> @@ -196,6 +196,19 @@ CONFIGURING the kernel:
> values to 'n' as much as possible.
> "make randconfig" Create a ./.config file by setting symbol
> values to random values.
> + "make scriptconfig SCRIPT=<path to script>" Run a Kconfiglib
> + script (see scripts/kconfig/kconfiglib.py). This
> + can be used to programatically generate a
> + ./.config, and for applications that need to
> + extract information from Kconfig files.
> + "make iscriptconfig" Launch an interactive Python shell
> + for running Kconfiglib on the architecture's
> + Kconfig configuration. The kconfiglib and sys
> + (for sys.argv[1] - the base Kconfig file) modules
> + will be imported automatically, and a Config
> + instance 'c' will be created for the architecture
> + (using c = kconfiglib.Config(sys.argv[1])).
> +
>
> You can find more information on using the Linux kernel config tools
> in Documentation/kbuild/kconfig.txt.
Hi Ulf,
This is interesting. I just wish I could read it. ;)
I'll get over it.
1. It would be really Good to have "make scriptconfig SCRIPT=<path to script>"
and "make iscriptconfig" (similar to above, but shortened) in "make help" output.
2. My first test (using your ex1.py script) failed because I used O=xx64 (build
directory):
rddunlap@chimera:lnx-2638-rc3> make O=xx64 scriptconfig SCRIPT=~/pkg/kconfiglib/ex1.py
GEN /lnx/src/lnx-2638-rc3/xx64/Makefile
Traceback (most recent call last):
File "/home/rddunlap/pkg/kconfiglib/ex1.py", line 1, in <module>
import kconfiglib
ImportError: No module named kconfiglib
make[2]: *** [scriptconfig] Error 1
make[1]: *** [scriptconfig] Error 2
make: *** [sub-make] Error 2
Does kconfiglib support O=builddir generally? I can't tell that it does (yet)
since all ex[1-7].py fail in this manner.
It needs to support/allow O=builddir.
Thanks. I'll keep looking...
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
Rob Landley
>> The patch should be preferably be applied to a recent kernel, i.e. Linus's
>> (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the
>> kconfigtest.py test suite - which compares output character-for-character -
>> will indicate failure on older (a few months old) kernels versions even though
>> the outputs are functionally equivalent.
>>
>> Documentation/kbuild/kconfig-language.txt | 5 +
>> Documentation/kbuild/kconfig.txt | 8 +
>> README | 13 +
>> scripts/kconfig/Makefile | 26 +-
>> scripts/kconfig/kconfiglib.py | 3918 +++++++++++++++++++++++++++++
>> scripts/kconfig/kconfigtest.py | 396 +++
>> 6 files changed, 4365 insertions(+), 1 deletions(-)
>> create mode 100644 scripts/kconfig/kconfiglib.py
>> create mode 100644 scripts/kconfig/kconfigtest.py
Wait, I thought this was an extra standalone library. Are you saying
you want to make it so Linux will no longer compile on a build machine
that doesn't have Python installed?
If this is merely an extra developer tool ala bloat-o-meter and
checkpatch.pl then it's merely uninteresting to me. (I myself wrote a
quick and dirty http://kernel.org/doc/make/menuconfig2html.py to
generate http://kernel.org/doc/menuconfig/x86.html and friends years
ago, and I still run it to update that once in a while. There's not
much to it.)
But adding new prerequities to a build machine would be really annoying
for my use cases.
Rob
Filip Honckiewicz
I'm working on pretty same thing also in python... Now I'm sad,
because you made it and I don't, and also my code looks like dung in
comparison to yours. How long did you make this?
BTW. "SLOW_WORK" symbol used in drivers/gpu/drm/Kconfig in "menucofnig
DRM" as one of selects isn't an unnecessary symbol? I can't find any
SLOW_WORK entry.
Filip
Ulf Magnusson
I have added short descriptions to "make help", mostly referring people
to the script itself. Do you think I should explain it more fully? I
felt bad about adding big blobs of text when the "make help"
descriptions for all the other *config targets are short oneliners :)
>
> 2. My first test (using your ex1.py script) failed because I used O=xx64 (build
> directory):
>
> rddunlap@chimera:lnx-2638-rc3> make O=xx64 scriptconfig SCRIPT=~/pkg/kconfiglib/ex1.py
> GEN /lnx/src/lnx-2638-rc3/xx64/Makefile
> Traceback (most recent call last):
> File "/home/rddunlap/pkg/kconfiglib/ex1.py", line 1, in <module>
> import kconfiglib
> ImportError: No module named kconfiglib
> make[2]: *** [scriptconfig] Error 1
> make[1]: *** [scriptconfig] Error 2
> make: *** [sub-make] Error 2
>
> Does kconfiglib support O=builddir generally? I can't tell that it does (yet)
> since all ex[1-7].py fail in this manner.
> It needs to support/allow O=builddir.
I hadn't considered O. I'll see if I can cook up a patch.
/Ulf
Ulf Magnusson
> On 02/04/2011 04:35 PM, Randy Dunlap wrote:
> >> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> >> (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the
> >> kconfigtest.py test suite - which compares output character-for-character -
> >> will indicate failure on older (a few months old) kernels versions even though
> >> the outputs are functionally equivalent.
> >>
> >> Documentation/kbuild/kconfig-language.txt | 5 +
> >> Documentation/kbuild/kconfig.txt | 8 +
> >> README | 13 +
> >> scripts/kconfig/Makefile | 26 +-
> >> scripts/kconfig/kconfiglib.py | 3918 +++++++++++++++++++++++++++++
> >> scripts/kconfig/kconfigtest.py | 396 +++
> >> 6 files changed, 4365 insertions(+), 1 deletions(-)
> >> create mode 100644 scripts/kconfig/kconfiglib.py
> >> create mode 100644 scripts/kconfig/kconfigtest.py
>
> Wait, I thought this was an extra standalone library. Are you saying
> you want to make it so Linux will no longer compile on a build machine
> that doesn't have Python installed?
No - it's completely standalone, and should have no effect on things
that do not use Kconfiglib. The only changes besides adding the script
itself is to add two (well, three - there's one that's only used by
kconfigtest.py) targets to scripts/kconfig/Makefile that make it easier
to use the library by ensuring the environment is set up correctly.
>
> If this is merely an extra developer tool ala bloat-o-meter and
> checkpatch.pl then it's merely uninteresting to me. (I myself wrote a
> quick and dirty http://kernel.org/doc/make/menuconfig2html.py to
> generate http://kernel.org/doc/menuconfig/x86.html and friends years
> ago, and I still run it to update that once in a while. There's not
> much to it.)
>
> But adding new prerequities to a build machine would be really annoying
> for my use cases.
>
> Rob
You would only need to have Python installed if you are going to use
Kconfiglib directly (hard to get around with a Python library).
/Ulf
Ulf Magnusson
> Hello Ulf!
>
> I'm working on pretty same thing also in python... Now I'm sad,
> because you made it and I don't, and also my code looks like dung in
> comparison to yours. How long did you make this?
You didn't see the early versions ;)
I was a bit afraid that someone would come out with something similar
before it was done. Now you've given me a bad conscience :(
I've been working on it on and off in my spare time for six months or so.
Originally it was a project for automatically generating a minimal kernel for a
given system by automatically turning off options one by one and testing the
resulting kernel in an emulator (yeah, the minimal kernel you get out of that
process is broken in many ways, but you get some idea of what's needed and
what's not at least). That then grew into a general-purpose library.
>
> BTW. "SLOW_WORK" symbol used in drivers/gpu/drm/Kconfig in "menucofnig
> DRM" as one of selects isn't an unnecessary symbol? I can't find any
> SLOW_WORK entry.
iscriptconfig says
A Config instance 'c' for the architecture (i386) has been created.
>>> c["SLOW_WORK"].is_defined()
False
so yes, it it's undefined on i386 at least. Grepping through the kernel shows
it only appears in drivers/gpu/drm/Kconfig, so it's in fact undefined on all
arches.
/Ulf
Ulf Magnusson
(original patch + all fixes) is available at
http://dl.dropbox.com/u/10406197/kconfiglib.patch (apply with 'git am').
Changelog:
v2:
- Now supports alternative output directories (O=).
- $-references were expanded as environment variables in some
contexts ('source', 'mainmenu', and 'defconfig') where they should
have been expanded as symbol values - fixed. The reason this broke
so little is that all symbols whose value come from an environment
variable are currently called the same thing as that variable.
- Added the internal special symbol UNAME_RELEASE, used by
DEFCONFIG_LIST. Previously get_defconfig_filename() failed to find
.configs whose DEFCONFIG_LIST entry involved UNAME_RELEASE - now
works.
- get_defconfig_filename() now searches relative to $srctree before
looking in the current directory, just like the C implementation.
- Updated example 1 to work regardless of build directory.
- Precompiled a few regexes.
Signed-off-by: Ulf Magnusson <ulfaliz...@gmail.com>
---
Convenience links:
Compound patch (original + fixes):
http://dl.dropbox.com/u/10406197/kconfiglib.patch
Latest documentation (generated with pydoc -w kconfiglib):
http://dl.dropbox.com/u/10406197/kconfiglib.html
Latest example files (ex1.py updated in v2):
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
scripts/kconfig/Makefile | 16 ++++---
scripts/kconfig/kconfiglib.py | 105 ++++++++++++++++++++++++++++++----------
scripts/kconfig/kconfigtest.py | 2 +-
3 files changed, 90 insertions(+), 33 deletions(-)
diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
index cfffe87..0044933 100644
--- a/scripts/kconfig/Makefile
+++ b/scripts/kconfig/Makefile
@@ -41,19 +41,21 @@ scriptconfig:
$(Q)if [ ! -n "$(SCRIPT)" ]; then \
echo 'No script argument provided; use "make scriptconfig SCRIPT=<path to script>".'; \
else \
- PYTHONPATH="$(obj):$$PYTHONPATH" "$(PYTHONCMD)" "$(SCRIPT)" "$(Kconfig)"; \
+ PYTHONPATH="$(srctree)/$(src):$$PYTHONPATH" \
+ "$(PYTHONCMD)" "$(SCRIPT)" $(srctree)/$(Kconfig); \
fi
iscriptconfig:
- $(Q)PYTHONPATH="$(obj):$$PYTHONPATH" "$(PYTHONCMD)" -i -c \
- "import kconfiglib; \
- import sys; \
- c = kconfiglib.Config(sys.argv[1]); \
- print \"A Config instance 'c' for the architecture ({0}) has been created.\".format(c.get_arch())" "$(Kconfig)"
+ $(Q)PYTHONPATH="$(srctree)/$(src):$$PYTHONPATH" "$(PYTHONCMD)" -i -c \
+ "import kconfiglib; \
+ import sys; \
+ c = kconfiglib.Config(sys.argv[1]); \
+ print \"A Config instance 'c' for the architecture ({0}) has been created.\".format(c.get_arch())" \
+ $(srctree)/$(Kconfig)
# Used by kconfigtest.py to prevent an 'option defconfig' .config from being loaded
kconfiglibtestconfig: $(obj)/conf
- $(Q)$< --defconfig=.config $(Kconfig)
+ $(Q)$< --defconfig=.config $(srctree)/$(Kconfig)
# if no path is given, then use src directory to find file
ifdef LSMOD
diff --git a/scripts/kconfig/kconfiglib.py b/scripts/kconfig/kconfiglib.py
index 84e70c3..fa3d5ee 100644
--- a/scripts/kconfig/kconfiglib.py
+++ b/scripts/kconfig/kconfiglib.py
@@ -74,8 +74,11 @@ import sys
# these can be created -- the library has no global state).
conf = kconfiglib.Config(sys.argv[1])
-# Load values from a .config file.
-conf.load_config("arch/x86/configs/i386_defconfig")
+# Load values from a .config file. 'srctree' is an environment variable set by
+# the Linux makefiles to the top-level directory of the kernel tree. It needs
+# to be used here for the script to work with alternative build directories
+# (specified e.g. with O=).
+conf.load_config("$srctree/arch/x86/configs/i386_defconfig")
# Print some information about a symbol (the Config class implements
# __getitem__() to provide a handy syntax for getting symbols).
@@ -415,7 +418,7 @@ class Config():
def __init__(self,
filename = "Kconfig",
- base_dir = ".",
+ base_dir = "$srctree",
print_warnings = True,
print_undef_assign = False):
"""Creates a new Config object, representing a Kconfig configuration.
@@ -429,9 +432,15 @@ class Config():
kconfiglib via 'make scriptconfig' the filename of the
correct Kconfig will be in sys.argv[1].
- base_dir (default: ".") -- The base directory relative to which
- 'source' statements within Kconfig files will work. For Linux
- this should be the top-level directory of the kernel tree.
+ base_dir (default: "$srctree") -- The base directory relative to which
+ 'source' statements within Kconfig files will work. For the
+ Linux kernel this should be the top-level directory of the
+ kernel tree. $-references to environment variables will be
+ expanded.
+
+ The environment variable 'srctree' is set by the Linux makefiles
+ to the top-level kernel directory. A default of "." would not
+ work if an alternative build directory is used.
print_warnings (default: True) -- Set to True if warnings related to
this configuration should be printed to stderr. This can
@@ -473,6 +482,9 @@ class Config():
register_special_symbol(TRISTATE, "m", "m")
register_special_symbol(TRISTATE, "y", "y")
+ # DEFCONFIG_LIST uses this
+ register_special_symbol(STRING, "UNAME_RELEASE", os.uname()[2])
+
self.m = self.syms["m"]
# Maps a symbol to its directly dependent symbols (any symbol whose
@@ -487,8 +499,13 @@ class Config():
# See Symbol.get_arch()
self.arch = os.environ.get("ARCH")
+ # See Config.__init__(). We need this for get_defconfig_filename().
+ self.srctree = os.environ.get("srctree")
+ if self.srctree is None:
+ self.srctree = "."
+
self.filename = filename
- self.base_dir = _strip_trailing_slash(base_dir)
+ self.base_dir = _strip_trailing_slash(os.path.expandvars(base_dir))
# The 'mainmenu' text
self.mainmenu_text = None
@@ -532,7 +549,11 @@ class Config():
def load_config(self, filename, reset = True):
"""Loads symbol values from a file in the familiar .config format.
- filename -- The .config file to load.
+ filename -- The .config file to load. $-references to environment
+ variables will be expanded. For scripts to work even
+ when an alternative build directory is used with the
+ Linux kernel, you need to refer to the top-level kernel
+ directory with "$srctree".
reset (default: True) -- True if the configuration should replace
the old configuration; False if it should add to it."""
@@ -544,6 +565,8 @@ class Config():
filename,
linenr)
+ filename = os.path.expandvars(filename)
+
# Put this first so that a missing file doesn't screw up our state
line_feeder = _FileFeed(_get_lines(filename), filename)
@@ -563,7 +586,7 @@ class Config():
def is_header_line(line):
return line.startswith("#") and \
- not re.match(unset_re, line)
+ not unset_re.match(line)
first_line = line_feeder.get_next()
@@ -605,7 +628,7 @@ class Config():
line = line.strip()
- set_re_match = re.match(set_re, line)
+ set_re_match = set_re.match(line)
if set_re_match:
name, val = set_re_match.groups()
val = _strip_quotes(val, line, filename, linenr)
@@ -635,7 +658,7 @@ class Config():
linenr)
continue
- unset_re_match = re.match(unset_re, line)
+ unset_re_match = unset_re.match(line)
if unset_re_match:
name = unset_re_match.group(1)
if name in self.syms:
@@ -688,15 +711,20 @@ class Config():
"""Returns the text of the 'mainmenu' statement (with environment
variables expanded to the value they had when the Config was created),
or None if the configuration has no 'mainmenu' statement."""
- return self.mainmenu_text
+ return self._expand_sym_refs(self.mainmenu_text)
def get_defconfig_filename(self):
- """Returns the name of the defconfig file, which is the first
- existing file in the list given in a symbol having 'option
- defconfig_list' set. $-references to environment variables will be
- expanded. Returns None in case of no defconfig file. Setting 'option
- defconfig_list' on multiple symbols currently results in undefined
- behavior."""
+ """Returns the name of the defconfig file, which is the first existing
+ file in the list given in a symbol having 'option defconfig_list' set.
+ $-references to symbols will be expanded ("$FOO bar" -> "foo bar" if
+ FOO has the value "foo"). Returns None in case of no defconfig file.
+ Setting 'option defconfig_list' on multiple symbols currently results
+ in undefined behavior.
+
+ If the environment variable 'srctree' was set when the Config was
+ created, get_defconfig_filename() will first look relative to that
+ directory before looking in the current directory. See
+ Config.__init__()."""
if self.defconfig_sym is None:
return None
@@ -704,9 +732,16 @@ class Config():
for (filename, cond_expr) in self.defconfig_sym.def_exprs:
cond_val = self._eval_expr(cond_expr)
if cond_val == "y":
- f = os.path.expandvars(filename)
- if os.path.exists(f):
- return f
+ filename = self._expand_sym_refs(filename)
+
+ # We first look in $srctree. os.path.join() won't work here as
+ # an absolute path in filename would override $srctree.
+ srctree_filename = os.path.normpath(self.srctree + "/" + filename)
+ if os.path.exists(srctree_filename):
+ return srctree_filename
+
+ if os.path.exists(filename):
+ return filename
return None
@@ -1295,7 +1330,7 @@ class Config():
elif t0 == T_SOURCE:
kconfig_file = tokens.get_next()
- f = os.path.join(self.base_dir, os.path.expandvars(kconfig_file))
+ f = os.path.join(self.base_dir, self._expand_sym_refs(kconfig_file))
if not os.path.exists(f):
raise IOError, ('{0}:{1}: sourced file "{2}" not found. Perhaps '
@@ -1319,7 +1354,7 @@ class Config():
filename,
linenr)
- self.mainmenu_text = os.path.expandvars(text)
+ self.mainmenu_text = text
else:
_parse_error(line, "unrecognized construct.", filename, linenr)
@@ -1896,6 +1931,23 @@ might be an error, and you should e-mail kconf...@gmail.com.
return "{0} (value: {1})".format(_expr_to_str(expr), _expr_to_str(val))
+ def _expand_sym_refs(self, s):
+ """Expands $-references to symbols in 's' to symbol values, or to the
+ empty string for undefined symbols."""
+
+ while True:
+ sym_ref_re_match = sym_ref_re.search(s)
+ if sym_ref_re_match is None:
+ return s
+
+ sym_name = sym_ref_re_match.group(0)[1:]
+ sym = self.syms.get(sym_name)
+ expansion = "" if sym is None else sym.calc_value()
+
+ s = s[:sym_ref_re_match.start()] + \
+ expansion + \
+ s[sym_ref_re_match.end():]
+
def _get_sym_or_choice_str(self, sc):
"""Symbols and choices have many properties in common, so we factor out
common __str__() stuff here. "sc" is short for "symbol or choice"."""
@@ -2246,8 +2298,11 @@ string_lex = (T_BOOL, T_TRISTATE, T_INT, T_HEX, T_STRING,
sym_chars = frozenset(string.ascii_letters + string.digits + "._/-")
# Regular expressions for parsing .config files
-set_re = r"CONFIG_(\w+)=(.*)"
-unset_re = r"# CONFIG_(\w+) is not set"
+set_re = re.compile(r"CONFIG_(\w+)=(.*)")
+unset_re = re.compile(r"# CONFIG_(\w+) is not set")
+
+# Regular expression for finding $-references to symbols in strings
+sym_ref_re = re.compile(r"\$[A-Za-z_]+")
# Integers representing symbol types
UNKNOWN, BOOL, TRISTATE, STRING, HEX, INT = range(0, 6)
diff --git a/scripts/kconfig/kconfigtest.py b/scripts/kconfig/kconfigtest.py
index 9d27dca..e6961a4 100644
--- a/scripts/kconfig/kconfigtest.py
+++ b/scripts/kconfig/kconfigtest.py
@@ -81,7 +81,7 @@ def get_arch_configs():
def add_arch(ARCH, res):
os.environ["SRCARCH"] = archdir
os.environ["ARCH"] = ARCH
- res.append(kconfiglib.Config())
+ res.append(kconfiglib.Config(base_dir = "."))
res = []
--
1.7.0.4
Ulf Magnusson
(original patch + all fixes) is available at
http://dl.dropbox.com/u/10406197/kconfiglib.patch (apply with 'git am').
Changelog:
v3:
- Forgot to update the documentation for get_mainmenu_text() in v2.
- s/effect/affect/ in example 7.
v2:
- Now supports alternative output directories (O=).
- $-references were expanded as environment variables in some
contexts ('source', 'mainmenu', and 'defconfig') where they should
have been expanded as symbol values - fixed. The reason this broke
so little is that all symbols whose value come from an environment
variable are currently called the same thing as that variable.
- Added the internal special symbol UNAME_RELEASE, used by
DEFCONFIG_LIST. Previously get_defconfig_filename() failed to find
.configs whose DEFCONFIG_LIST entry involved UNAME_RELEASE - now
works.
- get_defconfig_filename() now searches relative to $srctree before
looking in the current directory, just like the C implementation.
- Updated example 1 to work regardless of build directory.
- Precompiled a few regexes.
Signed-off-by: Ulf Magnusson <ulfaliz...@gmail.com>
---
Convenience links:
Latest documentation (generated with pydoc -w kconfiglib):
http://dl.dropbox.com/u/10406197/kconfiglib.html
Latest example files (ex1.py updated in v2):
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
scripts/kconfig/kconfiglib.py | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/scripts/kconfig/kconfiglib.py b/scripts/kconfig/kconfiglib.py
index fa3d5ee..964e5cb 100644
--- a/scripts/kconfig/kconfiglib.py
+++ b/scripts/kconfig/kconfiglib.py
@@ -323,7 +323,7 @@ while True:
sym.set_value(lower_bound)
- # We just changed the value of some symbol. As this may effect
+ # We just changed the value of some symbol. As this may affect
# other symbols, keep going.
done = False
@@ -708,9 +708,9 @@ class Config():
return self.arch
def get_mainmenu_text(self):
- """Returns the text of the 'mainmenu' statement (with environment
- variables expanded to the value they had when the Config was created),
- or None if the configuration has no 'mainmenu' statement."""
+ """Returns the text of the 'mainmenu' statement (with $-references to
+ symbols replaced by symbol values), or None if the configuration has no
+ 'mainmenu' statement."""
return self._expand_sym_refs(self.mainmenu_text)
def get_defconfig_filename(self):
Michal Marek
> Hi,
>
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems. It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
>
> For a much longer introduction including multiple examples, see
> arch/kconfig/kconfiglib.py.
Hi,
this looks like a very powerful tool, but I have a similar concern like
Arnaud had - being completely standalone, it reimplements most of the C
kconfig code. One option to reduce this duplication would be a swig
wrapper, another one would be to let the C code parse the Kconfig files
and write the required information in some digested form, that would be
easier to parse by scripts. Something like:
$ scripts/kconfig/conf_inspect --kconfig=Kconfig --eval='FOO || BAR'
y
$ scripts/kconfig/conf_inspect ... --dump-symbols
config FOO
type: bool
valule: m
visible: y
prompt: "zzz"
depends: X & Y
select: Z
..
$ scripts/kconfig/conf_inspect ... --dump-symbols \
--fields='depends,select,value'
config FOO
depends: X & Y
select: Z
value: m
config BAR
..
etc. The idea is that for instance instead of parsing the Kconfig files,
the Python code could fill it's data structures by reading the flat dump
provided by the C kconfig. There would be still lot to do in Python,
e.g. parsing and evaluating expressions, but it would be a small step
forward already. And people wanting to write quick&dirty scripts in
bash/awk/perl would make use of the C code as well.
Michal
Ulf Magnusson
> $ scripts/kconfig/conf_inspect ... --dump-symbols \
> --fields='depends,select,value'
> config FOO
> depends: X & Y
> select: Z
> value: m
>
> config BAR
>
> etc. The idea is that for instance instead of parsing the Kconfig files,
> the Python code could fill it's data structures by reading the flat dump
> provided by the C kconfig. There would be still lot to do in Python,
> e.g. parsing and evaluating expressions, but it would be a small step
> forward already. And people wanting to write quick&dirty scripts in
> bash/awk/perl would make use of the C code as well.
>
though some Kconfig subleties might complicate things. For example,
symbols that appear within choices aren't seen as real selectable choice
items if they depend on previous symbols in specific ways (see
_determine_actual_items() in kconfiglib.py), and capturing this
information in the output while at the same preserving the ordering of
the symbols could be messy. I guess you could sort this out within
Kconfiglib instead, though other tools might then have to reimplement
the same (messy) rules.
I'm not so sure about offloading anything else this way. For dynamically
updating symbol values in a script you would need to somehow preserve
state between invocations (or pass huge lists of assignments back and
forth), which could lead to an unwieldy protocol. You might need to go
the SWIG route instead.
/Ulf
Ulf Magnusson
in .config files are written out in the order symbols appear within the
Kconfig files), so you could probably just iterate over all items in
Kconfig order after parsing + implicit submenu creation (which causes
the weird choice behavior) to capture such stuff as well.