[Announce] Linux-tiny project revival
Tim Bird
Linux-tiny project. At OLS, I asked for interested parties
to volunteer to become the new maintainer for the Linux-tiny patchset.
A few candidates came forward, but eventually Michael Opdenacker
was selected as the new primary maintainer. A few other
people, including John Cooper of Wind River and myself
are working to support this effort.
Recently, many of the Linux-tiny patches have been brought up-to-date
and are now available for use with a 2.6.22 kernel. The intent
is to test these, and begin mainlining the most effective sub-patches,
in the next few months.
Some automated testing has already been set up, with some
preliminary results published at a CELF conference in Japan.
(See the linux-tiny page below for a link to the presentation.)
Hopefully, results publishing will also be automated soon.
We encourage anyone with interest in this project to get involved.
If you have ideas how to reduce the static or dynamic memory footprint
of Linux, or, even better, patches for this, please let us know about
them.
Please see http://elinux.org/Linux_Tiny
A related document: http://elinux.org/Kernel_Size_Tuning_Guide
is undergoing an update this week.
Thanks,
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Luis R. Rodriguez
> Recently, the CE Linux forum has been working to revive the
> Linux-tiny project. At OLS, I asked for interested parties
> to volunteer to become the new maintainer for the Linux-tiny patchset.
>
> A few candidates came forward, but eventually Michael Opdenacker
> was selected as the new primary maintainer. A few other
> people, including John Cooper of Wind River and myself
> are working to support this effort.
>
> Recently, many of the Linux-tiny patches have been brought up-to-date
> and are now available for use with a 2.6.22 kernel. The intent
> is to test these, and begin mainlining the most effective sub-patches,
> in the next few months.
>
> Some automated testing has already been set up, with some
> preliminary results published at a CELF conference in Japan.
> (See the linux-tiny page below for a link to the presentation.)
> Hopefully, results publishing will also be automated soon.
>
> We encourage anyone with interest in this project to get involved.
> If you have ideas how to reduce the static or dynamic memory footprint
> of Linux, or, even better, patches for this, please let us know about
> them.
>
> Please see http://elinux.org/Linux_Tiny
>
> A related document: http://elinux.org/Kernel_Size_Tuning_Guide
> is undergoing an update this week.
Will there be a separate git for testing? Is the idea to still keep
moving patches upstream?
Luis
Christian MICHON
> Linux-tiny project. At OLS, I asked for interested parties
> to volunteer to become the new maintainer for the Linux-tiny patchset.
>
> A few candidates came forward, but eventually Michael Opdenacker
> was selected as the new primary maintainer. A few other
> people, including John Cooper of Wind River and myself
> are working to support this effort.
>
congratulations to the new maintainer!
nice to see this subset of patches being revived.
;-)
--
Christian
--
http://detaolb.sourceforge.net/, a linux distribution for Qemu with Git inside !
Andi Kleen
> Recently, the CE Linux forum has been working to revive the
> Linux-tiny project. At OLS, I asked for interested parties
> to volunteer to become the new maintainer for the Linux-tiny patchset.
Not sure what the point is of a separate patchkit. If it's a reasonable
patch it should just be put into mainline. And hopefully all the patches
will be reasonable.
-Andi
Tim Bird
> Will there be a separate git for testing?
Right now we're publishing a quilt-able tarball.
Instructions for applying this are on the
http://elinux.org/Linux_Tiny page.
Note that some broken patches are kept around
in the patches/tiny directory, but commented
out in the series file, in case someone
wants to un-bitrot them.
> Is the idea to still keep
> moving patches upstream?
Yes, very much so. These patches have been
showing up in CE products, and this is part
of an overall effort to dust them off, test
them, and move them upstream.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
Tim Bird
> Tim Bird <tim....@am.sony.com> writes:
>
>
>> Recently, the CE Linux forum has been working to revive the
>> Linux-tiny project. At OLS, I asked for interested parties
>> to volunteer to become the new maintainer for the Linux-tiny patchset.
>
> Not sure what the point is of a separate patchkit. If it's a reasonable
> patch it should just be put into mainline. And hopefully all the patches
> will be reasonable.
I don't know the detailed history, but my understanding is that
for some of these, mainline attempts were made in the past.
The patchkit gives a place for things to live while they are out
of mainline, and still have multiple people use and work on them.
Optimally the duration of being out-of-mainline would be short, but
my experience is that sometimes what an embedded developer considers
reasonable to hack off the kernel is not considered so reasonable by
other developers (even with config options). Also, sometimes it
takes a while for a patch to mature to the point where it is
acceptable for general use, while still being usable for
special-purpose projects.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
Valdis.K...@vt.edu
> The patchkit gives a place for things to live while they are out
> of mainline, and still have multiple people use and work on them.
Is anybody working on testing that the patchkit "does no harm" for bigger
boxes (laptops, desktops, servers)? That would probably be helpful info
when pieces are pushed towards mainline....
(Was just looking at the patch details, there's at least a few that look
interesting for my purposes even though I'm not an embedded developer)...
Andrew Morton
Tim Bird <tim....@am.sony.com> wrote:
> Recently, the CE Linux forum has been working to revive the
> Linux-tiny project. At OLS, I asked for interested parties
> to volunteer to become the new maintainer for the Linux-tiny patchset.
I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.
Seriously, putting this stuff into some private patch collection should
be a complete last resort - you should only do this with patches which
you (and the rest of us) agree have no hope of ever getting into mainline.
Tim Bird
> On Wed, 19 Sep 2007 12:41:08 PDT, Tim Bird said:
>> The patchkit gives a place for things to live while they are out
>> of mainline, and still have multiple people use and work on them.
>
> Is anybody working on testing that the patchkit "does no harm" for bigger
> boxes (laptops, desktops, servers)?
only activated by config options. So my sense is that just
applying the patches should be harmless. But we'll need some
runtime testing to see what affect some of these have on
big iron.
Tim Bird
> On Wed, 19 Sep 2007 11:03:09 -0700
> Tim Bird <tim....@am.sony.com> wrote:
>
>> Recently, the CE Linux forum has been working to revive the
>> Linux-tiny project. At OLS, I asked for interested parties
>> to volunteer to become the new maintainer for the Linux-tiny patchset.
>
> I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.
>
> Seriously, putting this stuff into some private patch collection should
> be a complete last resort - you should only do this with patches which
> you (and the rest of us) agree have no hope of ever getting into mainline.
OK, I'll try to accelerate the effort to send these to you.
We'll still need some kind of bucket for the patches that
don't apply to recent kernels, but which no one has yet
had time to bring up-to-date (or evaluate for permanent
dismissal). And dribbling them out, fixing them up,
responding to issues - all take time that I can't
commit to personally for the next week or so.
I'll let Michael respond whether he can get to this
sooner rather than later, as planned.
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
Michael Opdenacker
> Andrew Morton wrote:
>
>> On Wed, 19 Sep 2007 11:03:09 -0700
>> Tim Bird <tim....@am.sony.com> wrote:
>>
>>
>>> Recently, the CE Linux forum has been working to revive the
>>> Linux-tiny project. At OLS, I asked for interested parties
>>> to volunteer to become the new maintainer for the Linux-tiny patchset.
>>>
>> I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.
>>
>> Seriously, putting this stuff into some private patch collection should
>> be a complete last resort - you should only do this with patches which
>> you (and the rest of us) agree have no hope of ever getting into mainline.
>>
>
> OK, I'll try to accelerate the effort to send these to you.
> We'll still need some kind of bucket for the patches that
> don't apply to recent kernels, but which no one has yet
> had time to bring up-to-date (or evaluate for permanent
> dismissal). And dribbling them out, fixing them up,
> responding to issues - all take time that I can't
> commit to personally for the next week or so.
> I'll let Michael respond whether he can get to this
> sooner rather than later, as planned.
>
Andrew, you're completely right... The patches should all aim at being
included into mainline or die.
I'm finishing a sequence of crazy weeks and I will have time to send you
patches one by one next week, starting with the easiest ones.
Thanks for your support.
Cheers,
Michael.
--
Michael Opdenacker
http://free-electrons.com
+33 621 604 642
Michael Opdenacker
> Valdis.K...@vt.edu wrote:
>
>> Is anybody working on testing that the patchkit "does no harm" for bigger
>> boxes (laptops, desktops, servers)?
>>
> Not to my knowledge. Most of the things it provides are
> only activated by config options. So my sense is that just
> applying the patches should be harmless. But we'll need some
> runtime testing to see what affect some of these have on
> big iron.
>
options.
However, that doesn't prevent us from proposing size reduction changes
impacting all kernel users, in cases a configuration option doesn't make
sense. Extensive testing in the -mm tree will then be required.
Cheers,
Michael.
--
Michael Opdenacker
http://free-electrons.com
+33 621 604 642
-
Andy Whitcroft
> Andrew, you're completely right... The patches should all aim at being
> included into mainline or die.
>
> I'm finishing a sequence of crazy weeks and I will have time to send you
> patches one by one next week, starting with the easiest ones.
Well thats good news. In response to the comments made about testing
the impact of these patches on big-iron I was going to suggest we ask
Andrew to include your patch set in -mm so that it firstly gets at least
compiled on big-iron, and secondly so we could think about how to test
with some of the options enabled on big-iron.
Knowing nothing about these options, from a test perspective it would
be nice if we were able to simply enable "the lot" so we can do "normal"
-mm runs and "tiny" -mm runs without any manual intervention?
-apw
Tim Bird
> Knowing nothing about these options, from a test perspective it would
> be nice if we were able to simply enable "the lot" so we can do "normal"
> -mm runs and "tiny" -mm runs without any manual intervention?
I agree completely.
I have been thinking for a while about how to make a "monster switch"
(the kind they always seem to have in Frankenstein movies) that
switches a whole bunch of settings at once. We currently have methods
in the kernel for:
* default (or recommended) config for a particular platform
* all yes - to build as much as possible
* all no - to build as little as possible
The problem with "allno" is that it rarely produces a usable
kernel.
There are three possible approaches that I can think of:
1) use a tool to start from default and turn off options
to make a small (but still usable) config
* I have a tool to do this now as part of my automated test
I haven't published it, but I can if anyone's interested.
2) use the kconfig dependency system to disable stuff automatically
if someone chooses the "make_it_small" option.
3) create defconfig_small files for the platforms that care about
size
3) is easiest to implement at first. It's trivial to make a new
defconfig, and we could easily come up with a convention for them.
However, they would be a pain to maintain (this would essentially
double the defconfig maintenance), and you'd have to convince
people that it's worth carrying these in the mainline tree.
I haven't looked at 2), so I'm not sure exactly what would be
involved. I'm not sure if you can centralize all the dependency
information in the "make_it_small" option, or if you'd have
to spread it out to the related configs. I'm not even sure which
arrangement of the info would be the easiest to maintain. Would
it be best to have a single size-conscious person maintain the
dependencies, or better for config authors to maintain this info
in parallel?
Anyway, those are just some thoughts on the subject.
Feedback on an acceptable solution would be welcome.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
Rob Landley
> Recently, the CE Linux forum has been working to revive the
> Linux-tiny project. At OLS, I asked for interested parties
> to volunteer to become the new maintainer for the Linux-tiny patchset.
>
> A few candidates came forward, but eventually Michael Opdenacker
> was selected as the new primary maintainer. A few other
> people, including John Cooper of Wind River and myself
> are working to support this effort.
>
> Recently, many of the Linux-tiny patches have been brought up-to-date
> and are now available for use with a 2.6.22 kernel. The intent
> is to test these, and begin mainlining the most effective sub-patches,
> in the next few months.
Cool!
Could you update http://www.selenic.com/linux-tiny/ to mention the new
maintainer and new URLs?
> Some automated testing has already been set up, with some
> preliminary results published at a CELF conference in Japan.
> (See the linux-tiny page below for a link to the presentation.)
> Hopefully, results publishing will also be automated soon.
>
> We encourage anyone with interest in this project to get involved.
> If you have ideas how to reduce the static or dynamic memory footprint
> of Linux, or, even better, patches for this, please let us know about
> them.
I've been playing with an idea for a while to improve the printk() situation,
but it's a more intrusive change than I've had time to bang on.
Right now, the first argument to printk() is a loglevel, but it's handled via
string concatenation. I'd like to change that to be an integer, and make it
an actual comma-separated first argument. (Mandatory, not optional.)
So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);
Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5
Then you can change the printk guts to do something vaguely like (untested):
#define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)
And so far no behavior has changed. But now the _fun_ part is, you can add a
config symbol for "what is the minimum loglevel I care about?" Set that as a
number from 0-9. And then you can define the printk to do:
#define printk(level, str, ...) \
do { \
if (level < CONFIG_PRINTK_DOICARE) \
actual_printk("<" #level ">" str, __VA_ARGS__); \
} while(0);
And viola (however you spell that, I think I'm using the stringed instrument
but it's french and I'm not trying to type a diacritical mark anyway), the
compiler's dead code eliminator zaps the printks you don't care about so they
don't bloat the kernel image. But this doesn't _completely_ eliminate
printks, so you can still get the panic() calls and such. You tweak precisly
how much bloat you want, using the granularity information that's already
there in the source code...
Opinions?
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Alexey Dobriyan
> I've been playing with an idea for a while to improve the printk() situation,
> but it's a more intrusive change than I've had time to bang on.
>
> Right now, the first argument to printk() is a loglevel, but it's handled via
> string concatenation. I'd like to change that to be an integer, and make it
> an actual comma-separated first argument. (Mandatory, not optional.)
>
> So instead of:
> printk(KERN_NOTICE "Fruit=%d\n", banana);
> It would now be:
> printk(KERN_NOTICE, "Fruit=%d\n", banana);
>
> Change the header from:
> #define KERN_NOTICE "<5>"
> to:
> #define KERN_NOTICE 5
>
> Then you can change the printk guts to do something vaguely like (untested):
> #define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)
>
> And so far no behavior has changed. But now the _fun_ part is, you can add a
> config symbol for "what is the minimum loglevel I care about?"
Given that
a) there're plenty of printks without any KERN_* bloat,
b) there're printks that SHOULD NOT have KERN_* bloat,
c) debugging-by-printk method is widely used and this will force
additional typing, head-scratching and swear words
d) time wasted on pointless discussions whether some particular
printk ALERT or CRIT
e) flag day for printk,
I think that this idea is not worth it.
> #define printk(level, str, ...) \
> do { \
> if (level < CONFIG_PRINTK_DOICARE) \
> actual_printk("<" #level ">" str, __VA_ARGS__); \
> } while(0);
> Opinions?
Ick.
Alexey "ignore_loglevel" Dobriyan
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -52,6 +52,7 @@ extern const char linux_proc_banner[];
*/
#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
+#ifdef CONFIG_FOO
#define KERN_EMERG "<0>" /* system is unusable */
#define KERN_ALERT "<1>" /* action must be taken immediately */
#define KERN_CRIT "<2>" /* critical conditions */
@@ -59,6 +60,15 @@ extern const char linux_proc_banner[];
#define KERN_WARNING "<4>" /* warning conditions */
#define KERN_NOTICE "<5>" /* normal but significant condition */
#define KERN_INFO "<6>" /* informational */
+#else
+#define KERN_EMERG ""
+#define KERN_ALERT ""
+#define KERN_CRIT ""
+#define KERN_ERR ""
+#define KERN_WARNING ""
+#define KERN_NOTICE ""
+#define KERN_INFO ""
+#endif
#define KERN_DEBUG "<7>" /* debug-level messages */
extern int console_printk[];
Joe Perches
> And so far no behavior has changed. But now the _fun_ part is, you can add a
> config symbol for "what is the minimum loglevel I care about?" Set that as a
> number from 0-9. And then you can define the printk to do:
>
> #define printk(level, str, ...) \
> do { \
> if (level < CONFIG_PRINTK_DOICARE) \
> actual_printk("<" #level ">" str, __VA_ARGS__); \
> } while(0);
>
> And viola (however you spell that, I think I'm using the stringed instrument
> But this doesn't _completely_ eliminate
> printks, so you can still get the panic() calls and such. You tweak precisly
> how much bloat you want, using the granularity information that's already
> there in the source code...
> Opinions?
I'd rather take the opportunity to convert all the printks to
use pr_<level>. That way, you can pick'n'choose if you want
arbitrary combinations of KERN_<level> compiled in or not.
Tim Bird
> Given that
> a) there're plenty of printks without any KERN_* bloat,
> b) there're printks that SHOULD NOT have KERN_* bloat,
Just to clarify, which bloat are you concerned about?
I presume source code bloat (but maybe you mean
message size bloat, or object code bloat)?
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
Rob Landley
> Andy Whitcroft wrote:
> > Knowing nothing about these options, from a test perspective it would
> > be nice if we were able to simply enable "the lot" so we can do "normal"
> > -mm runs and "tiny" -mm runs without any manual intervention?
>
> I agree completely.
>
> I have been thinking for a while about how to make a "monster switch"
> (the kind they always seem to have in Frankenstein movies) that
> switches a whole bunch of settings at once. We currently have methods
> in the kernel for:
> * default (or recommended) config for a particular platform
> * all yes - to build as much as possible
> * all no - to build as little as possible
>
> The problem with "allno" is that it rarely produces a usable
> kernel.
Beyond that, allno doesn't come close to switching everything off.
1) You have to _enable_ CONFIG_EMBEDDED in order to go into that menu and
switch _off_ the stuff in there.
2) The stuff CONFIG_EMBEDDED reveals isn't all in that menu. CONFIG_BLOCK is
at the top level menu. CONFIG_VT and CONFIG_UNIX98_PTYS are buried down
under device drivers->character devices, and there's more sprinkled all over.
You have to track it all down and switch it off to get an _actual_
allnoconfig kernel.
(I cut the bit where you reinvent miniconfig. People keep doing this. I dig
it up and resubmit it every year or so, so Roman Zippel can shoot it down
again. Meanwhile, not only is Firmware Linux happily using it, but I even
wrote more documentation at
http://landley.net/code/firmware/new_platform.html although you have to
scroll down a bit to get to the stuff about miniconfig...)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Randy Dunlap
I noticed that too.
> it up and resubmit it every year or so, so Roman Zippel can shoot it down
> again. Meanwhile, not only is Firmware Linux happily using it, but I even
> wrote more documentation at
> http://landley.net/code/firmware/new_platform.html although you have to
> scroll down a bit to get to the stuff about miniconfig...)
I use it for daily build/boot/run-some-number-like-30-tests kernel testing.
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
Rob Landley
So define a level 0 that doesn't prepend any level to the string, and have the
macro filter that out at the same default level it counts as now.
(KERN_INFO, I think?) The tests are all on contants which should resolve at
compile time and the dead code eliminator should zap it, even if the macro
gets more complicated it shouldn't result in a bigger binary.
> c) debugging-by-printk method is widely used and this will force
> additional typing, head-scratching and swear words
Because we never change kernel internal APIs. Oh yeah. Never happens.
> d) time wasted on pointless discussions whether some particular
> printk ALERT or CRIT
Let me get this straight: you're objecting to actually making the printk
levels useful enough that developers start to care what they're set to,
because then they might be motivated to want some of them changed?
Make it useful, people might care, thus they might talk about it...
Sorry, I'm still missing the downside here.
> e) flag day for printk,
That's the main reason I haven't played with it so far, although it would be
easy to define a new symbol (dprintk or some such, although I note several
drivers are already using that) and transition gradually.
> I think that this idea is not worth it.
*Shrug*.
My problem is that switching off printk is the single biggest bloat cutter in
the kernel, yet it makes the resulting system very hard to support. It
combines a big upside with a big downside, and I'd like something in between.
> > #define printk(level, str, ...) \
> > do { \
> > if (level < CONFIG_PRINTK_DOICARE) \
> > actual_printk("<" #level ">" str, __VA_ARGS__); \
> > } while(0);
> >
> > Opinions?
>
> Ick.
>
> Alexey "ignore_loglevel" Dobriyan
But ignore_loglevel doesn't decrease the size of the _binary_. That's what
we're talking about here with the -tiny tree. Embedded developers want to
squeeze more code onto smaller flash/rom chips. Setting ignore_loglevel does
prevent these messages from ever being emitted, but they're still in the
kernel image as dead weight. It saves noise but doesn't save _space_.
I'm proposing allowing an ignore_loglevel to remove the unused messages at
compile time so they don't take up space. Doing that requires the levels to
be integers so they can be compared with < or >, and the remaining changes
follow logically. (To me, anyway...)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Jared Hulbert
Don't use the config option then....
> My problem is that switching off printk is the single biggest bloat cutter in
> the kernel, yet it makes the resulting system very hard to support. It
> combines a big upside with a big downside, and I'd like something in between.
It's not such a big downside IMHO. You can support a kernel without
printk. Need to debug the kernel without printk? Use a JTAG
debugger...
If you have a system that actually configures out printk's, chances
are you don't have storage and output mechanisms to do much with the
messages anyway. Think embedded _products_ here. Sure the
development boards have serial, ethernet, and all that jazz but tens
of millions of ARM based gadgets don't.
Rob Landley
> > > I think that this idea is not worth it.
>
> Don't use the config option then....
>
> > My problem is that switching off printk is the single biggest bloat
> > cutter in the kernel, yet it makes the resulting system very hard to
> > support. It combines a big upside with a big downside, and I'd like
> > something in between.
>
> It's not such a big downside IMHO. You can support a kernel without
> printk. Need to debug the kernel without printk? Use a JTAG
> debugger...
I don't actually own a jtag. (I do use qemu's gdb support to debug the target
kernel, but it's darn awkward and has limited hardware support.)
> If you have a system that actually configures out printk's, chances
> are you don't have storage and output mechanisms to do much with the
> messages anyway. Think embedded _products_ here.
I plead the fifth.
> Sure the
> development boards have serial, ethernet, and all that jazz but tens
> of millions of ARM based gadgets don't.
I wonder if that "Monsoon" gadget does? (Sorry, just on my mind today...)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Indan Zupancic
> I've been playing with an idea for a while to improve the printk() situation,
> but it's a more intrusive change than I've had time to bang on.
>
> Right now, the first argument to printk() is a loglevel, but it's handled via
> string concatenation. I'd like to change that to be an integer, and make it
> an actual comma-separated first argument. (Mandatory, not optional.)
>
> So instead of:
> printk(KERN_NOTICE "Fruit=%d\n", banana);
> It would now be:
> printk(KERN_NOTICE, "Fruit=%d\n", banana);
>
> Change the header from:
> #define KERN_NOTICE "<5>"
> to:
> #define KERN_NOTICE 5
You have to jump through less hoops if you do:
#define KERN_NOTICE 5,
But the problem remains that there are printk's which don't have
a KERN_* as the first argument. Those are also impossible to get
rid off in this way, as the loglevel is unknown (and you don't want
partially printed messages).
So adding the comma is really needed and in addition all printk's
without a loglevel should get one. Which clutters the code and may
increase codesize.
A quick scroll through a vmlinux binary shows that there are quite a
lot areas consisting only of some repeated pattern. Mostly 0x00, but
also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save
something between 50 and 100KB.
Greetings,
Indan
Tim Bird
> So instead of:
> printk(KERN_NOTICE "Fruit=%d\n", banana);
> It would now be:
> printk(KERN_NOTICE, "Fruit=%d\n", banana);
>
> Change the header from:
> #define KERN_NOTICE "<5>"
> to:
> #define KERN_NOTICE 5
>
> Then you can change the printk guts to do something vaguely like (untested):
> #define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)
> [then] the
> compiler's dead code eliminator zaps the printks you don't care about so they
> don't bloat the kernel image.
I agree in principal with the idea, but there are some major
practical wrinkles that would have to be worked through.
First, not all printks that are missing a log level should have one.
People do stuff like this:
printk(KERN_INFO "interesting info follows:");
..
printk("var5: %d\n", var5);
Or even things that evaluate to:
printk("");
The code inside printk currently has to examine the
strings, looking for line feeds and inserting log levels.
Given that there are about 60,000 printks in the kernel (and that's
not counting wrappers like dprintk() and other locally-defined
functions and macros) it would be a huge task to examine the code
and differentiate strings that really start a new log message
(and thus should have an attached log level) and strings
that don't.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
-
Rob Landley
> On Wed, 19 Sep 2007 11:03:09 -0700
>
> Tim Bird <tim....@am.sony.com> wrote:
> > Recently, the CE Linux forum has been working to revive the
> > Linux-tiny project. At OLS, I asked for interested parties
> > to volunteer to become the new maintainer for the Linux-tiny patchset.
>
> I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.
>
> Seriously, putting this stuff into some private patch collection should
> be a complete last resort - you should only do this with patches which
> you (and the rest of us) agree have no hope of ever getting into mainline.
History!
<computer historian hat on>
The -tiny tree started out as a separate patch kit of Matt Mackall's, which he
stopped updating circa 2.6.14 because he didn't think keeping them out of
tree was helping attract other developers, nor was it helping to get them
inline. He decided to focus on pushing the existing patches into mainline,
and stop maintaining the out of tree patcheset for new releases. His last
post on the subject (to the linux-tiny mailing list) was a year ago:
http://selenic.com/pipermail/linux-tiny/2006-March/000314.html
But what happened is that most of the abandoned patches stopped applying to
new kernels yet still weren't available in mainline a year later, so Tim and
Michael have stepped in to revive the -tiny tree. (Tim talked about this a
bit at the CELF BOF at OLS, which is more acronyms than should really show up
immediately after one another in any confersation, FYI.)
So yay new tree. Tried without it, didn't work. Broken up to make merging
easier, but mainline will probably never _fully_ catch up, any more than
it'll catch up with any of the other special-interest development trees.
Making -tiny an .hg tree would be really really nice, though... :)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Rob Landley
> Rob Landley wrote:
> > So instead of:
> > printk(KERN_NOTICE "Fruit=%d\n", banana);
> > It would now be:
> > printk(KERN_NOTICE, "Fruit=%d\n", banana);
> >
> > Change the header from:
> > #define KERN_NOTICE "<5>"
> > to:
> > #define KERN_NOTICE 5
> >
> > Then you can change the printk guts to do something vaguely like
> > (untested): #define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">"
> > arg2, __VA_ARGS__)
>
>
> > [then] the
> > compiler's dead code eliminator zaps the printks you don't care about so
> > they don't bloat the kernel image.
>
> I agree in principal with the idea, but there are some major
> practical wrinkles that would have to be worked through.
>
> First, not all printks that are missing a log level should have one.
> People do stuff like this:
>
> printk(KERN_INFO "interesting info follows:");
> printk("var5: %d\n", var5);
>
> Or even things that evaluate to:
> printk("");
>
> The code inside printk currently has to examine the
> strings, looking for line feeds and inserting log levels.
>
> Given that there are about 60,000 printks in the kernel (and that's
> not counting wrappers like dprintk() and other locally-defined
> functions and macros) it would be a huge task to examine the code
> and differentiate strings that really start a new log message
> (and thus should have an attached log level) and strings
> that don't.
Hmmm. The hard part isn't making printk(0,blah) mean the same as not having a
log level message now, because the current logic already handles it. The
problem is that filtering continuations of previous messages involves knowing
what log level the previous message was so you know whether or not to filter
it.
Yeah, that would take some doing to untangle. An incremental switchever (easy
printks first, I.E. the ones that currently specify a loglevel) seems more
strongly indicated...
That said, I started this by noting I haven't personally had time to bang on
this since I thought of it. You did ask for ideas. :)
> -- Tim
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Joe Perches
> Given that there are about 60,000 printks in the kernel (and that's
> not counting wrappers like dprintk() and other locally-defined
> functions and macros) it would be a huge task to examine the code
> and differentiate strings that really start a new log message
> (and thus should have an attached log level) and strings
> that don't.
I've converted most all of that treewide.
printk(KERN_<level> to pr_<level>(
It's pretty automated.
$ cat pr_alert.sh
#!/bin/sh
egrep -r -w --include=*.[ch] -l "printk[[:space:]]*\([[:space:]]*KERN_ALERT" * | \
xargs perl ../cvt_pr.pl KERN_ALERT pr_alert
$ cat cvt_pr.pl
if ($#ARGV < 3) {
print "usage: KERN_<level> pr_<level> files...\n";
exit;
}
for ($i=2; $i<$#ARGV; $i++) {
PrintkSearchReplace($ARGV[$i], $ARGV[0], $ARGV[1]);
}
sub PrintkSearchReplace{
my($file, $search, $replace) = @_;
my $content = "";
local( $/ );
open( my $fh, $file ) or die "File not found '$file'\n";
$content = <$fh>;
close(my $fh);
my $orig = $content;
$content =~ s/\bprintk[[:space:]]*\([[:space:]]*${search}[[:space:]]*([^\"]*)\"([^\\]*)\\n\"/${replace}\(\1 \"\2\"/mgs;
$content =~ s/\b${replace}\( /${replace}\(/mgs;
if ($orig ne $content)
{
open(my $fh, ">${file}") or die "Could not open '$file'\n";
print $fh $content;
close(my $fh);
Gross, Mark
What about getting even more hard core?
Use compiler tricks to remove ALL the static printk string from the
kernel and replace the printk with something that outputs an decimal
index followed by tuples, of zero to N, hex-strings on a single line.
Then have the syslogd or some other utility take this cryptic output and
convolve it with a table (created at compile time) to re-create what
would have been dumped to the sys-log ring buffer. This way you strip
out most of the static text from the kernel and yet can still re-create
the kernlog output.
At least as a post processing operation....
Is this an old idea? I'm guessing this has been at least proposed
before....
--mgross
the
>_______________________________________________
>Celinux-dev mailing list
>Celin...@tree.celinuxforum.org
>http://tree.celinuxforum.org/mailman/listinfo/celinux-dev
Rob Landley
> On Thu, September 20, 2007 22:38, Rob Landley wrote:
> > I've been playing with an idea for a while to improve the printk()
> > situation, but it's a more intrusive change than I've had time to bang
> > on.
> >
> > Right now, the first argument to printk() is a loglevel, but it's handled
> > via string concatenation. I'd like to change that to be an integer, and
> > make it an actual comma-separated first argument. (Mandatory, not
> > optional.)
> >
> > So instead of:
> > printk(KERN_NOTICE "Fruit=%d\n", banana);
> > It would now be:
> > printk(KERN_NOTICE, "Fruit=%d\n", banana);
> >
> > Change the header from:
> > #define KERN_NOTICE "<5>"
> > to:
> > #define KERN_NOTICE 5
>
> You have to jump through less hoops if you do:
>
> #define KERN_NOTICE 5,
Less change to the source, but the result is less obvious about what it's
doing. I'd personally rather have the churn than wind up with magic
syntax...
> But the problem remains that there are printk's which don't have
> a KERN_* as the first argument. Those are also impossible to get
> rid off in this way, as the loglevel is unknown (and you don't want
> partially printed messages).
>
> So adding the comma is really needed and in addition all printk's
> without a loglevel should get one. Which clutters the code and may
> increase codesize.
It's ok to _explicitly_ not have a loglevel, and thus take a known default.
The problem is printing out less than a full line, continuing it later, and
not making obvious at compile time what the level of this chunk is.
> A quick scroll through a vmlinux binary shows that there are quite a
> lot areas consisting only of some repeated pattern. Mostly 0x00, but
> also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save
> something between 50 and 100KB.
Worse, if you feed an absolute path to O= when you build the kernel out of
tree, then it uses absolute paths for all the __FILE__ strings and that makes
kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way
to keep the __FILE__ strings compressed at runtime and gunzip them as needed
like busybox does with help messages... :)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Indan Zupancic
> On Thursday 20 September 2007 4:26:13 pm Indan Zupancic wrote:
>> A quick scroll through a vmlinux binary shows that there are quite a
>> lot areas consisting only of some repeated pattern. Mostly 0x00, but
>> also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save
>> something between 50 and 100KB.
>
> Worse, if you feed an absolute path to O= when you build the kernel out of
> tree, then it uses absolute paths for all the __FILE__ strings and that makes
> kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way
> to keep the __FILE__ strings compressed at runtime and gunzip them as needed
> like busybox does with help messages... :)
I suspect that can be fixed by changing the built system. How can using O=
change the source file path anyway? That seems unnecessary.
It seems to be worse, full pathnames are also used when giving a relative path.
(I'm using O=../obj/).
On the other hand, it doesn't seem to cause that much bloat here:
$ strings vmlinux | grep /home/ |wc
119 181 6400
CC'ing Sam Ravnborg, perhaps he has some ideas.
Greetings,
Indan
Rob Landley
> On Thu, 2007-09-20 at 14:58 -0700, Tim Bird wrote:
> > Given that there are about 60,000 printks in the kernel (and that's
> > not counting wrappers like dprintk() and other locally-defined
> > functions and macros) it would be a huge task to examine the code
> > and differentiate strings that really start a new log message
> > (and thus should have an attached log level) and strings
> > that don't.
>
> I've converted most all of that treewide.
>
> printk(KERN_<level> to pr_<level>(
>
> It's pretty automated.
Perl, being a write-only language, does not help my poor little brain
understand what's going on. You convert printk(KERN_INFO, blah) to
pr_INFO(blah)? I'm not finding pr_INFO with a grep on the files in
2.6.23-rc7. Is this something you added?
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Joe Perches
> You convert printk(KERN_INFO, blah) to pr_INFO(blah)?
more or less.
printk(KERN_INFO foo) to pr_info(foo)
printk(KERN_EMERG foo) to pr_emerge(foo)
etc.
> I'm not finding pr_INFO with a grep on the files in
> 2.6.23-rc7.
I haven't submitted them.
There's a lot of sensible resistance to what appears
to be churn. Some of the resistance is historical,
merging large changes used to be much more painful
pre git, some is just simple resistance to change.
I started with submitting an add of pr_err to kernel.h
which Andrew Morton picked up for awhile, but dropped.
I've got a local tree with those changes.
for example:
KERN_EMERG -> pr_emerg is ~100KB
KERN_ALERT -> pr_alert is ~80KB
KERN_CRIT -> pr_crit is ~200KB
KERN_NOTICE -> pr_notice is ~400KB
KERN_INFO -> pr_info is ~2500KB
I intended to strip the "\n" trailer from the messages.
Back to the scripts:
In this case, there are multiple files.
A script that finds all the files that contain a search string,
and a perl script that effectively s/search/replace/g on those files.
sed didn't work as well as perl here because I wanted to
play with perl a bit and many printk(KERN_<level> foo)
function calls are split across multiple lines.
I've still got to show a real use to this change set.
I believe controlling the interleaving of log messages
by having multiple statement printks have a start and end,
choosing specific message levels in compiled code,
and choosing to print file/function/line per compiled
code block might do, but more utility to the changes
is probably necessary before it could be applied.
cheers, Joe
Oleg Verych
* X-MimeOLE: Produced By Microsoft Exchange V6.5
[]
>>*Shrug*.
>>
>>My problem is that switching off printk is the single biggest bloat
> cutter
>>in
>>the kernel, yet it makes the resulting system very hard to support. It
>>combines a big upside with a big downside, and I'd like something in
>>between.
>
> What about getting even more hard core?
>
> Use compiler tricks to remove ALL the static printk string from the
> kernel and replace the printk with something that outputs an decimal
> index followed by tuples, of zero to N, hex-strings on a single line.
Not all, but critical info, that must exist in human-readable form of
course.
> Then have the syslogd or some other utility take this cryptic output and
> convolve it with a table (created at compile time) to re-create what
> would have been dumped to the sys-log ring buffer. This way you strip
> out most of the static text from the kernel and yet can still re-create
> the kernlog output.
>
> At least as a post processing operation....
Sure, but a little problem is, that many kernel developers do C (mostly)
and Perl (occasionally), i.e. very few do non-trivial userspace (even
userspace do too much C and Perl sometimes [:
<http://thread.gmane.org/gmane.comp.lib.glibc.alpha/12739>)
> Is this an old idea? I'm guessing this has been at least proposed
> before....
Seriously. When in the Windows there are only messages like:
"Error (Code:0x00002012)".
In the Linux... well, embedded targets, this can be turned in a very
efficient thing by means of userspace. On other setups this can be nice
and pleasant one, with yet another L10N project, recently promoted by
README translations. But,,, see problem above.
____
Sam Ravnborg
> On Fri, September 21, 2007 01:18, Rob Landley wrote:
> > On Thursday 20 September 2007 4:26:13 pm Indan Zupancic wrote:
> >> A quick scroll through a vmlinux binary shows that there are quite a
> >> lot areas consisting only of some repeated pattern. Mostly 0x00, but
> >> also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save
> >> something between 50 and 100KB.
> >
> > Worse, if you feed an absolute path to O= when you build the kernel out of
> > tree, then it uses absolute paths for all the __FILE__ strings and that makes
> > kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way
> > to keep the __FILE__ strings compressed at runtime and gunzip them as needed
> > like busybox does with help messages... :)
>
> I suspect that can be fixed by changing the built system. How can using O=
> change the source file path anyway? That seems unnecessary.
__FILE__ is passed the filename supplied as argument to gcc.
Try:
echo "char *s = __FILE__;" > sam.c
gcc -E sam.c
This gives you:
# 1 "sam.c"
# 1 "<built-in>"
# 1 "<command line>"
# 1 "sam.c"
char *s="sam.c";
gcc -E ~/sam.c
This gives you:
# 1 "/home/sam/sam.c"
# 1 "<built-in>"
# 1 "<command line>"
# 1 "/home/sam/sam.c"
char *s="/home/sam/sam.c";
So __FILE__ expand differently depending on the path on
the gcc command line.
I once posted a patch to fix up on this, especialy for BUG_ON and friends.
The solution was to let kbuild generate the filename and to use
this define in the source code.
But a quick grep for __FILE__ in the kernel source made me chicken out.
Simply too much chrunch at that time to justify it.
Googeling a bit I found it here: http://lkml.org/lkml/2006/7/8/22
The better approach would be to use at least the patch inside
the kernel.
This patch should be easy to update to latest kernel if anyone
is up to play with it.
I recall that there was some problems with the path used.
But I cannot remember the details.
Andrew had some inputs from his testing IIRC and google should
be able to tell the full story.
Sam
Christian MICHON
> (I cut the bit where you reinvent miniconfig. People keep doing this. I dig
> it up and resubmit it every year or so, so Roman Zippel can shoot it down
> again. Meanwhile, not only is Firmware Linux happily using it, but I even
> wrote more documentation at
> http://landley.net/code/firmware/new_platform.html although you have to
> scroll down a bit to get to the stuff about miniconfig...)
DetaolB (see signature) is also using miniconfig, which is good
existing code. It can actually be pushed further by storing a
/proc/miniconfig.gz rather than the full /proc/config.gz
You can find patches on how to perform this inside the iso of DetaolB v0.6
--
Christian
--
http://detaolb.sourceforge.net/, a linux distribution for Qemu with Git inside !
Valdis.K...@vt.edu
> Worse, if you feed an absolute path to O= when you build the kernel out of
> tree, then it uses absolute paths for all the __FILE__ strings and that makes
> kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way
> to keep the __FILE__ strings compressed at runtime and gunzip them as needed
> like busybox does with help messages... :)
What about something *really* hardcore ugly like:
#ifdef __FILE__
#undef __FILE__
#define __FILE__ ""
#endif
(or similar preprocessor blecherousness) if you want to *really* shrink
that binary down?
Joe Perches
> What about something *really* hardcore ugly like:
> #ifdef __FILE__
> #undef __FILE__
> #define __FILE__ ""
> #endif
> (or similar preprocessor blecherousness) if you want to *really* shrink
> that binary down?
I prefer removing all __FILE__, __FUNCTION__, __LINE__ uses
from printks and defining something that modifies pr_<level>.
Something like:
#define PR_FILE
#define PR_FUNCTION
#define PR_LINE
#if defined PR_FILE && defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%s:%u) "
#define PR_ARG , __FILE__ , __FUNCTION__ , __LINE__
#elif defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s:%s) "
#define PR_ARG , __FILE__ , __FUNCTION__
#elif defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%u) "
#define PR_ARG , __FILE__ , __LINE__
#elif defined PR_FILE && !defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s) "
#define PR_ARG , __FILE__
#elif !defined PR_FILE && defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%u) "
#define PR_ARG , __FUNCTION__ , __LINE__
#elif !defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s) "
#define PR_ARG , __FUNCTION__
#elif !defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%u) "
#define PR_ARG , __LINE__
#else
#define PR_FMT
#define PR_ARG
#endif
#define pr_info(fmt, arg) printk(KERN_INFO PR_FMT fmt PR_ARG, ##arg)
Alexey Dobriyan
> Alexey Dobriyan wrote:
> > Given that
> > a) there're plenty of printks without any KERN_* bloat,
> > b) there're printks that SHOULD NOT have KERN_* bloat,
>
> Just to clarify, which bloat are you concerned about?
> I presume source code bloat (but maybe you mean
> message size bloat, or object code bloat)?
Users of ignore_loglevel still has "<[0-7]>" prefixes in kernel image,
yes. On source level, too -- as someone who never saw value in them,
KERN_* are just needless characters, making code harder to read.
Rob Landley
> On Thu, Sep 20, 2007 at 01:22:45PM -0700, Tim Bird wrote:
> > Alexey Dobriyan wrote:
> > > Given that
> > > a) there're plenty of printks without any KERN_* bloat,
> > > b) there're printks that SHOULD NOT have KERN_* bloat,
> >
> > Just to clarify, which bloat are you concerned about?
> > I presume source code bloat (but maybe you mean
> > message size bloat, or object code bloat)?
>
> Users of ignore_loglevel still has "<[0-7]>" prefixes in kernel image,
> yes. On source level, too -- as someone who never saw value in them,
> KERN_* are just needless characters, making code harder to read.
Yeah, I bumped into this when I implemented dmesg for busybox. (Had to parse
and filter it out.) That said, userspace kind of expects it now...
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Rob Landley
> >-----Original Message-----
> >From: Oleg Verych [mailto:ole...@flower.upol.cz]
> >Sent: Thursday, September 20, 2007 5:58 PM
> >To: Gross, Mark
> >Cc: Rob Landley; Alexey Dobriyan; Michael Opdenacker; linux-
> >ti...@selenic.com; CE Linux Developers List; linux kernel
> >Subject: Message codes (Re: [Announce] Linux-tiny project revival)
> >
> >* Thu, 20 Sep 2007 15:15:47 -0700
> >* X-MimeOLE: Produced By Microsoft Exchange V6.5
> >[]
> >
> >>>*Shrug*.
> >>>
> >>>My problem is that switching off printk is the single biggest bloat
> >>
> >> cutter
> >>
> >>>in
> >>>the kernel, yet it makes the resulting system very hard to support.
>
> It
>
> >>>combines a big upside with a big downside, and I'd like something in
> >>>between.
> >>
> >> What about getting even more hard core?
> >>
> >> Use compiler tricks to remove ALL the static printk string from the
> >> kernel and replace the printk with something that outputs an decimal
> >> index followed by tuples, of zero to N, hex-strings on a single line.
> >
> >Not all, but critical info, that must exist in human-readable form of
> >course.
>
> to reduce the communication bandwidth required between the remote
> customer and the support organization.
>
> In fact there is a good argument that you don't what the remote customer
> to know enough to start guessing.
Don't use Linux then. Open source is a horrible fit for the way you think.
I'm sympathetic to "shrink the binary size" arguments. I'm not really
sympathic to "keep the customer in the dark intentionally" arguments, whether
the justification is "because they're stupid", "to increase dependency on our
support staff", or any other reason.
> >Seriously. When in the Windows there are only messages like:
> >
> > "Error (Code:0x00002012)".
>
> Now it's been ~8 years since I did any serious windows work, but if I
> recall correctly ALL THE FRICKING TIME!!! When was the last time you've
> seen a bug check on windows? This is about all you get.
I believe he was holding it up as a bad example, and definitely not something
we want to emulate.
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Kyle Moffett
> On Thursday 20 September 2007 4:26:13 pm Indan Zupancic wrote:
>> But the problem remains that there are printk's which don't have a
>> KERN_* as the first argument. Those are also impossible to get rid
>> off in this way, as the loglevel is unknown (and you don't want
>> partially printed messages).
>>
>> So adding the comma is really needed and in addition all printk's
>> without a loglevel should get one. Which clutters the code and may
>> increase codesize.
>
> It's ok to _explicitly_ not have a loglevel, and thus take a known
> default. The problem is printing out less than a full line,
> continuing it later, and not making obvious at compile time what
> the level of this chunk is.
That's actually a fairly straightforward problem to solve. Really we
ought to be able to guarantee that lines from different CPUs are not
intermixed. This can end up in a log like this:
info: Current device state: <1>netdev watchdog timeout: eth0
reg1=0xABCD reg2=0xDEADBEEF
Clearly unpleasant, no? Really any logging which wants to print out
a bunch of stuff with multiple printk()s should use an API something
like this:
Option 1: Buffer allocated with kmalloc(). Sleep-ish context, can
handle failures easily
> struct printk_queue qpk;
> if (qprintk_kmalloc(&qpk, level, GFP_KERNEL))
> return -ENOMEM;
>
> qprintk(&qpk, "Some string here:");
> qprintk(&qpk, " %d, %s, %d", 4, some_string, 42);
> qprintk_finish(&qpk);
Option 2: Preallocated per-cpu buffer. preempt_disable()d
> struct printk_queue qpk;
> qprintk_percpu(&qpk, level);
> [...]
> qprintk_finish(&qpk);
Option 3: Preallocated per-cpu interrupts-only buffer. Only from
code which may interrupt a preempt_disable() section
> struct printk_queue qpk;
> qprintk_irq(&pqk, level)
> [...]
> qprintk_finish(&qpk);
For all of the above, the final "qprintk_finish()" call would
essentially take the printk spinlock and write the contents of the
qpk->buffer into the dmesg ringbuffer, inserting ("<%d>", level) at
the beginning and after each newline. The buffers would all be some
useful fixed size (a page?); if you need more than that then you are
probably trying to queue too much and excess data would be truncated.
Since the level would be a constant passed to qprintk_
{kmalloc,percpu,irq} in almost every case, you could easily do
something like this:
static inline int qprintk_kmalloc(struct printk_queue *qpk,
unsigned int level, gfp_t gfp)
{
if (level > CONFIG_MAX_LOG_LEVEL) {
qpk->type = QPRINTK_TYPE_NONE; /* also 0 */
qpk->buffer = NULL;
return 0;
}
_qprintk_kmalloc(qpk, level, gfp);
}
#define qprintk(QPK, FMT...) do { \
if ((QPK)->type)
_qprintk(QPK, FMT);
} while(0)
With a bit more glue that would cause GCC to notice that for a given
qprintk_kmalloc the "qpk->type" is always zero because the level is
too high, and therefore it would optimize out *ALL* of the
_qprintk_kmalloc(), _qprintk(), and _qprintk_finish() calls.
Cheers,
Kyle Moffett
Joe Perches
> With a bit more glue that would cause GCC to notice that for a given
> qprintk_kmalloc the "qpk->type" is always zero because the level is
> too high, and therefore it would optimize out *ALL* of the
> _qprintk_kmalloc(), _qprintk(), and _qprintk_finish() calls.
A negative is that lockup conditions swallow partial messages.
Another approach that doesn't require any new buffering is:
id = printk_block_start();
printk_block(id, fmt, ...)
printk_block_end(id)
and have print_block output the id when multiple IDs are
concurrently issued.
This requires a trivial tool to post-process the log
when messages are interleaved.
Gross, Mark
I'll do what I like, thank you. I'll continue to use Linux, I think
it's a fine fit for the way *I* think.
>
>I'm sympathetic to "shrink the binary size" arguments. I'm not really
>sympathic to "keep the customer in the dark intentionally" arguments,
>whether
>the justification is "because they're stupid", "to increase dependency
on
>our
>support staff", or any other reason.
You are now talking religion at this point. Do you have a technical or
even experience based point to make?
I have experience that has shown that providing too much information in
a production product can is confusing, harmful and costly when dealing
with consumers. Your opinions won't change that.
I proposed a mechanism for keeping all the printk data and saving space
buy doing some table based compressions that has the side effect of
making the syslog not human readable. You proposed a mechanism for
no-oping out complete log-levels.
Which way hides more from the user? No-oping the log-levels is the
easier to implement.
>
>> >Seriously. When in the Windows there are only messages like:
>> >
>> > "Error (Code:0x00002012)".
>>
>> Now it's been ~8 years since I did any serious windows work, but if I
>> recall correctly ALL THE FRICKING TIME!!! When was the last time
you've
>> seen a bug check on windows? This is about all you get.
>
>I believe he was holding it up as a bad example, and definitely not
>something
>we want to emulate.
There is a time and place for many things. Even error codes.
--mgross
Joe Perches
> Use compiler tricks to remove ALL the static printk string from
> the kernel and replace the printk with something that outputs a
> I proposed a mechanism for keeping all the printk data and saving space
> buy doing some table based compressions that has the side effect of
> making the syslog not human readable. You proposed a mechanism for
> no-oping out complete log-levels.
How about compiler tricks to compress the static printk strings?
These could be expanded at runtime to use as the format.
Timothy Miller suggested something similar awhile ago.
Gross, Mark
>-----Original Message-----
>From: Joe Perches [mailto:j...@perches.com]
>Sent: Friday, September 21, 2007 3:33 PM
>To: Gross, Mark
>Cc: Rob Landley; Oleg Verych; Alexey Dobriyan; Michael Opdenacker;
linux-
>ti...@selenic.com; CE Linux Developers List; linux kernel
>Subject: RE: Message codes (Re: [Announce] Linux-tiny project revival)
>
>On Fri, 2007-09-21 at 15:12 -0700, Gross, Mark wrote:
>> Use compiler tricks to remove ALL the static printk string from
>> the kernel and replace the printk with something that outputs a
>> decimal index followed by tuples, of zero to N, hex-strings on
>
>> I proposed a mechanism for keeping all the printk data and saving
space
>> buy doing some table based compressions that has the side effect of
>> making the syslog not human readable. You proposed a mechanism for
>> no-oping out complete log-levels.
>
>How about compiler tricks to compress the static printk strings?
>These could be expanded at runtime to use as the format.
You would have to hold the text table (compressed) in memory to do this
at run time. That would still be pretty large hunk of memory.
Kyle Moffett
> On Fri, 2007-09-21 at 17:34 -0400, Kyle Moffett wrote:
>> With a bit more glue that would cause GCC to notice that for a
>> given qprintk_kmalloc the "qpk->type" is always zero because the
>> level is too high, and therefore it would optimize out *ALL* of
>> the _qprintk_kmalloc(), _qprintk(), and _qprintk_finish() calls.
>
> A negative is that lockup conditions swallow partial messages.
But typically you don't care if a "partial line" gets swallowed
regardless. The only reason people really use partial lines is when
they're accumulating a variable number of things into a single line
and so a single printk() won't do, and in that case it's really not a
problem to "lose" the first half of the line in event of a crash.
And hell, if it matters that much you could just make the qprintk_
{kmalloc,percpu,irq} functions chain the qpk variables on a little
linked list and stuff an smp_wmb() in the _gprint() function after
writing the text and before writing the size. That way any panic
could very carefully look at the messages being queued during the
crash and attempt to write out partial buffers.
It's a technique which in combination with looking at the first 3
characters of the arguments to printk() would let you elide 99% of
the non-critical printks pretty easily while only needing to change
the much smaller proportion of the printk()s which are partial
lines. Furthermore it's pretty easy to grep for the partial-line
printk()s and you can even have it emit warnings when you hit a
partial-line printk() (it doesn't start with "<"[0-9]">") in -mm to
help fix up the last few users and keep people from adding new ones.
Cheers,
Kyle Moffett
Rob Landley
> On Fri, 2007-09-21 at 13:16 -0400, Valdis.K...@vt.edu wrote:
> > What about something *really* hardcore ugly like:
> > #ifdef __FILE__
> > #undef __FILE__
> > #define __FILE__ ""
> > #endif
> > (or similar preprocessor blecherousness) if you want to *really* shrink
> > that binary down?
>
> I prefer removing all __FILE__, __FUNCTION__, __LINE__ uses
> from printks and defining something that modifies pr_<level>.
pr_level doesn't exist in mainline.
> #define pr_info(fmt, arg) printk(KERN_INFO PR_FMT fmt PR_ARG, ##arg)
Do we really need another layer of indirection?
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Joe Perches
> > from printks and defining something that modifies pr_<level>.
> pr_level doesn't exist in mainline.
pr_info and pr_debug do.
pr_alert, pr_emerg, pr_crit, pr_err, and pr_warn could be added.
> > #define pr_info(fmt, arg) printk(KERN_INFO PR_FMT fmt PR_ARG, ##arg)
> Do we really need another layer of indirection?
It'd make file/function/line cost free for embedded use.
Oleg Verych
[]
> > >Not all, but critical info, that must exist in human-readable form of
> > >course.
> >
> > I disagree. For a production product the you want minimal information
> > to reduce the communication bandwidth required between the remote
> > customer and the support organization.
> >
> > In fact there is a good argument that you don't what the remote customer
> > to know enough to start guessing.
>
> Don't use Linux then. Open source is a horrible fit for the way you think.
>
> I'm sympathetic to "shrink the binary size" arguments. I'm not really
> sympathic to "keep the customer in the dark intentionally" arguments, whether
> the justification is "because they're stupid", "to increase dependency on our
> support staff", or any other reason.
{1}
> > >Seriously. When in the Windows there are only messages like:
> > >
> > > "Error (Code:0x00002012)".
> >
> > Now it's been ~8 years since I did any serious windows work, but if I
> > recall correctly ALL THE FRICKING TIME!!! When was the last time you've
> > seen a bug check on windows? This is about all you get.
>
> I believe he was holding it up as a bad example, and definitely not something
> we want to emulate.
I tried to show, that keeping users in compete information vacuum is a
bad thing. Even without sources, _configuration_ makes another area of
mis-working and bugs, usually addressed by reinstalling.
That may be bad example, because here talk is about developers and
testers, who are not just ordinary users. And by applying Torvalds's Law,
all users are such in some degree. That's why {1} in your reply, Rob,
makes perfect sense.
If Mark have a bad experience with lusers only, then i just can say: what
a pity! AFAIK nobody can read somebody's plain-bin OOP output.
Anyway, anything must be opted by config options, even schedulers. But
maintenance and flame wars rule otherwise :).
What i can propose is form of binary-only "printk", where all info:
diagnostic, error, bug, statistics messages (in not debugging
environment, of course), is just fed right to output buffer (see,
pa, no kmallocs). Info itself must have structured content, that makes it
easy to extract and locate human-readable representation of both message
and data.
This doesn't address loglevels, though.
Implementation (seems) as easy as feeding output to `od` to have
unambiguous form of various troublesome bytes, like "0x00" and "0x0A",
Structuring, who is printing, i.e. arch code, fs, driver whatever, must be
agreed:
* Profiles[0]: originator's ID of a message is a byte (or word, or double word)
0x01 - arch, 0x02 - fs, 0x03 - net, 0x04 - hw drivers, etc.
* Data itself can be sent in form of [0]
[0] Banana -- extendable protocol for sending and receiving s-expressions
http://twistedmatrix.com/projects/core/documentation/specifications/banana.html
and having shell script with functions, that have names that correspond
to actual structured content:
_*_
olecom@flower:/tmp$ sh banana.sh < banana.c >bb
olecom@flower:/tmp$ sh -c '. ./bb ; _07080'
start
olecom@flower:/tmp$ sh -c '. ./bb ; _07081'
ti_startup - product 0000, num configurations 0, configuration value 0
olecom@flower:/tmp$ sh -c '. ./bb ; _07082'
not reached
olecom@flower:/tmp$
olecom@flower:/tmp$ sh -c '. ./bb ; _07081 777 7 8'
ti_startup - product 0x0309, num configurations 7, configuration value 8
olecom@flower:/tmp$
_(banana.c and banana.sh can be found in the ftp /upload on my server)_
From file linux/drivers/usb/serial/ti_usb*c with
[...]
dbg("%s - product 0x%4X, num configurations %d, configuration value %d",
__FUNCTION__, le16_to_cpu(dev->descriptor.idProduct),
dev->descriptor.bNumConfigurations,
dev->actconfig->desc.bConfigurationValue);
[...]
lets tacke one particular function (transformed a little bit):
_*_
#include <stdio.h>
#define dbg printf
#define ti_startup(foo) main (int argc, char **argv)
#define dev_descriptor_idProduct 3
#define dev_descriptor_bNumConfigurations 4
#define dev_actconfig_desc_bConfigurationValue 5
/* declaration */
int ti_startup(void);
/* implementation */
int ti_startup(void)
{
dbg("start\n");
return dbg("%s - product %#.4x, num configurations %d, "
"configuration value %d\n",
__FUNCTION__, dev_descriptor_idProduct,
dev_descriptor_bNumConfigurations,
dev_actconfig_desc_bConfigurationValue);
/* bla bla */
dbg("not reached\n");
}
_*_
* Process this file with this script: *
_*_
# just as an example
USB_SERIAL_ID=07
TI_USB_ID=08
__FILE__="ti_usb_3410_5052.c" # possible
i=0
sed -n '
# finding function body
/^[[:alpha:]]/{
# found, print it for __FUNCTION__ keyword
s_[^ ]* *\([^ (]*\).*[^;]$_\1_p;
t_func ; b ;
# walking inside of a function
:_func;
# load next string from the input
n;
# while not the end of a function
/^}/{
# insert delimiter, i.e \n\n is the end of a function
s_.*_\n_p;
# go out
b;
};
# find "dbg" (or printk, whatever) inside
/dbg *(/{
:_dbg;
# while not the end of argumet list, i.e ";\n", add other line
/;$/!{N ; b_dbg}
# change C to sh
s_dbg *(_@printf _;
# clear beginnig of the string
s_^[^@]*@__;
# if there is a __FUNCTION__ or __func__ keyword
/__[Ff]/{
# NOTE: then first %s in format is replaced with $FUNC
s_%s_$FUNC_
}
# strip "\n\tabs"
s_\n[[:blank:]]*__g;
# clear the end, i.e |..."format"/,.*//| or |.."format"/).*//|
s_"[,)].*_"_;
# print result
p;
}
b_func;
}'| while read FUNC
do
printf "_$USB_SERIAL_ID$TI_USB_ID(){
FUNC=$FUNC
}
"
FUNC=_$USB_SERIAL_ID$TI_USB_ID
while read -r PRINT
do test "$PRINT" && printf "$FUNC$i() {
$FUNC
%s "'"$@"'"
}
" "$PRINT"
i=$(($i+1))
done
done
_*_
Now switch on imagination and see how printk just puts
'\007\010\params'
and userspace after some banana `od`dity just calls shell functions.
Yet another thing to export to the userspace. But surely only for those
who whats that.
--
-o--=O`C
#oo'L O
<___=E M
Adrian Bunk
>...
> So __FILE__ expand differently depending on the path on
> the gcc command line.
> I once posted a patch to fix up on this, especialy for BUG_ON and friends.
> The solution was to let kbuild generate the filename and to use
> this define in the source code.
> But a quick grep for __FILE__ in the kernel source made me chicken out.
> Simply too much chrunch at that time to justify it.
Although it's not a big problem, we have the same issue with gcc
warnings and errors displaying the full path with O= builds making the
error messages quite long.
The root is that for O= builds we are feeding gcc the full path
We might perhaps attack it at this point?
The simplest solution that comes into my mind would be to create links
for the source file in the output dir before calling gcc and then give
gcc the link as input file.
> Sam
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Geert Uytterhoeven
> On Thu, 2007-09-20 at 15:38 -0500, Rob Landley wrote:
> > And so far no behavior has changed. But now the _fun_ part is, you can add a
> > config symbol for "what is the minimum loglevel I care about?" Set that as a
> > number from 0-9. And then you can define the printk to do:
> >
> > #define printk(level, str, ...) \
> > do { \
> > if (level < CONFIG_PRINTK_DOICARE) \
> > actual_printk("<" #level ">" str, __VA_ARGS__); \
> > } while(0);
> >
> > And viola (however you spell that, I think I'm using the stringed instrument
>
> > But this doesn't _completely_ eliminate
> > printks, so you can still get the panic() calls and such. You tweak precisly
> > how much bloat you want, using the granularity information that's already
> > there in the source code...
> > Opinions?
>
> I'd rather take the opportunity to convert all the printks to
> use pr_<level>. That way, you can pick'n'choose if you want
> arbitrary combinations of KERN_<level> compiled in or not.
Or:
if ((1 << level) & CONFIG_PRINTK_MASK)
actual_printk("<" #level ">" str, __VA_ARGS__);
But it would indeed be nice to have all of pr_<level> (and not only pr_info()
and pr_debug()) anyway.
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Network and Software Technology Center Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uyt...@sonycom.com
Internet: http://www.sony-europe.com/
Sony Network and Software Technology Center Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis Bank Zaventem · Swift GEBABEBB08A · IBAN BE39001382358619
Rob Landley
> The simplest solution that comes into my mind would be to create links
> for the source file in the output dir before calling gcc and then give
> gcc the link as input file.
The way I've been building various packages out-of-tree (including the Linux
kernel) is:
cp -sR /path/to/actual/source/tree newdir
cd newdir
configure
make
install
cd ..
rm -rf newdir
The cp -sR creates a new tree of symlinks to the original source, and
everything I've tried so far happily builds in such a directory.
(Yes, this breaks on cygwin. Ask me if I care.)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Arnd Bergmann
> So instead of:
> printk(KERN_NOTICE "Fruit=%d\n", banana);
> It would now be:
> printk(KERN_NOTICE, "Fruit=%d\n", banana);
>
> Change the header from:
> #define KERN_NOTICE "<5>"
> to:
> #define KERN_NOTICE 5
>
> Then you can change the printk guts to do something vaguely like (untested):
> #define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)
>
> And so far no behavior has changed. But now the _fun_ part is, you can add a
> config symbol for "what is the minimum loglevel I care about?" Set that as a
> number from 0-9. And then you can define the printk to do:
>
> #define printk(level, str, ...) \
> do { \
> if (level < CONFIG_PRINTK_DOICARE) \
> actual_printk("<" #level ">" str, __VA_ARGS__); \
> } while(0);
>
Assuming that we want to go down that road, I think you can do better with
more evil macro magic, by using something along the lines of
#define KERN_NOTICE "<5>",
#define PRINTK_CONTINUED "",
#define printk(level, str, ...) \
do { \
if (sizeof(level) == 1) /* continued printk */\
actual_printk(str, __VA_ARGS__); \
else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
actual_printk(level str, __VA_ARGS__); \
} while(0);
Then you don't have to change every single printk in the kernel, but
only those that don't currently come with a log level. More importantly,
you can do the conversion without a flag day, by spreading (an empty)
PRINTK_CONTINUED in places that do need a printk without a log level.
Arnd <><
Indan Zupancic
> Assuming that we want to go down that road, I think you can do better with
> more evil macro magic, by using something along the lines of
>
> #define KERN_NOTICE "<5>",
>
> #define PRINTK_CONTINUED "",
>
> #define printk(level, str, ...) \
> do { \
> if (sizeof(level) == 1) /* continued printk */\
> actual_printk(str, __VA_ARGS__); \
> else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
> actual_printk(level str, __VA_ARGS__); \
> } while(0);
>
> Then you don't have to change every single printk in the kernel, but
> only those that don't currently come with a log level. More importantly,
> you can do the conversion without a flag day, by spreading (an empty)
> PRINTK_CONTINUED in places that do need a printk without a log level.
The problem is, how do you know whether to print a continued printk or not?
It depends on the loglevel of the first printk.
So besides compile-time parsing of the source code, replacing printk with
loglevel specific alternatives (one way or the other) seems the only option.
Greetings,
Indan
Arnd Bergmann
> > Then you don't have to change every single printk in the kernel, but
> > only those that don't currently come with a log level. More importantly,
> > you can do the conversion without a flag day, by spreading (an empty)
> > PRINTK_CONTINUED in places that do need a printk without a log level.
>
> The problem is, how do you know whether to print a continued printk or not?
> It depends on the loglevel of the first printk.
Those need to be looked at individually. You can normally see easily from
the context whether the missing log level was an accident, or the author
actually has multiple printk statements for a single line. In one case,
you would add a log level, in the other case, you can add PRINTK_CONTINUED,
or something similar. An alternative to PRINTK_CONTINUED might be a new
function, e.g. printk_continued() or similar that does not expect a log
level.
> So besides compile-time parsing of the source code, replacing printk with
> loglevel specific alternatives (one way or the other) seems the only option.
That would mean replacing all of them, not just those that currently lack
a loglevel.
Arnd <><
Rob Landley
> #define KERN_NOTICE "<5>",
>
> #define PRINTK_CONTINUED "",
>
> #define printk(level, str, ...) \
> do { \
> if (sizeof(level) == 1) /* continued printk */\
> actual_printk(str, __VA_ARGS__); \
> else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
> actual_printk(level str, __VA_ARGS__); \
> } while(0);
>
> Then you don't have to change every single printk in the kernel, but
> only those that don't currently come with a log level. More importantly,
> you can do the conversion without a flag day, by spreading (an empty)
> PRINTK_CONTINUED in places that do need a printk without a log level.
The "change every printk in the kernel" suggestion came from me trying to
figure out how to get the printk() calls below a certain log level to
optimize out and not take up space in the binary.
The above doesn't address the original cause of the thread, as far as I can
tell.
> Arnd <><
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
Dick Streefland
| The "change every printk in the kernel" suggestion came from me trying to
| figure out how to get the printk() calls below a certain log level to
| optimize out and not take up space in the binary.
|
| The above doesn't address the original cause of the thread, as far as I can
| tell.
It does, provided you change the macro definition to:
#define _printk(level, str, ...) \
do { \
if (sizeof(level) == 1) /* continued printk */\
actual_printk(str, __VA_ARGS__); \
else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
actual_printk(level str, __VA_ARGS__); \
} while(0);
#define printk(level_str, ...) _printk(level_str, __VA_ARGS__)
Gcc will do constant folding on the string subscripts, and remove the
code and the string constants for calls above the desired level.
This is basically the same as I proposed in my earlier message [*],
but with the disadvantage that you need to modify all printk() calls
without an explicit level and add the ugly comma to the KERN_* macros.
Just compile the code in my message with -O and -S and you will see
that the KERN_NOTICE call and the corresponding string literal are
optimized away.
[*] http://lkml.org/lkml/2007/9/21/151
--
Dick Streefland //// Altium BV
dick.st...@altium.nl (@ @) http://www.altium.com
--------------------------------oOO--(_)--OOo---------------------------
Jörn Engel
>
> If think you misunderstood:
> Say, you compile out everything of DEBUG level.
> Say, you have a continued printk() after each and every pr_debug().
>
> Then how is the macro supposed to know (at compile-time) that the
> continued printk() actually belongs to a DEBUG printk (and not another
> one)?
#if CONFIG_PRINTK_DOICARE >= 3
#define PRINTK_DEBUG_CONTINUED "",
#else
#define PRINTK_DEBUG_CONTINUED PRINTK_DEBUG
#endif
..
#define printk(level, str, ...) \
do { \
if (sizeof(level) == 1) /* continued printk */\
actual_printk(str, __VA_ARGS__); \
else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
actual_printk(level str, __VA_ARGS__); \
} while(0);
Now you can mark a continued KERN_ERR call as such. And depending on
CONFIG_PRINTK_DOICARE it will either be printed as always or turned into
a stand-alone printk() that gets optimized away. Do the same for all
the other levels.
Open question remains: do we want to go down that road? I have no idea.
Jörn
--
I've never met a human being who would want to read 17,000 pages of
documentation, and if there was, I'd kill him to get him out of the
gene pool.
-- Joseph Costello