RE: plugin hooks
Grigori Fursin
By the way, just to add to Zbigniew comment that one the additional reasons for plugins and ICI research hooks
is actually to help clean up the compiler in a long run. I am working with a student to clean up pass sequences
(remove duplicates and add some dependence and performance info) using ICI and hope to have some
results to share with the community by December. And it is not anymore just for fun or research but is a necessity
since there are so many additional passes appearing with complex interaction that we have to be able to better balance/adapt
compilation time/performance/code size for different architectures/code ... Such flexibility will be available in the next
generations of other compilers (or already available) and GCC will need to have it too to be at least competitive or actually
to lead innovation ...
Also, I hope that we will start collaborating with Joern Rennecke in a few weeks to update the ICI and GSOC’09
developments based on the recent feedback to see if we can move it to the mainline ...
We will keep in touch about that...
Cheers,
Grigori
From: Zbigniew Chamski
[mailto:zbigniew...@gmail.com]
Sent: Tuesday, October 27, 2009 6:56 PM
To: Richard Guenther
Cc: Basile STARYNKEVITCH; Ian Lance Taylor; GCC Mailing List; Grigori
Fursin; Albert Cohen
Subject: Re: plugin hooks
All,
The primary purpose of ICI is to enable fast-and-easy (-easier...)
experimentation with the compilation flow. Once a new feature/change is
stabilised and verified in plugin mode, it can and should be moved to the
"static", compiled-in part of the compiler if deemed sufficiently
relevant.
As a result, the pass manager of ICI provides very simple primitives:
* register a pass in the pass pool (implemented using a hash table); once
registered, passes can be referenced by name
* list the names of all passes registered in pool
* run a pass from the pool by name.
The logic of pass management is by design left to the plugin implementor, so
that he/she can
* choose/alter the ordering of passes (obviously, not all pass orderings will
make sense)
* insert a completely new pass into a pass list
* override a default (compiled-in) implementation of a pass with plugin-provided
version, etc.
The pass chains are then altered by calling two kinds of plugin events:
* override of the default gate of a pass in execute_one_pass (force-disable /
force-enable a pass)
* pass list replacement: instead of calling 'execute_pass_list (<some_pass_list>)',
follow the plugin-provided pass list logic. It is up to the plugin to
construct and traverse a meaningful pass sequence.
Re: twisted code: there is indeed a whole lot twisted dependencies between
passes, state hidden in global vars etc. The infrastructure to improve
this aspect exists already in struct opt_pass (properties_* and todo_flags_*),
but it needs to be used more systematically than it was until know.
However, this state of affairs will only change "by side effect" -
if/when people start experimenting heavily with pass ordering and feel the need
to set the twisted stuff straight.
Cheers,
Zbigniew
2009/10/27 Richard Guenther <richard....@gmail.com>
On Tue, Oct 27, 2009 at 4:48
PM, Basile STARYNKEVITCH
<bas...@starynkevitch.net>
wrote:
> Richard Guenther wrote:
>>
>> The fix is not to add a hook to replace the pass manager but instead
>> to make the GCC pass manager more flexible itself.
>
> I leave that task to my ICI friends (in CC). They have already an
> implementation, and I believe they probably did propose some patches, or
are
> working on it.
>
> However, it is not a small patch.
>
> Also, I still believe that a hook for tha pass manager may make sense to
> those brave souls wanting to experiment new pass managers in plugins, etc.
> Having a hook for that is a way to encourage such crazy experimentations.
The point is that compilation is not driven by the pass
manager
but by a bunch of twisted code througout the compiler. Better
fix that first before adding more twists.
Richard.
--
dr Zbigniew Chamski - Infrasoft IT Solutions
ul. Oskara Kolberga 33, 09-407 PŁOCK, Poland
mobile: +48 668 773 916, phone: +48 24 262 0166
NIP 526-286-69-21, REGON: 141082454
Grigori Fursin
Grigori Fursin
I think, that's very reasonable.
Just as we agreed, wait a few days until you are covered by the INRIA agreement
to transfer copyright to FSF ... I should be able to tell you more within a few
days ...
Also, I added Yuanjie, Liang (they did GSOC developments) and Yuri in CC since
they will help with testing and extending the ICI and GSOC developments...
Cheers,
Grigori
> -----Original Message-----
> From: Joern Rennecke [mailto:amy...@spamcop.net]
> Sent: Monday, November 02, 2009 1:50 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-d...@googlegroups.com; Yuri Kashnikoff
> Subject: RE: plugin hooks
>
> Quoting Grigori Fursin <grigori...@inria.fr>:
> > Also, I hope that we will start collaborating with Joern Rennecke in
> > a few weeks to update the ICI
> > and GSOC'09
> >
> > developments based on the recent feedback to see if we can move it
> > to the mainline ...
>
> We still need a branch name for that. Since GCC is currently in phase 3,
> I suppose we should plan on re-baselining at least once - re-baselining
> from 4.6 experimental mainline after 4.5 is branched - so instead of
> a gcc version number a branch date seems more suitable.
>
> I.e. if the branch were to be created today, I propose to use the name
> branches/ici-20091102-branch for the branch.
Grigori Fursin
Just wanted to announce that we will now collaborative with Joern until the end of
December to sync the GCC GSOC'09 developments and ICI with the mainline GCC. Joern
can transfer all the code to FSF since he is covered by INRIA agreement
until the end of December at least.
As a reminder, the goal of our developments is to sync our Interactive Compilation
Interface with the current plugin framework to enable already established research community
continue working with ICI directly in the official GCC without any changes. We would
also like to continue attracting more researchers to GCC and enable quick prototyping
of research ideas in a real production quality compiler while avoiding a long process
of hacking it ;) ... This in turn should help us move useful research ideas back to
GCC much faster to improve program performance, compilation time, code size, etc ...
Furthermore, this can eventually help us clean and modularize compiler, and hopefully
eventually provide a clean interface and full documentation of all modules (I think we
all agreed on that in the last few years and there are already multiple on-going projects
in that direction ...)
Our current plan of work:
* Create ICI branch, create related Wiki page on GCC website (or reuse/update the current one),
agree on testing methodology, agree on plugin location
* Sync current ICI and pass manager hooks with the branch (Zbigniew can advise on that).
This will allow users select/deselect/reorder current passes to continue research on adaptive compilation
technology to improve performance, code size, compilation time or add new passes (generic function cloning,
feature extraction from the MILEPOST project, analysis and instrumentation routines, passes written in MELT, etc).
Our latest plugins also use XML to dump current info about executed passes, change them and put them back to the compiler...
* Check that ICI uses the same calling convention as current plugins (compiler flags, etc)
* Sync function cloning and basic program instrumentation (GSOC'09 extensions by Liang and Yuanjie - they can help with that).
This will allow users create multiversioning applications to adapt to different architectures or program behavior at run-time,
or even create multi-target applications such as GPU/CELL/accelerators, etc. It is implemented as a generic IP pass that can
be called on demand through ICI and the clones can be optimized differently selecting/deselecting/reordering passes through ICI...
All the above is already in the reasonably stable stage, requires minimal modifications to GCC and mainly requires testing and
moving to the mainline. Me and Yuri (in CC) are testing pass selection and reordering mechanism in GCC...
* Check fine-grain optimization selection in GCC through ICI (another GCC GSOC'09 development by Yunajie and Liang). The idea
is to enable selection and tuning of individual transformations such as unrolling, vectorization, tiling, inlining, etc through
plugins. The prototype is available, however it seems that it slows down the compiler so we have to check what to do if we have
too many hooks in the compiler - we need some simple testing mechanism that will turn them off if not used ...
So, before moving those developments, we have to thoroughly investigate this matter since we can't afford to slow down GCC ;) ..
* Check/add MILEPOST feature extractor - it just a pass that is not invoked unless called by ICI so it is simple to add but
we need to show the useful usage cases ...
The latest version of ICI and GSOC developments for GCC 4.4.0 is available at SVN:
http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
(gcc-4.4.0-ici-2.0-adapt)
The stable version of MILEPOST GCC with ICI and feature extractor from IBM is available at:
http://sourceforge.net/projects/gcc-ici/files/
The latest documentation for ICI and GSOC developments is available here:
http://ctuning.org/wiki/index.php/CTools:ICI:Documentation
http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Function_cloning_and_program_instrumentation
http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Fine_grain_tuning
The idea is to move all the these developments and the documentation to the GCC branch and GCC site now
By the way, just a small reminder that the submission deadline for the GROW'10 workshop
is coming soon: http://ctuning.org/workshop-grow10
We hope to have a 1-hour open discussion at the end of the GROW workshop about plugins, LTO and
further R&D opportunities in GCC ...
Also, until the end of December, the main contact point about these developments will be Joern ...
Will keep in touch,
Grigori
> -----Original Message-----
> From: Joern Rennecke [mailto:amy...@spamcop.net]
> Sent: Monday, November 02, 2009 1:50 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-d...@googlegroups.com; Yuri Kashnikoff
> Subject: RE: plugin hooks
>
> Quoting Grigori Fursin <grigori...@inria.fr>:
> > Also, I hope that we will start collaborating with Joern Rennecke in
> > a few weeks to update the ICI
> > and GSOC'09
> >
> > developments based on the recent feedback to see if we can move it
> > to the mainline ...
>
Grigori Fursin
Normally, there should be no new changes in the ICI2.0 API during GSOC'09 developments
(i.e. in adapt branch), just new hooks.
Yuanjie and Liang, can you please confirm that ASAP?..
Also, I am trying to figure out if GCC 4.5 branch is still open or not?..
The thing is that I think Zbigniew sent a first patch of the ICI2.0 for GCC 4.5
but I don't remember now what happened with that since I was not available for a few months ...
Since the changes are really minimal on top of plugin branch on GCC 4.5 and we had been discussing
that with Diego for a while (in CC), maybe we can still resubmit the ICI2.0 to the GCC 4.5 or check
what was the status of the previous patch and do all the rest (function cloning/program instrumentation)
for GCC 4.6?..
If the GCC 4.5 is officially closed, we can start working on the next version
and as soon as Yuanjie and Liang reply we take either gcc-4.4.0-ici-2.0 or gcc-4.4.0-ici-2.0-adapt ...
Cheers,
Grigori
> -----Original Message-----
> From: Joern Rennecke [mailto:amy...@spamcop.net]
> Sent: Thursday, November 05, 2009 2:26 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-d...@googlegroups.com; 'Yuri Kashnikoff';
> Yuanjie Huang; Liang Peng; anno...@hipeac.net; ctuning-...@googlegroups.com;
> do...@il.ibm.com; Mircea Namolaru; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
> Quoting Grigori Fursin <grigori...@inria.fr>:
> > The latest version of ICI and GSOC developments for GCC 4.4.0 is
> > available at SVN:
> > http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
> > (gcc-4.4.0-ici-2.0-adapt)
>
> Can I start with gcc-4.4.0-ici-2.0 or is this now outdated relative
> to gcc-4.4.0-ici-2.0-adapt ?
Grigori Fursin
By the way, I think you still didn't reply which fine-grain
optimizations did you manage to add, i.e. you could see
that they properly transform the code ...
Thanks again,
Grigori
> -----Original Message-----
> From: huangy...@gmail.com [mailto:huangy...@gmail.com] On Behalf Of Yuanjie Huang
> Sent: Thursday, November 05, 2009 3:34 PM
> To: Grigori Fursin
> Cc: Joern Rennecke; Liang Peng; Zbigniew Chamski; GCC Mailing List; ctuning-
> discu...@googlegroups.com; Yuri Kashnikoff; Diego Novillo
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
> Hi all,
>
> There is a change in the parameter API, which is used to add a type
> information to the parameters.
> I've document the changes here:
> http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Fine_grain_tuning#ICI_Parameter_H
> andling_APIs
>
> Cheers,
> Yuanjie
> --
> Yuanjie HUANG
Yuanjie HUANG
loop unrolling, strip mine and blocking in the GSoC project.
Cheers,
Yuanjie
--
Yuanjie HUANG
Grigori Fursin
The ICI changes are very small and seems that they already had been pre-reviewed.
Joern is taking care of that now.
As for GSOC'09 developments, it seems that they will require more changes
(though improvement to generic function cloning may be fairly small) so will
keep them for the next version ...
Cheers,
Grigori
> -----Original Message-----
> From: Ian Lance Taylor [mailto:ia...@google.com]
> Sent: Friday, November 06, 2009 12:34 AM
> To: Grigori Fursin
> Cc: 'Joern Rennecke'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List';
> ctuning-d...@googlegroups.com; 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
> "Grigori Fursin" <grigori...@inria.fr> writes:
>
> > Also, I am trying to figure out if GCC 4.5 branch is still open or not?..
>
> gcc 4.5 is in stage 3, which normally means that no new features are
> permitted. That said, plugins are a new feature in gcc 4.5, so the
> release managers are likely to grant some leeway for changing them.
> But major rearrangements or modifications of existing code are not
> appropriate at this time.
>
> Ian
Joern Rennecke
> Sure, thanks for the info, Ian.
>
> The ICI changes are very small and seems that they already had been
> pre-reviewed.
> Joern is taking care of that now.
After the pass-name changes are taken out (this is the pre-reviewed bit
I'm currently trying to contribute to mainline), there is about 65 KB of
unidiff left from the ICI 2.0 branch. That code needs a bit of GNU
reformatting, so I doubt it has been reviewed in its entirety.
As it looks like the pass-name changes can go into mainline soon, I want
to hold off creating a branch for the rest of the code till the former
changes are in mainline.
Grigori Fursin
By the way, the ICI itself should not touch too much of the GCC code -
it's an API on top of current plugin API. After resolving the issue
with the pass names that is necessary for ICI to work properly,
could you actually please check what will be all the necessary current
GCC modifications to add ICI ... If it really modifies some critical
parts, we can leave it for the next version then ...
Grigori
> -----Original Message-----
> From: ctuning-d...@googlegroups.com [mailto:ctuning-d...@googlegroups.com] On
> Behalf Of Joern Rennecke
> Sent: Friday, November 06, 2009 3:54 PM
> To: Grigori Fursin
> ctuning-d...@googlegroups.com; 'Yuri Kashnikoff'; 'Diego Novillo'
Joern Rennecke
>
> Ok, that's fine ...
>
> By the way, the ICI itself should not touch too much of the GCC code -
> it's an API on top of current plugin API. After resolving the issue
> with the pass names that is necessary for ICI to work properly,
> could you actually please check what will be all the necessary current
> GCC modifications to add ICI ... If it really modifies some critical
> parts, we can leave it for the next version then ...
I think things like sticking -export-dynamic into make files rules are
a portability showstopper.
There should be an autoconf test to check if the option is available.
Grigori Fursin
so maybe we do not need this option anymore ?..
Cheers,
Grigori
> -----Original Message-----
> From: ctuning-d...@googlegroups.com [mailto:ctuning-d...@googlegroups.com] On
> Behalf Of Joern Rennecke
> To: ctuning-d...@googlegroups.com; Grigori Fursin
> Cc: 'Ian Lance Taylor'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List';
> 'Yuri Kashnikoff'; 'Diego Novillo'
Basile STARYNKEVITCH
My suggestion would be to try pushing the minimal missing plugin hooks needed for ICI into the trunk (that could perhaps
happen in gcc 4.5 timeframe), and adjust your ICI 2.x branch to make it a plugin (perhaps dlopen-ing itself other code).
What I am thinking of is just adding a few more PLUGIN_* events. Even adding a single event take much time in practice.
http://gcc.gnu.org/ml/gcc/2009-10/msg00547.html http://gcc.gnu.org/ml/gcc/2009-10/msg00548.html
You'll need to
1.) convince the GCC community that your event is useful. This is sometimes not as easy as it seems.
(and I Basile did try a bit to explain what possible hooks could be useful to *ICI* & also MELT).
http://gcc.gnu.org/ml/gcc/2009-10/msg00549.html
2.) propose a small patch to add it
2.1) add your PLUGIN_* event in gcc/gcc-plugin.h & gcc/plugin.c
2.2) just a call to invoke_plugin_callbacks (at the place you need).
2.3) document it in gcc/doc/plugins.texi
2.4) make a testcase [this means understanding dejagnu]
3.) wait for your patch to be accepted.
My feeling is that ICI should have a small steps strategy to add the minimal hooks required, and be [like MELT] mostly a
plugin and/or an experimental branch nobody really cares about.
I am not sure it is realistic to try to push (at least for gcc 4.5, and perhaps even gcc 4.6) a multithousand lines ICI
patch.... Better push several smaller patches, each adding a PLUGIN_* event, and make most of ICI a plugin itself.
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Grigori Fursin
Well, the point of ICI is actually not to hardwire specific events at the beginning
but allow users/researchers easily add their own hooks referenced by any string
to simplify prototyping of new ideas. Then if they see that it is useful and they
can persuade the community about that, those hooks go to the mainline and are hardwired
in the PLUGIN_* event ...
We found such usage very convenient in the last few years and during that
time got a considerable feedback about ICI - it is also being used at the moment
in multiple projects that's why we are now trying to add it to the mainline
to help our colleagues use it directly in the mainline GCC and continue research...
Again, several thousands of lines of code in ICI have been related just to manage
event referencing by the string (with hashing, etc) + clean API but it doesn't really
change GCC too much. Actually, I am thinking that maybe we should actually
make ICI as a library and provide only a few necessary hooks/changes to GCC
to support it.
Joern and Zbigniew, what do you think about that?..
Cheers,
Grigori
> -----Original Message-----
> From: ctuning-d...@googlegroups.com [mailto:ctuning-d...@googlegroups.com] On
> Sent: Friday, November 06, 2009 4:55 PM
> To: Joern Rennecke
> Cc: Grigori Fursin; 'Ian Lance Taylor'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski';
> 'GCC Mailing List'; ctuning-d...@googlegroups.com; 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
>
Basile STARYNKEVITCH
> Hi Basile,
>
> Well, the point of ICI is actually not to hardwire specific events at the beginning
> but allow users/researchers easily add their own hooks referenced by any string
> to simplify prototyping of new ideas. Then if they see that it is useful and they
> can persuade the community about that, those hooks go to the mainline and are hardwired
> in the PLUGIN_* event ...
My intuition was that ICI need a single (or probably a few) hook to replace the entire pass manager....
Then the ICI plugin would call the existing GCC passes in the order it wants....
Grigori Fursin
since we use it to manipulate passes, add new analysis passes of generic function cloning
(during GSOC'09), etc. So we will need a hard-wired hook to substitute pass manager.
Other hooks such as fine-grain tuning for another GSOC'09 project
are still experimental and are not hardwired, but just referenced by some strings ...
Cheers,
Grigori
> -----Original Message-----
> From: Basile STARYNKEVITCH [mailto:bas...@starynkevitch.net]
> Sent: Friday, November 06, 2009 6:00 PM
> To: Grigori Fursin
> Cc: ctuning-d...@googlegroups.com; 'Joern Rennecke'; 'Ian Lance Taylor'; 'Yuanjie
> Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List'; 'Yuri Kashnikoff'; 'Diego
> Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
Joern Rennecke
In gcc 4.5, passes.c:register_pass is used to register a pass in a
specific place in the pass tree.
In gcc-4.4.0-ici, pass-manager.c:register_pass is used to register a
pass by name.
Grigori Fursin
Actually, I think I realized why you still see many differences in ICI 2.0 and GCC 4.5 -
my memory is not working well after being away for 2 months and I am trying to remember
what happened with ICI in summer - sorry about that.
So, if I remember correctly, when we saw the new GCC plugin branch open and that it overlaps
considerably with ICI we decided to change all our function names that had the same functionality
as plugin branch to sync it later with plugin branch goes to the mainline (and that is happening
now in GCC 4.5) while we continued using baseline GCC 4.4.0 for all our further experiments.
In parallel Zbigniew prepared a patch that synchronized ICI with the plugin branch
and if I am correct, he removed all the duplicating functionality so the changes became
quite minimal. BUT we did not use that branch for GSOC projects, so what you see
in the gcc-4.4.0-ici2.0-adapt is the ICI with duplicate functionality of the old plugin branch
and normally the current mainline GCC 4.5. That's I think you start having conflict names.
I sent another private email to Zbigniew and you to check if I am right and if so,
I think you should look at the Zbigniew patch from June and try to update it according
to the current GCC 4.5 mainline plus add small extension from GSOC'09 about types of
the plugin parameters. After that we will synchronize GSOC'09 branches with this new
ICI and with the mainline GCC 4.5 but not versa!!!
Please, tell me if it makes sense. If someone thinks that I am still wrong (particularly
Zbigniew, Yuanjie and Liang) please respond to that so that we could decide on
our work plan ...
By the way, by accident I added 2 emails to the CC due to auto completing of emails:
anno...@hipeac.net and ctuning-...@googlegroups.com - I removed them
since they are not related to this discussion. Replying to this message is now fine
but if you reply to other messages, please remove those emails ...
I am sorry about that...
Cheers,
Grigori
> -----Original Message-----
> From: ctuning-d...@googlegroups.com [mailto:ctuning-d...@googlegroups.com] On
> Sent: Friday, November 06, 2009 6:14 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Yuanjie Huang; Liang Peng; anno...@hipeac.net; ctuning-...@googlegroups.com;
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
>
Joern Rennecke
...
> In parallel Zbigniew prepared a patch that synchronized ICI with the
> plugin branch
> and if I am correct, he removed all the duplicating functionality so
> the changes became
> quite minimal.
Where is that patch?
> BUT we did not use that branch for GSOC projects, so what you see
> in the gcc-4.4.0-ici2.0-adapt is the ICI with duplicate
> functionality of the old plugin branch
> and normally the current mainline GCC 4.5. That's I think you start
> having conflict names.
The two register_pass functions are quite distinct in their functionality
(except that logically the passes.c one should call - directly or
indirectly - the pass_manager.c one). Both are an interface exposed to
plugins, so we can't simply rename an internal function. One of the
interfaces has to change.
Grigori Fursin
> > In parallel Zbigniew prepared a patch that synchronized ICI with the
> > plugin branch
> > and if I am correct, he removed all the duplicating functionality so
> > the changes became
> > quite minimal.
>
> Where is that patch?
After looking at all my email I think we only had internal copies of the patch
so it has never been sent out. It seems like marriage can break memory completely -
so I am really sorry for those memory blackouts :( ...
This means that we are getting back to the same task of syncing the ICI2.0 from adapt
SVN to the current GCC. Actually, could you please check what is duplicate functionality
between current GCC 4.5 in the ICI and if we can remove it from ICI ...
It should naturally relate mainly to the plugin functions ...
Only when it's done we can see if ICI will be intrusive or not ...
> > BUT we did not use that branch for GSOC projects, so what you see
> > in the gcc-4.4.0-ici2.0-adapt is the ICI with duplicate
> > functionality of the old plugin branch
> > and normally the current mainline GCC 4.5. That's I think you start
> > having conflict names.
>
> The two register_pass functions are quite distinct in their functionality
> (except that logically the passes.c one should call - directly or
> indirectly - the pass_manager.c one). Both are an interface exposed to
> plugins, so we can't simply rename an internal function. One of the
> interfaces has to change.
Well, ICI is not in the mainline so it has to change and adapt to the
current mainline... Can you please check if there are other similar cases apart
from this to decide on our changes in one go since we will need to alter
all our plugins due to that (unless we provide some legacy wrapper but
I think at this stage I would prefer to have a clean ICI in GCC)...
Thanks a lot,
Grigori
Joern Rennecke
I created
svn://gcc.gnu.org/svn/gcc/branches/ici-20091108-branch
and merged in the patches from:
http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/patch-gcc-4.4.0-ici-2.0
Could you please check that this contains the functionality that we want to
integrate in the first step.
FWIW I know that the code does not conform to the GNU coding standard yet.
I've changed register_pass to register_pass_name to resolve the name clash.
I'm not sure if it should be called register_pass_by_name or something else,
opinions welcome.
Both the gcc 4.5 code and the ICI patches have the concept of events, but
the implementations are so different that the functionality is pretty much
orthogonal.
4.5 has a real C-style interface with an enum to identify the event and
a single pointer for the data. I.e. low overhead, but rigid typing,
and the different parts of the program presumably find their data by virtue
of using the same header files.
Multiple plugins can register a callback for any event, and all will get
called. However, since the set of events is hard-coded by an enum
declaration, you can't have two plugins communicating using events that
the gcc binary doesn't know about.
The ICI events feel much more like TCL variables and procedure calls.
Events are identified by strings, and parameters are effectively global
variables found in a hash table. This is very flexible and can allow
a good deal of ABI stability, but costs a lot of processor cycles as
before an event call the parameters are looked up to be entered in the hash
table, and afterwards they are looked up to be removed, even if no callback
is registered for the event.
Also, when a plugin registers a callback for an ICI event, it overrides any
previous callback registered by another (or even the same) plugin.
I think we could have the ICI event flexibility/stability with lower
overhead if the event sender requests an event identifier number (which
can be allocated after the numbers of the gcc 4.5 static event enum values)
for an event name at or before the first event with that name, and then
sends this identifier number with one or more pointers, which might point
to internal gcc data structures, and a pointer to a function to look up
the address of a named parameter. The event sender site source code can
then provide information to build the lookup functions at build time,
e.g. using gperf.
I.e.:
/* Call an event with number ID, which is either a value of enum plugin_event,
or a number allocated for a named event. If the event named parameters,
the first parameter after id should be as if declared
void * (*lookup_func) (const char *, va_list) .
LOOKUP_FUNC can be passed the name of a parameter as its first argument,
and a va_list as its second argument, which will be the list of parameters
after LOOKUP_FUNC, to find the named parameter. */
void
call_plugin_event (int id, ...)
{
struct callback_info *callback;
va_list ap;
gcc_assert (id < event_id_max);
callback = plugin_callbacks[id];
va_start (ap, id);
for (callback = plugin_callbacks[id]; callback; callback = callback->next)
(*callback->func) ((void *) ap, callback->user_data);
va_end (ap);
}
Grigori Fursin
> After checking in the patch to provide unique pass names for all passes,
> I created
>
> svn://gcc.gnu.org/svn/gcc/branches/ici-20091108-branch
>
> and merged in the patches from:
>
> http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/patch-gcc-4.4.0-ici-2.0
>
> Could you please check that this contains the functionality that we want to
> integrate in the first step.
Thanks a lot, Joern!
I downloaded it and will be gradually checking it.
In the mean time, Yuanjie, Liang and Yuri - could you check this version ASAP
and check that the functionality provided during GSOC'09 developments/pass reordering
work is correct in this version!.. The reason is that since there will be some small
changes, our plugins will have to slightly change as well (see register_pass change)...
By the way, we should keep the track of such changes on the GCC Wiki for ICI ...
> FWIW I know that the code does not conform to the GNU coding standard yet.
>
> I've changed register_pass to register_pass_name to resolve the name clash.
> I'm not sure if it should be called register_pass_by_name or something else,
> opinions welcome.
I think register_pass_by_name will be better to show what it does now ;) ...
> Both the gcc 4.5 code and the ICI patches have the concept of events, but
> the implementations are so different that the functionality is pretty much
> orthogonal.
>
> 4.5 has a real C-style interface with an enum to identify the event and
> a single pointer for the data. I.e. low overhead, but rigid typing,
> and the different parts of the program presumably find their data by virtue
> of using the same header files.
> Multiple plugins can register a callback for any event, and all will get
> called. However, since the set of events is hard-coded by an enum
> declaration, you can't have two plugins communicating using events that
> the gcc binary doesn't know about.
>
> The ICI events feel much more like TCL variables and procedure calls.
> Events are identified by strings, and parameters are effectively global
> variables found in a hash table. This is very flexible and can allow
> a good deal of ABI stability, but costs a lot of processor cycles as
> before an event call the parameters are looked up to be entered in the hash
> table, and afterwards they are looked up to be removed, even if no callback
> is registered for the event.
> Also, when a plugin registers a callback for an ICI event, it overrides any
> previous callback registered by another (or even the same) plugin.
That's very true. Our current idea is that for prototyping of ideas, it is
often fine to slow down the compiler slightly but as soon as development
matures and there are some interesting results, the developers will try
to persuade the GCC community to add the event permanently...
I have been discussing that with Zbigniew some months ago and I think it can be reasonable
to add such functionality on top of current ICI. ICI users will still prototype their ideas
using events referenced by name, however if it works fine and before potential lengthy approval
to get such an event, they can speed up their plugins using this extended functionality...
The good thing is that their plugins will still be compatible if we will decide to keep associated
names with the hardwired event number ...
Cheers,
Grigori
Grigori Fursin
> I created
>
> svn://gcc.gnu.org/svn/gcc/branches/ici-20091108-branch
>
> and merged in the patches from:
>
> http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/patch-gcc-4.4.0-ici-2.0
By the way, not to forget - we should compile/test GCC with ICI with the following libraries
(orthogonal to ICI but we need them for our experiments):
* gmp & mpfr (for fortran)
* ppl & cloog (for GRAPHITE)
i.e. I configure GCC with the following flags:
configure --enable-languages=c,c++,fortran --with-mpfr=$BUILD_DIR --with-gmp=$BUILD_DIR --with-ppl=$BUILD_DIR --with-cloog=$BUILD_DIR
I used it for the GCC 4.4.0 - maybe some GRAPHITE related flags changed ...
The idea is to have the same setup that we used for our local developments ...
Also, we have been using ICI with C and Fortran a lot, but never checked C++ -
it will be important to check it too ...
Cheers,
Grigori
Joern Rennecke
> I think we could have the ICI event flexibility/stability with lower
> overhead if the event sender requests an event identifier number (which
> can be allocated after the numbers of the gcc 4.5 static event enum values)
> for an event name at or before the first event with that name, and then
> sends this identifier number with one or more pointers, which might point
> to internal gcc data structures, and a pointer to a function to look up
> the address of a named parameter. The event sender site source code can
> then provide information to build the lookup functions at build time,
> e.g. using gperf.
I thought a bit more about this, and decided that using gperf-generated hash
tables is probably overkill.
It is useful to have provisions for the event generator and the event
callback being somewhat out of step, but do we really have to cater
for multiple sources of the same event providing their parameters in
a different way?
If there is only one way to find a parameter with a particular name for
a particular event (for a given combination of compiler binary and plugins),
that this can be defined with an accessor function, which would
generally be defined in the same module which raises the event.
Actually, if we passed around the dso which raises the event, we could
dynamically look up the accessor function to allow co-existence of different
accessor functions for the same event::parameter tuple, but I don't think
there is a need for that.
Here is an example of how I think we can reduce the overhead while keeping
a good amount of flexibility; in loop-unroll.c we currently have:
/* Code for loop-unrolling ICI decision enabling. */
register_event_parameter ("loop->num", &(loop->num));
register_event_parameter ("loop->ninsns", &(loop->ninsns));
register_event_parameter ("loop->av_ninsns", &(loop->av_ninsns));
register_event_parameter ("loop->lpt_decision.times",
&(loop->lpt_decision.times));
register_event_parameter ("loop->lpt_decision.decision",
&(loop->lpt_decision.decision));
register_event_parameter ("loop->lpt_decision.unroll_runtime",
loop->lpt_decision.decision == LPT_UNROLL_RUNTIME ?
(void *) 1 : (void *) 0);
register_event_parameter ("loop->lpt_decision.unroll_constant",
loop->lpt_decision.decision == LPT_UNROLL_CONSTANT ?
(void *) 1 : (void *) 0);
call_plugin_event("unroll_feature_change");
unregister_event_parameter ("loop->num");
unregister_event_parameter ("loop->ninsns");
unregister_event_parameter ("loop->av_ninsns");
unregister_event_parameter ("loop->lpt_decision.times");
unregister_event_parameter ("loop->lpt_decision.decision");
Instead we could have:
invoke_plugin_va_callbacks (PLUGIN_UNROLL_FEATURE_CHANGE, loop);
and then accessor functions:
int
plugin_unroll_feature_change_param_loop_num (va_list va)
{
struct loop *loop = va_arg (va, struct loop *);
return loop->num;
}
unsigned
plugin_unroll_feature_change_param_loop_ninsns (va_list va)
{
struct loop *loop = va_arg (va, struct loop *);
return loop->ninsns;
}
unsigned
plugin_unroll_feature_change_param_loop_av_ninsns (va_list va)
{
struct loop *loop = va_arg (va, struct loop *);
return loop->av_ninsns;
}
...
bool
plugin_unroll_feature_change_param_loop_lpt_decision_unroll_runtime
(va_list va)
{
struct loop *loop = va_arg (va, struct loop *);
return loop->lpt_decision.decision == LPT_UNROLL_RUNTIME;
}
...
There is still another practical issue: as I change the ICI infrastructure
to fit better with the existing gcc 4.5 plugin infrastructure,
the ICI plugins must be adapted to keep working.
As we are trying to have something working in a short time frame, I think
I should pick one plugin and modify it in lock-step with the infrastructure
to demonstrate how it is supposed to work.
Do you think the adapt.c plugin is suitable for that purpose?
Grigori Fursin
I am a bit confused about your above example - you suggest to add
this functionality on top of current ICI or substitute it?
If you want to substitute it, I personally disagree. We spent a very long time
with many colleagues and real ICI users discussing how to simplify the usage of
events for people who are not programming professionals,
so I really prefer to keep the current structure as it is ...
However, if it is the way to speedup slow prototype plugins and addition to ICI,
it may be fine but I need to think about it more.
In both cases, I think it is not critical for now and should be the second step
after current ICI is synced with the mainline. However, suggestions from others
are very welcome ;) !..
> There is still another practical issue: as I change the ICI infrastructure
> to fit better with the existing gcc 4.5 plugin infrastructure,
> the ICI plugins must be adapted to keep working.
> As we are trying to have something working in a short time frame, I think
> I should pick one plugin and modify it in lock-step with the infrastructure
> to demonstrate how it is supposed to work.
>
> Do you think the adapt.c plugin is suitable for that purpose?
Yes, adapt.c is the latest plugin that we use for our experiments ...
Cheers,
Grigori
Joern Rennecke
> I am a bit confused about your above example - you suggest to add
> this functionality on top of current ICI or substitute it?
The idea was to replace it. The current event implementation has two
issues:
- It is very different from the existing GCC 4.5 events which makes it
confusing to have it side-by-side. I think if we can make the ICI code
more of a natural extension of the current event code, it would be more
likely to be acceptable.
- The parameter registering / unregistering and the event call itself
introduce a number of hashing operations which are performed irrespective
of the presence of a plugin. This makes these events intrusive if placed
in places with a high execution frequency.
> If you want to substitute it, I personally disagree. We spent a very
> long time
> with many colleagues and real ICI users discussing how to simplify
> the usage of
> events for people who are not programming professionals,
> so I really prefer to keep the current structure as it is ...
Were these discussions done face-to-face or on a news group/mailing list?
If the latter, could you point me where I can read up on the discussion
so that I better understand the issues involved.
Would these people who are not programming professionals both insert
event raising points as well as write event handling code?
If we would use C++, some things would get easier, i.e. we could have an
event class with subclasses for the separate event types, and then have
the parameter accessors be member functions. This would remove the need
to repeatedly type the event name when accessing the parameters.
However, it would require to build GCC with C++, so I'd say this
significantly reduces the chances of having ICI integrated into the
gcc mainline and having release GCC binaries shipped with the
functionality enabled.
If plugin-side run time is not important, we can have register_plugin_event
as a wrapper for register_callback and use user_data to squirrel away the
event name and the actual callback function; then we can have a wrapper
callback function which sets a thread-local variable (can be non-exported
global inside the plugin as long as no threads are used) to the plugin name,
and make get_event_parameter use a dynamic function name lookup by stringing
together the event name with the parameter name.
This way, we could look the ICI interface on the plugin side pretty much
what it looks now, except that we should probably restrict the event and
parameter names to identifier characters, lest we need name mangling
to translate them to function names.
I had a look at the adapt branch to gauge if the there is really a point for
having the varargs stuff, i.e. events with parameters that are not readily
available in a single struct.
The unroll_parameter_handler / graphite_parameter_handler events are somewhat
odd because they have a varying set of parameters. So they really have a list
of parameters. We could do this with somehing like:
invoke_plugin_va_callbacks (PLUGIN_UNROLL_PARAMETER_HANDLER,
"_loop.id", EP_INT, loop->num,
"_loop.can_duplicate_loop_p",
EP_UNSIGNED_CHAR,
ici_can_duplicate_loop_p);
And then have the plugin pick through the va_list. Or, if preferred,
have a helper function - possibly invoked implicitly by the ICI
infrastructure - go through the va_list and build the hash table of arguments
from it so that the current parameter query interface can be used.
In fact, we could use such a mechanism in general, so if we go back to
passing pointers to parameters instead of parameters, you'd have
backward
compatibility on the plugin side.
OTOH, does using pointers to parameters really make this interface easy to
use - it seems you had to use this to avoid type compatibility issues, which
are sidestepped by using a va_list. If we pass parameter values instead, you
could have a query function that tells you if a parameter is set. Possibly
also with a pointer argument to assign the value if the parameter is set.
tests if a parameter exists - if you like
> However, if it is the way to speedup slow prototype plugins and
> addition to ICI,
> it may be fine but I need to think about it more.
Having a single call to invoke an event by number (even if it is a
varargs call) is certainly much faster than having all these
register_parameter / unregister_parameter calls, particularly in the
case that no plugin has registered a callback for the event.
> In both cases, I think it is not critical for now and should be the
> second step
> after current ICI is synced with the mainline. However, suggestions
> from others
> are very welcome ;) !..
I thought you were hoping to push some ICI infrastructure into mainline
in time for 4.5 . In order to do that
- Time is of the essence.
- We should reduce the impact that ICI has on the rest of GCC.
Grigori Fursin
> > I am a bit confused about your above example - you suggest to add
> > this functionality on top of current ICI or substitute it?
>
> The idea was to replace it. The current event implementation has two
> issues:
> - It is very different from the existing GCC 4.5 events which makes it
> confusing to have it side-by-side. I think if we can make the ICI code
> more of a natural extension of the current event code, it would be more
> likely to be acceptable.
> - The parameter registering / unregistering and the event call itself
> introduce a number of hashing operations which are performed irrespective
> of the presence of a plugin. This makes these events intrusive if placed
> in places with a high execution frequency.
for more tech. details to be sure that we are in sync before doing
further changes in ICI ...
> > If you want to substitute it, I personally disagree. We spent a very
> > long time
> > with many colleagues and real ICI users discussing how to simplify
> > the usage of
> > events for people who are not programming professionals,
> > so I really prefer to keep the current structure as it is ...
>
> Were these discussions done face-to-face or on a news group/mailing list?
> If the latter, could you point me where I can read up on the discussion
> so that I better understand the issues involved.
> Would these people who are not programming professionals both insert
> event raising points as well as write event handling code?
to GCC in 2006 and then had discussions with Albert and Diego at
one of the HiPEAC tutorials about having plugins in GCC. However,
since it was a taboo for GCC at that time, I continued extending
it in the MILEPOST project and within the HiPEAC network of excellence.
We had multiple discussions during MILEPOST and HiPEAC face-to-face
meetings and private mailing lists. Recent emails at GCC mailing lists
and comparison of plugin systems at GCC Wiki are here:
http://www.mail-archive.com/g...@gcc.gnu.org/msg41368.html
http://gcc.gnu.org/wiki/GCC_PluginComparison
> If we would use C++, some things would get easier, i.e. we could have an
> event class with subclasses for the separate event types, and then have
> the parameter accessors be member functions. This would remove the need
> to repeatedly type the event name when accessing the parameters.
> However, it would require to build GCC with C++, so I'd say this
> significantly reduces the chances of having ICI integrated into the
> gcc mainline and having release GCC binaries shipped with the
> functionality enabled.
the functionality works (i.e. pass selection and reordering + fine-grain
selection of optimization and function cloning for c++ programs) when using
g++ ...
at the ICI code for some time and have a few deadlines coming this and
next week...
>
> > However, if it is the way to speedup slow prototype plugins and
> > addition to ICI,
> > it may be fine but I need to think about it more.
>
> Having a single call to invoke an event by number (even if it is a
> varargs call) is certainly much faster than having all these
> register_parameter / unregister_parameter calls, particularly in the
> case that no plugin has registered a callback for the event.
>
> > In both cases, I think it is not critical for now and should be the
> > second step
> > after current ICI is synced with the mainline. However, suggestions
> > from others
> > are very welcome ;) !..
>
> I thought you were hoping to push some ICI infrastructure into mainline
> in time for 4.5 . In order to do that
> - Time is of the essence.
> - We should reduce the impact that ICI has on the rest of GCC.
I think oppositely is barely touches the internals, except the pass manager ...
Again, let's sync offline ...
Cheers,
Grigori
Grigori Fursin
Just a small update, that after some discussions with Joern we think that based on our
time constraints and the current state of GCC, instead of trying to push full ICI into GCC
we start from the opposite approach:
We take all our plugins (support pass selection and reordering from MILEPOST;
generic function cloning and fine-grain optimizations from GSOC'09) and trying
to see which low-level GCC functionality is missing to support them.
Then we provide a few hooks to support them, provide a few small updates to GCC and rewrite
our plugins to support low-level plugin system. Joern will continue communicating
about a few extensions to the plugin system we need to make it happen.
This is a pragmatic step and should require minimal changes in GCC and will help us already
use current plugin system for our work. However, I think there is still a benefit
of ICI in separating GCC and plugins when using internal data structures, i.e. currently
the referencing of data structures in GCC is hardwired in plugins. If one day these
data structures change, we will need to rewrite all plugins. Using referencing mechanism
in ICI (data is used in plugins indirectly through parameter registering) allows us
to insure plugins compatibility but with the performance degradation. We can discuss
that later after GCC 4.5 release and when we get some more experience from the users
about plugins ...
By the way, due to that, I think that maybe besides documenting all the data structures
we should maybe also start providing info if they are used in some plugins. This maybe will
help clearning up the internals of the compiler and will prevent careless changes of the data
structures in GCC to keep plugins compatible?..
Anyway, Joern will continue communicating about the progress and extensions to the plugin
system ...
Take care and have a good weekend,
Grigori
Basile STARYNKEVITCH
g...@gcc.gnu.org could be interested, feel free to make a reply there (even citing this mail if you want) if you believe
this won't start a useless flamewar [I currently believe it will start a useless flamewar, and I don't want to do that].
But I am also adding Rafael Espindola to the recipient list, because he is a plugin maintainer and very nicely impproved
all my plugin patches and approved them.
My feeling is that most of the GCC community don't care much about extra plugin infrastructure like ICI (or MELT) and is
afraid of plugins in general. There might be some hostility to that from several top-level GCC masters (I mean people
with reviewer priviledge, or guys more equal than others in the Orwellian sense http://en.wikipedia.org/wiki/Animal_Farm
I am explicitly classifiying myself in the "less equal" camp.).
My feeling was (and is still) that ICI should become a {mega,meta}-plugin [*] of *unmodified* gcc-4.5 (not gcc 4.6 which
means 2011 probably!) and therefore that ICI should explicitly try *very hard* to push the *few* plugin hooks (I
precisely mean some more members of enum plugin_event in gcc/gcc-plugin.h and a few more lines added carefully inside
some few gcc/*.c files which just call invoke_plugin_callbacks appropriately.
The point is that ICI should push small patches *quickly* to 4.5 (not 4.6), and for each additional plugin_event you
push there, you should also probably
* argue about its usefulness on @gcc.gnu.org lists with appropriate details and explain why ICI can run *today* with
that additional event.
Never forget that the "theoretical" word does not have the same meaning in the GCC community and in academia, and
that within the GCC community, "theory" is never a compliment!
* document it (as a patch to gcc/plugins.texi)
* patch gcc/gcc-plugin.h, gcc/plugin.c, perhaps gcc/plugins.h (but try to avoid needing to patch it)
* patch with a few lines adding invoke_plugin_callbacks some gcc/*.c file.
* add some small testcase in gcc/testsuite/*
You could also try improving the GCC pass manager but I don't feel that could go into GCC 4.5.
Grigori's first paragraph "We take all our plugins..." makes me think ICI want to go the way I am describing above, but
Grigori's second paragraph "This is a pragmatic step..." make me think the opposite. The last sentence "We can discuss
that later after GCC 4.5 release" is confusing me (but I am not as fluent in English as Grigori).
My intuition is that aiming 4.5 is vital for ICI. In my sad intuition, if ICI cannot fit into an *unmodified* 4.5 gcc
release, the whole ICI/MilePost/Ctuning effort could be practically dead. I hope to be too pessimistic!
Aiming 4.5 means probably avoid replacing the whole pass manager, but just inject inside the few hooks required.
So I really think you should try *hard* to push the few events useful to you. And in my MELT experience, this is almost
a bitter effort. The point is that the GCC community is by principle always right -whatever that means exactly-. This
probably means that you might have to remove (with an #ifndef) some very interesting features from ICI to make it fit
into GCC 4.5. And I insist that adding a few lines to GCC core is in my feeling much more hard than coding thousands of
lines in my MELT branch (or inside the MELT plugin). This means that if ICI would need some significant adaptation to
fit into GCC 4.5, it is worth the effort to make them in my eyes. And pushing even a small plugin event into GCC takes a
big lot of time & effort - much more than what Grigori imagines! A "small-steps" policy is preferable to sending a big
patch nobody will review.
Maybe I don't understand well ICI (I admit I did not glance inside the ICI code recently), but I would like ICI too
succeed in been usable with GCC. Because in my experience, nobody cares about GCC variants or branches. ICI need to be
delivered as a GCC plugin to be used.
I am a bit sorry for such a harsh email, it is only a discussion and I definitely hope I am not offending anyone. But I
don't understand the strategy of ICI (maybe I should not even try to understand it).
So a question for ICI gurus. Can ICI (or a large subset of it) fit into a weakly modified gcc 4.5 (by adding only few
plugin events & their invocation)? I guess that yes, it could fit! If yes, try to list the few additional plugin events
& where they should be invoked inside GCC! If you need to restrict ICI to do that, I believe doing that restriction is
worth the effort.
But don't forget that pushing even one single plugin_event into GCC will take you significant time & effort, so better
start now!
###
Regards
Note * about {mega,meta}-plugins to GCC:
In that respect, MELT & ICI are similar: MELT is and ICI should in my opinion become a big plugin of gcc-4.5 (hence the
mega adjective) which itself loads (thru dlopen) other "modules" (or call them otherwise, I mean technically any shared
object passed as argument to dlopen with defined hooking conventions implemented by dlsym). Also, since MELT and ICI are
themelselves code which call dlopen I call them (quite pedantically) meta-plguins. Hence the {mega,meta}-plugin
notation (with braces parsed like the unix shells do).
Grigori Fursin
Basile,
Ideally, I would like to have ICI in GCC, but I personally can't continue
programming it, Zbigniew has been very busy since May and didn't manage to finish
moving ICI to GCC and Joern is working with us until the end of December, so we
just physically don't have time to prepare ICI to move to the mainline.
So I am trying to get at least minimal functionality in GCC 4.5 to support our current
plugins and research. Whenever it's done, we will evaluate and discuss next steps ...
We will keep in touch about the progress ;) ...
Grigori
> -----Original Message-----
> From: Albert Cohen [mailto:Albert...@inria.fr]
> Sent: Sunday, November 15, 2009 3:43 PM
> To: Basile STARYNKEVITCH
> Cc: Grigori Fursin; 'Joern Rennecke'; ctuning-d...@googlegroups.com; 'Grigori Fursin';
> 'Zbigniew Chamski'; Diego Novillo; Rafael Espindola; 'Yuri Kashnikoff'; 'Yuanjie Huang';
> 'Liang Peng'; do...@il.ibm.com
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
> > I am *not* replying to g...@gcc.gnu.org because I don't want to start
> > there a flamewar. If you believe that g...@gcc.gnu.org could be
> > interested, feel free to make a reply there (even citing this mail if
> > you want) if you believe this won't start a useless flamewar [I
> > currently believe it will start a useless flamewar, and I don't want to
> > do that].
> > But I am also adding Rafael Espindola to the recipient list, because he
> > is a plugin maintainer and very nicely impproved all my plugin patches
> > and approved them.
> >
> > My feeling is that most of the GCC community don't care much about extra
> > plugin infrastructure like ICI (or MELT) and is afraid of plugins in
> > general. There might be some hostility to that from several top-level
> > GCC masters (I mean people with reviewer priviledge, or guys more equal
> > than others in the Orwellian sense
> > http://en.wikipedia.org/wiki/Animal_Farm I am explicitly classifiying
> > myself in the "less equal" camp.).
>
>
> Nobody is more equal. It is much simpler: some people are working hard
> to improve GCC and fix its bugs. Any project conflicting with those
> people's goals and complicating their life is a risky/dangerous project.
> We can all understand this. Plugins are not welcome if they damage the
> future of GCC. Of course, plugins bring much more opportunities than
> risks, and I hope that MELT will be one clear demonstration of this.
>
> I have the feeling that the ICI has not been well positionned so far.
> The goal is to improve GCC, and ultimately to be part of the compiler
> itself rather than a plugin. Recent research in adaptive and statistical
> approaches to compilation demonstrate the benefits of something like the
> ICI. It is just as important as LTO, and likewise, provides major
> performance improvements, but also serious deployment difficulties.
> > My feeling was (and is still) that ICI should become a
> > {mega,meta}-plugin [*] of *unmodified* gcc-4.5 (not gcc 4.6 which means
> > 2011 probably!) and therefore that ICI should explicitly try *very hard*
> > to push the *few* plugin hooks (I precisely mean some more members of
> > enum plugin_event in gcc/gcc-plugin.h and a few more lines added
> > carefully inside some few gcc/*.c files which just call
> > invoke_plugin_callbacks appropriately.
>
> hope 4.6 will be released no later than summer 2010, and second ICI will
> still be used only by researchers and people doing compiler tuning for
> the time being. Our ambition should be that some companies doing GCC
> performance tuning would start using it internally first, before
> releasing their own performance-tuning plugins to 4.6 later. This does
> not seem to make a 4.5 merge that critical.
> > The point is that ICI should push small patches *quickly* to 4.5 (not
> > 4.6), and for each additional plugin_event you push there, you should
> > also probably
> > * argue about its usefulness on @gcc.gnu.org lists with appropriate
> > details and explain why ICI can run *today* with that additional event.
> > Never forget that the "theoretical" word does not have the same
> > meaning in the GCC community and in academia, and that within the GCC
> > community, "theory" is never a compliment!
>
> > * document it (as a patch to gcc/plugins.texi)
> > * patch gcc/gcc-plugin.h, gcc/plugin.c, perhaps gcc/plugins.h (but
> > try to avoid needing to patch it)
> > * patch with a few lines adding invoke_plugin_callbacks some gcc/*.c
> > file.
> > * add some small testcase in gcc/testsuite/*
>
> has 2 months to work on this (full time at least).
> > You could also try improving the GCC pass manager but I don't feel that
> > could go into GCC 4.5.
>
> moment, it seems unrealistic to shoot for 4.5 merge.
> > Grigori's first paragraph "We take all our plugins..." makes me think
> > ICI want to go the way I am describing above, but Grigori's second
> > paragraph "This is a pragmatic step..." make me think the opposite. The
> > last sentence "We can discuss
> > that later after GCC 4.5 release" is confusing me (but I am not as
> > fluent in English as Grigori).
>
> merge :-).
> > My intuition is that aiming 4.5 is vital for ICI. In my sad intuition,
> > if ICI cannot fit into an *unmodified* 4.5 gcc release, the whole
> > ICI/MilePost/Ctuning effort could be practically dead. I hope to be too
> > pessimistic!
> > Aiming 4.5 means probably avoid replacing the whole pass manager, but
> > just inject inside the few hooks required.
>
> be able to work on ICI, and possibly even convincing Intel to invest
> into it in the next 2 years. Second, if Grigori is really out of luck
> with Intel, I will take over together with our IBM colleagues to make
> sure ICI does not just die out.
> > So I really think you should try *hard* to push the few events useful to
> > you. And in my MELT experience, this is almost a bitter effort. The
> > point is that the GCC community is by principle always right -whatever
> > that means exactly-. This probably means that you might have to remove
> > (with an #ifndef) some very interesting features from ICI to make it fit
> > into GCC 4.5. And I insist that adding a few lines to GCC core is in my
> > feeling much more hard than coding thousands of lines in my MELT branch
> > (or inside the MELT plugin). This means that if ICI would need some
> > significant adaptation to fit into GCC 4.5, it is worth the effort to
> > make them in my eyes. And pushing even a small plugin event into GCC
> > takes a big lot of time & effort - much more than what Grigori imagines!
> > A "small-steps" policy is preferable to sending a big patch nobody will
> > review.
>
>
> To ease the process, the ICI should only appear as a small set of
> patches and new plugin hooks. It should not be more than this. There is
> no way ICI can become a big plugin. It is meant to facilitate new
> interaction means for third-party, small plugins to tune existing passes
> of GCC and the pass manager itself.
> > Maybe I don't understand well ICI (I admit I did not glance inside the
> > ICI code recently), but I would like ICI too succeed in been usable with
> > GCC. Because in my experience, nobody cares about GCC variants or
> > branches. ICI need to be delivered as a GCC plugin to be used.
>
> enabling new kinds of interactions that plugins cannot currently offer.
> But this may itself be too disruptive to be realistic, and should be
> implemented step by step.
> > I am a bit sorry for such a harsh email, it is only a discussion and I
> > definitely hope I am not offending anyone. But I don't understand the
> > strategy of ICI (maybe I should not even try to understand it).
> >
> > So a question for ICI gurus. Can ICI (or a large subset of it) fit into
> > a weakly modified gcc 4.5 (by adding only few plugin events & their
> > invocation)? I guess that yes, it could fit! If yes, try to list the few
> > additional plugin events & where they should be invoked inside GCC! If
> > you need to restrict ICI to do that, I believe doing that restriction is
> > worth the effort.
> >
> >
> > But don't forget that pushing even one single plugin_event into GCC will
> > take you significant time & effort, so better start now!
> >
> >
> >
> > ###
> >
> > Regards
> >
> > Note * about {mega,meta}-plugins to GCC:
> > In that respect, MELT & ICI are similar: MELT is and ICI should in my
> > opinion become a big plugin of gcc-4.5 (hence the mega adjective) which
> > itself loads (thru dlopen) other "modules" (or call them otherwise, I
> > mean technically any shared object passed as argument to dlopen with
> > defined hooking conventions implemented by dlsym). Also, since MELT and
> > ICI are themelselves code which call dlopen I call them (quite
> > pedantically) meta-plguins. Hence the {mega,meta}-plugin notation (with
> > braces parsed like the unix shells do).
>
> small in terms of functionality. DragonEgg is clearly not the kind of
> plugin we should push for. It was not the original intent of the plugin
> system. But it does not harm to have such big plugins either... as long
> as the main developments are not harmed by these big plugins hanging around.
>
> Albert
Joern Rennecke
http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/gcc-4.4.0-ici-2.0-ad
apt/gcc/ipa-inline.c ,
with the log message:
change name of fapi-tools.c, fix a bug, delete old scripts, please use
scripts u
nder directory branches/ici-script-tools
I have taken the liberty to reformat the change as attached below.
Questions:
- Does this fix a bug that exists independently of ICI ?
- How important is this fix for ICI ?
- Is it intentional that when DECL_FUNCTION_SPECIFIC_OPTIMIZATION is set
for the node with optimize > 1, inline_indirect_intraprocedural_analysis
will be called irrespective of the setting of flag_indirect_inlining ?
- Do you have a testcase?
Grigori Fursin
ICI sometime has been using environment variables inside GCC with its own invocation flags
(-fici) and dynamic library loading.
Naturally, Joern will remove duplicate dynamic library handling and invocation flags from ICI to
use the plugin functionality from GCC 4.5.
As for environment variables inside GCC - we needed it for transparent program analysis
and optimizations but I remember that there were several concerns about that (I think
Richard mentioned several times that it complicated debugging if there are problems),
so we will remove them too. We will use current plugin flags to pass parameters to
the plugins (we can pass a configuration file I guess that will be parsed by our plugins
if there is lots of information to pass) and if we need to do transparent program
analysis and optimization we will use a script and a wrapper around GCC that translated
our environment variables into flags. I already did that recently for MILEPOST GCC so it
should be easy.
But for now we assume that it's fine to use environment variables (such as to control
verbose output or pass some parameters) in plugins themselves ... However, eventually
we should also use some configuration files for plugins that can be easily shared with the community
if there is a bug in plugins ...
Cheers,
Grigori
lpeng
I made this modification trying to fix a
bug(http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41150) that I found when
implementing function cloning for GCC4.4.0. I was not very sure about
this solution.
> - Does this fix a bug that exists independently of ICI ?
Best regards,
Liang Peng
lpeng
Best regards,
Liang Peng
>
> You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
> To post to this group, send email to ctuning-d...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=.
>
>
>
>
Joern Rennecke
PLUGIN_ , whereas the ICI event names are lower case and
have no particular start prefix (e.g. "unroll_parameter_handler",
"all_passes_start").
Would it be a problem if we renamed all the ICI event names to
fit the current gcc scheme?
We have alternatives, but they are a bit messy:
- Try to push the current names. The downside is that the
inconsistency could count against the patch.
- Remap event names inside ICI. This could be a lookup table,
or if we get event names that all are just prepending PLIGIN_
and have the captalized version of the current name, ICI could
use toupper to convert the legacy ICI event names on the fly.
Yuanjie Huang
As far as I know only three environment variables are used for the
summer of code projects of Liang and I: one is used to control the
record/reuse behavior
of adapt plugin, one to control the verbose level of adapt plugin, and
the last is used to set the path to library file which contains the
select or instrument function for Liang's clone/instrument project. I
think the first two can be removed by adding an extra call to
initialize plugin with a path to setting file, and the last one can be
moved to our XML files.
Cheers,
Yuanjie
>
> You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
> To post to this group, send email to ctuning-d...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=.
>
>
>
Yuanjie HUANG
Grigori Fursin
I think at this stage we can afford renaming of all events
to make a smooth transition to GCC 4.5.
We should provide a match table which we will later
put on the Wiki so that ICI users can rewrite plugins ...
At this stage it should not yet be too difficult ...
Cheers,
Grigori
Grigori Fursin
we will need to sync all plugins with the new style ...
Cheers,
Grigori
> -----Original Message-----
> From: huangy...@gmail.com [mailto:huangy...@gmail.com] On Behalf Of Yuanjie Huang
> Sent: Tuesday, November 17, 2009 4:51 PM
> To: ctuning-d...@googlegroups.com
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>
Grigori Fursin
Joern Rennecke
headers to build new plugins, but also install all plugins that
have been contributed up to the code freeze for the release.
First, it would make testing the plugin interface and the plugins easier.
Second, if the version of a plugin available with the release is good enough
for the needs of a user, (s)he can try out without having to install -
or having to have installed - the plugins separately.
Third, when someone uses an experimental plugin for a while and then upgrades
some time later to a newer gcc release, it is useful when an updated plugin
version that works with the new gcc release comes with the new gcc
distribution.
Diego Novillo
> What do people think about making install-plugin not only install
> headers to build new plugins, but also install all plugins that
> have been contributed up to the code freeze for the release.
would be beneficial to include a tutorial plugin somewhere that shows
the basics. I have no opinion on where in the tree.
Diego.
Basile STARYNKEVITCH
We could simply adapt a plugin from our testsuite, and install it...
The interesting question is: do we have an installed plugins directory? (We might have already discussed that, I forgot
the details and the context, probably more than a year ago). I wish we had one:
We might even consider that invoking [sorry to be so selfish and take MELT as a plugin example]
gcc-4.5 -fplugin=melt ....
doing the same as
gcc-4.5 -fplugin=$(gcc-trunk --print-file-name=plugin)/libexec/melt.so ....
That is, a plugin only specified with XXX [only letters or digits or underscores, no dots, so no .so extensions, no
slashes] being searched in some well known directory like
/usr/local/lib/gcc-trunk/gcc/x86_64-unknown-linux-gnu/4.5.0/plugin/libexec/ . If I recall correctly, most other
pluginable software have their "standard plugin directory".
I would be delighted by having in practice short -fplugin=XXX program argument to GCC. In my understanding, the current
one is almost always long in practice [because in practice the plugin directory should depend of the GCC version].
The issues are:
* do we agree that having a well defined directory to contain some installed plugins is desirable?
* what is that standard plugins directory?
And then, we have to implement & document that.
In the current state of the trunk, it seems that plugins are the only point where environment variables matter, and that
environment variable is LD_LIBRARY_PATH ... (because dlopen mandates that behavior).
BTW, I think that both MELT & ICI are dlopen-ing their own shared objects (in MELT I call these modules), and that MELT
(& probably ICI) have some mechanism for some standard paths for these.
Regards.
Joern Rennecke
> The interesting question is: do we have an installed plugins
> directory? (We might have already discussed that, I forgot
> the details and the context, probably more than a year ago). I wish
> we had one:
subdirectory of an otherwise empty plugin directory:
[amylaar@laria gcc]$ /user/inria/bin/gcc --print-file-name=libgcc.a
/user/inria/lib/gcc/i686-pc-linux-gnu/4.5.0/libgcc.a
[amylaar@laria gcc]$ /user/inria/bin/gcc --print-file-name=plugin
/user/inria/lib/gcc/i686-pc-linux-gnu/4.5.0/plugin
[amylaar@laria gcc]$ ls -l `!!`
ls -l `/user/inria/bin/gcc --print-file-name=plugin`
total 4
drwxrwxr-x. 7 amylaar amylaar 4096 2009-11-18 17:48 include
[amylaar@laria gcc]$ /user/inria/bin/gcc --version
gcc (GCC) 4.5.0 20091108 (experimental)
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Joern Rennecke
to show a use for every hook added, and leave it up yo the reviewers
if they only accept the infrastructure in GCC proper (of course there
is no guarantee that we get even that, but it is the main objective of
the current work), or also all the plugins so that they can be built
alongside GCC on hosts where this is supported.
However, before I can even put code into a GCC branch, I must check that
Copyright assignments are in place for all the code. I know that INRIA
has a Copyright assignment in place, put there is code from other
contributers.
E.g. is all the Google Summer of Code for the code adaptation project covered
by proper Copyright assignments? I tried to look this up on the
Summer of Code
web sites, but came up empty; likewise, there is no Copyright header in any of
the files in the adapt plugin directory.
Likewise for all the rest of the plugins-ici-2.0 directory, except for
the MiBench sources.
In fact, although I approached the code in
http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/gcc-4.4.0-ici-2.0-adapt
with the assumption that Copyright assignments for all of it are in place -
after all, all the files bear FSF Copyright headers - I would like to
double-check that this is in fact the case.
This concerns not only the added new files, but also every patch to a
pre-existing file.
Also, even if Copyright assignments are in place, AFAIK these generally
operate on the basis of assigning Copyright to code that is actually submitted
to the covered projects. So, if such a submission has not taken place yet, do
the current Copyright holders agree with submitting the code?
Finally, when it comes to submitting the patch, I will need to write a
consolidated ChangeLog entry for the patch.
Although the GNU coding standard says that only the person checking in code
is relevant, in the GCC project we have the convention of listing all the
authors of a patch submission. So, can I rely on the file headers or on
sourceforge svn annotate to get a list of authors? If neither of these,
how do I get such a list?
How do I find the matching email addresses?
Are all the authors OK with being named in the patch submission and ChangeLog
entry, and with their email addresses being published there?
If individuals want their name and/or email address left out, I can do that
and still use the code as long as we can tell the FSF who the previous
Copyright holders were and assignments are in place, filed, and acknowledged,
to cover all the to-be-submitted code.
Yuanjie HUANG
gcc/events.c: add event_parameter_type to event_parameter
gcc/feature-list.c: add feature struct feature_main_input_filename and get_main_input_filename function, modified init_features
gcc/highlev-plugin-internal.h: add event_parameter_type enumerate type and declaration for run_ipa_pass, initialize_ici_pass_list, insert_ici_pass_list, run_ici_pass_list, run_ici_pass_list_ipa_summary, run_ipa_pass_list_per_function and delete_ici_pass_list
gcc/highlev-plugin.h: add event_parameter_type enumerate type and declaration for run_ipa_pass, initialize_ici_pass_list, insert_ici_pass_list, run_ici_pass_list, run_ici_pass_list_ipa_summary, run_ipa_pass_list_per_function and delete_ici_pass_list
gcc/pass-manager.h: declaration for run_ipa_pass, initialize_ici_pass_list, insert_ici_pass_list, run_ici_pass_list, run_ici_pass_list_ipa_summary, run_ipa_pass_list_per_function and delete_ici_pass_list
gcc/pass-manager.c: add run_ipa_pass, initialize_ici_pass_list, insert_ici_pass_list, run_ici_pass_list, run_ici_pass_list_ipa_summary, run_ipa_pass_list_per_function and delete_ici_pass_list function
And I'd prefer huangy...@gmail.com as my email
Cheers,
Yuanjie
>
> You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
> To post to this group, send email to ctuning-d...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=.
>
>
>
Yuanjie HUANG
lpeng
here is a list of contributers of adapt GCC4.4.0 branch. I hope it will
be helpful for you.
Best regards,
Liang Peng
Joern Rennecke
> Hi Joern,
> here is a list of contributers of adapt GCC4.4.0 branch. I hope it will
> be helpful for you.
The FSF will assume that ICT (I presume that it is a Chinese university or
similar institution of learning, with a name like that, a web search is not
very useful) will have rights to the code you wrote while you were a student
there or working there.
If that is the case, the FSF will require a Copyright assignment from ICT.
If that is not the case, the FSF will require a disclaimer from ICT that they
don't own these Copyrights, and a Copyright assignment from you, or if the
Copyright vested in a different organization, from that organization.
Have such documents been submitted to the FSF and their receipt and
successful filing been acknowledged?
Could whoever know about the relevant details about Copyright
ownership / assignment for the other contributers also please share
this information.
If we cannot confirm the existence of valid copyright assignments for any
contributer(s), we cannot use their code in a patch submission for GCC.
Grigori Fursin
for the Google Summer of Code that if you work in this program
and paid by it, the copyright which would belong to GSOC goes
to FSF??? It was summer developments which means that the students
do not officially work for ICT during that time and can do anything
they want so during that time copyright should not belong to ICT?..
Uff, these are tricky questions ...
Also, maybe we should ask it in the GCC mailing list because they
were several other people at GSOC'09 that may have the same issue ...
Cheers,
Grigori
> -----Original Message-----
> From: Joern Rennecke [mailto:amy...@spamcop.net]
> Sent: Friday, November 20, 2009 2:02 PM
> To: ctuning-d...@googlegroups.com; lpeng
> Cc: Grigori Fursin; 'Grigori Fursin'; 'Albert Cohen'; 'Basile STARYNKEVITCH'; 'Zbigniew
> Chamski'; 'Diego Novillo'; 'Rafael Espindola'; 'Yuri Kashnikoff'; 'Yuanjie Huang';
Chengyong Wu
--
You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
To post to this group, send email to ctuning-d...@googlegroups.com.
To unsubscribe from this group, send email to ctuning-discuss...@googlegroups.com.
lpeng
You are right. Here is a question on GSOC FAQ page.
"Who owns the code produced by student developers?
Each student (or her/his mentoring organization) must license all
student GSoC code under a license palatable to the mentoring
organization. Some organizations will require students to assign
copyright to them, but many will allow them to retain copyright. If
Google is a student's sponsoring organization, then the student keeps
copyright to her/his code. "
So I think it depends on GCC, right?
Best regards,
Liang Peng
Grigori Fursin
Well, unfortunately, I am not a specialist on those topics (yet ;) )...
But it seems that we shouldn't have a problem.
Maybe together with copyrights from you and Yuanjie, we should
add that these were GSOC'09 developments - it should clear up
things for the reviewers?..
Joern, what do you think about that?..
And any feedback from GCC people ;) ?...
> discussions?hl=.
> >
> >
> >
> >
>
>
> You received this message because you are subscribed to the Google Groups "ctuning-
> discussions" group.
> To post to this group, send email to ctuning-d...@googlegroups.com.
> To unsubscribe from this group, send email to ctuning-
Joern Rennecke
> Hi Grigori,
> You are right. Here is a question on GSOC FAQ page.
>
> "Who owns the code produced by student developers?
>
> Each student (or her/his mentoring organization) must license all
> student GSoC code under a license palatable to the mentoring
> organization. Some organizations will require students to assign
> copyright to them, but many will allow them to retain copyright. If
> Google is a student's sponsoring organization, then the student keeps
> copyright to her/his code. "
>
> So I think it depends on GCC, right?
At any rate, for the copyright to be assigned to the FSF you would have to
have signed something - either a direct Copyright assignment to the FSF, or
something like a contract that made the GSOC code work for for hire for an
organization that had already filed or did then file a Copyright Assignment
to the FSF.
So, did you sign anything that could fit that description?
Yuanjie Huang
--
You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
To post to this group, send email to ctuning-d...@googlegroups.com.
To unsubscribe from this group, send email to ctuning-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=.
--
Yuanjie HUANG
Joern Rennecke
participants get some payment for their work. Is there a written
agreement?
Does it have provisions about the copyright of code produced in the
course of the GSOC work?
Yuanjie Huang
--
You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
To post to this group, send email to ctuning-d...@googlegroups.com.
To unsubscribe from this group, send email to ctuning-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=.
--
Yuanjie HUANG
Grigori Fursin
and from what I know there was code moved to GCC mainline from those programs before.
So, I guess there will be no problem with that.
We just need to clearly mark the code done under GSOC program.
We can check it IF reviewers of the patch have a problem with that, but
considering that it's not the first GSOC participation, I don't think so.
Maybe you can ask directly a few people responsible for that.
I think Ian Lance Taylor was an official GCC representative of GSOC program -
maybe you should ask him about that ?..
Cheers,
Grigori
> -----Original Message-----
> From: Joern Rennecke [mailto:amy...@spamcop.net]
> Sent: Friday, November 20, 2009 4:36 PM
> To: ctuning-d...@googlegroups.com; Yuanjie Huang
> Cc: lpeng; Grigori Fursin; Albert Cohen; Basile STARYNKEVITCH; Zbigniew Chamski; Diego
> Novillo; Rafael Espindola; Yuri Kashnikoff; do...@il.ibm.com
> Subject: Re: [plugins-ici-cloning-instrumentation] Contribution scope, authors and copyright
> assignments
>
Joern Rennecke
> Also, Joern, I think GCC participates in GSOC program for the second
> time (or maybe even 3rd)
> and from what I know there was code moved to GCC mainline from those
> programs before.
> So, I guess there will be no problem with that.
remedy the situation, but that will be too late for GCC 4.5 .
> We just need to clearly mark the code done under GSOC program.
> We can check it IF reviewers of the patch have a problem with that, but
> considering that it's not the first GSOC participation, I don't think so.
at the technical quality of the submission an to make sure that it fits well
with the current and desired future code base. The submitter is supposed
to have checked the presence of Copyright assignments beforehand.
Moreover, only code with valid Copyright assignments should be in the
FSF gcc repository, no matter if in main line or on a branch.
> Maybe you can ask directly a few people responsible for that.
>
> I think Ian Lance Taylor was an official GCC representative of GSOC program -
> maybe you should ask him about that ?..
Grigori Fursin
Grigori
> -----Original Message-----
> From: Joern Rennecke [mailto:amy...@spamcop.net]
> Sent: Friday, November 20, 2009 6:29 PM
> To: Grigori Fursin
> Cc: ctuning-d...@googlegroups.com; 'Yuanjie Huang'; 'lpeng'; 'Albert Cohen'; 'Basile
> STARYNKEVITCH'; 'Zbigniew Chamski'; 'Diego Novillo'; 'Rafael Espindola'; 'Yuri Kashnikoff';
> do...@il.ibm.com
> Subject: RE: [plugins-ici-cloning-instrumentation] Contribution scope, authors and copyright
> assignments
>
Joern Rennecke
I see that only integer types are passed. In fact, event_parameter_type
only has values for integer types. Thus, it should be sufficient to describe
size and signedness of the type - where these coincide for multiple basic
types, there is no actual problem on the hosts we care about for plugins if
we mix up these equivalent types when accessing data via pointers.
The size and signedness can be automatically encoded; likewise, we can
automatically attach source file and line of the plugin callback
invocation.
Then ICI can construct parameter names from source file name, line, and
parameter number.
I have attached patches for gcc and ICI to implement this new scheme for
the unroll heuristics; we can do graphite similarily once we know what
heuristics we want to target in the current version.
Does this scheme make sense to you?
Yuanjie Huang
To my understanding, this new patch may cause different XML files
generated by the same version of compiler on different systems, I am
afraid this is not a good news.
Cheers,
Yuanjie
> --
>
> You received this message because you are subscribed to the Google Groups "ctuning-discussions" group.
> To post to this group, send email to ctuning-d...@googlegroups.com.
> To unsubscribe from this group, send email to ctuning-discuss...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=en.
>
>
>
--
Yuanjie HUANG
Joern Rennecke
> Hi Joern,
>
> To my understanding, this new patch may cause different XML files
> generated by the same version of compiler on different systems, I am
> afraid this is not a good news.
You always had the factor that different targets will can get
you different passes and parameters.
I suppose you are referring here to the different parameter names
depending on how and where the compiler is built.
I generally build gcc in a sibling directory of the gcc toplevel,
so I get xml records like:
<parameter param_name="../../gcc-param/gcc/loop-init.c:337:0"
param_type="int">0
</parameter>
As long as using the same relative pathnames for configuring, you
should get the same relative pathnames in the xml file.
But of course, as soon as you use a different relative pathname
for configuring, or an absolute pathname, the parameter name will
change.
We could reduce the file name to its basename - or use
__BASE_FILE__ in the first place - but that would
cause ambiguities in the multi-targeted compiler:
(gdb) break note_stores
Breakpoint 3 at 0x6bf88c: file ../../srcw/gcc/rtlanal.c, line 1456.
(gdb) break mxp_elf::note_stores
Breakpoint 4 at 0x84fd14: file ../../../srcw/gcc/rtlanal.c, line 1456.
I suppose we can handle this by adding a target field to the
struct heuristics, and constructing the parameter name from
target_name:base_name:lineno:parm_number
Joern Rennecke
tags in the XML file - independent of the build location and the way
absolute / relative pathnames are used for configuring.
Joern Rennecke
some proposed new plugin callbacks would also be interesting to have as
target hooks. Therefore, I would like target hooks to become writeable
by plugins, and make it much easier to add new target hooks in the GCC
sources.
Right now, to make a new target hook, you have to add a new field in
target.h, define a new default in target-def.h, place the new macro
in exactly the right position there of the right initializer macro,
describe the new hook in tm.texi, and if you need a new function with
a bunch of parameters returning a constant, you have to add this to
hooks.c .
I would like to be able to do all this by adding a single entry in a
new definition file; and the information should also be usable by
plugin sources so that they can automatically make wrappers for all
function-type hooks.
Most of the ICI unroll_parameter_handler / graphite_parameter_handler
callbacks can the be made into target hooks.
Joern Rennecke
> Right now, to make a new target hook, you have to add a new field in
> target.h, define a new default in target-def.h, place the new macro
> in exactly the right position there of the right initializer macro,
> describe the new hook in tm.texi, and if you need a new function with
> a bunch of parameters returning a constant, you have to add this to
> hooks.c .
>
> I would like to be able to do all this by adding a single entry in a
> new definition file; and the information should also be usable by
> plugin sources so that they can automatically make wrappers for all
> function-type hooks.
I've attached what I have so far.
There is an issue that the struct gcc_target member names don't always agree
with the TARGET_* macro names.
Should I rather change one or the other to make them agree, or add an extra
parameter in the definition file macros to specify these names independently?
Richard Guenther
I didn't look at the patch yet but I'd simply change one or the other to
make the names consistent.
Richard.
>
>
Joern Rennecke
> On Wed, 23 Dec 2009, Joern Rennecke wrote:
>
>> I've attached what I have so far.
>
> If you want to have documentation extracted from source files, you need to
> engage with the SC and FSF at an early stage to get suitable license
> exception wording to permit the relevant text to be used in the manuals as
> well as the main (GPL) sources.
I suppose I'm reaching the steering commitee by posting to g...@gcc.gnu.org.
What email address should I use to engage the FSF?
I want to move the hook descriptions from tm.texi into target.def .
gentarget.c includes target.def, and gentarget (re-)generates
$(srcdir)/doc/targhooks.texi (the makefile dependency would be keyed on
--enable-maintainer-mode) ; targhooks.texi would then include the hook
descriptions from target.def .
As the person who regenerates targhooks.texi is in possession of target.def ,
I think the (s)he can use the license on target.def to generate a GFDL
(terms according to gcc.texi) targhooks.texi which includes pieces of
target.def, without requiring the gentarget executable to have a special
license; the executable is only a means to create a (documentation) work
based on target.def .
I've looked at the bison skeleton for inspiration on the license blurb
for target.def .
This is what I have come up with:
/* Target hook definitions.
Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010
Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 3, or (at your option) any
later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>.
In other words, you are welcome to use, share and improve this program.
You are forbidden to forbid anyone else to use, share and improve
what you give them. Help stamp out software-hoarding! */
/* As a special exception, you may create documentation that contains
part or all of this target hooks definitions file and distribute that
work under the copying terms outlined in doc/gcc.texi .
Alternatively, if you modify or redistribute this target hooks
definitions file itself, you may (at your option) remove this special
exception, which will cause the target hooks definitions file (and any
program output which incorporates parts of this file) to be licensed
under the GNU General Public License without this special exception. */
> Of course, the ordering and (especially)
> section divisions in the internals manual are human-designed not arbitrary
> so you need to retain the ability to put the documentation of each hook in
> an appropriate position in an appropriate section of the manual.
Yes, I plan to allow any number of explicit placements of hooks descriptions
in the resulting documentation, and any hooks that are not explicitly placed
will appear as successor of their predecessor in target.def .
Joern Rennecke
> If you want to have documentation extracted from source files, you need to
> engage with the SC and FSF at an early stage to get suitable license
> exception wording to permit the relevant text to be used in the manuals as
> well as the main (GPL) sources.
I haven't gotten any reply to my proposal from the 5th of January
http://gcc.gnu.org/ml/gcc/2010-01/msg00082.html
Is the GCC mailing list no longer the right place to reach the
steering committee?
> Of course, the ordering and (especially)
> section divisions in the internals manual are human-designed not arbitrary
> so you need to retain the ability to put the documentation of each hook in
> an appropriate position in an appropriate section of the manual.
I have posted my current patch here:
http://gcc.gnu.org/ml/gcc-patches/2010-01/msg00559.html
> If you can make this work it should reduce the risk of people not
> documenting new hooks
We actually have a number of instances where the target hook documentation is
out of sync with the sources. I have attached a file with my notes on this.
> , but for it to work the FSF agreement is critical
So who do I have to talk to for this?
> as
> is ensuring it genuinely works as well as the existing system without
> regressions (including working in update_web_docs).
tm.texi is half a megabyte, so it would be nice not to have it as a generated
file in the repository. Also, it'd be annoying to have to regenerate & check
it in for every target.def change even though the build works fine with
the generated file in the build directory.
Will it be acceptable to have update_web_docs build a generator program
(written in C) to rebuild tm.texi?
Joseph S. Myers
> Quoting "Joseph S. Myers" <jos...@codesourcery.com>:
> > If you want to have documentation extracted from source files, you need to
> > engage with the SC and FSF at an early stage to get suitable license
> > exception wording to permit the relevant text to be used in the manuals as
> > well as the main (GPL) sources.
>
> I haven't gotten any reply to my proposal from the 5th of January
> http://gcc.gnu.org/ml/gcc/2010-01/msg00082.html
>
> Is the GCC mailing list no longer the right place to reach the steering
> committee?
It is the right place, but you should start a new thread with just the
relevant issue and an appropriate subject line marked for the attention of
the SC. You should expect it to take at least several months for the FSF
to prepare an exception (worded as additional permissions under Section 7
of GPL version 3, like COPYING.RUNTIME, *not* as an old-style "As a
special exception"), maybe years, if the SC persuades them that there
should be such an exception.
> > Of course, the ordering and (especially)
> > section divisions in the internals manual are human-designed not arbitrary
> > so you need to retain the ability to put the documentation of each hook in
> > an appropriate position in an appropriate section of the manual.
>
> I have posted my current patch here:
> http://gcc.gnu.org/ml/gcc-patches/2010-01/msg00559.html
>
> > If you can make this work it should reduce the risk of people not
> > documenting new hooks
>
> We actually have a number of instances where the target hook documentation is
> out of sync with the sources. I have attached a file with my notes on this.
Please note that your initial change to implement automatic doc extraction
should not result in any changes to the Texinfo content of the manual.
Such fixes should all go in either before or after the automatic doc
extraction change, but not at the same time; the doc extraction change
should result in identical text in the manual, but with the Texinfo files
produced in a different way. I recommend sending such fixes before the
automatic doc extraction change, since they do not depend on the FSF doing
anything.
> > , but for it to work the FSF agreement is critical
>
> So who do I have to talk to for this?
The SC. Not in the middle of a technical thread, but a thread on its own
marked for the SC and indicating exactly what you want them to raise with
the FSF.
> tm.texi is half a megabyte, so it would be nice not to have it as a generated
> file in the repository. Also, it'd be annoying to have to regenerate & check
> it in for every target.def change even though the build works fine with
> the generated file in the build directory.
> Will it be acceptable to have update_web_docs build a generator program
> (written in C) to rebuild tm.texi?
It already builds a generator program in Ada for the Ada manual, so yes.
--
Joseph S. Myers
jos...@codesourcery.com
Joern Rennecke
> Please note that your initial change to implement automatic doc extraction
> should not result in any changes to the Texinfo content of the manual.
> Such fixes should all go in either before or after the automatic doc
> extraction change, but not at the same time; the doc extraction change
> should result in identical text in the manual, but with the Texinfo files
> produced in a different way. I recommend sending such fixes before the
> automatic doc extraction change, since they do not depend on the FSF doing
> anything.
Duplicating all these changes separately by hand seems nigh impossible.
I think the best approach is then to take the auto-generated tm.texi as
the new tm.texi, and packages it up as a patch together with the
struct member / hook name changes that I made for consistency.
There is only one issue with using the current auto-generated tm.texi:
Unless special formatting was in force (e.g. @smallexample), I've removed
intra-paragraph newlines. This should work in principle just as will
as with these newlines for producing output, but it looks somewhat daft
in tm.texi when you consider it as a source file.
Putting newlines in the input file would make it harder to read & edit,
since the documentation comes as C strings - and the GNU multiline string
extension has been deprecated some time ago.
I could pipe the documentation through fold -s, but that would also fold
extra-long lines outside the hook documentation, e.g.:
@@ -28,7 +28,8 @@
@menu
* Target Structure:: The @code{targetm} variable.
* Driver:: Controlling how the driver runs the
compilation passes.
-* Run-time Target:: Defining @samp{-m} options like
@option{-m68000} and @option{-m68020}.
+* Run-time Target:: Defining @samp{-m} options like @option{-m68000} and
+@option{-m68020}.
* Per-Function Data:: Defining data structures for per-function
information.
* Storage Layout:: Defining sizes and alignments of data.
* Type Layout:: Defining sizes and properties of basic user
data types.
So I suppose I'll have to add some fold mechanism into the documentation
output code.
Joseph S. Myers
> Duplicating all these changes separately by hand seems nigh impossible.
> I think the best approach is then to take the auto-generated tm.texi as
> the new tm.texi, and packages it up as a patch together with the
> struct member / hook name changes that I made for consistency.
>
> There is only one issue with using the current auto-generated tm.texi:
> Unless special formatting was in force (e.g. @smallexample), I've removed
> intra-paragraph newlines. This should work in principle just as will
> as with these newlines for producing output, but it looks somewhat daft
> in tm.texi when you consider it as a source file.
I am not particularly concerned about newlines, if the file is identical
apart from whitespace. But your text file lists things such as "was
undocumented" and "Fixed return value description". Each such change
needs its own review, by someone familiar with the relevant part of the
compiler, and needs its own explanation of the problem posted. Remember,
a patch should not contain multiple changes that can logically be
considered separately, and in this case I expect many different people to
be appropriate reviewers for changes relating to different hooks, so it's
important not to mix changes relating to hooks in different areas of the
compiler.