Fwd: unbounded dependency examples
Deniz Yuret
From: Laura Rimell <laura....@cl.cam.ac.uk>
Date: Mon, Mar 1, 2010 at 11:48 AM
Subject: Re: unbounded dependency examples
To: Deniz Yuret <dyu...@ku.edu.tr>
Hi Deniz,
We've been looking over the dataset and it's quite interesting. A
quick question - I see you've posted a sample script for obtaining
entailments from CoNNL style output. Is any such script going to be
part of the evaluation, or will you simply be evaluating on the yes/no
answers to the hypotheses regardless of how they are obtained?
Many thanks,
Laura
Deniz Yuret
From: Deniz Yuret <dyu...@ku.edu.tr>
Date: Mon, Mar 1, 2010 at 12:00 PM
Subject: Re: unbounded dependency examples
Hi Laura,
The script is not going to be part of the evaluation - I just posted
it as an example. I do not want to tie people to a particular
formalism. I will only evaluate the yes/no answers regardless of how
they are obtained.
best,
deniz
Deniz Yuret
From: Laura Rimell <laura....@cl.cam.ac.uk>
Date: Mon, Mar 1, 2010 at 12:51 PM
Subject: Re: unbounded dependency examples
Hi Deniz,
Thanks. One other point - where you list publicly available parsers on
the web site, you might want to include some of the other major ones
like C&C, Enju, and Rasp, especially if people don't need to be tied
to parsers that produce CoNLL or PTB style output. These three were
all part of the unbounded dependency evaluation and in fact C&C and
Enju scored the highest on that evaluation. (Also, we weren't sure
what you meant by saying that participants don't need to use a parser
at all, since it seemed the original purpose of the task was parser
evaluation!)
If we find any problems in the data are you interested in hearing
about them at this point, or is it too late to make any changes?
Best,
Laura
Deniz Yuret
From: Deniz Yuret <dyu...@ku.edu.tr>
Date: Mon, Mar 1, 2010 at 1:56 PM
Subject: Re: unbounded dependency examples
Hi Laura,
Thank you for the parser references. I will add them ASAP.
My contoversial statement is supposed to encourage original
approaches. The task is not just for comparing current parsing
approaches. Any new idea that claims to replicate human syntactic
performance would be welcome. What I have in mind are unsupervised
approaches, or RTE, IR type tricks, finite state approaches etc. So
maybe the choice of task name was unfortunate.
Please send me any feedback you have on the dataset. The test set has
not been released yet so some issues may still be addressed.
thanks
deniz
Deniz Yuret
From: Laura Rimell <laura....@cl.cam.ac.uk>
Date: Fri, Mar 5, 2010 at 5:05 PM
Subject: Feedback on PETE dev data
To: Deniz Yuret <dyu...@ku.edu.tr>
Hi Deniz,
Here are the issues that occurred to us as we were looking at the dev
data. It is a very interesting task (both the data collection and the
evaluation). I'm not sure any of these would be possible or desirable
to change for the test data, but perhaps they may be helpful as points
for discussion or clarification.
1) We wondered about the addition of determiners in H where the
corresponding noun in T is determinerless. This has a variety of
effects:
<pair id="2082.N" entailment="YES">
<t>Trading in AMR shares was suspended shortly after 3
p.m. EDT Friday and did n't resume .</t>
<h>The trading didn't resume.</h>
</pair>
Here, "Trading didn't resume" would have been a grammatical sentence.
The use of the determiner thus seems to add a needless complication
when checking for the entailment. One small complexity it introduces
in this particular example is changing upper to lower case in
"trading". But more broadly, the system needs to have some kind of
rule for ignoring the determiner in H even though this makes H deviate
syntactically from T.
<pair id="5032" entailment="YES">
<t>Thus committed , action might follow .</t>
<h>The action might follow.</h>
</pair>
This case is slightly more problematic. I'm not sure H is strictly an
entailment of T (semantically as well as syntactically), because of
the definite determiner - there's no single, definite action which the
speaker of T necessarily had in mind. So again, you need to have a
rule about ignoring the contribution of determiners in H. (And again,
"Action might follow" would have been grammatical.)
<pair id="2007.N" entailment="YES">
<t>And many in the young cast bear striking
resemblances to American TV and movie personalities known for light
roles .</t>
<h>Somebody bears the resemblances.</h>
</pair>
Even though this one is marked as a positive example of entailment,
the addition of the determiner has made H fully ungrammatical. I'm
unsure of the role of ungrammatical H's as negative examples (see next
comment), but it does strike me that positive examples of entailment
should always be grammatical.
Here's another example where the substitution of the determiner
(compounded by the deletion of the PP) creates a quite unnatural, if
not perhaps fully ungrammatical, H:
<pair id="5062" entailment="YES">
<t>He turned his attention to the workbench .</t>
<h>The attention was turned.</h>
</pair>
If part of your system for whatever reason used a language model to
check the probability of H (not saying that's necessarily a good
strategy), you'd have a problem with these examples.
I'm pretty ignorant about RTE tasks in general, though, so maybe the
treatment of determiners in this way is a given in RTE and I'm not
aware of it.
2) We were wondering about the role of ungrammatical H's.
<pair id="6098" entailment="NO">
<t>Moreland sat brooding for a full minute , during
which I made each of us a new drink .</t>
<h>Somebody sat brooding I made each.</h>
</pair>
Given that a certain percentage of the negative examples are
ungrammatical, you could probably write a really basic script that
gave a negative judgement if you couldn't parse H. (Though this won't
work if there are ungrammatical positive examples as well, and of
course you can often get an implausible parse of an ungrammatical
sentence.) What do you see as the role of the ungrammatical examples
in parser evaluation? Perhaps they are actually important for teasing
out some of the common parser attachment errors.
3) We were also wondering about the role of transformations that
change the grammatical category of one of the crucial words.
<pair id="1032.N" entailment="NO">
<t>Mr. Hahn attributes the gains to the philosophy of
concentrating on what a company knows best .</t>
<h>Somebody philosophizes what a company knows best.</h>
</pair>
I don't think the noun-to-verb transformation will cause much of a
problem in this example because it's a negative example for other
reasons (the object of "philosophize" is incorrect). However, if H
had been e.g. "Somebody philosophizes about concentrating on what a
company knows best" - would that be an entailment or not? It seems to
me this may involve a judgment about meaning preservation under the
philosophy-philosophize transformation rather than a strictly
syntactic entailment. (Unless you intend any H in which one of the
lexical items changes its major grammatical category to be a negative
example.)
4) We were wondering about the benefits of using the passive
transformation. Is it preferable to use a passive H derived from an
active T, rather than extracting a simple active H - e.g. "He saw the
stream" or "Someone saw the stream" rather than "The stream had been
seen" in this example?
<pair id="2004" entailment="YES">
<t>He could also hear the stream which he had seen from
his position .</t>
<h>The stream had been seen.</h>
</pair>
Of course, a system that recognizes entailments should be able to
recognize passive ones, but from the point of view of parser
evaluation, does it add an unnecessary layer of complexity?
5) Just a minor error on this sentence:
<pair id="4015.N" entailment="YES">
<t>And , says Mr. Dinkins , he did n't know the man his
campaign paid for a get-out-the-vote effort had been convicted of
kidnapping .</t>
<h>The campaign paid for somebody.</h>
</pair>
The correct entailment here is "The campaign paid somebody" -- I
believe the bracketing is
[the man [his campaign paid [<t_man>] [for a get-out-the-vote effort]]]
As it happens, I think the entailment as given is still semantically
OK but not syntactically.
6) Substituting a pronoun for a clausal complement, creating an ungrammatical H.
<pair id="4026.N" entailment="YES">
<t>After the first set of meetings two months ago ,
some U.S. officials complained that Japan had n't come up with
specific changes it was prepared to make .</t>
<h>The officials complained something.</h>
</pair>
H is fully ungrammatical here since you can't substitute "something"
for the clausal complement of "complain". Could be a problem if your
system checked the grammaticality of H.
Also, it could be a bit difficult to rule the above example in but this one out:
<pair id="4022.N" entailment="NO">
<t>After the first set of meetings two months ago ,
some U.S. officials complained that Japan had n't come up with
specific changes it was prepared to make .</t>
<h>That was complained.</h>
</pair>
Since "that" could be a pronoun as well as a complementizer, it seems
like this is potentially equally OK - unless you simply have a fixed
list of which pronouns are allowed as substitutions.
In contrast, here's an example of an OK substitution of "something"
(though still probably difficult to get right, since you need to know
that "something" is acceptable as the complement of "say").
<pair id="6095" entailment="YES">
<t>`` Furiouser and furiouser '' , I said .</t>
<h>I said something.</h>
</pair>
7) Substituting a pronoun for a noun phrase, creating an unnatural H.
<pair id="4116" entailment="YES">
<t>The U.S. wants the removal of what it perceives as
barriers to investment ; Japan denies there are real barriers .</t>
<h>Somebody denies there are something.</h>
</pair>
I find the substitution of "something" for the plural noun in this
construction pretty unnatural.
8) What role does argument structure play in the judgements? For example:
<pair id="1048" entailment="YES">
<t>Index-arbitrage trading is `` something we want to
watch closely , '' an official at London 's Stock Exchange said .</t>
<h>We want to watch.</h>
</pair>
Here "watch" permits a null understood object (as opposed to if the
verb had been e.g. "examine"). However compare this example:
<pair id="5038" entailment="NO">
<t>`` That 'll get you started .</t>
<h>You started something.</h>
</pair>
One way of ruling it out is that "start" has an object in H but none
in T. (Also the tensing of the verb "start", but this is subtle, and
as there are many positive examples in the corpus which include a
change at least of voice on the verb, it would be much simpler to rule
this out based on the argument structure.)
Again, here is a positive example:
<pair id="5087" entailment="YES">
<t>If he was sober , which was doubtful , he 'd have
him get in touch with Mr. Crombie .</t>
<h>Somebody would have him get in touch.</h>
</pair>
This is only grammatical since "with Mr. Crombie" is omissible in the
"get in touch" construction. Again, it seems to me to be related to
the question of whether we can assume positive examples of entailment
are grammatical. Can we assume that if the verb in H has a reduced
argument structure compared to T it's always OK?
Best,
Laura
--
Laura Rimell
Research Assistant
University of Cambridge Computer Laboratory
Deniz Yuret
Thank you for your careful analysis. Below, you will find my detailed
responses. To summarize, the general idea is: correct entailments
should always be grammatical, incorrect entailments (since they are
based on bad parser decisions) sometimes may not be grammatical. I
will release another version of the dataset this week which normalizes
spacing and punctuation. A final version will be released within
another week that fixes grammatical errors.
best,
deniz
On Fri, Mar 5, 2010 at 5:05 PM, Laura Rimell <laura....@cl.cam.ac.uk> wrote:
>
> Hi Deniz,
>
> Here are the issues that occurred to us as we were looking at the dev data. It is a very interesting task (both the data collection and the evaluation). I'm not sure any of these would be possible or desirable to change for the test data, but perhaps they may be helpful as points for discussion or clarification.
>
>
> 1) We wondered about the addition of determiners in H where the
> corresponding noun in T is determinerless. This has a variety of
> effects:
>
> <pair id="2082.N" entailment="YES">
> <t>Trading in AMR shares was suspended shortly after 3 p.m. EDT Friday and did n't resume .</t>
> <h>The trading didn't resume.</h>
> </pair>
>
> Here, "Trading didn't resume" would have been a grammatical sentence.
> The use of the determiner thus seems to add a needless complication
> when checking for the entailment. One small complexity it introduces
> in this particular example is changing upper to lower case in
> "trading". But more broadly, the system needs to have some kind of
> rule for ignoring the determiner in H even though this makes H deviate
> syntactically from T.
If these sentences appeared one after the other, wouldn't the <h> need
the definite noun phrase to refer back to the specific type of
training introduced in <t>?
>
> <pair id="5032" entailment="YES">
> <t>Thus committed , action might follow .</t>
> <h>The action might follow.</h>
> </pair>
>
> This case is slightly more problematic. I'm not sure H is strictly an
> entailment of T (semantically as well as syntactically), because of
> the definite determiner - there's no single, definite action which the
> speaker of T necessarily had in mind. So again, you need to have a
> rule about ignoring the contribution of determiners in H. (And again,
> "Action might follow" would have been grammatical.)
This time I agree, we need to lose the determiner in <h>.
>
> <pair id="2007.N" entailment="YES">
> <t>And many in the young cast bear striking resemblances to American TV and movie personalities known for light roles .</t>
> <h>Somebody bears the resemblances.</h>
> </pair>
>
> Even though this one is marked as a positive example of entailment,
> the addition of the determiner has made H fully ungrammatical. I'm
> unsure of the role of ungrammatical H's as negative examples (see next
> comment), but it does strike me that positive examples of entailment
> should always be grammatical.
I fully agree -- positive entailment examples should always be
grammatical. Is "Somebody bears resemblances." a grammatical sentence
by itself? Or do you think we need to add "... to somebody else." at
the end to complete it?
>
> Here's another example where the substitution of the determiner
> (compounded by the deletion of the PP) creates a quite unnatural, if
> not perhaps fully ungrammatical, H:
>
> <pair id="5062" entailment="YES">
> <t>He turned his attention to the workbench .</t>
> <h>The attention was turned.</h>
> </pair>
>
> If part of your system for whatever reason used a language model to
> check the probability of H (not saying that's necessarily a good
> strategy), you'd have a problem with these examples.
The problem here seems to be the idiomatic nature of "turn attention".
For example if we had "He sold his watch" instead, "The watch was
sold" would be perfectly OK. "Somebody turned his attention"
expresses the connection between "turn" and "attention", but as with
the previous example I am not sure this sentence is complete without a
"to ..." phrase to complete it.
>
> I'm pretty ignorant about RTE tasks in general, though, so maybe the treatment of determiners in this way is a given in RTE and I'm not aware of it.
No, it is just that we Turks do not have determiners in our language
and always mess up their placement in English :)
>
> 2) We were wondering about the role of ungrammatical H's.
>
> <pair id="6098" entailment="NO">
> <t>Moreland sat brooding for a full minute , during which I made each of us a new drink .</t>
> <h>Somebody sat brooding I made each.</h>
> </pair>
>
> Given that a certain percentage of the negative examples are
> ungrammatical, you could probably write a really basic script that
> gave a negative judgement if you couldn't parse H. (Though this won't
> work if there are ungrammatical positive examples as well, and of
> course you can often get an implausible parse of an ungrammatical
> sentence.) What do you see as the role of the ungrammatical examples
> in parser evaluation? Perhaps they are actually important for teasing
> out some of the common parser attachment errors.
You are exactly right. We did not make up these bad entailments.
They were actually created from bad parses given by state of the art
parsers. When we saw two different parses of a sentence by different
parsers, we tried to express both parser decisions using entailments.
This is usually difficult to do grammatically when the given parse is
nonsensical. I will reanalyze the bad parse this entailment is based
on and see if there is a more natural way to express it.
>
>
> 3) We were also wondering about the role of transformations that
> change the grammatical category of one of the crucial words.
>
> <pair id="1032.N" entailment="NO">
> <t>Mr. Hahn attributes the gains to the philosophy of concentrating on what a company knows best .</t>
> <h>Somebody philosophizes what a company knows best.</h>
> </pair>
>
> I don't think the noun-to-verb transformation will cause much of a
> problem in this example because it's a negative example for other
> reasons (the object of "philosophize" is incorrect). However, if H
> had been e.g. "Somebody philosophizes about concentrating on what a
> company knows best" - would that be an entailment or not? It seems to
> me this may involve a judgment about meaning preservation under the
> philosophy-philosophize transformation rather than a strictly
> syntactic entailment. (Unless you intend any H in which one of the
> lexical items changes its major grammatical category to be a negative
> example.)
This is a mistake on our part and it shouldn't happen. As you
mentioned noun-to-verb transformations are difficult to judge and this
is certainly not a purely syntactic matter. We will check the data
again to make sure at least the content words are used as they appear
in the original sentence.
>
>
> 4) We were wondering about the benefits of using the passive
> transformation. Is it preferable to use a passive H derived from an active T, rather than extracting a simple active H - e.g. "He saw the stream" or "Someone saw the stream" rather than "The stream had been seen" in this example?
>
> <pair id="2004" entailment="YES">
> <t>He could also hear the stream which he had seen from his position .</t>
> <h>The stream had been seen.</h>
> </pair>
>
> Of course, a system that recognizes entailments should be able to
> recognize passive ones, but from the point of view of parser
> evaluation, does it add an unnecessary layer of complexity?
We had to make a decision on how to test verb-object dependence. One
option is to use passive: "object was verb-ed", the other is to insert
a dummy subject "someone verb-ed object". The first one appealed to
me because I wanted to minimize the number of dummy words we
introduced to make grammatical entailments. The active-passive
transformation is purely syntactic (unlike the noun-to-verb
transformation which also requires semantic judgement). I think
systems should be able to handle both types of entailment without too
much difficulty.
>
>
> 5) Just a minor error on this sentence:
>
> <pair id="4015.N" entailment="YES">
> <t>And , says Mr. Dinkins , he did n't know the man his campaign paid for a get-out-the-vote effort had been convicted of kidnapping .</t>
> <h>The campaign paid for somebody.</h>
> </pair>
>
> The correct entailment here is "The campaign paid somebody" -- I believe the bracketing is
>
> [the man [his campaign paid [<t_man>] [for a get-out-the-vote effort]]]
>
> As it happens, I think the entailment as given is still semantically
> OK but not syntactically.
>
You are right, it should be "The campaign paid somebody."
>
> 6) Substituting a pronoun for a clausal complement, creating an ungrammatical H.
>
> <pair id="4026.N" entailment="YES">
> <t>After the first set of meetings two months ago , some U.S. officials complained that Japan had n't come up with specific changes it was prepared to make .</t>
> <h>The officials complained something.</h>
> </pair>
>
> H is fully ungrammatical here since you can't substitute "something"
> for the clausal complement of "complain". Could be a problem if your
> system checked the grammaticality of H.
In this example I think "The officials complained." would be
acceptable as the clausal complement is not mandatory. Can you think
of cases when it is mandatory, and do you see a way to express it
without introducing too many dummy words?
>
> Also, it could be a bit difficult to rule the above example in but this one out:
>
> <pair id="4022.N" entailment="NO">
> <t>After the first set of meetings two months ago , some U.S. officials complained that Japan had n't come up with specific changes it was prepared to make .</t>
> <h>That was complained.</h>
> </pair>
>
> Since "that" could be a pronoun as well as a complementizer, it seems
> like this is potentially equally OK - unless you simply have a fixed
> list of which pronouns are allowed as substitutions.
I do not consider "that" a content word, so this entailment also
violates the "two content words per entailment" principle. I will
look at the parse again.
>
> In contrast, here's an example of an OK substitution of "something"
> (though still probably difficult to get right, since you need to know
> that "something" is acceptable as the complement of "say").
>
> <pair id="6095" entailment="YES">
> <t>`` Furiouser and furiouser '' , I said .</t>
> <h>I said something.</h>
> </pair>
The only purpose of adding words like "somebody" and "something" is to
make the entailment grammatical while testing the dependence of two
content words. In this particular case we are testing to see if "I"
is the subject of "said" and since "I said" is not a grammatical
sentence, we add "something" to make it complete. However you are
right that judging whether "somebody" or "something" is an acceptable
complement is difficult. Thus my recommendation is to ignore that
problem, and think of "somebody" and "something" as generic noun
phrases. Concentrate on the relations in the rest of the entailment.
>
>
> 7) Substituting a pronoun for a noun phrase, creating an unnatural H.
>
> <pair id="4116" entailment="YES">
> <t>The U.S. wants the removal of what it perceives as barriers to investment ; Japan denies there are real barriers .</t>
> <h>Somebody denies there are something.</h>
> </pair>
>
> I find the substitution of "something" for the plural noun in this
> construction pretty unnatural.
>
Again, this example violates my "two content words per entailment"
principle, it should be "Somebody denies there are barriers." or at
least "Somebody denies there is something".
>
> 8) What role does argument structure play in the judgements? For example:
>
> <pair id="1048" entailment="YES">
> <t>Index-arbitrage trading is `` something we want to watch closely , '' an official at London 's Stock Exchange said .</t>
> <h>We want to watch.</h>
> </pair>
>
> Here "watch" permits a null understood object (as opposed to if the verb had been e.g. "examine").
Here the entailment should be "We want to watch something."
> However compare this example:
>
> <pair id="5038" entailment="NO">
> <t>`` That 'll get you started .</t>
> <h>You started something.</h>
> </pair>
>
> One way of ruling it out is that "start" has an object in H but none
> in T. (Also the tensing of the verb "start", but this is subtle, and as there are many positive examples in the corpus which include a change at least of voice on the verb, it would be much simpler to rule this out based on the argument structure.)
>
In this example "you" is the object of start, not the subject. The
"NO" answer should be based on the relation between these two words,
not only whether one of the words is missing a particular argument.
> Again, here is a positive example:
>
> <pair id="5087" entailment="YES">
> <t>If he was sober , which was doubtful , he 'd have him get in touch with Mr. Crombie .</t>
> <h>Somebody would have him get in touch.</h>
> </pair>
>
> This is only grammatical since "with Mr. Crombie" is omissible in the
> "get in touch" construction. Again, it seems to me to be related to
> the question of whether we can assume positive examples of entailment
> are grammatical. Can we assume that if the verb in H has a reduced
> argument structure compared to T it's always OK?
A reduced argument structure (if grammatical) is OK only if the rest
of the arguments for the verb have their correct roles in <h>.
>
>
> Best,
>
> Laura
>
> --
> Laura Rimell
> Research Assistant
> University of Cambridge Computer Laboratory
>
>
Thank you,
deniz
Laura Rimell
Thanks for the explanations. I'm looking forward to seeing the new
version of the data. A few brief comments:
> > 1) We wondered about the addition of determiners in H where the
> > corresponding noun in T is determinerless. This has a variety of
> > effects:
>
> > <pair id="2082.N" entailment="YES">
> > <t>Trading in AMR shares was suspended shortly after 3 p.m. EDT Friday and did n't resume .</t>
> > <h>The trading didn't resume.</h>
> > </pair>
>
> > Here, "Trading didn't resume" would have been a grammatical sentence.
> > The use of the determiner thus seems to add a needless complication
> > when checking for the entailment. One small complexity it introduces
> > in this particular example is changing upper to lower case in
> > "trading". But more broadly, the system needs to have some kind of
> > rule for ignoring the determiner in H even though this makes H deviate
> > syntactically from T.
>
> If these sentences appeared one after the other, wouldn't the <h> need
> the definite noun phrase to refer back to the specific type of
> training introduced in <t>?
Is the ability of the sentences to appear sequentially a criterion for
a good entailment? Even if so, if the original text discussed
multiple types of trading, then simply adding the definite determiner
would not be enough to single out one of them.
> > <pair id="2007.N" entailment="YES">
> > <t>And many in the young cast bear striking resemblances to American TV and movie personalities known for light roles .</t>
> > <h>Somebody bears the resemblances.</h>
> > </pair>
>
> > Even though this one is marked as a positive example of entailment,
> > the addition of the determiner has made H fully ungrammatical. I'm
> > unsure of the role of ungrammatical H's as negative examples (see next
> > comment), but it does strike me that positive examples of entailment
> > should always be grammatical.
>
> I fully agree -- positive entailment examples should always be
> grammatical. Is "Somebody bears resemblances." a grammatical sentence
> by itself? Or do you think we need to add "... to somebody else." at
> the end to complete it?
In this case there's a number agreement problem with plural
"resemblances" - you need something like "Some people bear
resemblances to some people" or "Some bear resemblances to some". I
also think that it needs the to-phrase.
On the issue of grammaticality judgements in general, aside from the
problem which Stephan mentioned about using a parsing system which may
be participating in the task, I'm not sure how far you'll get trying
to do this automatically. Many broad-coverage parsers won't make
these judgements since they assume grammatical input and will often
give an (implausible) analysis for ungrammatical sentences, and
systems for automatic grammaticality judgements are still far from
perfect. I think you might be better off relying on human native
speaker judgements in this instance.
> > 6) Substituting a pronoun for a clausal complement, creating an ungrammatical H.
>
> > <pair id="4026.N" entailment="YES">
> > <t>After the first set of meetings two months ago , some U.S. officials complained that Japan had n't come up with specific changes it was prepared to make .</t>
> > <h>The officials complained something.</h>
> > </pair>
>
> > H is fully ungrammatical here since you can't substitute "something"
> > for the clausal complement of "complain". Could be a problem if your
> > system checked the grammaticality of H.
>
> In this example I think "The officials complained." would be
> acceptable as the clausal complement is not mandatory. Can you think
> of cases when it is mandatory, and do you see a way to express it
> without introducing too many dummy words?
It's tough to see how to do this - it sounds like you'd need a subcat
lexicon indicating which verbs can take clausal/NP/PP/null
complements, in order to make the decision for each sentence. For
example "complain" is OK with a null complement, "convince" can have a
null clausal complement as long as it has a direct object, "say" can
take an NP complement (hence "something"), but perhaps "hope" is an
example that has to take a PP complement ("hoped that x" => "hoped for
something"). I suppose this falls under the same general problem area
as identifying idioms, selected prepositions, etc. ("bear a
resemblance to x") in T in order to generate a grammatical H - a very
interesting problem.
Best,
Laura