Measures of Performance and Effectiveness (ATTN: The Marine References: <9eelm2$99m$5@slb2.atl.mindspring.net> <9ef0qo$2v19$1@news.thFollowup-To:
"John ...@oemcomputer.foo.bar
In article <9ef0qo$2v19$1...@news.tht.net>, maenad wrote:
>"John wrote:
...
>> I understand this, but I can definitely see the practical
>> management problem, here. The taxpayers are paying for
>> their kids to be taught.
>
><nitpick> The taxpayers are paying for THE kids to be taught. </nitpick>
Sorry, quite right. The taxpayers and voters want to know
that their money is being spent on actually teaching
children, rather than, say, building shiny new buildings or
hiring more administrative staff as a form of political
patronage. Standardized testing of kids is one way to
measure performance. Accreditation boards, teacher
licensing requirements, and teacher testing are all ways to
try to ensure that at least the raw materials are there for
the kids to be taught. I think it's much better to measure
actual performance, since otherwise, you're measuring what
you *think* will lead to good results, rather than measuring
some of those results. Also, some licensing requirements
may not be for the benefit of the students, but instead for
the benefit of the state university's school of education,
or the existing teachers, or for some other group with an
interest in deciding which people get to be teachers.
I'd make one nitpick, though: Any teacher teaching an
advanced subject in high school ought to have to pass
(hopefully pretty easily) the standardized test for that
field *before he starts teaching the class*. This would
both keep teachers from claiming abilities they don't have,
and would also (more importantly, I think) keep school
administrators from filling a gap with a teacher who says
"but I don't *know* anything about algebra," but ends up
teaching algebra anyway.
>> How do they know whether they're
>> getting a good deal? How do they know whether what they're
>> paying for is worthwhile? How do they even know whether
>> their teacher is trying, or is just serving time until he
>> retires?
>Interestingly enough, the generic taxpayer will have a hard
>time judging this, because people look at them funny if they
>volunteer to hang around schools.
So, this seems like a pretty good reason to want to see
performance measured in schools. If kids are passing your
algebra class, but cannot, in fact, solve any algebra
problems when they're done, this ought to be dealt with. If
it's not dealt with by the school administration, then it
ought to be dealt with by putting in a new administration.
It mustn't be dealt with by ignoring the problem, in hopes
that maybe none of those kids needed to learn algebra
anyway.
I understand that this can lead to teaching to the test, and
there's a certain amount of inevitability to that. It's
probably not such a bad thing in math classes, where it's
pretty easy to define what sorts of things you ought to be
able to do at the end of a basic algebra, geometry, or
calculus class. It's more of a problem for a history class,
where a teacher that cares about nothing but her
standardized test scores at the end of the year will
probably force-feed her kids dates and names to be
memorized. But what's the alternative? There needs to be
someone outside the administrative loop who makes an
objective measurement of how the teacher is doing, and who
has the power to *do* something about it, or there will be
no feedback to tell teachers and administrators whether
they're spending our money, and their time, and our kids
time, in worthwhile ways.
>Those taxpayers who are also parents, however, can do all
>of these things, by the simple method of GETTING INVOLVED.
>Go to the PTA meetings, call the kid's teachers, or
>volunteer in the classroom. Ask the kids what they think.
>Help them with their homework, see what you think of it. The
>answers to all of your questions above will be revealed.
Well, knowledge without the power do apply it isn't worth
much. I intend (assuming various things go as I plan) to
send our children to Catholic schools. But I also plan to
be involved, and if I don't like what I see going on there,
I'll try to find someplace better to send them. I think
giving some version of this power to public school parents
would be a *huge* win, regardless of whether there's
standardized testing. Whether that's done with an education
voucher ($N for educating your kid in any accredited
school, for some reasonable value of N), or by allowing
parents to transfer their kids to any public school in the
area that will take them, seems less important than making
sure that involved parents can change their kids' schools,
in general, without having to cough up several thousand
dollars of their own to do so, or spend untold hours in
paperwork and red tape. There are various details that have
to be gotten right to make such a system work, naturally.
However, there's also a role for standardized testing.
I'd like to think that I will be able to judge whether my
children are getting a decent education, but it's certainly
possible for a school to spend a lot of resources impressing
parents instead of teaching the kids. Standardized tests
have to be used intelligently (e.g., if you automatically
fire the teachers whenever these scores are too low, you'll
cause far more problems than you cure), but they can make it
clear when there are problems to be addressed. And it's not
all that hard to analyze test scores based on enough data
that you don't penalize teachers for teaching poor kids, or
kids with behavior problems, or whatever.
>Now, I know parents are busy. I know they generally don't
>have time to do all of the above. But if they don't have
>enough time to do enough of the above to satisfy themselves
>as to the quality of their kids' teachers, I have to wonder
>what they were thinking when they decided to have kids.
Hmmm. I more-or-less agree with this, but it is important
to remember that unexpected things happen: Death of a
spouse, divorce, illness, financial emergencies, etc., can
easily leave a parent with no spare time or energy for
spending time at the school, trying to decide whether their
kids are learning anything.
One one hand, I think that if we're going to have
government-run schools, it ought not to be a total mystery
whether they're going to actually teach the kids sent to
them, or just stand guard over them for eight hours a day
before sending them home. On the other hand, I suspect that
if you count on the government to make sure your kids are
getting an education, you're probably going to be
disappointed with the outcome.
>> I suspect that simple testing can't distinguish an adequate
>> teacher from a great one, but it seems likely that it can
>> help distinguish an adequate teacher from an incompetent
>> one.
><sarcasm>
>So, fire the incompetent ones, while at the same time
>lowering the morale of the adequate ones, and generally
>failing to reward the excellent ones.
>Test again, repeat as necessary, and bemoan the lack of
>good educators in the country.
></sarcasm>
>How exactly are we improving the system with this method?
How are we improving the system if there's no difference in
a teacher's career, whether he does a stellar job teaching
his students, a mediocre job, or a terrible job? How do you
think we should deal with schools where half the graduates
can't read at an eighth grade level? Or where kids who
passed algebra can't solve simple algebra problems? Or
where kids who passed english can't write a complete
sentence?
It's pretty clear that the current system produces some
pretty decent outcomes, and some really, really awful ones.
It doesn't seem to me that this is inevitable, nor that it's
inevitable that schools in poor neighborhoods will often not
do much teaching, but schools in middle-class suburbs
usually will.
>maenad, cynical
--John Kelsey
k.e.l.s.e.y.(dot).j.(at).i.x.(dot).n.e.t.c.o.m.(dot).c.o.m
PGP: 5D91 6F57 2646 83F9 6D7F 9C87 886D 88AF
``So long ago, when we were taught, that for whatever
kind of problem you've got, you just put the right formula
in, a solution for every fool....''
--Indigo Girls, ``Least Complicated''
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.1 Int.
iQCVAwUBOxHzTEHx57Ag8goBAQHTzgP/TclNrvxOnxsNJLKRrYtm824hoGM2pJdf
gEad5HPEoA41HLCa6U0ba8AZ70NiTePLrWvCOLfTX835XWb64Om2hwddLdVuPWYt
vjy/kMrVUbBCFW9lvgCgGoXul1tjR8eTHkI2hMlH45na9jEGLrAbYLJXYxXops0S
EnYQ2qNhAYM=
=yIcl
-----END PGP SIGNATURE-----
"John ...@oemcomputer.foo.bar
In article <Pine.GSU.4.05.101052...@psyche.the-wire.com>,
Austin Ziegler wrote:
>On Fri, 25 May 2001, Kris Overstreet wrote:
...
>> Still, far, far, FAR too many teachers are opposed even to
>> -standardized testing of students- as a metric.
><<I can think of a number of reasons why this would be a crappy metric.
>First, all standardised tests are centered around certain sets of
>assumptions about where children should be at a given point. In
>particular, the SAT has been pointed as having a bit of a white
>middle-class bias in previous years (this may not be true anymore).
What we want from public education is, IMO, centered around
certain assumptions about where children should be at a
certain point. Kids who have passed the eighth grade ought
to be, for the most part, functioning on an eighth grade
level academically. They ought to be able to read, write,
spell, and do math at a certain level. There will be
variations in each class, and variations among individuals,
but that's what we expect. Kids who graduate high school
ought to have a whole set of skills that twelve years of
mandatory government-provided education has given them. If
very many of them don't, that ought to be a big, red warning
flag.
><<Second, performance on a standardised test can vary significantly
>based on how one feels on a given day. I'm told of a star student who,
>because she forgot to go to the bathroom during lunch, did very poorly
>on a test that other independent sources had shown she was very good
>at. Why? Because she couldn't concentrate on the test.
So, when you're using standardized test scores to evaluate a
teacher's performance, you're looking at the distribution of
scores. In fact, if you're halfway serious about it, you're
looking at an expected set of scores based on previous
years' scores, grades, household income, IQ, etc., and then
you're evaluating the teacher on how they've done relative
to what could have been expected. So occasional cases of
kids who have a cold that day, or who forgot to eat
breakfast, or whatever, shouldn't have that big an impact.
And if you're really serious about it, you'll also test more
than once, and you'll make sure the people supervising the
test aren't directly employed by the school being evaluated.
Testing to decide whether someone graduates is a little more
complicated, though it's clearly possible to allow make-up
tests or retests for kids who have a bad day. I'm a little
less comfortable with testing to decide whether you can
graduate high school, say, because there really are people
who have the needed skills, but who may have a hard time
passing a complicated test. But there are also people who
graduate high school without being able to read very well,
and that needs to stop.
><<Third, standardised testing is only one way -- and a poor
>one, IMO -- to measure a small subset of skills that are
>necessary for a person in life. Just as folks have different
>ways of learning, folks also have different ways of
>expressing that learning. There are smart people I know who
>can't effectively take tests at all, but can express their
>knowledge in other fora without problems.
Okay, but the problem is that we (as voters and taxpayers
and parents) need to determine whether the schools are doing
their jobs. Because if some teacher, or some school, isn't
doing its job, it's imposing a huge cost on all of us, in
terms of kids it's graduating who have to take lots of
remedial classes in high school, or who show up to work
barely able to read. So while I understand that
standardized testing isn't some magical potion you pour over
a school to make it better, it's hard for me to see that
there's no place for it at all.
How should we, given limited money and time, figure out
whether a given high school is making good use of the money
we give them, and good use of their students' time? How
should we find out when some teacher isn't competent, or
isn't bothering to teach anything?
><<Fourth, you end up teaching to the test. I'll address
>this further below, but this is *clearly* a bad thing
>because it encourages solely the regurgitation of facts and
>not independent thinking.>>
Hmmm. So, I agree, and I think it would also be worthwhile
to send independent teams around to audit abilities that
simple standardized tests can't measure. But what I want
from these tests is to catch major problems when they're
happening, and to react to them. If you passed your basic
trig class, but you can't define what a sine function is in
terms of the unit circle, or you can't work out how to
determine the height of a tree from the length of its shadow
and the angle of the sun, then this ought to raise some
questions about whether you really learned the material.
And if most of your classmates also can't do these things,
then it probably means that the teacher didn't get much done
this year. That means that we're paying someone to do a job
they didn't do, and that needs to be fixed, both to avoid
wasting money, and to avoid wasting opportunities to teach
kids stuff they need to know.
...
><<How do you measure the knowledge and ability -- especially the
>ability? That's what teachers certifications are for, Redneck. In some
>jurisdictions, noncertified teachers can teach in certain situations
>but have to be working toward a certification.
I'm curious how much the certification requirements have to
do with actual quality of teaching. I've heard the opinion
(from my stepfather, who was a teacher, and later a
principal) that a lot of the certification requirements in
Missouri have more to do with keeping education classes full
than with improving the quality of education. And licensing
requirements in many fields end up being controlled by the
people already in the field, who have a really different
goal in mind than the public might. (Just look at the AMA,
which restricts the number of doctors through more-or-less
controlling how many students will be admitted to various
medical schools.)
><<You want to make it better? Make the wages better and
>then start placing serious review requirements on the
>teachers. Don't make it something like a salesperson (e.g.,
>how many happy 'customers'), but make it like a software
>developer's review (as an example). It's a very *soft* thing
>to review; people know if you're doing well or poorly, but
>it's often very hard to quantify because it's a creative
>process. Most software companies also do a 360 degree/peer
>review, and that's where input from the parents and students
>can enter the picture: validation of what's observed
>otherwise.>>
This would be really cool, if it worked. My impression is
that most teachers in a school know who the really good
teachers are, and who the really lousy ones are. But it
seems like reviews from other teachers in the same school
would end up having a huge amount of office politics
involved. There are probably a lot of details involved in
getting this right.
I think there are two additional things that would help:
a. Give the parents some kind of choice about where their
kids go, so that when they're unhappy with what's going on
at their school, they don't have to try to influence some
huge bureacracy to help their kid out. Parents won't always
know what's best, but they're far more likely to have their
kids' interests at heart than teachers or administrators.
And parents will often know or suspect that something
unpleasant is happening at the school, by the way their kids
are reacting. It should be made easy for them to act on
that knowledge in a way that helps their kid, and that also,
if done by enough parents, acts as a signal that there's a
problem with the school.
b. Find a way to allow effective feedback from colleges and
universities where the graduates of high schools end up.
There have been complaints for years from universities,
about how many high school graduates with good grades needed
to take remedial math and english classes, just to be
capable of keeping up with the minimum expectations for
college. Maybe a student who passed algebra in high school,
but still has to take a remedial algebra class in college,
ought to be able to send their high school a bill for part
of the tuition. Or at least send them a nasty-gram, that
gets heard by the state board of education or whomever is
overseeing them.
>-f
--John Kelsey
k.e.l.s.e.y.(dot).j.(at).i.x.(dot).n.e.t.c.o.m.(dot).c.o.m
PGP: 5D91 6F57 2646 83F9 6D7F 9C87 886D 88AF
``So long ago, when we were taught, that for whatever
kind of problem you've got, you just put the right formula
in, a solution for every fool....''
--Indigo Girls, ``Least Complicated''
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.1 Int.
iQCVAwUBOxH/3EHx57Ag8goBAQEGmQQAiJCrsXGSoYw1jDS6909GlButmjxghokT
s1iKV6CxL/Fp0Z2oNZSqnm65L/GrQbFnnEFHAKpdBLkjxriFxdh+1tkf8MwQBn5Z
QVf8VNgFpsNJTYemNQLpVjwLC/hkKyuJzy8ORDKhWqazfAzGfQd/aYIqNcJUjYbC
rkcSA4PNX44=
=QIJA
-----END PGP SIGNATURE-----
Austin Ziegler
> I wrote:
> Austin Ziegler wrote:
>> On Fri, 25 May 2001, Kris Overstreet wrote:
>
> ...
>>> Still, far, far, FAR too many teachers are opposed even to
>>> -standardized testing of students- as a metric.
>> <<I can think of a number of reasons why this would be a crappy metric.
>> First, all standardised tests are centered around certain sets of
>> assumptions about where children should be at a given point. In
>> particular, the SAT has been pointed as having a bit of a white
>> middle-class bias in previous years (this may not be true anymore).
> What we want from public education is, IMO, centered around
> certain assumptions about where children should be at a
> certain point.
<<To a very limited point, I agree with what you just said. There are a
number of problems with that, though. The logical conclusion of your
statement leads to the English school system, which isn't exactly
great. (As I said in another post, "if it's the first Tuesday in
November, it must be triangles.") This puts a strain on external
resources, allows little or no variation for levels of students.
<<What defines a completing eigth grader should know changes over time;
by using standardized tests, you essentially put that definition in the
hands of either (a) bureaucrats, (b) politicians, or (c) private test
making corporations. I consider (c) the most likely result, and the
least acceptable. (If the education is government mandated, the tests
should be government written.)
middle/upper-class students rather than anyone else.
<<Standardized tests have a number of problems besides that. They don't
differentiate for the fact that different jurisdictions have different
curricula. This would logically lead to national control of education
(which could be good, but more likely very bad). I've previously
brought up the bias of most standardized tests toward white
middle/upper-class students rather than anyone else.
<<Finally, most standardized tests measure particular items well, but
they are misused by nearly everone involved with them, especially
politicians. (Parents often don't understand the meaning of these
tests, either.) They're not really the panacea that people seem to
think they are -- and if they're not that good for students, why should
student results be good for measuring teacher performance?>>
>> <<Second, performance on a standardised test can vary significantly
>> based on how one feels on a given day. I'm told of a star student who,
>> because she forgot to go to the bathroom during lunch, did very poorly
>> on a test that other independent sources had shown she was very good
>> at. Why? Because she couldn't concentrate on the test.
> So, when you're using standardized test scores to evaluate a
> teacher's performance, you're looking at the distribution of
> scores. In fact, if you're halfway serious about it, you're
> looking at an expected set of scores based on previous
> years' scores, grades, household income, IQ, etc., and then
> you're evaluating the teacher on how they've done relative
> to what could have been expected. So occasional cases of
> kids who have a cold that day, or who forgot to eat
> breakfast, or whatever, shouldn't have that big an impact.
> And if you're really serious about it, you'll also test more
> than once, and you'll make sure the people supervising the
> test aren't directly employed by the school being evaluated.
<<You've just eaten up a lot of instructional days with what you're
talking about (Ontario is giving its standardized tests these days, and
it's between five and fifteen days, apparently). You're also being
pretty invasive into students' privacy without good reason. You've also
just added significant expense by requiring non-employed supervisors
(who also have to know how to handle kids).
<<I think that you've just added more complexity than you want.>>
> Testing to decide whether someone graduates is a little more
> complicated, though it's clearly possible to allow make-up
> tests or retests for kids who have a bad day. I'm a little
> less comfortable with testing to decide whether you can
> graduate high school, say, because there really are people
> who have the needed skills, but who may have a hard time
> passing a complicated test. But there are also people who
> graduate high school without being able to read very well,
> and that needs to stop.
<<I'll agree with your last statement unreservedly. There's a concept
in schools of an IEP (independent education plan) for students that are
in need of help. Frankly, we should make it so that teachers are paid
well *and* that they have a small-enough class size such that all
students have IEPs to better involve the parents, teachers, and the
student hirself in a *meaningful* learning contract. Ultimately, I
think that we have to get away from the factory-model of education.>>
>> <<Third, standardised testing is only one way -- and a poor
>> one, IMO -- to measure a small subset of skills that are
>> necessary for a person in life. Just as folks have different
>> ways of learning, folks also have different ways of
>> expressing that learning. There are smart people I know who
>> can't effectively take tests at all, but can express their
>> knowledge in other fora without problems.
> Okay, but the problem is that we (as voters and taxpayers
> and parents) need to determine whether the schools are doing
> their jobs. Because if some teacher, or some school, isn't
> doing its job, it's imposing a huge cost on all of us, in
> terms of kids it's graduating who have to take lots of
> remedial classes in high school, or who show up to work
> barely able to read. So while I understand that
> standardized testing isn't some magical potion you pour over
> a school to make it better, it's hard for me to see that
> there's no place for it at all.
<<I haven't said that standardized tests have no place in schools. I
have said that they (a) don't have a place in determining whether
someone graduates or advances, and (b) don't have a place in judging
teacher effectiveness. They *do* give a sense of where the student lies
in a certain set of skills, but they don't give the whole picture. IMO,
they don't even give enough of the picture to measure (a) and (b). (As
a note, from sixth grade on, I tested at a reading level of "beyond
high school" or "college.")>>
> How should we, given limited money and time, figure out
> whether a given high school is making good use of the money
> we give them, and good use of their students' time? How
> should we find out when some teacher isn't competent, or
> isn't bothering to teach anything?
<<I've given several suggestions that, in fact, match most of what is
done by private industry where there aren't clear things to measure
like "number of widgets sold.">>
>> <<Fourth, you end up teaching to the test. I'll address
>> this further below, but this is *clearly* a bad thing
>> because it encourages solely the regurgitation of facts and
>> not independent thinking.>>
> Hmmm. So, I agree, and I think it would also be worthwhile
> to send independent teams around to audit abilities that
> simple standardized tests can't measure. But what I want
> from these tests is to catch major problems when they're
> happening, and to react to them. If you passed your basic
> trig class, but you can't define what a sine function is in
> terms of the unit circle, or you can't work out how to
> determine the height of a tree from the length of its shadow
> and the angle of the sun, then this ought to raise some
> questions about whether you really learned the material.
> And if most of your classmates also can't do these things,
> then it probably means that the teacher didn't get much done
> this year. That means that we're paying someone to do a job
> they didn't do, and that needs to be fixed, both to avoid
> wasting money, and to avoid wasting opportunities to teach
> kids stuff they need to know.
<<Agreed to the point that I don't think everyone should be able to
define such functions, but they should know how to use them and
generally what they mean. That said, I think that most parents and
principles know when a teacher is being ineffective, but are blocked
from taking action of any sort for a whole host of reasons, mostly union
contracts[1].>>
>> <<How do you measure the knowledge and ability -- especially the
>> ability? That's what teachers certifications are for, Redneck. In some
>> jurisdictions, noncertified teachers can teach in certain situations
>> but have to be working toward a certification.
> I'm curious how much the certification requirements have to
> do with actual quality of teaching. I've heard the opinion
> (from my stepfather, who was a teacher, and later a
> principal) that a lot of the certification requirements in
> Missouri have more to do with keeping education classes full
> than with improving the quality of education.
<<To a degree, your stepfather is right. One of the key points of most
certification programs is in-classroom teaching experience under
supervision of an existing teacher.>>
>> <<You want to make it better? Make the wages better and
>> then start placing serious review requirements on the
>> teachers. Don't make it something like a salesperson (e.g.,
>> how many happy 'customers'), but make it like a software
>> developer's review (as an example). It's a very *soft* thing
>> to review; people know if you're doing well or poorly, but
>> it's often very hard to quantify because it's a creative
>> process. Most software companies also do a 360 degree/peer
>> review, and that's where input from the parents and students
>> can enter the picture: validation of what's observed
>> otherwise.>>
> This would be really cool, if it worked. My impression is
> that most teachers in a school know who the really good
> teachers are, and who the really lousy ones are. But it
> seems like reviews from other teachers in the same school
> would end up having a huge amount of office politics
> involved. There are probably a lot of details involved in
> getting this right.
<<Absolutely. But *business *has *already *done *this. There's no
reason that the methodology involved with a peer review cannot be
adapted to a teaching environment. Note that peer review in the case of
teachers includes input from parents and students as well as legitimate
peers -- but it is *not* a determining factor. It is a corroboration of
what is observed by the professional evaluator (often, a principle).>>
> I think there are two additional things that would help:
> a. Give the parents some kind of choice about where their
> kids go, so that when they're unhappy with what's going on
> at their school, they don't have to try to influence some
> huge bureacracy to help their kid out. Parents won't always
> know what's best, but they're far more likely to have their
> kids' interests at heart than teachers or administrators.
> And parents will often know or suspect that something
> unpleasant is happening at the school, by the way their kids
> are reacting. It should be made easy for them to act on
> that knowledge in a way that helps their kid, and that also,
> if done by enough parents, acts as a signal that there's a
> problem with the school.
<<Actually, I'll disagree that parents will have their kids' interest
at heart more often than teachers will. This varies significantly from
parent to parent and from teacher to teacher. One of the key points is
that the parent will also be concerned only for hir child, whereas a
teacher is concerned for all of the children in hir class(es), by and
large. What if one child is actually responsible for most of the
problems in the class, and the teacher can't get the parents involved
enough to make a difference?>>
> b. Find a way to allow effective feedback from colleges and
> universities where the graduates of high schools end up.
> There have been complaints for years from universities,
> about how many high school graduates with good grades needed
> to take remedial math and english classes, just to be
> capable of keeping up with the minimum expectations for
> college. Maybe a student who passed algebra in high school,
> but still has to take a remedial algebra class in college,
> ought to be able to send their high school a bill for part
> of the tuition. Or at least send them a nasty-gram, that
> gets heard by the state board of education or whomever is
> overseeing them.
<<I think that the way to do this is to send an aggregate result of the
first semester of the freshman year to the schools involved. Difficult
if only one student went to the school -- you lose privacy, which IMO
is important -- but probably a good thing overall.>>
-f
[1] I'll note that on the whole, I'm not enamoured of unions, but
recognise just what value they provide. I also know that had it not
been for the union's influence and the contract, my girlfriend
might actually have to teach a class of 38 fifth graders next year.
There *has* to be a middle ground where principles-as-managers
don't end up screwing the teachers in order to please the parents,
but teachers who are doing fuckall year after year get the boot.
--
austin ziegler * Ni bhionn an rath ach mar a mbionn an smacht
Toronto.ON.ca * (There is no Luck without Discipline)
-----------------* I speak for myself alone