Theory: summarising webpages
Jorn Barger
was so bad it inspired a lot of schadenfreude when it was first
released. It used a statistical approach based on sentence
lengths and word-frequencies, I think, so it often returned
utter nonsense, and I used to argue that using the first sentence
of each paragraph instead ought to work better...
The statistical approach to summarisation goes back to the
earliest days of digitised document-search-- 1958, when Luhn
tried to generate keyword-lists for indexing by eliminating all
but the least-common words.
I came across that reference while researching my timeline of
knowledge representation (kr)--
http://www.robotwisdom.com/ai/timeline.html --which has now
ballooned past 200Kb. Since Google stops paying attention at
101Kb, I'm trying to break this up into three subpages, but my
design-theory disapproves of such fragmentation, because it
reduces the visitor's freedom to quickly scan thru it.
To compensate, I'm trying to create a top page that summarises
the sub-fragments, that one can read thru quickly and then use
a dozen-or-so #-links for each subpage to jump close to their
desired spot, without too much disorientation from arriving in
the middle.
But just how to summarise the timeline is a puzzle.
My first thought was to winnow out a shorter timeline, by
focusing on the most interesting items. I may end up going
this route, but if the shorter version is going to be at all
representative, it will still be so long it can't be easily
scanned, and the visitor will be forced to re-read a lot of
the same content when they jump down from the summary to the
full timeline.
One problem here is that a major theme of my timeline is the
parallel evolution of _all_ domains of kr-- the various
physical sciences, social sciences, business, literature,
etc. So my second thought was to write a prose summary that
explicitly traces these parallel paths. This could add
value, but has the danger of becoming too abstract and
driving people away before they _see_ the timeline.
Another option is interleaving these two approaches-- short
prose summaries for each time period, followed by short
segments of timeline highpoints.
A third possibility is to break out the parallel domains
into separate summaries-- one for automatic translation, one
for military simulations, one for flight-sims, etc. This
has a big disadvantage in places where domains overlap--
the item must be mentioned twice, and the overall sense of
shared-evolution is hidden. (Also, the top 'summary' page
will need a lot more subpage #-links this way, or else the
jumping will be much more poorly targeted.)
One last thing: as I try composing summaries for the first
segements of timeline I notice myself 'keeping one eye on'
the last sections of the timeline, because subtleties of
word-choice and emphasis can help prepare readers and
smooth the overall effect. Which is a long way from the
MS-Word approach!
Tom Osborn
This summariser isn't too bad. The "quality" of the output
depends a lot on how well (and standardised) the input text
is. It likes journalistic writing (ie, mostly in inverted pyramid
style - see http://www.tpub.com/journalist/11.htm). [My wife's
teaching a course at the moment which includes both creative
writing and journalistic writing, and needs to keep the creatives
from frothing up their "fact" pieces, and also keep the hacks
from hyper-structuring their story-telling. Message, direct
writing is easier to summaries and gets the point across
quickly, but unless you're interested before you start, it
may be dull as dogshit].
The summariser:
http://www.lextek.com/cgi-bin/brevity.exe
What it did to Jorn's post below:
"It used a statistical approach based on sentence lengths and
word-frequencies, I think, so it often returned utter nonsense,
and I used to argue that using the first sentence of each paragraph
instead ought to work better. The statistical approach to summarisation
goes back to the earliest days of digitised document-search-- 1958,
when Luhn tried to generate keyword-lists for indexing by eliminating
all but the least-common words. I came across that reference while
researching my timeline of knowledge representation (kr)--
http://www.robotwisdom.com/ai/timeline.html --which has now
ballooned past 200Kb. My first thought was to winnow out a shorter
timeline, by focusing on the most interesting items".
Sense got across. Some details got munged towards the end...
Why was I looking at this stuff? I had to look at summarising
messages in groupware. Problem was NO SOLUTION looked
viable for writers who less good were!!! AND often the problems
in groups stem from the poor writing of lazy or poor writers
(and some of them become senior manager!!!).
Tom.
--
Dr Tom Osborn
Director AI/Modelling
NUIX Pty Ltd
Level 9 143 York St
Sydney NSW 2000
--
This message is intended for only for the named recipient. If you are not the
intended recipient you are notified that disclosing, copying, distributing or
taking any action in reliance on the content of this information is strictly
prohibited.
"Jorn Barger" <jo...@enteract.com> wrote in message
news:16e613ec.03020...@posting.google.com...
Jorn Barger
> [...] my second thought was to write a prose summary that
> explicitly traces these parallel paths.
I decided to pick one major path and try writing such a summary.
It came out a lot longer than I expected, but other than that
it seems promising. (Apologies for the choppy style, which I'll
try to smooth out eventually.) ((The content is not why I'm
posting it, remember, but rather its utility as a hypertext
summary/index, and the nature of the language-processing
required to compile it from a timeline.))
The history of knowledge-representation is probably
reflected best in the evolution of military
simulations. Fighting, defending, and also _hunting
intelligent prey_ are vital skills that go back a
billion years. Kittens when they wrestle are
unconsciously simulating more-serious grownup fights.
The memetic-evolutionary process of 'externalising'
these skills in representations may have started with
the gestural sign-language of elders, training youths
for coordinated group attacks. Gradually this
guidance was verbalised, and it became possible to
plan group-strategies in advance, and retell war
stories at night around the fire.
The shaping of weapons reflected increasing
sophistication in anticipating their effectiveness in
battle. Heroes must have been rewarded with special
body-ornaments. With the invention of written
language c3500BC, leaders quickly took to praising
their own deeds on monuments. Treaties were also
written out formally.
The first written number-systems allowed for much more
efficient inventories of the growing armies' weaponry
and provisions. The earliest theoretical treatment of
warfare was Sun Tzu's "Art of War" c500BC. The
earliest historical treatment was probably Herodotus
c440BC, and Greek playwrights explored the psychology
and ethics of fighting. (Although Aristotle tutored
Alexander the Great, his inventories of human
knowledge omitted military matters, except indirectly
in the 'Politics' and 'Ethics'.)
Weaponry continued to evolve, with the help of spoken
and written language, but it wasn't until 1537 that
mathematics became critical, with Tartaglia's
invention of the gunner's quadrant, and his
publication of the first 'firing tables' for different
weapons. Galileo then worked out the mathematics of
parabolic ballistics in 1609.
The industrial revolution c1750 introduced increasing
mathematical precision into the construction of
cannons and other weapons. Simultaneously, miltary
training and planning began to use models and games.
(By 1898, wargames were popular enough among civilians
to inspire the creation of Jane's first reference
work, "All the World's Fighting Ships".)
Between the Boer War and World War I, wargames for
training and planning began to reveal both the
strengths and weaknesses of their limited models.
Logistics-models (eg) could suggest more efficient
strategies for supplying the front, but simply knowing
the size, weaponry, and disposition of the armies was
not enough to predict the winner-- various human
factors (eg sabotage, diplomacy, and morale) also had
to be taken into account.
During WW1, the USA experimented with punchcard
technology to manage various resources. The first
psychological tests were invented to better fit
inductees to their assignments. In 1916, Lanchester
tried to use mathematical techniques to analyse air-
warfare, and in 1919 the Quaker pacifist Lewis Fry
Richardson tried to model the "Mathematical Psychology
of War".
German wargames after WW1 employed experts to more-
accurately fill the roles of diplomats, journalists,
etc. Flight simulators evolved during the 1930s to
train pilots quickly, cheaply, and safely.
Moulton in 1926 introduced much more complicated
formulae for ballistics, which demanded vastly
improved calculating machines. Vannevar Bush's analog
differential analyzer began filling this role in 1932.
The British invented 'operations research' c1937,
applying mathematical analysis to tactical planning.
Experts were assigned during WW2 to search for ways of
undermining the enemy's economy.
The German rocket program put Konrad Zuse to work,
building some of the first computers for aerodynamic
calculations, while the British assigned Alan Turing
to break codes. WW2 also saw increasing mathematical
sophistication in designing bombs, airplanes, and
radar. And the atomic bomb program of course required
massive amounts of computation. Simulators were
designed to train soldiers in various technical
skills. Analog devices including gyroscopes were
explored to help aim longrange guns.
After WW2, the prospects of nuclear war inspired the
Pentagon to create the RAND thinktank, using wargames
and computers to completely rethink strategies and
systems for the 'Cold War'. Social scientists at RAND
were encouraged to create mathematical models that
often bore only a faint resemblance to reality. Von
Neumann's game theory required endlessly sophisticated
math, but offered only dubious wisdom.
The USA's offense focused on guided missiles, their
defense on radar detection of Russian planes and
missiles via Jay Forrester's massive SAGE project
(begun in 1950). The Pentagon quickly embraced
digital computers for day-to-day logistics, so RAND
also explored mundane issues like optimising
inventories.
Awkward attempts were made to translate Russian
scientific literature automatically, and to store it
in searchable databanks. Civilian wargames achieved
new levels of authenticity during the 1950s and 60s
under the leadership of Avalon Hill. RAND tried to
introduce computers into wargaming as early as 1954,
and slowly but surely these models increased in
accuracy and utility.
Sputnik in 1957 turned the missile-race into a
spacerace, with new emphasis on basic scientific
research at ARPA. JFK and Robert McNamara in 1961
introduced the RAND approach thruout the Pentagon, and
LBJ would expand this optimistically to every
department. (The Vietnam War eventually demonstrated
some realworld limitations of RAND-style modeling.)
Computer models were used to test the designs of
complex weapons like tanks, and they became
increasingly useful in wargames. By the late 1970s,
civilian innovations in wargame-design were being
adapted by the Pentagon.
1982 saw the US Air Force exploring AI for mission
planning, and the same year brought RAND wargames in
which the computer itself first did the enemy's
planning. By 1988, a networked simulation 'SimNet'
had been created in which each computer took the role
of a single combat vehicle.
Chris Crawford in 1985 offered the geopolitical sim
"Balance of Power", which tried to model the
diplomatic challenges of avoiding nuclear Armageddon.
Bengt Richter
>I wrote, re: http://www.robotwisdom.com/ai/timeline.html
>> [...] my second thought was to write a prose summary that
>> explicitly traces these parallel paths.
>
>I decided to pick one major path and try writing such a summary.
>It came out a lot longer than I expected, but other than that
>it seems promising. (Apologies for the choppy style, which I'll
>try to smooth out eventually.) ((The content is not why I'm
>posting it, remember, but rather its utility as a hypertext
>summary/index, and the nature of the language-processing
>required to compile it from a timeline.))
>
>
[...]
The summary reminded me of a TV show from the past, called "Connections"
Any connection? ;-)
Regards,
Bengt Richter
Mark Watson
from news articles - that is a form of sumarization.
Check out http://www.markwatson.com/demos and
look at the last demo link on the page.
-Mark