Re: CG: Re: Controlled Natural Language vs. Natural Language
Rick and Norbert,
I'd like to mention an interesting case study about
the human factors of CNLs, which unfortunately dates
to the dark ages (i.e., pre-web days) when reports
and articles were only available on paper.
It's the TQA (Transformational Question Answering)
system, which was developed at IBM during the 1970s
and terminated in the late 1980s when it was clear
that it was not going to become a product. In those
days, it was called a natural language system, but
it could more properly be called a controlled NL.
To test the system on real data, they used the land-use
planning database for the city of White Plains, which
was the closest mid-sized city to IBM Yorktown. After
they had it working, IBM installed a 3270 display in
the White Plains city hall and connected it by a leased
phone line to the mainframe computer at Yorktown. And
the short story is that the users loved it.
Before IBM installed TQA, the planners had no way to
query the database by themselves. The city owned a
minicomputer running RPG (Report Program Generator),
which was a language for searching files and printing
reports. Whenever the planners needed some information,
they had to tell the RPG programmer what they needed
and had to wait a day or two before the programmer got
around to writing a program, running it, and printing
the report.
With TQA, however, the planners could walk over to
the terminal, type in a question, see the answer, and
push a button to have it printed. They could also try
different questions to check various options before
deciding which one they needed or preferred.
Contrary to claims that users might attribute more
intelligence to the system than it actually had, they
were really much too conservative. Stan Petrick, who
had written his PhD dissertation at MIT in the 1960s,
was justly proud of the coverage of English syntax in
TQA. However, the users rarely tried more than a small
number of syntactic constructions. When they found
something that worked, they tended to stick with it
rather than try anything new. They preferred to type
several short questions than one complex sentence with
multiple conditions expressed in different clauses.
After a two-year evaluation period, IBM pulled the plug
on the leased line, and the city planners were very
unhappy. IBM was willing to donate the software to the
city, but it only ran on a mainframe. So the users had
to go back to something more primitive, which they hated.
The reasons why TQA never became a product are very
simple:
1. TQA worked very well for land-use planning
because several people with PhDs in linguistics
customized the vocabulary for that task and
mapped the words to the database fields.
2. Very few clients have some linguists in their
back room who could do such careful customizing
of the vocabulary and DB mapping.
3. Fred Damerau, one of the IBM linguists, designed
a tool to simplify the task of customizing the
dictionary and the mapping to the database.
4. However, Fred's tool still required quite a lot
of expertise, which would require someone who
could at least remember some high-school grammar,
have a good feel for the logical issues involved,
and have the time and patience to learn how to
map words to database fields.
These issues are just as significant today as they
were for TQA twenty years ago. A lot of people have
designed fairly decent systems, but the major drawback
to wider use is the cost of customizing the vocabulary
and DB mapping. The only thing that has changed since
then is that we now have a good buzzword: ontology.
Rick wrote
RW> Fortunately, though, people are used to interacting
> with computers a lot. So they will perhaps be more
> willing to modify their language in order to achieve
> the benefit of being able to put complex information
> into a knowledge base.
Actually, people are most experienced with search engines
that ignore all syntax and just look for keywords. That
is not helpful because most NL parsers take advantage
of all the little words like "a", "the", "of", and "is",
which search engines ignore.
Even with TQA, the users first tried to "simplify" their
English by typing their idea of how a robot would speak
without using "a" and "the", but that was not helpful.
NF> A recurring topic in this thread is the – (depending
> on the contributor) potential, probable or certain –
> confusion between controlled natural language and
> natural language.
The land-use planners who used TQA never confused its
version of English with unrestricted NL. In fact,
they tended to restrict their own range of English
constructions more than they needed to.
Does anyone have any actual data to show that users might
overestimate the computer's ability? I've heard people
make that claim, but never with any data to support it.
John