Re: Software to mark short textual responses
Jean-Luc,
JLD> Sorry about my "synthese/foutaise" remark.
> I guess that due to my usual abrasive style you
> took it personally, it was not meant to be a
> personal attack...
That didn't bother me at all. Actually, Kant was
the first to use the thesis-antithesis-synthesis
terminology. And Hegel only used those words once
in his entire collected works. (He did use a lot
of trichotomies, but not with those words.)
But I wanted to clarify some points:
1. The project I mentioned for evaluating textual
responses was definitely *not* generating a full
logical form. It was generating CGs, but not at
the level of mapping them to an exact logical form.
2. LSA is useful for information retrieval, but it
is a "bag of words" approach that can only be
applied to large texts -- at least a large
paragraph and preferably several pages or more.
3. Please reread the article on analogical reasoning
that I cited:
http://www.jfsowa.com/pubs/analog.htm
Note that analogical reasoning is a more primitive
and more fundamental method than logic. As I point
out in that article, logic is a more disciplined
and more specialized version of analogy.
4. What we used for evaluating the student answers
was analogy -- not logic.
Some comments on your comments:
JLD> Anyway, we obviously disagree about the role of logic
> in semantic.
I don't think we do. I have at least as many serious doubts
about the ability to map unrestricted NL to logic as you do.
If anything, I suspect that I have more doubts than you do.
JLD> Though techniques like LSA are severely lacking in this
> respect they *do* have overwhelming advantages for spotting
> the underlying "semantic flavor" of seemingly unrelated
> texts if they have been *trained on the right corpus.*
I am very familiar with LSA and related techniques. I first
started using vector and cluster methods for information
retrieval in the mid 1960s. The primary innovation of LSA
(singular value decomposition) reduces the size of the vector
spaces to about 300 dimensions instead of several thousand
dimensions. That is useful for improving performance, but
it still does nothing more than approximating the flavor
of the knowledge soup.
JLD> This is absolutely the opposite of your remark:
JFS>> It is also incapable of recognizing correct answers
>> that happen to use different words.
Yes, I realize that LSA, cluster analysis, and many related
technologies do derive a similar "flavor" even when restated
in different words -- but only if they have large texts to
work with. Even then, they can only detect the "flavor".
They cannot tell whether the text addresses a particular
question -- much less tell whether it answers it correctly.
But for that project of evaluating textual responses, which
I discussed in my previous note, we applied the VivoMind
Analogy Engine *after* a previous group had attempted to
use LSA for the same project. And they failed -- not just
partially, they totally and miserably failed to do anything.
They could not even approximately distinguish correct from
incorrect answers.
And that group, by the way, was a large consulting company
that had professionals who were very familiar with the LSA
technology and had developed commercial software for using
LSA to score student essays. But their technology was
absolutely worthless for evaluating single sentences.
John