Message

Marina –

 

Please excuse the delay in responding. I do not know if anyone responded – but here is my interpretation.

 

The Kappa statistic includes measures of class accuracy within an overall measurement of classifier accuracy. It is a better measure of classifier accuracy than overall accuracy because it considers inter-class agreement.

 

Consider the two-class confusion matrix:

 

a          b

------------------

Ta         Fa  | a

Fb        Tb  | b

 

Your equation for K is correct, given that

 

P(A) = (Ta + Tb)/N   (Sum of diagonal in m-class case.)

 

P(E) = (Ta + Fa) x (Ta + Fb)/N2    + (Tb + Fb) x (Tb + Fa)/ N2

 

To generalize P(E) to the m-class case, the terms are Sum_Row_1 x Sum_Col_1 + Sum_Row_2 x Sum_Col_2…

 

One author I found suggests that K < 0.7 is “poor” and that K > 0.8 is “good.”

 

Hope this helps.

 

John Hinchman

BAE SYSTEMS

San Diego, CA

 

 

-----Original Message-----
From: Marina Santini (ITRI) [mailto:Marina.Santini <at> itri.brighton.ac.uk]
Sent: Sunday, November 23, 2003 5:00 AM
To: wekalist <at> list.scms.waikato.ac.nz
Subject: [Wekalist] Kappa Statistic

 

Hallo Everybody,

 

does anyone know if the

Kappa statistic included in the output of

"weka.classifiers.NaiveBayes" (see my

output below) is computed using the following formula:

 

K = P(A) -P(E)

      _________

       1 - P(E)

 

?

 

Thanks a lot for any suggestions, Marina

 

 

^^^^^^^^^^^MyOutput^^^^^^^^^^^^^^^^

=== Stratified cross-validation ===

 

Correctly Classified Instances         132               88      %
Incorrectly Classified Instances        18               12      %
Kappa statistic                          0.8667
Mean absolute error                      0.0246
Root mean squared error                  0.1547
Relative absolute error                 13.6408 %
Root relative squared error             51.4973 %
Total Number of Instances              150    

 

=== Confusion Matrix ===

 

  a  b  c  d  e  f  g  h  i  j   <-- classified as
  9  1  0  4  0  1  0  0  0  0 |  a = interview
  0 15  0  0  0  0  0  0  0  0 |  b = debate
  0  0 15  0  0  0  0  0  0  0 |  c = scripted
  1  0  0 14  0  0  0  0  0  0 |  d = conversation
  0  0  0  0 15  0  0  0  0  0 |  e = advert
  0  0  0  0  0 14  0  1  0  0 |  f = biography
  0  0  0  0  0  0 15  0  0  0 |  g = lore
  0  0  0  0  0  0  0 15  0  0 |  h = reportage
  0  0  0  0  1  0  1  1 10  2 |  i = instructional
  0  1  0  0  0  0  0  1  3 10 |  j = academic