9 Jan 2004 10:09

## Re: Kappa Statistic

Regards, Marina

Hinchman, John H (US SSA) wrote:
Message

Marina

Please excuse the delay in responding. I do not know if anyone responded – but here is my interpretation.

The Kappa statistic includes measures of class accuracy within an overall measurement of classifier accuracy. It is a better measure of classifier accuracy than overall accuracy because it considers inter-class agreement.

Consider the two-class confusion matrix:

a          b

------------------

Ta         Fa  | a

Fb        Tb  | b

Your equation for K is correct, given that

P(A) = (Ta + Tb)/N   (Sum of diagonal in m-class case.)

P(E) = (Ta + Fa) x (Ta + Fb)/N2    + (Tb + Fb) x (Tb + Fa)/ N2

To generalize P(E) to the m-class case, the terms are Sum_Row_1 x Sum_Col_1 + Sum_Row_2 x Sum_Col_2…

One author I found suggests that K < 0.7 is “poor” and that K > 0.8 is “good.”

Hope this helps.

John Hinchman

BAE SYSTEMS

San Diego, CA

-----Original Message-----
From: Marina Santini (ITRI) [mailto:Marina.Santini <at> itri.brighton.ac.uk]
Sent:
Sunday, November 23, 2003 5:00 AM
To: wekalist <at> list.scms.waikato.ac.nz
Subject: [Wekalist] Kappa Statistic

Hallo Everybody,

does anyone know if the

Kappa statistic included in the output of

"weka.classifiers.NaiveBayes" (see my

output below) is computed using the following formula:

K = P(A) -P(E)

_________

1 - P(E)

?

Thanks a lot for any suggestions, Marina

^^^^^^^^^^^MyOutput^^^^^^^^^^^^^^^^

=== Stratified cross-validation ===

Correctly Classified Instances         132               88      %
Incorrectly Classified Instances        18               12      %
Kappa statistic                          0.8667
Mean absolute error                      0.0246
Root mean squared error                  0.1547
Relative absolute error                 13.6408 %
Root relative squared error             51.4973 %
Total Number of Instances              150

=== Confusion Matrix ===

a  b  c  d  e  f  g  h  i  j   <-- classified as
9  1  0  4  0  1  0  0  0  0 |  a = interview
0 15  0  0  0  0  0  0  0  0 |  b = debate
0  0 15  0  0  0  0  0  0  0 |  c = scripted
1  0  0 14  0  0  0  0  0  0 |  d = conversation
0  0  0  0 15  0  0  0  0  0 |  e = advert
0  0  0  0  0 14  0  1  0  0 |  f = biography
0  0  0  0  0  0 15  0  0  0 |  g = lore
0  0  0  0  0  0  0 15  0  0 |  h = reportage
0  0  0  0  1  0  1  1 10  2 |  i = instructional
0  1  0  0  0  0  0  1  3 10 |  j = academic

```_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
```

Gmane