Favicon

FW: Re-evaluate model on test data

 

Hi All,

 

I go through the following set of steps in WEKA:

 

1)     Load training dataset

2)     Choose classifier (say multilayer perceptron)

3)     Chose test option (cross validation x10)

4)     Start

5)     View classification output

6)     Save model

7)     Load same model

8)     Set test dataset to original training dataset

9)     Re-evaluate model on test datatset

10)  The classification result I see is different from the view 5) and is the same as if I had chosen “Use training set” in original classification instead of cross validation

 

Can anybody explain this?

 

Cheers, Dave

 

David A. Wilkinson,
Chevron ETC.
6001 Bollinger Canyon Rd.
San Ramon,
CA. 94583

Phone # 925 842-6200
Fax # 925 842-2076

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist <at> list.scms.waikato.ac.nz
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
David Sharpe | 5 Aug 2012 21:49
Favicon

Re: FW: Re-evaluate model on test data

On Sun, Aug 5, 2012 at 12:24 PM, Wilkinson, Dave (DWLK) (DavidAWilkinson) <DavidAWilkinson <at> chevron.com> wrote:

 

Hi All,

 

I go through the following set of steps in WEKA:

 

1)     Load training dataset

2)     Choose classifier (say multilayer perceptron)

3)     Chose test option (cross validation x10)

4)     Start

5)     View classification output

6)     Save model

7)     Load same model

8)     Set test dataset to original training dataset

9)     Re-evaluate model on test datatset

10)  The classification result I see is different from the view 5) and is the same as if I had chosen “Use training set” in original classification instead of cross validation

 

Can anybody explain this?

 

Cheers, Dave

 

David A. Wilkinson,
Chevron ETC.
6001 Bollinger Canyon Rd.
San Ramon,
CA. 94583

Phone # 925 842-6200
Fax # 925 842-2076


_______________________________________________
Wekalist mailing list
Send posts to: Wekalist <at> list.scms.waikato.ac.nz
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


Hi David,

The results you describe make perfect sense.

Without looking at your code, I assume the model output by cross-validation is trained on the entire set.

Cross-validation is a technique for estimating test performance. 10-fold cross-validation produces 10 models, but none of these models are output by cross-validation, and these models cannot be averaged. Instead, the output model is trained on the entire set.


--
David Sharpe
Software Developer
Seeker Solutions Inc.

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist <at> list.scms.waikato.ac.nz
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

Gmane