Thanks very much for the quick reply Jessica, yes this makes sense.

I think it would be great though if a future version of PRoNTo would allow for random partitioning as a hard coded option.

Kind regards,
Oliver



____________________________________
Oliver Baumann, PhD
Cognitive and Behavioral Neuroscience
Queensland Brain Institute
University of Queensland
[log in to unmask]
Tel: (+61 7) 33466326
________________________________________
From: PRoNTo users <[log in to unmask]> on behalf of Jessica Schrouff <[log in to unmask]>
Sent: Wednesday, 16 November 2016 12:33 PM
To: [log in to unmask]
Subject: Re: Random partitioning for cross validation

Hi Oliver,

PRoNTo does not provide such a random partitioning of the data. It will
typically allow for one cross-validation scheme (e.g. 10-folds). This
partitioning, if based on k=5 or k=10 folds, is believed to provide a
good estimation of the generalization error (e.g. Hastie, Tibshirani &
Friedman, 2009).

If you do want to perform random CVs, the 'custom' CV option could be
used. You would have to first create a custom CV based on e.g. a
10-folds CV. Then you can save the matrix and create 99 random versions
of it (permuting the rows). Then you can script the batch to create one
hundred models, each based on a matrix file. You'll need to load the PRT
and average the accuracy across all output models. It is hence feasible
but requires a little coding.

Does that make sense?

HTH

Best

Jessica


On 11/15/2016 5:46 PM, Oliver Baumann wrote:
> Hi,
>
> I wonder whether there is a way in PRONTO to randomise the partitioning of cross validation folds and perform the analysis multiple times (e.g. 100 random 10-fold CV)?
>
> Based on my current understanding of partition noise, it is important to get an average, i.e., the expected CV error rate over all possible ways in which the test and training data could fall in the partitioning regime (number of folds, stratified vs non-stratified, ..etc).
>
> Thanks very much,
>
> Oliver
>
>
> ____________________________________
> Oliver Baumann, PhD
> Cognitive and Behavioral Neuroscience
> Queensland Brain Institute
> University of Queensland
> [log in to unmask]
> Tel: (+61 7) 33466326