 |
 |
Hi, We tried redoing ICA+FIX, with the same results (~30% TPR, 0% TNR using HCP’s classifier). Like was mentioned above, is it possible that FIX is working correctly, it’s just failing at the accuracy testing stage? I’m assuming we still probably want to know how accurate the classifier is.. I asked more about versions used in the FIX install: The R version we use is 3.4.1. All the R packages I installed in the docker are: mvtnorm_1.0-6.tar.gz modeltools_0.2-21.tar.gz lattice_0.20-35.tar.gz zoo_1.8-0.tar.gz sandwich_2.4-0.tar.gz strucchange_1.5-1.tar.gz MASS_7.3-47.tar.gz TH.data_1.0-8.tar.gz multcomp_1.4-6.tar.gz coin_1.2-1.tar.gz party_1.0-25.tar.gz e1071_1.6-8.tar.gz class_7.3-14.tar.gz bitops_1.0-6.tar.gz gtools_3.5.0.tar.gz gdata_2.18.0.tar.gz caTools_1.17.1.tar.gz gplots_3.0.1.tar.gz ROCR_1.0-5.tar.gz randomForest_4.6-12.tar.gz (the version names are at the end). The FSL version is: fsl-core=5.0.9-4~nd14.04+1 FIX version: fix1.065 Connectome Workbench version: connectome-workbench=1.2.3-1~nd14.04+1 MCR version: MCR_2014a OS version: Ubuntu 14.04 LTS Thanks, Antoni > On Jan 27, 2018, at 4:11 PM, Glasser, Matthew <[log in to unmask]> wrote: > > Did you try running ICA+FIX again on a clean folder with just the input > data and see if it works properly then? > > Matt. > > On 1/27/18, 6:09 PM, "Antoni Kubicki" <[log in to unmask]> wrote: > >> No smoothing, but temporal filtering using a highpass filter of 2000s >> FWHM. >> >> Antoni >> >> >> >>> On Jan 27, 2018, at 4:05 PM, Glasser, Matthew <[log in to unmask]> >>> wrote: >>> >>> Are you doing any smoothing, filtering, or anything like that, or is it >>> just straight from the output of the pipelines? >>> >>> Matt. >>> >>> On 1/27/18, 6:02 PM, "Antoni Kubicki" <[log in to unmask]> wrote: >>> >>>> Okay that makes sense, thanks for clarifying. >>>> >>>> We are using the HCP minimal preprocessing pipeline (with fsl & melodic >>>> 5.0.9.3~nd14.04+1, fix 1.065), so I am fairly confident files are in >>>> the >>>> correct formats/locations, etc.. >>>> >>>> Best, >>>> Antoni >>>> >>>> >>>> >>>>> On Jan 27, 2018, at 3:32 PM, Glasser, Matthew <[log in to unmask]> >>>>> wrote: >>>>> >>>>> No I would expect that. The things that matter more are large changes >>>>> to >>>>> the protocol (very different voxel size, TR, scanner, etc). We have >>>>> cleaned a bunch of the HCP task fMRI data in the paper I sent you >>>>> using >>>>> the resting state training and the performance on a subselected group >>>>> that >>>>> was hand classified was equivalent to what we got with resting state >>>>> data >>>>> originally. I take this to mean you can combine resting state and >>>>> task >>>>> in >>>>> multi-run ICA+FIX and we plan to do this for the Lifespan HCP. >>>>> >>>>> Maybe you can be more explicit on how you are processing the data in >>>>> case >>>>> there are any gotchas. >>>>> >>>>> Matt. >>>>> >>>>> On 1/27/18, 5:27 PM, "Antoni Kubicki" <[log in to unmask]> >>>>> wrote: >>>>> >>>>>> That’s good to know, I appreciate the insight and the paper. >>>>>> >>>>>> We are not concatenating runs, I will give multi-run a try and see >>>>>> how >>>>>> it >>>>>> goes. You wouldn’t expect variations in sources/types of noise/signal >>>>>> between rsfMRI and task fMRI, for example? Does the multi-run version >>>>>> account for this? >>>>>> >>>>>> Thanks for all your help! >>>>>> Antoni >>>>>> >>>>>> >>>>>> >>>>>>> On Jan 27, 2018, at 3:02 PM, Glasser, Matthew <[log in to unmask]> >>>>>>> wrote: >>>>>>> >>>>>>> Well the question will depend on whether you want to say anything >>>>>>> about >>>>>>> the individuals in the study. If you only wish to make statements >>>>>>> about >>>>>>> groups and have enough subjects (e.g. like the biobank) it can be >>>>>>> reasonable though it means the data have less general utility >>>>>>> (though >>>>>>> if >>>>>>> you have 3-4 sessions and analyzed the data all together, that might >>>>>>> work >>>>>>> nicely for some things). >>>>>>> >>>>>>> We tried ICA+FIX on short runs a long time ago and found that it >>>>>>> could >>>>>>> decrease stats in some cases, so we developed the multi-run approach >>>>>>> instead. This is discussed in this bioaRXiv that hopefully will be >>>>>>> out >>>>>>> soon: >>>>>>> >>>>>>> https://www.biorxiv.org/content/early/2017/12/13/193862 >>>>>>> >>>>>>> As for your problem, you aren’t just simply concatenating runs >>>>>>> without >>>>>>> demean/detrend are you? I would just give the multi-run FIX a try >>>>>>> with >>>>>>> the default training and see if that works better. >>>>>>> >>>>>>> Matt. >>>>>>> >>>>>>> On 1/27/18, 4:52 PM, "Antoni Kubicki" <[log in to unmask]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Matt, >>>>>>>> >>>>>>>> I think the rationale behind doing this originally was that we were >>>>>>>> getting only 30% accuracy for TPR (and 0 for TNR) using the HCP’s >>>>>>>> classifier, which now seems to be caused by a different issue.. I >>>>>>>> will >>>>>>>> double-check the install/R versions/etc however, I just installed a >>>>>>>> clean >>>>>>>> version of FIX yesterday that works fine (~100% TPR/TNR) with the >>>>>>>> HCP-provided data, but still gives 0% TNR for our data. >>>>>>>> >>>>>>>> We discussed using multi-run in the beginning, but ended up making >>>>>>>> our >>>>>>>> own classifier for each task and phase encoding direction using the >>>>>>>> single run FIX. I can see how multi-run may be an advantage for >>>>>>>> shorter >>>>>>>> runs though, will give this a try as well and see if still getting >>>>>>>> 0.. >>>>>>>> >>>>>>>> I joined long after the protocol was established, would this still >>>>>>>> be >>>>>>>> an >>>>>>>> issue in a longitudinal study? Most of our subjects are getting >>>>>>>> scanned >>>>>>>> 3-4 times. >>>>>>>> >>>>>>>> Thanks for the input! >>>>>>>> Antoni >>>>>>>> >>>>>>>> >>>>>>>>> On Jan 27, 2018, at 2:06 PM, Glasser, Matthew <[log in to unmask]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Antoni, >>>>>>>>> >>>>>>>>> I wouldn¹t recommend that and we are not doing that for the >>>>>>>>> lifespan >>>>>>>>> HCP. >>>>>>>>> Instead, I would recommend cleaning all of your fMRI together >>>>>>>>> using >>>>>>>>> the >>>>>>>>> new Multi-run ICA+FIX pipeline, which is more effective for >>>>>>>>> shorter >>>>>>>>> fMRI >>>>>>>>> runs. As I said on the list e-mail, I expect there is something >>>>>>>>> wrong >>>>>>>>> with the FIX install/R/etc rather than with your data or your FIX >>>>>>>>> training >>>>>>>>> (or the HCP¹s FIX training which should work fine on your data). >>>>>>>>> >>>>>>>>> I am afraid you will be very limited in what you can do with a >>>>>>>>> total >>>>>>>>> of >>>>>>>>> 13 >>>>>>>>> minutes of resting state data at the individual subject level. >>>>>>>>> See >>>>>>>>> for >>>>>>>>> example Figure 4 of this publication: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://www.sciencedirect.com/science/article/pii/S0896627315006005 >>>>>>>>> #f >>>>>>>>> ig >>>>>>>>> 4 >>>>>>>>> where you really need to get at least 30mins to avoid the bad part >>>>>>>>> of >>>>>>>>> the >>>>>>>>> slope because of instability in connectivity in short time >>>>>>>>> windows. >>>>>>>>> You >>>>>>>>> may be able to combine resting state and task for some other >>>>>>>>> analyses, >>>>>>>>> but >>>>>>>>> I would recommend reconsidering that study design if it isn¹t too >>>>>>>>> late. >>>>>>>>> 26 minutes, while not a lot, should be much better than 13 >>>>>>>>> minutes. >>>>>>>>> >>>>>>>>> Matt. >>>>>>>>> >>>>>>>>> On 1/27/18, 3:52 PM, "Antoni Kubicki" <[log in to unmask]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Matt, >>>>>>>>>> >>>>>>>>>> We decided to manually label and train a study-specific >>>>>>>>>> classifier >>>>>>>>>> for >>>>>>>>>> our resting state, and two tasks separately, but I agree that the >>>>>>>>>> HCP >>>>>>>>>> classifier should be be working much better, and something is >>>>>>>>>> going >>>>>>>>>> wrong... >>>>>>>>>> >>>>>>>>>> We are getting 13 min total of resting state for patients (26min >>>>>>>>>> for >>>>>>>>>> controls), in addition to 9 min of an in-house facematching task >>>>>>>>>> (AP+PA), >>>>>>>>>> and 4min of CARIT (PA). >>>>>>>>>> >>>>>>>>>> Thanks, I appreciate any input! >>>>>>>>>> Antoni >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Jan 27, 2018, at 1:41 PM, Glasser, Matthew >>>>>>>>>>> <[log in to unmask]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Antoni, >>>>>>>>>>> >>>>>>>>>>> It should not be necessary to retrain the FIX classifier for >>>>>>>>>>> that >>>>>>>>>>> data, >>>>>>>>>>> so >>>>>>>>>>> something is clearly going wrong. I hope you are getting more >>>>>>>>>>> than >>>>>>>>>>> a >>>>>>>>>>> total of 13 minutes of fMRI, right? >>>>>>>>>>> >>>>>>>>>>> Peace, >>>>>>>>>>> >>>>>>>>>>> Matt. >>>>>>>>>>> >>>>>>>>>>> On 1/27/18, 3:33 PM, "Antoni Kubicki" <[log in to unmask]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you for the replies. >>>>>>>>>>>> >>>>>>>>>>>> We are using a similar protocol as the lifespan HCP (3T Prisma >>>>>>>>>>>> with >>>>>>>>>>>> 2mm >>>>>>>>>>>> isotropic voxel size, 800ms TR, 37ms TE, 8x multiband accel. >>>>>>>>>>>> factor, >>>>>>>>>>>> 208mm FOV, 72 slices, 488 volumes) with both AP and PA phase >>>>>>>>>>>> encoding >>>>>>>>>>>> directions for resting state. This example was run on both >>>>>>>>>>>> directions >>>>>>>>>>>> included in the same classifier, but even when we run them >>>>>>>>>>>> separately, >>>>>>>>>>>> still getting 0% for TNR. >>>>>>>>>>>> >>>>>>>>>>>> When we use the HCP-generated training-weights files on our >>>>>>>>>>>> data, >>>>>>>>>>>> we >>>>>>>>>>>> get >>>>>>>>>>>> lower TPR (~30%) but still zero for TNR. (Attaching below). I >>>>>>>>>>>> am >>>>>>>>>>>> also >>>>>>>>>>>> sharing an example .ica folder from our data, along with the >>>>>>>>>>>> hand-labeled >>>>>>>>>>>> text file of noise components. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://drive.google.com/open?id=11XKawhqbg_YJqISmiUm0n8X3P_kH2L >>>>>>>>>>>> gQ >>>>>>>>>>>> >>>>>>>>>>>> We tested the HCP-generated classifer on the HCP-provided data >>>>>>>>>>>> it >>>>>>>>>>>> was >>>>>>>>>>>> trained on, and got ~100% accuracy for both TPR and TNR, so FIX >>>>>>>>>>>> itself >>>>>>>>>>>> seems to be working correctly. >>>>>>>>>>>> >>>>>>>>>>>> I have double checked the hand labels and they seem to be in >>>>>>>>>>>> the >>>>>>>>>>>> correct >>>>>>>>>>>> format with square brackets and commas. >>>>>>>>>>>> >>>>>>>>>>>> Also doing another check to look at components that FIX is >>>>>>>>>>>> classifying >>>>>>>>>>>> as >>>>>>>>>>>> "noise" and comparing them to our hand labels for a few scans. >>>>>>>>>>>> Will >>>>>>>>>>>> report back on the results of that.. >>>>>>>>>>>> >>>>>>>>>>>> Thanks again, we're having difficulty pinning down the problem. >>>>>>>>>>>> Antoni Kubicki >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Results of HCP-trained classifier tested on our data (26 rs >>>>>>>>>>>> runs >>>>>>>>>>>> with >>>>>>>>>>>> AP/PA phase encoding) >>>>>>>>>>>> 34.5 0.0 32.8 0.0 27.6 0.0 25.9 0.0 24.1 0.0 24.1 0.0 >>>>>>>>>>>> 24.1 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 22.4 0.0 >>>>>>>>>>>> 43.4 0.0 39.6 0.0 37.7 0.0 35.8 0.0 32.1 0.0 28.3 0.0 >>>>>>>>>>>> 28.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 28.3 0.0 >>>>>>>>>>>> 18.2 0.0 18.2 0.0 15.9 0.0 13.6 0.0 12.5 0.0 12.5 0.0 >>>>>>>>>>>> 12.5 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 10.2 0.0 >>>>>>>>>>>> 27.7 0.0 27.7 0.0 20.6 0.0 12.8 0.0 11.3 0.0 11.3 0.0 >>>>>>>>>>>> 11.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 11.3 0.0 >>>>>>>>>>>> 21.3 0.0 20.2 0.0 18.0 0.0 16.9 0.0 16.9 0.0 16.9 0.0 >>>>>>>>>>>> 16.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 15.7 0.0 >>>>>>>>>>>> 23.9 0.0 23.9 0.0 18.5 0.0 14.1 0.0 13.0 0.0 13.0 0.0 >>>>>>>>>>>> 13.0 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 12.0 0.0 >>>>>>>>>>>> 34.3 0.0 34.3 0.0 31.4 0.0 31.4 0.0 31.4 0.0 28.6 0.0 >>>>>>>>>>>> 27.1 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 27.1 0.0 >>>>>>>>>>>> 47.4 0.0 44.7 0.0 35.5 0.0 31.6 0.0 28.9 0.0 27.6 0.0 >>>>>>>>>>>> 26.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 26.3 0.0 >>>>>>>>>>>> 30.7 0.0 30.7 0.0 29.3 0.0 28.0 0.0 28.0 0.0 25.3 0.0 >>>>>>>>>>>> 25.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 22.7 0.0 >>>>>>>>>>>> 29.1 0.0 29.1 0.0 26.7 0.0 24.4 0.0 23.3 0.0 23.3 0.0 >>>>>>>>>>>> 20.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 18.6 0.0 >>>>>>>>>>>> 41.2 0.0 41.2 0.0 32.4 0.0 32.4 0.0 30.9 0.0 29.4 0.0 >>>>>>>>>>>> 27.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 26.5 0.0 >>>>>>>>>>>> 47.1 0.0 45.6 0.0 42.6 0.0 38.2 0.0 35.3 0.0 32.4 0.0 >>>>>>>>>>>> 32.4 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 30.9 0.0 >>>>>>>>>>>> 13.5 0.0 13.5 0.0 10.1 0.0 6.7 0.0 5.6 0.0 4.5 0.0 >>>>>>>>>>>> 2.2 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 1.1 0.0 >>>>>>>>>>>> 11.5 0.0 11.5 0.0 10.3 0.0 9.0 0.0 6.4 0.0 6.4 0.0 >>>>>>>>>>>> 2.6 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 1.3 0.0 >>>>>>>>>>>> 34.5 0.0 32.7 0.0 27.3 0.0 27.3 0.0 27.3 0.0 25.5 0.0 >>>>>>>>>>>> 23.6 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 23.6 0.0 >>>>>>>>>>>> 29.4 0.0 26.5 0.0 20.6 0.0 19.1 0.0 19.1 0.0 16.2 0.0 >>>>>>>>>>>> 13.2 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 13.2 0.0 >>>>>>>>>>>> 16.9 0.0 16.9 0.0 15.5 0.0 15.5 0.0 15.5 0.0 14.1 0.0 >>>>>>>>>>>> 14.1 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 14.1 0.0 >>>>>>>>>>>> 21.4 0.0 19.0 0.0 16.7 0.0 16.7 0.0 16.7 0.0 15.5 0.0 >>>>>>>>>>>> 15.5 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 14.3 0.0 >>>>>>>>>>>> 60.3 0.0 37.9 0.0 17.2 0.0 17.2 0.0 17.2 0.0 17.2 0.0 >>>>>>>>>>>> 15.5 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 13.8 0.0 >>>>>>>>>>>> 27.7 0.0 24.6 0.0 23.1 0.0 20.0 0.0 16.9 0.0 16.9 0.0 >>>>>>>>>>>> 15.4 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 15.4 0.0 >>>>>>>>>>>> 38.2 0.0 38.2 0.0 38.2 0.0 36.4 0.0 36.4 0.0 36.4 0.0 >>>>>>>>>>>> 32.7 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 32.7 0.0 >>>>>>>>>>>> 44.4 0.0 42.6 0.0 38.9 0.0 38.9 0.0 38.9 0.0 37.0 0.0 >>>>>>>>>>>> 33.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 33.3 0.0 >>>>>>>>>>>> 42.7 0.0 39.0 0.0 37.8 0.0 37.8 0.0 35.4 0.0 35.4 0.0 >>>>>>>>>>>> 34.1 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 34.1 0.0 >>>>>>>>>>>> 46.8 0.0 45.5 0.0 42.9 0.0 41.6 0.0 40.3 0.0 39.0 0.0 >>>>>>>>>>>> 39.0 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 36.4 0.0 >>>>>>>>>>>> 17.5 0.0 15.5 0.0 15.5 0.0 15.5 0.0 15.5 0.0 15.5 0.0 >>>>>>>>>>>> 15.5 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 15.5 0.0 >>>>>>>>>>>> 29.6 0.0 28.4 0.0 23.5 0.0 19.8 0.0 17.3 0.0 14.8 0.0 >>>>>>>>>>>> 14.8 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 13.6 0.0 >>>>>>>>>>>> 33.3 0.0 28.2 0.0 24.4 0.0 21.8 0.0 17.9 0.0 16.7 0.0 >>>>>>>>>>>> 15.4 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 12.8 0.0 >>>>>>>>>>>> 28.9 0.0 27.6 0.0 25.0 0.0 22.4 0.0 19.7 0.0 19.7 0.0 >>>>>>>>>>>> 18.4 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 17.1 0.0 >>>>>>>>>>>> 29.7 0.0 29.7 0.0 29.7 0.0 27.0 0.0 24.3 0.0 21.6 0.0 >>>>>>>>>>>> 18.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 18.9 0.0 >>>>>>>>>>>> 33.3 0.0 31.7 0.0 30.0 0.0 30.0 0.0 26.7 0.0 25.0 0.0 >>>>>>>>>>>> 23.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 20.0 0.0 >>>>>>>>>>>> 16.9 0.0 14.5 0.0 9.6 0.0 7.2 0.0 6.0 0.0 6.0 0.0 >>>>>>>>>>>> 4.8 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 4.8 0.0 >>>>>>>>>>>> 36.6 0.0 30.5 0.0 19.5 0.0 12.2 0.0 8.5 0.0 8.5 0.0 >>>>>>>>>>>> 8.5 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 7.3 0.0 >>>>>>>>>>>> 28.8 0.0 28.8 0.0 25.8 0.0 25.8 0.0 22.7 0.0 22.7 0.0 >>>>>>>>>>>> 21.2 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 18.2 0.0 >>>>>>>>>>>> 33.3 0.0 33.3 0.0 31.9 0.0 27.5 0.0 24.6 0.0 23.2 0.0 >>>>>>>>>>>> 23.2 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 23.2 0.0 >>>>>>>>>>>> 31.8 0.0 29.5 0.0 26.1 0.0 25.0 0.0 25.0 0.0 25.0 0.0 >>>>>>>>>>>> 22.7 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 22.7 0.0 >>>>>>>>>>>> 36.7 0.0 32.9 0.0 26.6 0.0 25.3 0.0 24.1 0.0 22.8 0.0 >>>>>>>>>>>> 21.5 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 21.5 0.0 >>>>>>>>>>>> 37.9 0.0 37.9 0.0 33.3 0.0 30.3 0.0 30.3 0.0 30.3 0.0 >>>>>>>>>>>> 28.8 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 27.3 0.0 >>>>>>>>>>>> 34.1 0.0 30.7 0.0 26.1 0.0 25.0 0.0 20.5 0.0 19.3 0.0 >>>>>>>>>>>> 18.2 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 18.2 0.0 >>>>>>>>>>>> 40.4 0.0 38.6 0.0 36.8 0.0 33.3 0.0 29.8 0.0 29.8 0.0 >>>>>>>>>>>> 29.8 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 28.1 0.0 >>>>>>>>>>>> 44.4 0.0 42.6 0.0 38.9 0.0 37.0 0.0 33.3 0.0 31.5 0.0 >>>>>>>>>>>> 29.6 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 27.8 0.0 >>>>>>>>>>>> 38.3 0.0 36.2 0.0 29.8 0.0 27.7 0.0 27.7 0.0 25.5 0.0 >>>>>>>>>>>> 23.4 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 21.3 0.0 >>>>>>>>>>>> 35.9 0.0 35.9 0.0 34.4 0.0 29.7 0.0 28.1 0.0 26.6 0.0 >>>>>>>>>>>> 25.0 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 23.4 0.0 >>>>>>>>>>>> 31.8 0.0 30.6 0.0 28.2 0.0 28.2 0.0 28.2 0.0 28.2 0.0 >>>>>>>>>>>> 24.7 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 23.5 0.0 >>>>>>>>>>>> 39.7 0.0 38.5 0.0 30.8 0.0 30.8 0.0 26.9 0.0 26.9 0.0 >>>>>>>>>>>> 26.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 24.4 0.0 >>>>>>>>>>>> 26.9 0.0 23.9 0.0 23.9 0.0 23.9 0.0 23.9 0.0 23.9 0.0 >>>>>>>>>>>> 23.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 23.9 0.0 >>>>>>>>>>>> 28.3 0.0 28.3 0.0 26.7 0.0 25.0 0.0 21.7 0.0 20.0 0.0 >>>>>>>>>>>> 16.7 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 16.7 0.0 >>>>>>>>>>>> 38.8 0.0 37.3 0.0 32.8 0.0 28.4 0.0 26.9 0.0 25.4 0.0 >>>>>>>>>>>> 23.9 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 23.9 0.0 >>>>>>>>>>>> 40.3 0.0 38.7 0.0 31.1 0.0 26.1 0.0 22.7 0.0 17.6 0.0 >>>>>>>>>>>> 16.0 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 15.1 0.0 >>>>>>>>>>>> 25.3 0.0 22.9 0.0 20.5 0.0 18.1 0.0 14.5 0.0 13.3 0.0 >>>>>>>>>>>> 13.3 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 13.3 0.0 >>>>>>>>>>>> 25.0 0.0 23.9 0.0 18.5 0.0 16.3 0.0 15.2 0.0 14.1 0.0 >>>>>>>>>>>> 12.0 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 12.0 0.0 >>>>>>>>>>>> 26.5 0.0 22.1 0.0 20.6 0.0 20.6 0.0 19.1 0.0 19.1 0.0 >>>>>>>>>>>> 19.1 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 19.1 0.0 >>>>>>>>>>>> 29.4 0.0 27.9 0.0 23.5 0.0 22.1 0.0 22.1 0.0 20.6 0.0 >>>>>>>>>>>> 19.1 >>>>>>>>>>>> 0.0 >>>>>>>>>>>> 16.2 0.0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> set of thresholds is: 1 2 5 10 20 30 40 50 >>>>>>>>>>>> [TPR,TNR,(3*TPR+TNR)/4] pairs of results (averaged over >>>>>>>>>>>> datasets, >>>>>>>>>>>> one >>>>>>>>>>>> pair per threshold): >>>>>>>>>>>> >>>>>>>>>>>> mean >>>>>>>>>>>> 32.4 30.4 26.5 24.5 22.8 21.7 20.5 19.5 >>>>>>>>>>>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >>>>>>>>>>>> 24.3 22.8 19.9 18.4 17.1 16.3 15.4 14.6 >>>>>>>>>>>> >>>>>>>>>>>> median >>>>>>>>>>>> 32.6 30.5 26.6 25.2 23.6 22.8 21.1 19.0 >>>>>>>>>>>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >>>>>>>>>>>> 24.4 22.9 20.0 18.9 17.7 17.1 15.8 14.3 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >