Thanks, Matthew, that led me to the solution. I tried a divisor of 5,
but had no statistically significant change in the character of
failures. However, reducing the divisor to 1 enabled all processes to
finish and produce results.
Odd that the $howlong limit seemingly sabotages the successful running
of the processing stream. I suppose this is good if one is charged for
CPU cycles, but our grid is dedicated to fMRI data processing and the
cost allocation has long since been lost to history. That is, the grid
is bought and paid for.
This may have implications in processing data produced from our new
multi-band sequences, which provide 2mm^2 x 72 slice resolution vs 3mm^2
x 32 slice resolution. The failures were typically while processing
nmaskvoxels > 4100.
Chuck
On 02/22/2014 04:51 AM, Matthew Webster wrote:
> Hello Chuck,
> It sounds like the estimation of how long the flameo sub-jobs will take to complete is failing for your data. One solution is to edit
>
> $FSLDIR/bin/feat
>
> and change the line
>
> if { $fmri(robust_yn) } { set howlong [ expr int( $fmri(npts) * $fmri(evs_real) / 10 ) ] }
>
> to be more conservative
>
> e.g.
>
> if { $fmri(robust_yn) } { set howlong [ expr int( $fmri(npts) * $fmri(evs_real) / 5 ) ] }
>
> will double the time estimate.
>
> Kind Regards
> Matthew
>> Hello,
>>
>> I have a study with 32 participants whose data has already passed through first-level analysis and has been denoised. I've got 22 lower-level contrasts successfully processed for each subject. When I try to run the higher-level analysis, it fails for about half the participants with a mode illustrated in the output below. Essentially, flameo fails to create a res4d.nii.gz file with the correct number of slices, determined using nifti_tool. Note copes 18, 21 and 22 are correct, but copes 19 and 20 are short 3 and 1 slice, respectively:
>>
>> cope18
>> mask.nii.gz: dim 40 8 3 91 109 91 1 1 1 1
>> res4d.nii.gz: dim 40 8 4 91 109 91 32 1 1 1
>> cope19
>> mask.nii.gz: dim 40 8 3 91 109 91 1 1 1 1
>> res4d.nii.gz: dim 40 8 4 91 109 88 32 1 1 1
>> cope20
>> mask.nii.gz: dim 40 8 3 91 109 91 1 1 1 1
>> res4d.nii.gz: dim 40 8 4 91 109 90 32 1 1 1
>> cope21
>> mask.nii.gz: dim 40 8 3 91 109 91 1 1 1 1
>> res4d.nii.gz: dim 40 8 4 91 109 91 32 1 1 1
>> cope22
>> mask.nii.gz: dim 40 8 3 91 109 91 1 1 1 1
>> res4d.nii.gz: dim 40 8 4 91 109 91 32 1 1 1
>>
>> Below is an excerpt from the report_log file showing a typical failure ("87Log directory is: stats0054") followed by a successful processing. Finally, the FATAL ERROR results from a bad res4d file. The same behavior occurs when using a runmode of flame1.
>>
>> Processing is done on a small cluster of Core i7 machines with 3 GB RAM per core, submitted through SGE.
>>
>> I would appreciate any help in tracking down this failure and seeing the processing through to the end. FYI, reportedly, if "Use automatic outlier de-weighting" is unchecked (set fmri(robust_yn) 0), then processing finishes without issue. I will start another run to confirm this.
>>
>> /usr/share/fsl/5.0/bin/flameo --cope=tmpcope0000 --vc=tmpvarcope0000 --mask=tmpmask0000 --ld=stats0000 --dm=design.mat --cs=design.grp --tc=design.con --io --runmode=ols
>>
>> . . .
>>
>> No f contrasts
>> nevs=3
>> ntpts=32
>> ngs=1
>> nvoxels=4254
>> Running:
>> nmaskvoxels=4254
>> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87Log directory is: stats0054
>> Setting up:
>> ntptsing=32.000000
>>
>> evs_group=1.000000
>> 1.000000
>> 1.000000
>>
>> No f contrasts
>> nevs=3
>> ntpts=32
>> ngs=1
>> nvoxels=4175
>> Running:
>> nmaskvoxels=4175
>> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
>> nmaskvoxels=4175
>> Saving results
>>
>> Log directory was: stats0054
>> Log directory is: stats0055
>> Setting up:
>> ntptsing=32.000000
>>
>> . . .
>>
>> /bin/mv stats0000 stats ; /bin/rm -rf stats?* tmp*
>>
>> /bin/rm -f stats/zem* stats/zols* stats/mask* ; /bin/mv dof stats
>>
>> /usr/share/fsl/5.0/bin/smoothest -d 29 -m mask -r stats/res4d > stats/smoothness
>> Mask and Data (residuals/zstat) volumes MUST be the same size!
>>
>> Post-stats
>>
>> /usr/share/fsl/5.0/bin/fslmaths stats/zstat1 -mas mask thresh_zstat1
>> Image Exception : #3 :: Attempted to multiply images/ROIs of different sizes
>> terminate called after throwing an instance of 'RBD_COMMON::BaseException'
>>
>> FATAL ERROR ENCOUNTERED
>> terminate called after throwing an instance of 'RBD_COMMON::BaseException'
>> while executing
>> "if { [ catch {
>>
>>
>>
>> for { set argindex 1 } { $argindex < $argc } { incr argindex 1 } {
>> switch -- [ lindex $argv $argindex ] {
>>
>> -I {
>> incr argin..."
>> (file "/usr/share/fsl/5.0/bin/feat" line 119)
>>
>> --
>> Chuck Theobald
>> System Administrator
>> The Robert and Beverly Lewis Center for Neuroimaging
>> University of Oregon
>> P: 541-346-0343
>> F: 541-346-0345
--
Chuck Theobald
System Administrator
The Robert and Beverly Lewis Center for Neuroimaging
University of Oregon
P: 541-346-0343
F: 541-346-0345
|