Hello,
Are you using a version of fsl_sub modified for SLURM?
Kind Regards
Matthew
--------------------------------
Dr Matthew Webster
FMRIB Centre
John Radcliffe Hospital
University of Oxford
> On 12 Sep 2017, at 15:14, Tony Goldstone <[log in to unmask]> wrote:
>
> Dear FSLers
> We have recently upgraded our departmental computing network cluster to one which uses the SLURM environment in a Centos operating system.
>
> We have discovered that we are unable to run parallel Feat analyses using FSL v5.0.10 as each individual Feat grinds to a halt at the Stats stage.
> This seems to be due to a timing out issue as per the output of the squeue enquiry below as the feat3 stage never seems to complete.
> The Feat Report Log Page point at which it stops is also below.
> So parallel Feat analyses seem to be incompatible with the SLURM environment.
>
> Note that these Feat analyses run serially without any problems on a desktop.
>
> We have tried to solve this by turning off the automatic fsl_sub partitioning by Feat using either of these 2 commands in our scripts, and than adding sbatch or fsl_sub to the command line in front of the "feat design.fsf" command, without success (informed from some previous internet threads including on FSL listserv):
>
> export NO_FSL_JOBS=true
> export FSLSUBALREADYRUN
>
> Does anyone have any solutions or can identify what the problem may be?
>
> Many thanks
> Tony Goldstone
> Head PsychoNeuroEndocrinology Research Group, Senior Clinical Research Fellow,
> Neuropsychopharmacology Unit, Centre for Psychiatry, and Computational, Cognitive and Clinical Neuroimaging Laboratory,
> Division of Brain Sciences, Imperial College London
>
>
> squeue output:
>
> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
> 705464 long feat4_po tgoldsto PD 0:00 1 (DependencyNeverSatisfied)
> 705465 long feat4_po tgoldsto PD 0:00 1 (DependencyNeverSatisfied)
> 705466 long feat4_po tgoldsto PD 0:00 1 (DependencyNeverSatisfied)
> 705467 long feat5_st tgoldsto PD 0:00 1 (DependencyNeverSatisfied)
> 705468 long feat5_st tgoldsto PD 0:00 1 (DependencyNeverSatisfied)
> 705469 long feat5_st tgoldsto PD 0:00 1 (DependencyNeverSatisfied)
>
> Feat Report Log Page:
>
> -------------------------------------------------------------------------
> Stats
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/onsets/mid/version_a/win_cue.txt custom_timing_files/ev1.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/onsets/mid/version_a/neutral_cue.txt custom_timing_files/ev2.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/onsets/mid/version_a/loss_cue.txt custom_timing_files/ev3.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/Y12/10182379/func/mid/mid_outcomes_win_hit.txt custom_timing_files/ev4.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/Y12/10182379/func/mid/mid_outcomes_win_miss.txt custom_timing_files/ev5.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/Y12/10182379/func/mid/mid_outcomes_neutral_hit.txt custom_timing_files/ev6.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/Y12/10182379/func/mid/mid_outcomes_neutral_miss.txt custom_timing_files/ev7.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/Y12/10182379/func/mid/mid_outcomes_loss_hit.txt custom_timing_files/ev8.txt
>
> mkdir -p custom_timing_files ; /apps/software/fsl/5.0.10/bin/fslFixText /group/hormone/CIF/Y12/10182379/func/mid/mid_outcomes_loss_miss.txt custom_timing_files/ev9.txt
>
> /apps/software/fsl/5.0.10/bin/film_gls --in=filtered_func_data --rn=stats --pd=design.mat --thr=1000.0 --sa --ms=5 --con=design.con
> Log directory is: stats
> paradigm.getDesignMatrix().Nrows()=245
> paradigm.getDesignMatrix().Ncols()=9
> sizeTS=245
> numTS=46581
> Calculating residuals...
> Completed
> Estimating residual autocorrelation...
> Calculating raw AutoCorrs... Completed
> mode = 9962.71
> sig = 873
> Spatially smoothing auto corr estimates
> ..............
> Completed
> Tukey M = 15
> Tukey estimates... Completed
> Completed
> Prewhitening and Computing PEs...
> Percentage done:
> 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,Completed
> Saving results...
> Completed
>
> /apps/software/fsl/5.0.10/bin/smoothest -d 236 -m mask -r stats/res4d > stats/smoothness
|