Dear FSL users and developers,

In order to speed up the tracking of probabilistic white matter fiber tracts the GPU acceleration function provided by probtrackx2_gpu seems very useful. For a project we would like to build an extensive network of brain regions, which are connected with each other via a structurally connected network. So first the regions of interest (ROIs) were defined and compasses the entire cortex and several subcortical areas. A total of 250 regions was selected. Next the output from bedpostx was used to model the white matter fiber tracts. This was done in a network fashion in order to check for connections between each of the ROIs. Therefor probtrackx2 runs in network mode.

Hereby the detail about the GPU system that was "made for the job"; it holds several nodes including GeForce GTX 1080Ti, Titan V and Titan RTX. This system is embedded in the national super computer facility. The system admin build FSL 6.0.1 from the 64-bit source and this was successful according to the log;
probtrackx2_gpu: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, not stripped

g++ -O2 -ftree-vectorize -O2 -march=sandybridge -fno-math-errno -std=c++11 -Wall -pedantic -Wno-long-long      -m64  -O2 -ftree-vectorize -O2 -march=sandybridge -m64  -I/sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/CUDA/10.0.130-GCC-7.3.0-2.30/include
... [whole bunch of includes/links] ...
-I/sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/include  -c -o probtrackx_gpu.o probtrackx_gpu.cc


A first try to only do tracking from one selected ROI was successful. The command simply loaded the merged samples from bedpostx, included a whole brain mask and was seeding from the subcortical ROI. Overall the application of probtrackx2_gpu resulted in very fast processing of this part. Awesome!

So the next step was to see if the network based processing would work. The call for probtrackx2 included the --network flag, list of seeds in a .txt file, bedpostx merged samples, the brain mask, -S 2000, -P 5000, and other settings at default. This routine would run with the regular probtrackx2 command on the system, just as a sanity check. But this takes 16 days...

For the GPU processing the module FSL/6.0.1-foss-2018b-Python-2.7.15-CUDA-10.0.130 was loaded via SLURM, which includes FSL 6.0.1, FOSS, Python and CUDA. The processing with probtrackx2_gpu fails unfortunately and provides the error; “terminate called after throwing an instance of 'std::bad_array_new_length'”. This error seems to be related to a mismatch in the array length and indexing and shows up after the message "Running in network mode". Note that the number of seeds is 997548.

How can this error be resolved? The system admin came up with the suggestion of explicitly building on the 64-bits system, while this might be done automatically.

Thanks in advance.
Kind regards,

Stijn Michielse, PhD
Department of Neurosurgery
School for Mental Health and Neuroscience | Maastricht University

########################################################################

To unsubscribe from the FSL list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=FSL&A=1