Hi Moises,
I'm actually the one who compiled FSL from sources for Stijn. I'm a consultant at the Dutch National Supercomputing centre and Stijn wants to run this on one our systems. Since these are big, shared systems, we typically compile from sources so that we can link to our own versions of shared libraries and optimise builds for our instruction sets.
I agree with you that it would definitely be a good test to check if Stijn can reproduce the same problem with the binary installation. At the same time, I'd be curious to know what could be the potential differences between your build an my build.
In order to compile the GPU codes that are part of FSL, I altered the following to the sources I downloaded (FSL-6.0.1):
====
diff -Nru fsl.orig/src/eddy/fslconfig fsl/src/eddy/fslconfig
--- fsl.orig/src/eddy/fslconfig 2019-10-11 13:50:36.315062983 +0200
+++ fsl/src/eddy/fslconfig 2019-10-11 18:09:10.016835149 +0200
@@ -1,3 +1 @@
-if [ `hostname` == "caper.fmrib.ox.ac.uk" -o `hostname` == "osx-10-9.fmrib.ox.ac.uk" -o `hostname` == "jalapeno19.fmrib.ox.ac.uk" ]; then
- export MAKEOPTIONS="${MAKEOPTIONS} cuda=1 cpu=1" ;
-fi
+export MAKEOPTIONS="${MAKEOPTIONS} cuda=1 cpu=1" ;
diff -Nru fsl.orig/src/fdt/fslconfig fsl/src/fdt/fslconfig
--- fsl.orig/src/fdt/fslconfig 2019-10-11 13:50:36.336062960 +0200
+++ fsl/src/fdt/fslconfig 2019-10-11 18:09:38.260806371 +0200
@@ -1,3 +1 @@
-if [ `hostname` == "caper.fmrib.ox.ac.uk" -o `hostname` == "jalapeno19.fmrib.ox.ac.uk" ]; then
- export MAKEOPTIONS="${MAKEOPTIONS} COMPILE_GPU=1" ;
-fi
+export MAKEOPTIONS="${MAKEOPTIONS} COMPILE_GPU=1" ;
diff -Nru fsl.orig/src/ptx2/CUDA/tractographyInput.cc fsl/src/ptx2/CUDA/tractographyInput.cc
--- fsl.orig/src/ptx2/CUDA/tractographyInput.cc 2019-10-11 21:07:49.408644181 +0200
+++ fsl/src/ptx2/CUDA/tractographyInput.cc 2019-10-11 18:08:32.851872933 +0200
@@ -492,20 +492,20 @@
void tractographyInput::csv_tri_crossed_voxels( float tri[3][3],
vector<ColumnVector>& crossed)
{
- int minx=(int)round(tri[0][0]);
- int miny=(int)round(tri[0][1]);
- int minz=(int)round(tri[0][2]);
+ int minx=(int)MISCMATHS::round(tri[0][0]);
+ int miny=(int)MISCMATHS::round(tri[0][1]);
+ int minz=(int)MISCMATHS::round(tri[0][2]);
int maxx=minx,maxy=miny,maxz=minz;
crossed.clear();
int i=0;int tmpi;
do{
- tmpi=(int)round(tri[i][0]);
+ tmpi=(int)MISCMATHS::round(tri[i][0]);
minx=tmpi<minx?tmpi:minx;
maxx=tmpi>maxx?tmpi:maxx;
- tmpi=(int)round(tri[i][1]);
+ tmpi=(int)MISCMATHS::round(tri[i][1]);
miny=tmpi<miny?tmpi:miny;
maxy=tmpi>maxy?tmpi:maxy;
- tmpi=(int)round(tri[i][2]);
+ tmpi=(int)MISCMATHS::round(tri[i][2]);
minz=tmpi<minz?tmpi:minz;
maxz=tmpi>maxz?tmpi:maxz;
i++;
diff -Nru fsl.orig/src/ptx2/fslconfig fsl/src/ptx2/fslconfig
--- fsl.orig/src/ptx2/fslconfig 2019-10-11 13:50:36.310062989 +0200
+++ fsl/src/ptx2/fslconfig 2019-10-11 18:09:19.874825110 +0200
@@ -1,3 +1 @@
-if [ `hostname` == "caper.fmrib.ox.ac.uk" -o `hostname` == "jalapeno19.fmrib.ox.ac.uk" ]; then
- export MAKEOPTIONS="${MAKEOPTIONS} COMPILE_GPU=1" ;
-fi
+export MAKEOPTIONS="${MAKEOPTIONS} COMPILE_GPU=1" ;
====
I'm thinking only the last 2 files in the diff affect the probtrackx2_gpu build. I had to replace the call to round by one from an explicit namespace because the compiler was complaining about the function call being ambiguous (apparently there were two 'round' functions in my includes that both qualified). However, I'm 99% certain this is unrelated to Stijn's issue :)
FYI: my compile line for probtrackx_gpu.o looks like this (I'll leave out a major part of the -I and -L's to just focus on the compiler options)
===
g++ -fno-math-errno -std=c++11 -Wall -pedantic -Wno-long-long -ftree-vectorize -O2 -march=sandybridge -m64
...
-I/nfs/home4/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/FSL/6.0.1-foss-2018b-Python-2.7.15-CUDA-10.0.130/fsl/extras/include/armawrap/armawrap -DARMA_USE_LAPACK -DARMA_USE_BLAS -DARMA_64BIT_WORD -
...
-c -o probtrackx_gpu.o probtrackx_gpu.cc
===
The link line for probtrackx2_gpu looks like this (again leaving out -I and -L's):
===
g++ -fno-math-errno -std=c++11 -Wall -pedantic -Wno-long-long -ftree-vectorize -O2 -march=sandybridge -m64
...
-I/nfs/home4/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/FSL/6.0.1-foss-2018b-Python-2.7.15-CUDA-10.0.130/fsl/extras/include/armawrap/armawrap -DARMA_USE_LAPACK -DARMA_USE_BLAS -DARMA_64BIT_WORD
...
-lopenblas -o probtrackx2_gpu link_gpu.o saveResults_ptxGPU.o probtrackx_gpu.o CUDA/tractographyInput.o CUDA/tractographyData.o probtrackxOptions.o csv.o csv_mesh.o tractography_gpu.o -lnewmeshclass -lwarpfns -lbasisfield -lfslsurface -lfslvtkio -lmeshclass -lnewimage -lutils -lmiscmaths -lnewmat -lnewran -lNewNifti -lgiftiio -lexpat -lfirst_lib -lznz -lcprob -lutils -lprob -lm -lz -I/sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/CUDA/10.0.130-GCC-7.3.0-2.30/include -lcudart -lcudadevrt -lcuda -lnvToolsExt
===
Does anything stand out to you as different compared to your own builds in terms of the command line options?
(btw, kudo's for providing builds against so many different versions of CUDA - you wouldn't believe how annoying it is if you have to install a gazillion ancient CUDA versions because ppl want to use prebuilt binaries that are linked against only one specific CUDA version...)
Cheers,
Caspar
########################################################################
To unsubscribe from the FSL list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=FSL&A=1
|