Hello,
I have a Titan Z that contain two GTK 110 graphics processors for my processing. When I run bedpostx_gpu or probtrackx2_gpu, only one of the processors within the Titan Z. I can toggle which one it uses by setting the CUDA_VISIBLE_DEVICES in my bash script. So either the first or second gpu is used. However, for a single job I cannot get both GPU to work on the same job. An easy workaround is to just launch two separate bedpost for separate subjects, but this is not possible with probtrack due to the ram usage. I am aware that probtrackx2_gpu is memory hungry and a sparse matrix workaround is being developed. For now each subject takes around 150 GB of RAM which I only have 128 GB, but I have a fast hard drive and just set 128 GB to swap which enables the probtrack to run (I am surprised it processed so fast running in swap). Bedpost takes around 28 minutes and probtrack takes around 2 hours. Which are huge improvements over the cluster I was working on, but I am still only using 1/2 the power of this graphics card for probtrack. Is there an easy fix to access both processors within the Titan Z. I tried setting up a cluster using gridengine following the wiki page, but it still does not use both processors. Plus, it seems odd that I would have to cluster them since it is a single card. I feel that I am missing something.
Here are my commands for calling bedpost and probtrack
bedpostx_gpu ${DATADIR} --nf=3 --bi=5000 --fudge=1 --nj=1250 --se=25 --model=2 --cnonlinear --f0 --ardf0 --rician
probtrackx2_gpu --network -x ${DATADIR}/ROIlist.txt --modeuler --randfib=0 --fibst=1 --forcedir --opd --ompl -l --distthresh1=4.00 --onewaycondition --omatrix1 -c 0.2 -S 2000 --steplength=0.5 -P 5000 --fibthresh=0.01 --sampvox=1.00 -s ${BEDPOSTDIR}/merged -m ${DATADIR}/nodif_brain_mask --dir=${TRACTDIR}/${SUBJ}_Tractography_WholeBrainConnectome -V 2
I have tried adjusting the number of jobs for bedpost to no avail.
Thanks,
John
|