 |
 |
Hi Sjors, I agree I am being hampered by somethings. My workstation has dual 10- core Xeon processors which with hyperthreading should give me a total of 40 threads. Additionally it has 128GB of ram and a 8 harddrive raid 5 array giving over 1GB/s throughput, which according to 'top' isn't being taxed at all. An interesting observation is that when I use relion_refine, I have no problems of it running more than 2 threads (4, 8, 16, etc. verified using 'top'). However when I use the relion_refine_mpi, I am stuck at 2 threads per mpi process. Some additional info: 1) I have seen this issue on two different systems running Centos7 and Fedora 23 with 20 and 4 cores respectively. 2) I have see issue with my own compiled version of relion 1.4 and SBGrid.org's compiled version. 3) I suspect this might be a reason the program is not scaling as well as we would like on our cluster, a Cray XE6 system. Cheers, Bharat Reddy Post Doc University of Chicago On Thu, 2016-04-28 at 10:10 +0100, Sjors Scheres wrote: > Hi again, > The program is actually using 4 threads (as from the stdout). The > fact > top runs at 200% means that your threads are hampered by something > else. > This could for example be the reading of particles from the hard > disk, > which can become a bottle neck. Also: how many cores does > nsit-dhcp-148-090.bsd.uchicago.edu have? You're running 4 MPI > slaves, > each with 4 threads on it. The master also takes 1 core. Therefore, > your > machine should have 17 cores to do everything you ask for. If it has > fewer cores, then they'll just be in each others way. > HTH, > Sjors > > On 04/27/2016 08:18 PM, Baru Reddy wrote: > > > > Hi Sjors, > > 'top' says each mpi process is running at ~200%. This is the > > criteria by which I say it is only using 2 threads is based on the > > ~200%. The command I use and initial output I get is shown below. > > > > mpirun -n 5 ~/Downloads/relion-1.4/bin/relion_refine_mpi --o > > Class3D/run1_ct5 --continue Class3D/run1_it005_optimiser.star -- > > iter 25 --tau2_fudge 4 --solvent_mask proteasome_mask_150.mrc -- > > oversampling 1 --healpix_order 3 --offset_range 5 --offset_step 2 > > --j 4 & > > > > [reddybg@nsit-dhcp-148-090 gauto]$ === RELION MPI setup === > > + Number of MPI processes = 5 > > + Number of threads per MPI process = 4 > > + Total number of threads therefore = 20 > > + Master (0) runs on host = nsit-dhcp-148- > > 090.bsd.uchicago.edu > > + Slave 1 runs on host = nsit-dhcp-148- > > 090.bsd.uchicago.edu > > + Slave 2 runs on host = nsit-dhcp-148- > > 090.bsd.uchicago.edu > > + Slave 3 runs on host = nsit-dhcp-148- > > 090.bsd.uchicago.edu > > + Slave 4 runs on host = nsit-dhcp-148- > > 090.bsd.uchicago.edu > > > > Cheers,Bharat ReddyPost DocUniversity of Chicago > > > > From: Sjors Scheres <[log in to unmask]> > > To: Baru Reddy <[log in to unmask]> > > Cc: [log in to unmask] > > Sent: Wednesday, April 27, 2016 2:08 PM > > Subject: Re: [ccpem] Stuck at 2 Threads per MPI Process > > > > Hi Bharat, > > --j N should always launch N threads. You'll only see them as 1 > > process in > > 'top', but it may run up to ~N00%. Why do you say relion launches > > only 2 > > threads? How do you see this? Does it say so in the stdout? > > S > > > > > > > > Hi Everyone, > > > Currently we are trying to mobilize the power of threads as our > > > refinements have become more memory intensive and we have hit a > > > limit with > > > the number of MPI processes we can deploy. The problem is that > > > however > > > many threads I tell relion_refine_mpi to use (-j X where X is > > > 4,8,16,etc.), it only uses two threads. Is there a setting I am > > > missing, a > > > variable I am failing to define, or is this a limit of > > > relion_refine_mpi . > > > Cheers,Bharat ReddyPost DocUniversity of Chicago