Chris,
If you have 42k particles of 12 frames you should end up with 42kx12 lines
in your movie star file. If not, you've done something wrong.
The new cluster sounds good. Just ask them how to submit hydridly parallel
jobs: that's different on each system.
HTH,
S
> Sjors
>
> I agree that there is not enough memory on my cluster nodes, even if I
> divide the job in half, which is really hard to believe.
>
> It is so strange, that the process takes only 1 core on each node on my
> cluster and then starts to take more and more memory as it is "Expanding
> current model for movie frames... " until it breaks the system by taking
> all the memory and you get an error that dumps the job.
>
> Because of the way the program is written you cannot just restart it.
>
> I have tried running it on our big cluster at BU (the shared compute
> cluster) which has lots of memory per node, but it automatically uses all
> the cores on each node assigned to the job and then uses up the available
> memory and obviously this is not going to work.
>
> I do not believe that I can specify memory per thread with the movie
> refine command>
>
> so Not sure how to tell the BU cluster to set up the job with at most 2
> processes per node so that there will be enough memory.
>
> This is a big cluster, I am use the 16 core node queue and these nodes
> have 128 and some 256 Gb. So in fact I ought to be able to run between 2
> and 4 process per node depending upon the memory, but how to specify this?
>
> I still don't understand how my post doc got a job to run on our lab
> cluster at all, after looking through all of his notes.
>
> I am beginning to wonder if the movie particle extraction step has messed
> up, because I don't understand why the movie.star file would have over 7
> million lines, (the 3400 project movies have 12 frames and there are 42K
> ptcls)
>
> please explain.
>
> I did see that my postdocs job which ran on our cluster used a movie file
> with 4 million lines...so again very puzzled and now getting frazzled.
>
> C Akey
>
>
>
>
>> Expanding finds all movie frames for each particle based on filename
>> conventions, and then gives them the orientations from the previous
>> alignment.
>> If too large STAR files are a problem for you, you could also average
>> every 2nd movie frame. That's an option on the extract tab I think.
>> A cluster with better disc I/O (70hrs extracting movie particles is
>> ridiculous), more stable MPI connections, and in particular more RAM
>> would
>> be a good idea. It sounds your cluster isn't very up-to-date anymore.
>> 200Gb of space is very limited though. However, once you've done the
>> movie
>> particles, perhaps its let space?
>>
>> Good luck,
>> S
>>
>>
>>> Sjors
>>>
>>> Ok, yes I am using the 1.4beta >
>>>
>>> I had extracted my movie ptcls using the group star file as listed
>>> below
>>> and this star file has ONLY good ptcls in it ie
>>>
>>> grep @Part cl-2d-3d-2_all_group.star | wc -l > 42145 good ptcls
>>>
>>> So no I hadn't run your script before the movie ptcl align step.
>>>
>>> The extract step is listed below, which needed 1.4beta to run and give
>>> the
>>> movie.star file, I was stuck at this stage quite a while until your
>>> help.
>>>
>>> I would like to know what the expanding step is doing?
>>>
>>> While I still don't understand why a slightly smaller job ran under 1.3
>>> in
>>> some 15 hrs on our cluster, apparently using both big and small memory
>>> nodes, this job crashed out again last nite with the full data set on
>>> one
>>> of the small nodes.
>>>
>>> Its hard to think that the internal IO over the ethernet between nodes
>>> and
>>> the Master has changed inside the cluster; the data files are all on an
>>> internal RAID, as usual.
>>>
>>> --------------------------------------------------------------------------
>>> mpiexec noticed that process rank 11 with PID 5256 on node
>>> compute-0-2.local exited on signal 9 (Killed).
>>> --------------------------------------------------------------------------
>>>
>>> For the job that just crashed, I had gone back and clipped the movie
>>> frames in the mrcs from 348 to 320, since this saves 15% memory and I
>>> plan
>>> to process two data sets together, one previous from over holes and one
>>> on
>>> c-film to minimize a preferred orientation problem, and the previous
>>> data
>>> were processed at 320 box size.
>>>
>>> The total size of 320 frame movie ptcl mrcs is 200Gb.
>>>
>>> I am thinking to try and move this to our BU SCC machine, but the disk
>>> limit there is 200Gb, but I can ask for more temporarily and try the
>>> job
>>> there. If I do this, should I install rel 1.4-beta on the cluster and
>>> use
>>> it since you indicate that it is faster?
>>>
>>> Just stuck in the movie processing phase here for quite some time now,
>>> and
>>> a grant renewal looming....
>>>
>>>
>>> thanks for all your time.
>>>
>>> C Akey
>>>
>>>
>>>
>>> in/home/guettler/relion-1.4-beta-1 > make install <cr> can now run with
>>> new options etc.
>>>
>>> `which relion_preprocess` --o cl_1-3 --mic_star
>>> cl-2d-3d-2_all_group.star
>>> --coord_suffix _cl-1-3_e2box.star --extract --extract_size 348
>>> --extract_movies --movie_rootname movie --first_movie_frame 1
>>> --last_movie_frame 12 --norm --bg_radius 150 --white_dust -1
>>> --black_dust
>>> -1 --invert_contrast
>>>
>>> 78.64/78.64 hrs
>>> ............................................................~~(,_,">
>>> Joining all metadata in one STAR file...
>>> Written out STAR file with all particles in cl_1-3_movie.star
>>> Done!
>>>
>>>>>> cl_1-3_movie.star 2.1G May 3 !!!!!
>>>
>>> grep @Particle cl_1-3_movie.star | wc -l > 7578396
>>>
>>>
>>> _________________________________________________________________________
>>>
>>>> Hi Chris,
>>>> Have you tried the script on the Wiki that speeds up the movie
>>>> expansion
>>>> step?
>>>> http://www2.mrc-lmb.cam.ac.uk/relion/index.php/FAQs#The_expansion_of_my_movie_frames_takes_forever._What_can_I_do.3F
>>>> HTH,
>>>> S
>>>>
>>>>> Sjors and users:
>>>>>
>>>>> With regards to memory issues and movie ptcl processing.
>>>>>
>>>>> We recently ran a movie align job > realign_movie_frames as shown in
>>>>> the
>>>>> command below with about 35000 ptcls at 320 box size and it ran on
>>>>> our
>>>>> cluster in ~15 hrs on all the cores without a memory problem for the
>>>>> full
>>>>> job!
>>>>>
>>>>> mpiexec -bynode -n 12 `which relion_refine_mpi` --o
>>>>> Refine3D/run1_ct26
>>>>> --continue Refine3D/lsu+70S/run1_ct12_it026_optimiser.star
>>>>> --oversampling
>>>>> 1 --healpix_order 2 --auto_local_healpix_order 4 --offset_range 5
>>>>> --offset_step 2 --realign_movie_frames e2multi1+3_movie.star
>>>>> --movie_frames_running_avg 5 --sigma_off 2 --skip_rotate
>>>>> --skip_maximize
>>>>> --j 8
>>>>>
>>>>> in the .out file>
>>>>>
>>>>> === RELION MPI setup ===
>>>>> + Number of MPI processes = 12
>>>>> + Number of threads per MPI process = 8
>>>>> + Total number of threads therefore = 96
>>>>> .
>>>>> .
>>>>> .
>>>>> Expanding current model for movie frames...
>>>>> Auto-refine: Iteration= 27
>>>>> Auto-refine: Resolution= 9.67442 (no gain for 0 iter)
>>>>> Auto-refine: Changes in angles= 999 degrees; and in offsets= 999
>>>>> pixels
>>>>> (no gain for 0 iter)
>>>>> Auto-refine: Refinement has converged, entering last iteration where
>>>>> two
>>>>> halves will be combined...
>>>>> Auto-refine: Angular step= 15 degrees; local searches= false
>>>>> Auto-refine: Offset search range= 6 pixels; offset step= 1.5 pixels
>>>>> CurrentResolution= 9.67442 Angstroms, which requires
>>>>> orientationSampling
>>>>> of at least 3.07692 degrees for a particle of diameter 360 Angstroms
>>>>> Oversampling= 0 NrHiddenVariableSamplingPoints= 49
>>>>> OrientationalSampling= 15 NrOrientations= 1
>>>>> TranslationalSampling= 1.5 NrTranslations= 49
>>>>> =============================
>>>>> Estimated memory for expectation step > 0.472081 Gb, available
>>>>> memory
>>>>> =
>>>>> 12 Gb.
>>>>> Estimated memory for maximization step > 0.669746 Gb, available
>>>>> memory
>>>>> =
>>>>> 12 Gb.
>>>>> Expectation iteration 27
>>>>> 14.39/14.39 hrs
>>>>> ............................................................~~(,_,">
>>>>> Auto-refine: Skipping maximization step, so stopping now...
>>>>>
>>>>> Hoever, I am running a job now with 42K ptcls at 348 box size with
>>>>> the
>>>>> same cluster and I had to divide the job in two halves (I made two
>>>>> independent movie.star files from the .star and movie.mrcs ptcl
>>>>> files)
>>>>> to
>>>>> get it to even run and the half job has now run 48 hrs with no sign
>>>>> of
>>>>> progressing past the expansion step:
>>>>>
>>>>> So something must be wrong but I am at a loss what to try. The job is
>>>>> using only one core on each of the 6 nodes, and this core is using
>>>>> 30Gb
>>>>> of
>>>>> total 32 Gb on each node for one thread/core. Any suggestions as to
>>>>> why
>>>>> there is such a big difference, at this rate it will take a very long
>>>>> time, there is of course no indication whatsoever of how far along
>>>>> the
>>>>> job
>>>>> is in the expansion phase (not sure either what it is doing at this
>>>>> step).
>>>>>
>>>>> === RELION MPI setup ===
>>>>> + Number of MPI processes = 6
>>>>> + Number of threads per MPI process = 12
>>>>> + Total number of threads therefore = 72
>>>>> + Master (0) runs on host = compute-0-11.local
>>>>> + Slave 1 runs on host = compute-0-6.local
>>>>> + Slave 4 runs on host = compute-0-8.local
>>>>> =================
>>>>> + Slave 5 runs on host = compute-0-10.local
>>>>> + Slave 2 runs on host = compute-0-7.local
>>>>> + Slave 3 runs on host = compute-0-9.local
>>>>> Expanding current model for movie frames...
>>>>>
>>>>> C Akey
>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 6 mai 2015 à 18:36, Sjors Scheres <[log in to unmask]> a
>>>>>>> écrit
>>>>>>> :
>>>>>>>
>>>>>>> Chris,
>>>>>>> The upcoming 1.4 release should be more conservative in memory use
>>>>>>> when
>>>>>>> reading in large STAR files of the movies.
>>>>>>> If you are the only person using each node, and only 1 job is
>>>>>>> running
>>>>>>> on
>>>>>>> it, then yes: you should be able to take the full 32Gb. For doing
>>>>>>> large
>>>>>>> refinements, you might consider finding a cluster with 64Gb nodes.
>>>>>>> That's what we typically use for our ribosome structures, so that
>>>>>>> should
>>>>>>> be enough. I think we've also seen problems using our 6-yrs old
>>>>>>> nodes
>>>>>>> with 32Gb.
>>>>>>> HTH,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>>> On 05/05/2015 04:13 PM, Christopher Akey wrote:
>>>>>>>> Users and Sjors:
>>>>>>>>
>>>>>>>> After looking at a post on the issue in March of 2015, it seemed
>>>>>>>> to
>>>>>>>> be
>>>>>>>> a memory issue.
>>>>>>>>
>>>>>>>> For the realign_movie_frames step even though specifying --j 12
>>>>>>>> threads
>>>>>>>> and 6 nodes
>>>>>>>> for a total of 72 threads, the job ran on 6 nodes using only 1
>>>>>>>> core
>>>>>>>> per
>>>>>>>> node, based on the top for each node,
>>>>>>>> so the amount of memory available per node was 32 Gb (2.7
>>>>>>>> Gb/core).
>>>>>>>>
>>>>>>>> However, since no intermediate files are produced, it is dealing
>>>>>>>> with
>>>>>>>> a
>>>>>>>> large amount of data since the input
>>>>>>>> movie star file has a huge number of entries/lines:
>>>>>>>>
>>>>>>>> grep @Particle cl_1-3_movie.star | wc -l
>>>>>>>> 7578396
>>>>>>>>
>>>>>>>> this seems very inefficient.
>>>>>>>>
>>>>>>>> I have 12 frames per movie at about 2.2 e/A2/frame so I am
>>>>>>>> averaging
>>>>>>>> over 3 frames as the ptcls are fairly dense and big.
>>>>>>>>
>>>>>>>> I don't see how to get any more memory for this job.
>>>>>>>>
>>>>>>>> Under these conditions is the job able to use all of the 32Gb per
>>>>>>>> node
>>>>>>>> since only one core on each node is active at 100% ?
>>>>>>>>
>>>>>>>> C Akey
>>>>>>
>>>>>> --
>>>>>> Sjors Scheres
>>>>>> MRC Laboratory of Molecular Biology
>>>>>> Francis Crick Avenue, Cambridge Biomedical Campus
>>>>>> Cambridge CB2 0QH, U.K.
>>>>>> tel: +44 (0)1223 267061
>>>>>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sjors Scheres
>>>> MRC Laboratory of Molecular Biology
>>>> Francis Crick Avenue, Cambridge Biomedical Campus
>>>> Cambridge CB2 0QH, U.K.
>>>> tel: +44 (0)1223 267061
>>>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Sjors Scheres
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue, Cambridge Biomedical Campus
>> Cambridge CB2 0QH, U.K.
>> tel: +44 (0)1223 267061
>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>
>
>
>
--
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres
|