JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCPEM Archives


CCPEM Archives

CCPEM Archives


CCPEM@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCPEM Home

CCPEM Home

CCPEM  May 2015

CCPEM May 2015

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: memory issue with refinement of movie particles? from C Akey

From:

Sjors Scheres <[log in to unmask]>

Reply-To:

Sjors Scheres <[log in to unmask]>

Date:

Wed, 13 May 2015 08:13:03 +0100

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (378 lines)

Chris,
If you have 42k particles of 12 frames you should end up with 42kx12 lines
in your movie star file. If not, you've done something wrong.
The new cluster sounds good. Just ask them how to submit hydridly parallel
jobs: that's different on each system.
HTH,
S

> Sjors
>
> I agree that there is not enough memory on my cluster nodes, even if I
> divide the job in half, which is really hard to believe.
>
> It is so strange, that the process takes only 1 core on each node on my
> cluster and then starts to take more and more memory as it is "Expanding
> current model for movie frames... " until it breaks the system by taking
> all the memory and you get an error that dumps the job.
>
> Because of the way the program is written you cannot just restart it.
>
> I have tried running it on our big cluster at BU (the shared compute
> cluster) which has lots of memory per node, but it automatically uses all
> the cores on each node assigned to the job and then uses up the available
> memory and obviously this is not going to work.
>
> I do not believe that I can specify memory per thread with the movie
> refine command>
>
> so Not sure how to tell the BU cluster to set up the job with at most 2
> processes per node so that there will be enough memory.
>
> This is a big cluster, I am use the 16 core node queue and these nodes
> have 128 and some 256 Gb. So in fact I ought to be able to run between 2
> and 4 process per node depending upon the memory, but how to specify this?
>
> I still don't understand how my post doc got a job to run on our lab
> cluster at all, after looking through all of his notes.
>
> I am beginning to wonder if the movie particle extraction step has messed
> up, because I don't understand why the movie.star file would have over 7
> million lines, (the 3400 project movies have 12 frames and there are 42K
> ptcls)
>
> please explain.
>
>  I did see that my postdocs job which ran on our cluster used a movie file
> with 4 million lines...so again very puzzled and now getting frazzled.
>
> C Akey
>
>
>
>
>> Expanding finds all movie frames for each particle based on filename
>> conventions, and then gives them the orientations from the previous
>> alignment.
>> If too large STAR files are a problem for you, you could also average
>> every 2nd movie frame. That's an option on the extract tab I think.
>> A cluster with better disc I/O (70hrs extracting movie particles is
>> ridiculous), more stable MPI connections, and in particular more RAM
>> would
>> be a good idea. It sounds your cluster isn't very up-to-date anymore.
>> 200Gb of space is very limited though. However, once you've done the
>> movie
>> particles, perhaps its let space?
>>
>> Good luck,
>> S
>>
>>
>>> Sjors
>>>
>>> Ok, yes I am using the 1.4beta >
>>>
>>> I had extracted my movie ptcls using the group star file as listed
>>> below
>>> and this star file has ONLY good ptcls in it ie
>>>
>>> grep @Part cl-2d-3d-2_all_group.star | wc -l > 42145 good ptcls
>>>
>>> So no I hadn't run your script before the movie ptcl align step.
>>>
>>> The extract step is listed below, which needed 1.4beta to run and give
>>> the
>>> movie.star file, I was stuck at this stage quite a while until your
>>> help.
>>>
>>> I would like to know what the expanding step is doing?
>>>
>>> While I still don't understand why a slightly smaller job ran under 1.3
>>> in
>>> some 15 hrs on our cluster, apparently using both big and small memory
>>> nodes, this job crashed out again last nite with the full data set on
>>> one
>>> of the small nodes.
>>>
>>> Its hard to think that the internal IO over the ethernet between nodes
>>> and
>>> the Master has changed inside the cluster; the data files are all on an
>>> internal RAID, as usual.
>>>
>>> --------------------------------------------------------------------------
>>> mpiexec noticed that process rank 11 with PID 5256 on node
>>> compute-0-2.local exited on signal 9 (Killed).
>>> --------------------------------------------------------------------------
>>>
>>> For the job that just crashed, I had gone back and clipped the movie
>>> frames in the mrcs from 348 to 320, since this saves 15% memory and I
>>> plan
>>> to process two data sets together, one previous from over holes and one
>>> on
>>> c-film to minimize a preferred orientation problem, and the previous
>>> data
>>> were processed at 320 box size.
>>>
>>> The total size of 320 frame movie ptcl mrcs is 200Gb.
>>>
>>> I am thinking to try and move this to our BU SCC machine, but the disk
>>> limit there is 200Gb, but I can ask for more temporarily and try the
>>> job
>>> there. If I do this, should I install rel 1.4-beta on the cluster and
>>> use
>>> it since you indicate that it is faster?
>>>
>>> Just stuck in the movie processing phase here for quite some time now,
>>> and
>>> a grant renewal looming....
>>>
>>>
>>> thanks for all your time.
>>>
>>> C Akey
>>>
>>>
>>>
>>> in/home/guettler/relion-1.4-beta-1 > make install <cr> can now run with
>>> new options etc.
>>>
>>> `which relion_preprocess` --o cl_1-3 --mic_star
>>> cl-2d-3d-2_all_group.star
>>> --coord_suffix _cl-1-3_e2box.star --extract --extract_size 348
>>> --extract_movies --movie_rootname movie --first_movie_frame 1
>>> --last_movie_frame 12 --norm --bg_radius 150 --white_dust -1
>>> --black_dust
>>> -1 --invert_contrast
>>>
>>> 78.64/78.64 hrs
>>> ............................................................~~(,_,">
>>>  Joining all metadata in one STAR file...
>>>  Written out STAR file with all particles in cl_1-3_movie.star
>>>  Done!
>>>
>>>>>> cl_1-3_movie.star  2.1G May  3  !!!!!
>>>
>>> grep @Particle cl_1-3_movie.star | wc -l > 7578396
>>>
>>>
>>> _________________________________________________________________________
>>>
>>>> Hi Chris,
>>>> Have you tried the script on the Wiki that speeds up the movie
>>>> expansion
>>>> step?
>>>> http://www2.mrc-lmb.cam.ac.uk/relion/index.php/FAQs#The_expansion_of_my_movie_frames_takes_forever._What_can_I_do.3F
>>>> HTH,
>>>> S
>>>>
>>>>> Sjors and users:
>>>>>
>>>>> With regards to memory issues and movie ptcl processing.
>>>>>
>>>>> We recently ran a movie align job > realign_movie_frames as shown in
>>>>> the
>>>>> command below with about 35000 ptcls at 320 box size and it ran on
>>>>> our
>>>>> cluster in ~15 hrs on all the cores without a memory problem for the
>>>>> full
>>>>> job!
>>>>>
>>>>> mpiexec -bynode -n 12 `which relion_refine_mpi` --o
>>>>> Refine3D/run1_ct26
>>>>> --continue Refine3D/lsu+70S/run1_ct12_it026_optimiser.star
>>>>> --oversampling
>>>>> 1 --healpix_order 2 --auto_local_healpix_order 4 --offset_range 5
>>>>> --offset_step 2 --realign_movie_frames e2multi1+3_movie.star
>>>>> --movie_frames_running_avg 5 --sigma_off 2 --skip_rotate
>>>>> --skip_maximize
>>>>> --j 8
>>>>>
>>>>> in the .out file>
>>>>>
>>>>> === RELION MPI setup ===
>>>>>  + Number of MPI processes             = 12
>>>>>  + Number of threads per MPI process  = 8
>>>>>  + Total number of threads therefore  = 96
>>>>> .
>>>>> .
>>>>> .
>>>>>  Expanding current model for movie frames...
>>>>>  Auto-refine: Iteration= 27
>>>>>  Auto-refine: Resolution= 9.67442 (no gain for 0 iter)
>>>>>  Auto-refine: Changes in angles= 999 degrees; and in offsets= 999
>>>>> pixels
>>>>> (no gain for 0 iter)
>>>>>  Auto-refine: Refinement has converged, entering last iteration where
>>>>> two
>>>>> halves will be combined...
>>>>>  Auto-refine: Angular step= 15 degrees; local searches= false
>>>>>  Auto-refine: Offset search range= 6 pixels; offset step= 1.5 pixels
>>>>>  CurrentResolution= 9.67442 Angstroms, which requires
>>>>> orientationSampling
>>>>> of at least 3.07692 degrees for a particle of diameter 360 Angstroms
>>>>>  Oversampling= 0 NrHiddenVariableSamplingPoints= 49
>>>>>  OrientationalSampling= 15 NrOrientations= 1
>>>>>  TranslationalSampling= 1.5 NrTranslations= 49
>>>>> =============================
>>>>>  Estimated memory for expectation step  > 0.472081 Gb, available
>>>>> memory
>>>>> =
>>>>> 12 Gb.
>>>>>  Estimated memory for maximization step > 0.669746 Gb, available
>>>>> memory
>>>>> =
>>>>> 12 Gb.
>>>>>  Expectation iteration 27
>>>>> 14.39/14.39 hrs
>>>>> ............................................................~~(,_,">
>>>>>  Auto-refine: Skipping maximization step, so stopping now...
>>>>>
>>>>> Hoever, I am running a job now with 42K ptcls at 348 box size with
>>>>> the
>>>>> same cluster and I had to divide the job in two halves (I made two
>>>>> independent movie.star files from the .star and movie.mrcs ptcl
>>>>> files)
>>>>> to
>>>>> get it to even run and the half job has now run 48 hrs with no sign
>>>>> of
>>>>> progressing past the expansion step:
>>>>>
>>>>> So something must be wrong but I am at a loss what to try. The job is
>>>>> using only one core on each of the 6 nodes, and this core is using
>>>>> 30Gb
>>>>> of
>>>>> total 32 Gb on each node for one thread/core. Any suggestions as to
>>>>> why
>>>>> there is such a big difference, at this rate it will take a very long
>>>>> time, there is of course no indication whatsoever of how far along
>>>>> the
>>>>> job
>>>>> is in the expansion phase (not sure either what it is doing at this
>>>>> step).
>>>>>
>>>>> === RELION MPI setup ===
>>>>>  + Number of MPI processes             = 6
>>>>>  + Number of threads per MPI process  = 12
>>>>>  + Total number of threads therefore  = 72
>>>>>  + Master  (0) runs on host            = compute-0-11.local
>>>>>  + Slave     1 runs on host            = compute-0-6.local
>>>>>  + Slave     4 runs on host            = compute-0-8.local
>>>>>  =================
>>>>>  + Slave     5 runs on host            = compute-0-10.local
>>>>>  + Slave     2 runs on host            = compute-0-7.local
>>>>>  + Slave     3 runs on host            = compute-0-9.local
>>>>>  Expanding current model for movie frames...
>>>>>
>>>>> C Akey
>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 6 mai 2015 à 18:36, Sjors Scheres <[log in to unmask]> a
>>>>>>> écrit
>>>>>>> :
>>>>>>>
>>>>>>> Chris,
>>>>>>> The upcoming 1.4 release should be more conservative in memory use
>>>>>>> when
>>>>>>> reading in large STAR files of the movies.
>>>>>>> If you are the only person using each node, and only 1 job is
>>>>>>> running
>>>>>>> on
>>>>>>> it, then yes: you should be able to take the full 32Gb. For doing
>>>>>>> large
>>>>>>> refinements, you might consider finding a cluster with 64Gb nodes.
>>>>>>> That's what we typically use for our ribosome structures, so that
>>>>>>> should
>>>>>>> be enough. I think we've also seen problems using our 6-yrs old
>>>>>>> nodes
>>>>>>> with 32Gb.
>>>>>>> HTH,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>>> On 05/05/2015 04:13 PM, Christopher Akey wrote:
>>>>>>>> Users and Sjors:
>>>>>>>>
>>>>>>>> After looking at a post on the issue in March of 2015, it seemed
>>>>>>>> to
>>>>>>>> be
>>>>>>>> a memory issue.
>>>>>>>>
>>>>>>>> For the realign_movie_frames step even though specifying --j 12
>>>>>>>> threads
>>>>>>>> and 6 nodes
>>>>>>>> for a total of 72 threads, the job ran on 6 nodes using only 1
>>>>>>>> core
>>>>>>>> per
>>>>>>>> node, based on the top for each node,
>>>>>>>> so the amount of memory available per node was 32 Gb (2.7
>>>>>>>> Gb/core).
>>>>>>>>
>>>>>>>> However, since no intermediate files are produced, it is dealing
>>>>>>>> with
>>>>>>>> a
>>>>>>>> large amount of data since the input
>>>>>>>> movie star file has a huge number of entries/lines:
>>>>>>>>
>>>>>>>> grep @Particle cl_1-3_movie.star | wc -l
>>>>>>>> 7578396
>>>>>>>>
>>>>>>>> this seems very inefficient.
>>>>>>>>
>>>>>>>> I have 12 frames per movie at about 2.2 e/A2/frame so I am
>>>>>>>> averaging
>>>>>>>> over 3 frames as the ptcls are fairly dense and big.
>>>>>>>>
>>>>>>>> I don't see how to get any more memory for this job.
>>>>>>>>
>>>>>>>> Under these conditions is the job able to use all of the 32Gb per
>>>>>>>> node
>>>>>>>> since only one core on each node is active at 100% ?
>>>>>>>>
>>>>>>>> C Akey
>>>>>>
>>>>>> --
>>>>>> Sjors Scheres
>>>>>> MRC Laboratory of Molecular Biology
>>>>>> Francis Crick Avenue, Cambridge Biomedical Campus
>>>>>> Cambridge CB2 0QH, U.K.
>>>>>> tel: +44 (0)1223 267061
>>>>>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sjors Scheres
>>>> MRC Laboratory of Molecular Biology
>>>> Francis Crick Avenue, Cambridge Biomedical Campus
>>>> Cambridge CB2 0QH, U.K.
>>>> tel: +44 (0)1223 267061
>>>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Sjors Scheres
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue, Cambridge Biomedical Campus
>> Cambridge CB2 0QH, U.K.
>> tel: +44 (0)1223 267061
>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>
>
>
>


-- 
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

For help and support help@jisc.ac.uk

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager