Dear all,
We are still struggling with this - it is very frustrating that with 496 pixel box the last maximization iteration in autorefine takes 2-3-4 days (and apparently nothing happens during this time, no progress output, though CPUs are used).
We have plenty of CPUs (usually we use ~17 MPIs with 15 threads = 255 threads per job) and memory (128 GB per node with 32 hyper-threaded cores), so there is no swapping to disk. Memory requested by Relion in the last iteration is about 30GB.
I wonder if people could share their examples of how long this iteration takes on their set-up, especially with large box of about 500 pixels?
And whether anybody resolved similar problem?
Many thanks!
>Hi Leo,
It also puts pixels until Nyquist back into the 3D transform, so will cost
more CPU than the other iterations.
HTH
Sjors
> Hi, still an important question for us -
> It does not look like overall I/O cluster load is a big issue and memory
> also is not an issue.
> What else can be done to speed up the last iteration in 3D autorefine (496
> box, 128 GB memory per node)?
> Now it takes up to several days so we really want to do something about
> it.
> Apart from using more memory per image, what else is different about the
> last 3D autorefine operation so that it is so slow?
>
> Many thanks!
>
>
>
> On our cluster we started to get exceedingly long times for the last
> iteration in 3D autorefine (with large box). There is definitely enough
> RAM so there is no swapping. Previously the same jobs run about 10X faster
> on our cluster, so I wonder if the problem is in general I/O bottlenecks
> in the cluster.
> Is there a lot of particle images reading in the final maximisation step
> (takes up to a day now)?
> Thanks!
>
|