Sjors-
Just a quick note, I had a problem when doing the subvolume extraction step in the Tomo workflow. I noticed a progressive
memory leak; I had to run on a node of the cluster with 256 Gb of memory (but slower IO than the Master)
and the used memory started quite low and quickly used up
about 230 Gb (10% of job done), however, the job did finish doing an extraction of 6400 subvols from the +100 tomograms.
To speed things up I broke the extraction into 4 chunks running on 4 nodes and after the jobs finished I had to
reboot the nodes and run: sge_execd to bring each node back to an active state in the queue (SGI).
The extraction job would not run on the Master for my system (it was very slow), since its memory is pretty maxed out at baseline
and I have to upgrade this at the next rebuild, hopefully when we get some gpu-cpu nodes.
C Akey
Prof.
|