One could add more EXTRA variables to the GUI by editing below line 212 
of the src/gui_jobwindow.cpp.
Alternatively, one could use a trick by using the standard linux command 
'cat' instead of 'qsub' (or the likes) for the "Queue submit command:". 
That way, when pressing "Run now" the script gets written to disc, the 
job gets incorporated into the pipeline, but the job actually does not 
get submitted. One can then edit the script and submit it manually. I 
think I saw this first being done by Tom Houweling in the Nogales lab.
HTH,
Sjors

On 01/26/2017 11:17 PM, Elad Binshtein wrote:
> Hi Daniel,
>
> I'm agree that more XXXextraNXXX variables will be nice.
> for know I have default val. for GPU that I'm edit before running the job
> (if I want more GPU).
> I can run more that 1 GPU node.
> you can also use the XXXdedicatedXXX as an extra variables
>
> Best,
>
> On Thu, Jan 26, 2017 at 4:43 PM, Daniel Larsson <[log in to unmask]>
> wrote:
>
>> Hi Elad,
>>
>> Thanks for your suggestion about the extra variables. I also figured I
>> should use the —cpus-per-task slurm variable to get the right number of
>> processes and threads. I now have a script that seems to work ok. However I
>> cannot run multiple nodes with GPUs. Is it just me (or our hardware), or is
>> that feature not implemented yet?
>>
>> In my mind, there is still room for additional XXXextraNXXX variables,
>> e.g. for setting memory usage and max wall time, like you did. (My guess is
>> that the GUI could only fit two more lines, so therefore we can only have 2
>> extra variables.) It would also be nice if one could toggle the numeric
>> value slider for the extra variables.
>>
>>
>> For posterity, this is what I ended up with:
>>
>> #!/bin/bash
>> #SBATCH --job-name=XXXnameXXX
>> #SBATCH -NXXXextra1XXX
>> #SBATCH --ntasks-per-node=XXXmpinodesXXX
>> #SBATCH --cpus-per-task=XXXthreadsXXX
>> #SBATCH -p XXXqueueXXX
>> #SBATCH --gres=gpu:XXXextra2XXX
>> #SBATCH --output=XXXnameXXXslurm-%j.out
>> mpirun XXXcommandXXX
>>
>> Using these environmental variables:
>>
>> export RELION_QSUB_EXTRA1="No. of nodes"
>> export RELION_QSUB_EXTRA2="No. of GPUs/node"
>> export RELION_QSUB_EXTRA1_DEFAULT="1"
>> export RELION_QSUB_EXTRA2_DEFAULT=“0"
>>
>>
>> Regards,
>> Daniel
>>
>>
>> On 25 Jan 2017, at 22:29, Elad Binshtein <[log in to unmask]> wrote:
>>
>> Hi Daniel,
>> my script look like this:
>>
>> #SBATCH --partition=  XXXqueueXXX
>> #SBATCH --account=accout_gpu
>> #SBATCH --gres=gpu:2
>> #SBATCH --ntasks=XXXmpinodesXXX
>> #SBATCH --cpus-per-task=XXXthreadsXXX
>> #SBATCH --time=XXXextra1XXX
>> #SBATCH --mem-per-cpu=XXXextra2XXX
>> #SBATCH -J XXXoutfileXXX
>> #SBATCH --error=XXXerrfileXXX
>> #SBATCH --output=XXXoutfileXXX
>> srun --mpi=pmi2 XXXcommandXXX
>>
>> you can add 2 extra variables and use them as you want.
>>
>> Best,
>>
>>
>>
>> On Wed, Jan 25, 2017 at 3:04 PM, Daniel Larsson <[log in to unmask]>
>> wrote:
>>
>>> Hi all,
>>>
>>> I have a few thoughts regarding batch jobs using multiple nodes. From my
>>> research, the mapping between the GUI parameters and the batch job
>>> variables seems to be:
>>>
>>> Number of MPI procs => XXXmpinodesXXX
>>> Number of MPI procs => XXXnodesXXX
>>> Number of threads => XXXthreadsXXX
>>> Number of MPI procs * Number of threads => XXXcoresXXX
>>> Minimum dedicated cores per node => XXXdedicatedXXX
>>> XXXthreadsXXX => sets the -j flag of the relion_refine command
>>>
>>> My current template scripts to run jobs on a single node on our SLURM
>>> cluster looks like this (for single node job):
>>>
>>> #!/bin/bash
>>> #SBATCH --job-name=XXXnameXXX
>>> #SBATCH -N1
>>> #SBATCH --ntasks-per-node=XXXcoresXXX
>>> #SBATCH -p c
>>> #SBATCH --gres=gpu:4
>>> #SBATCH --output=XXXnameXXX/slurm-%j.out
>>> mpirun XXXcommandXXX
>>>
>>> For two nodes I use this:
>>>
>>> #!/bin/bash
>>> #SBATCH --job-name=XXXnameXXX
>>> #SBATCH -N2
>>> #SBATCH --ntasks-per-node=XXXcoresXXX
>>> #SBATCH -p c
>>> #SBATCH --gres=gpu:4
>>> #SBATCH --output=XXXnameXXX/slurm-%j.out
>>> mpirun XXXcommandXXX
>>>
>>> The two-node version works I have one thread per MPI process. But for
>>> more, there is no way to separately define -j and XXXcoresXXX, since -j
>>> seems to implicitly be controlled by XXXthreadsXXX and XXXcoresXXX seems to
>>> be defined by XXXmpinodesXXX * XXXthreadsXXX. This causes "XXXmpinodesXXX *
>>> XXXthreadsXXX” MPI processes to be spawned on each node.
>>>
>>> Please suggest improvements to my script if there is something I have
>>> overlooked. Maybe I can use the XXXdedicatedXXX to set the variable
>>> "#SBATCH --ntasks-per-node” instead and thereby be able to control it using
>>> the “Minimum dedicated cores per node” slider in the GUI? That is a bit
>>> ugly according to me.
>>>
>>> My suggestions are that there should be:
>>> - independent ways to set XXXcoresXXX (which controls -j) and
>>> XXXthreadsXXX from the GUI
>>> - independent ways to set XXXnodesXXX and XXXmpinodesXXX from the GUI (so
>>> that I don’t have to use separate scripts for different N)
>>> - XXXmpinodesXXX should be renamed to XXXmpiprocsXXX.
>>>
>>> Regards,
>>> Daniel
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> ________________________________
>> Elad Binshtein, Ph.D.
>> Cryo EM specialist - staff scientist
>> Center for Structure Biology (CSB)
>> MCN Room 1207
>> Vanderbilt University
>> Nashville, TN
>> Office: +1-615-322-4671 <(615)%20322-4671>
>> Mobile: +1-615-481-4408 <(615)%20481-4408>
>> E-Mail: [log in to unmask]
>> ________________________________
>>
>>
>>
>

-- 
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres