Hello,
Vangelis Koukis wrote:
> I've been following Cal's very good instructions from
>
> http://goc.grid.sinica.edu.tw/gocwiki/MPI_Support_with_Torque
>
> in order to get a working Torque/mpiexec setup on our site,
> HG-01-GRNET, under LCG 2.4.0. Everything seems to be working
> OK, however there are a few points which I don't think I have
> understood completely regarding the way the middleware
> interacts with MPICH.
>
> I've installed the latest version of Torque, 1.2.0p3, which
> seems to work without any problems, based on Cal's original
> source RPMs. The changes needed were trivial, and the TMPDIR
> patch applies cleanly to this version. The resulting files
> can be found at:
>
> http://www.cslab.ece.ntua.gr/~vkoukis/torque-mpi
Did Cal's version not include the $TMPDIR patch already? What changes have you made exactly?
> I've also setup the submit filter and changed the LRMS type
> to 'pbs', as suggested in the Wiki, in order to work around
> the problem with the RB not liking the 'torque' LRMSType.
Torque should also be supported as LRMS type for MPI jobs, but for now setting pbs solves the problem. This should be fixed though.
> MPICH Job submission works without problems, when using a JDL like:
>
> Type = "job";
> JobType = "MPICH";
> NodeNumber = 16;
> Executable = "hello";
> #Arguments ="a1";
> StdOutput = "hello.out";
> StdError = "hello.err";
> InputSandbox = {"hello"};
> OutputSandbox = {"hello.out","hello.err"}; Requirements =
> other.GlueCEUniqueID ==
> "ce01.isabella.grnet.gr:2119/jobmanager-pbs-dteam"
>
> and submitting the job with just "edg-job-submit --vo dteam mpi.jdl".
> "hello" is the actual executable of the application, produced
> by "mpicc -o hello hello.c".
> The point is, I am not sure on exactly *who* is responsible
> for calling mpirun, in order to spawn the processes of the
> parallel application.
Neither am I. I thought the pbs.pm script was doing this, but changing the mpirun definition for Globus did not help. The mpirun command is in fact put in somewhere else. Does anyone know where?
> In the GOC Wiki, it is suggested (there are many other
> sources of documentation suggesting something similar, too)
> using a test job which declares a shell script as the
> executable. The shell script then calls
> mpirun/mpiexec/whatever, at its discretion, whenever it
> thinks it's necessary. However, from what I have seen both
> using LCG2.3.1 and LCG2.4.0, mpirun is not meant to be called
> by a shell script, but is called directly by Torque.
This is something that works using the mpich mpirun. It is not the recommended way of running MPI jobs though. In fact this may easily lead to problems in the future, because mpirun is not meant to run scripts Therefore it may not work in a future version of mpich. Furthermore nested mpiruns can be dangerous in themselves by spawning n*n processes, instead of n.
Using mpirun multiple times does, at the moment, not seem to give problems though.
> So, to provide support for mpiexec, I had to replace the
> MPICH-provided /usr/bin/mpirun with a very small, very simple
> wrapper which discards all mpirun-specific arguments, and
> then calls mpiexec, as Fokke Dijkstra suggests in:
>
> http://www.listserv.rl.ac.uk/cgi-bin/webadmin?A2=ind0503&L=LCG
> -ROLLOUT&P=R45745&I=-3
We have done the same for a while, but stopped doing this. When a script is run using mpiexec you cannot call mpiexec in the script itself. Therefore when using a wrapper script for mpirun scripts that call mpirun will call mpiexec a second time and fail. This is caused by the fact that pbs_mom is used to control the processes and it can only be used for one process using mpiexec.
> The way mpirun is inserted in the job description submitted
> to Torque can be seen in in
> /opt/globus/lib/perl/Globus/GRAM/JobManager/pbs.pm on the CE,
> which constructs the submission script for Torque/PBS:
>
> if($description->jobtype() eq 'mpi')
> {
> print JOB "$mpirun -np ", $description->count(), ' ';
> if($cluster) {
> print JOB " -machinefile \$PBS_NODEFILE "; }
> print JOB $description->executable(), " $args < ",
> $description->stdin(), "\n";
> }
Someone decided that the mpirun call should be done somewhere else. So the above is probably only done for globus-job-submit jobs. For LCG jobs the mpirun is inserted elsewhere.
> It seems that the user has no control over whether mpirun is
> called or not. In fact, when I tried submitting a JDL with
> Cal's MPItest.sh as the executable, the small /usr/bin/mpirun wrapper
> was still called. This means that MPItest.sh gets to be used as the
> parallel application, which is wrong... Depending on the
> platform-specific way MPICH uses to spawn its processes, this
> could lead to multiple instances of it running around the cluster.
This is indeed a problem like described earlier. We should probably think of a different way to run MPI jobs, that allows for user scripts. These are needed when one wants to do preprocessing and postprocessing work before and after running the mpi executable. Think of things like data transfers.
Furthermore when a shared directory is not available for the job, explicit copies of the input data for the job are needed. This could also be done by the middleware.
The last thing is that we also shouldn't depend on a specific batch system. The INFN solution does require a pbs variant because it makes use of things like $PBS_NODEFILE.
> So, the question is: As our installation is now, a user can
> submit an MPICH executable for execution, without doing
> anything special, and get back its results. Is this the usual
> way in which MPICH jobs are submitted, or is it necessary
> that the script-based methods are supported? And if yes,
> exactly how can this be done?
Script based methods should be supported. Right now when using mpich mpirun this works. This is not the recommended way of using mpirun though, so something else is probably needed.
Kind regards,
Fokke Dijkstra
--------
Fokke Dijkstra
High Performance Computing
SARA - Reken- en Netwerkdiensten http://www.sara.nl
Tel. +31 20 592 8004 Fax. +31 20 668 3167
|