Hello Fokke,
On Thu, May 19, 2005 at 11:20:01am +0200, Fokke Dijkstra wrote:
> >
> > I've installed the latest version of Torque, 1.2.0p3, which
> > seems to work without any problems, based on Cal's original
> > source RPMs. The changes needed were trivial, and the TMPDIR
> > patch applies cleanly to this version. The resulting files
> > can be found at:
> >
> > http://www.cslab.ece.ntua.gr/~vkoukis/torque-mpi
>
> Did Cal's version not include the $TMPDIR patch already? What changes have you made exactly?
>
Yes, of course it did, but Cal's RPMs are based on the source tarball
of Torque 1.2.0p1. All I did was update the spec file and the source
definitions so that the tarball for Torque 1.2.0p3 was used and noticed
that Cal's port of the TMPDIR patch for 1.2.0p1 could be applied to
1.2.0p3 without any changes on my part. So, all that was needed was
changing the version number wherever it was mentioned.
> > I've also setup the submit filter and changed the LRMS type
> > to 'pbs', as suggested in the Wiki, in order to work around
> > the problem with the RB not liking the 'torque' LRMSType.
>
> Torque should also be supported as LRMS type for MPI jobs, but for now setting pbs solves the problem. This should be fixed though.
>
Exactly, if Torque is setup automatically by yaim in recent versions of
LCG, it would make sense to also adapt the RB code to accept an LRMSType
of 'torque'.
> > The point is, I am not sure on exactly *who* is responsible
> > for calling mpirun, in order to spawn the processes of the
> > parallel application.
>
> Neither am I. I thought the pbs.pm script was doing this, but changing the mpirun definition for Globus did not help. The mpirun command is in fact put in somewhere else. Does anyone know where?
>
> > In the GOC Wiki, it is suggested (there are many other
> > sources of documentation suggesting something similar, too)
> > using a test job which declares a shell script as the
> > executable. The shell script then calls
> > mpirun/mpiexec/whatever, at its discretion, whenever it
> > thinks it's necessary. However, from what I have seen both
> > using LCG2.3.1 and LCG2.4.0, mpirun is not meant to be called
> > by a shell script, but is called directly by Torque.
>
> This is something that works using the mpich mpirun. It is not the recommended way of running MPI jobs though. In fact this may easily lead to problems in the future, because mpirun is not meant to run scripts Therefore it may not work in a future version of mpich. Furthermore nested mpiruns can be dangerous in themselves by spawning n*n processes, instead of n.
> Using mpirun multiple times does, at the moment, not seem to give problems though.
>
Yes, I am now able to understand why this works, based on some testing I
did with a small MPICH installation. From what I understand, this not
only is an MPICH-specific hack, it is a very ugly MPICH-over-TCP/IP-specific
hack! :-)
On a small cluster where MPICH is setup to run over the ch_p4 device,
which is used for communication over TCP/IP, I tried running the MPI
"executable" called /bin/echo:
vkoukis@kid1$ mpirun -v -np 8 /bin/echo
running /bin/echo on 8 LINUX ch_p4 processors
Created /home/users/vkoukis/PI7856
-p4pg /home/users/vkoukis/PI7856 -p4wd /usr/lib/mpich/bin
It seems that mpirun only creates the first process of the group, which
is instructed to spawn the remaining processes, consulting the procgroup
file PI7856, using a variant of rsh/ssh.
However, this method is specific *only* to ch_p4. So, it will not work
when using a port of MPICH over a different interconnect, for example
when using MPICH-GM, over Myrinet. On a Myrinet cluster of ours, I tried:
vkoukis@zealot1$ mpirun -np 4 -machinefile quad /bin/hostname
zealot2
zealot1
zealot3
zealot4
Because when the fabric used is Myrinet-GM, mpirun uses rsh/ssh to
spawn all processes directly, and passes the necessary data so that they
can join the group using environment variables.
Both behaviors of mpirun are acceptable, and, as you also said, it is
totally wrong to rely on each of them.
> > So, to provide support for mpiexec, I had to replace the
> > MPICH-provided /usr/bin/mpirun with a very small, very simple
> > wrapper which discards all mpirun-specific arguments, and
> > then calls mpiexec, as Fokke Dijkstra suggests in:
> >
> > http://www.listserv.rl.ac.uk/cgi-bin/webadmin?A2=ind0503&L=LCG
> > -ROLLOUT&P=R45745&I=-3
>
> We have done the same for a while, but stopped doing this. When a script is run using mpiexec you cannot call mpiexec in the script itself. Therefore when using a wrapper script for mpirun scripts that call mpirun will call mpiexec a second time and fail. This is caused by the fact that pbs_mom is used to control the processes and it can only be used for one process using mpiexec.
>
Yes, I also noticed that: mpiexec produces an error like 'not connected
to PBS", or something like that. That's the reason Cal's test script
failed with our installation.
I think that having to support this specific way of pre-processing
scripts is very constraining, and restricts severely the number of MPICH
implementations and the number of interconnects that can be used. Using
mpiexec is indeed a valid way to spawn the processes. Exposing the
way processes are created to the end user is the root of the problem,
and makes the whole Grid abstraction terribly leaky, leading to exactly
the problems you describe... They make it impossible for the system
administrator to select the best way of managing the processes to be
created, so that they can be managed and accounted for.
> > The way mpirun is inserted in the job description submitted
> > to Torque can be seen in in
> > /opt/globus/lib/perl/Globus/GRAM/JobManager/pbs.pm on the CE,
> > which constructs the submission script for Torque/PBS:
> >
> > if($description->jobtype() eq 'mpi')
> > {
> > print JOB "$mpirun -np ", $description->count(), ' ';
> > if($cluster) {
> > print JOB " -machinefile \$PBS_NODEFILE "; }
> > print JOB $description->executable(), " $args < ",
> > $description->stdin(), "\n";
> > }
>
> Someone decided that the mpirun call should be done somewhere else. So the above is probably only done for globus-job-submit jobs. For LCG jobs the mpirun is inserted elsewhere.
>
I will try to explore the code a bit more, and will come back later with
a more detailed description of where the call to mpirun is hardcoded...
but the point remains that mpirun will be called, when the JobType is
specified as "MPICH".
> > It seems that the user has no control over whether mpirun is
> > called or not. In fact, when I tried submitting a JDL with
> > Cal's MPItest.sh as the executable, the small /usr/bin/mpirun wrapper
> > was still called. This means that MPItest.sh gets to be used as the
> > parallel application, which is wrong... Depending on the
> > platform-specific way MPICH uses to spawn its processes, this
> > could lead to multiple instances of it running around the cluster.
>
> This is indeed a problem like described earlier. We should probably think of a different way to run MPI jobs, that allows for user scripts. These are needed when one wants to do preprocessing and postprocessing work before and after running the mpi executable. Think of things like data transfers.
> Furthermore when a shared directory is not available for the job, explicit copies of the input data for the job are needed. This could also be done by the middleware.
> The last thing is that we also shouldn't depend on a specific batch system. The INFN solution does require a pbs variant because it makes use of things like $PBS_NODEFILE.
>
I agree completely. I think the infrastructure for submitting MPI jobs
should be extended to allow the user to specify pre- and post-processing
scripts, which are to be run in a serial way. But the user should have
no control on how the parallel application is to be run.
He/she should just provide the executable, as is done currently with
uniprocessor jobs, and ask for the required number of processors. It is
the job of the specific MPICH-implementation used to determine how (e.g.
using mpiexec, or some other method in the future) the processes are to
be spawned.
Anothe very important issue here is that of binary compatibility,
because the user will have to compile the application for the specific
port of MPICH used. If it is to run over MPICH-GM, it has to be compiled
using MPICH-GM, if it is to run over MPICH-ch_p4 it has to be compiled
for MPICH-ch_p4, and so on. So, the best thing to do is to provide
(in the right way) a small pre-processing script which will compile
the application locally, with the right site-specific tools. This
executable can then be submitted for execution.
> > So, the question is: As our installation is now, a user can
> > submit an MPICH executable for execution, without doing
> > anything special, and get back its results. Is this the usual
> > way in which MPICH jobs are submitted, or is it necessary
> > that the script-based methods are supported? And if yes,
> > exactly how can this be done?
>
> Script based methods should be supported. Right now when using mpich mpirun this works. This is not the recommended way of using mpirun though, so something else is probably needed.
>
I understand that script-based methods should be supported, but abusing
an MPICH-over-TCP/IP peculiarity to do that goes a bit too far :), and
really constrains the Resource Centers which could support MPICH jobs...
As you say, we really need a standardized way to specify pre- and post-
processing scripts.
However, even as it is now, there is a guaranteed/standardized way to
spawn a parallel job. And that is to just specify the executable in the
JDL file, along with the required number of processors and have the target
LRMS do the rest. Would that suffice for the current MPI needs of VOs?
Best Regards,
Vangelis.
--
Vangelis Koukis, PhD candidate
Computing Systems Laboratory,
National Technical University of Athens.
Institute of Communication and Computer Systems (ICCS)
[log in to unmask]
