On Thu, Jan 20, 2005 at 03:15:06pm +0100, Fokke Dijkstra wrote:
> When I submit an MPI job to your site I still get the above error. We have not defined a jobmanager in our LCFGng config files. I'm not sure if something default is defined in that case.
> I get the same error when submitting to grid10.lal.in2p3.fr:2119/jobmanager-torque-dteam all other sites were able to run my test job.
> 

From what I can understand, based on my experimentation with MPICH-LCG
integration in the last few days, Torque provides native support for
MPICH-type jobs, and runs mpirun itself. Until now, most MPI jobs were
submitted along with a wrapper script, which undertook running mpirun,
after consulting the $PBS_NODEFILE, in order to know exactly which
processors were allocated to it by the local scheduler.

In our current installation in HG-01-GRNET, no such script is
necessary, since a pbs_mom process on one of the worker nodes will
execute mpirun itself, with the necessary arguments.

> 
> We have installed a shared /home filesystem for the worker nodes. This makes the authentication easier since it is possible to make use of an authorized key. It also makes it easier for MPI jobs to get access to their data. As far as I know executables and data are not copied to all the worker nodes when an MPI job starts.
> 

This is also the case with our worker nodes. In fact, if it was not the
case, a wrapper script would be necessary in order to copy the input
sandbox to all other worker nodes that are going to be used, using for
example rsync or scp. However, since Torque runs mpirun directly when it
sees a JDL of JobType="MPICH", I'm not sure how this could be
accomplished with the current implementation/installation.

> The LCG scripts that generate the job create the correct pbs job script for you. This has the disadvantage that it leaves not much freedom to the user to do other things in his or her job. You can only submit an executable with some data and not run scripts that do for example a file transfer.
> 
> Another problem running jobs using mpirun is that pbs_mom does not really keep track of what a job is doing on the other worker nodes. When a job crashes or is killed from outside this may  then only kill the processes on the master woker node leaving garbage processes on the other worker nodes involved. 
> 
> This can be solved by making use of mpiexec. Mpiexec spawns the mpi processes using pbs_mom like you already explained. I first have installed it on our machines. Then you should be able to run jobs using mpiexec using a local account (very easy to have for testing purposes!). If that works you have to find a way to convince grid jobs to use mpiexec instead of doing mpirun.
> 

I agree completely. However, with the current version of Torque it seems
it is not up to the user (the submitted grid job) to decide whether
mpirun or mpiexec is to be run, since it is now run directly. However,
there are patches, which we are soon going to experiment with, that
integrate mpiexec into Torque:

http://www.osc.edu/~pw/mpiexec/

So, to sum up, all that is needed to run an MPI job on our site is a JDL
of the form:

JobType = "MPICH";
NodeNumber = 20;
Executable = "hello";
StdOutput = "hello.out";
StdError = "hello.err";
InputSandbox = {"hello"};
OutputSandbox = {"hello.out","hello.err"};
Requirements = other.GlueCEUniqueID == "ce01.isabella.grnet.gr:2119/jobmanager-torque-dteam"


> 
> Our RB does not work either, and probably none will. It is possible to make use of a different job manager, but I'm not sure how that works using YAIM.
> 

This is the reason why, for the time being the lrms type needs to be
manually provided for the job to be submitted properly:

edg-job-submit --lrms pbs -r ce01.isabella.grnet.gr:2119/jobmanager-torque-dteam --vo dteam mpi.jdl

However, I believe this is actually a bug of either YAIM or the lnx1188
RB installation, which presented the original error message: YAIM set
the lrms value to "Torque" for our site, and the RB seems to only look
for the lrms values of "lsf" or "pbs" in order to run an MPI job, so
one of the two most probably needs to adapt :) And anyway, if I
understand correctly Torque is just a PBS variant, so it could be used
as such without problems.

-- 
Vangelis Koukis
[log in to unmask]
OpenPGP public key ID:
pub  1024D/1D038E97 2003-07-13 Vangelis Koukis <[log in to unmask]>
     Key fingerprint = C5CD E02E 2C78 7C10 8A00  53D8 FBFC 3799 1D03 8E97