On Thu, Oct 27, 2005 at 04:04:14PM +0200 or thereabouts, Stijn De Weirdt wrote:
> hi davis,
>
> thanks for the patch, but is this sufficient to make the job write it's
> output to that directory? and the cleaning afterwards?
> and how do you make the $TMPDIR value unique? (and how do you actually
> create this unique directory on the fly?)
It is done by torque itself, David's patch instructs pbs to create
and delete a configurable directory at job start end.
So we have
$tmpdir /pool
in /var/spool/pbs/mom_priv/config
this creates
/pool/<jobid>
I think it and then
LCG_SCRATCH_DIR=$TMPDIR
in the profile handles the LCG job case.
Of course this all assumes you have pbs.
Steve
>
> (i probably missed something...)
>
> stijn
>
> David Groep wrote:
> >Hi Stijn,
> >
> >Using the fact that TMPDIR points to a local (per-node) scratch directory,
> >we have been applying a patch to the original pbs manager to select,
> >based on the #nodfes requested by the job and the job type, the
> >current working directory of the job. Patch is attached to this mail
> >(it's just a few lines).
> >
> >What it does:
> >* if the job is of type "mpi", or if the type is "multiple" and the
> > number of requested nodes > 1, the bevahviour of the pbs job manager
> > is un-altered.
> >* if the job type is "single", or the type is "multiple" and the
> > job requests 0 or 1 nodes, the following statement is inserted
> > in the PBS job script, just before the user job is started:
> >
> > [ x"$TMPDIR" != x"" ] && cd $TMPDIR
> >
> >This patch is applied to the template for the pbs.pm job manager script
> >in /opt/globus/setup/globus/pbs.in, which then gets translated on
> >startup in /opt/globus/lib/perl/Globus/GRAM/JobManager/pbs.pm
> >
> >It has till now worked fine for all LCG jobs that at NIKHEF also go
> >through the "old" pbs JM. The jobs don't notice the difference and
> >we can use shared home dirs for all VOs, provided we also have a
> >per-node $TMPDIR location on local disk.
> >
> > Cheers,
> > DavidG.
> >
> >Stijn De Weirdt wrote:
> >
> >>i am looking for a way to set the SCRATCH_DIRECTORY value for jobs
> >>running on our cluster based on the vo.
> >>(we have pbs queues with nfs mounted home directories for mpi, but we
> >>also have jobs that might perform better when they can write directly
> >>to local disks.)
> >>
> >>there's an option for the globus-job-manager called -scratch-dir-base,
> >>but i don't know if that's the good way to change this. (and i also
> >>don't know how to make it vo dependent)
> >>
> >>or if someone has succesfully mixed pbs (nfs) and lcgpbs (local disk)
> >>queues on their site, that could also be a solution.
> >>
> >>many thanks
> >>
> >>stijn
> >
> >
> >
> >The patch
> >
> >*** pbs.in.orig 2005-05-20 12:56:32.000000000 +0200
> >--- pbs.in 2005-05-20 12:52:05.000000000 +0200
> >***************
> >*** 321,327 ****
> > }
> > print JOB "wait\n";
> > }
> >! elsif($description->jobtype() eq 'multiple')
> > {
> > my $count = $description->count;
> > my $cmd_script_url ;
> >--- 321,327 ----
> > }
> > print JOB "wait\n";
> > }
> >! elsif( ($description->jobtype() eq 'multiple') and
> >($description->count > 1 ) )
> > {
> > my $count = $description->count;
> > my $cmd_script_url ;
> >***************
> >*** 374,379 ****
> >--- 374,393 ----
> > }
> > else
> > {
> >+ # this is a simple single-node job that can use $TMPDIR
> >+ # unless the user has given one explicitly
> >+ # refer back to JobManager.pm, but currently it seems that
> >+ # $self->make_scratchdir uses "gram_scratch_" as a component
> >+ if ( ( $description->directory() =~ /.*gram_scratch_.*/ ) and
> >+ ( $description->host_count() <= 1 ) and
> >+ ( $description->count <= 1 )
> >+ ) {
> >+ print JOB '# user ended in a scratch directory, reset to
> >TMPDIR'."\n";
> >+ print JOB '[ x"$TMPDIR" != x"" ] && cd $TMPDIR'."\n";
> >+ } else {
> >+ print JOB '# user requested this specific directory'."\n";
> >+ }
> >+
> > print JOB $description->executable(), " $args <",
> > $description->stdin(), "\n";
> > }
> >
> >
--
Steve Traylen
[log in to unmask]
http://www.gridpp.ac.uk/
|