Hi Massimo,
Sorry, didn't mean to hit send then.
Nothing in either that seems to correspond. One thing I've just noticed is
that if I search in the CreamCE logs for that cream job id it was created a
couple of days ago and looking closer at the error logs, although there are
many errors in the gridftp-session.log file they actually relate to just two
jobs.
I've realised that we're bumping up against 80% usage of that pool group and
we're using Argus for auth but have lcg-expiregridmapdir running on all the
CreamCEs.
I'm wondering if something is getting wrongly cleaned somewhere.
I've disabled the cleanup jobs and am purging all that DNs jobs from the
system and we'll see what happens to new jobs.
Chris.
> -----Original Message-----
> From: Massimo Sgaravatto [mailto:[log in to unmask]]
> Sent: 30 May 2012 13:00
> To: LHC Computer Grid - Rollout
> Cc: Brew, Chris (STFC,RAL,PPD)
> Subject: Re: [LCG-ROLLOUT] CreamCE: Problem uploading Input Sandbox for
> one DN
>
> That directory is supposed to be created by CREAM (via sudo) when a job
> for that user is registered.
>
> Do you see some error messages in glite-ce-cream.log and/or in
> /var/log/secure ?
>
> Cheers, Massimo
>
>
> On 05/30/2012 01:47 PM, Chris Brew wrote:
> > Hi,
> >
> > I'm stumped by this.
> >
> > I have one CMS production DN that is having problems running jobs on
> one of
> > my CreamCEs reporting errors about proxies or sandboxes.
> >
> > I appear to have tracked down some linked errors in
> > /var/log/gridftp-session.log that look like:
> >
> > [13818] Wed May 30 12:29:41 2012 :: GFork functionality not enabled.:
> > globus_gfork: GFork error: Env not set
> >
> > [13818] Wed May 30 12:29:41 2012 :: Configuration read from
> > /etc/gridftp.conf.
> > [13818] Wed May 30 12:29:41 2012 :: Server started in inetd mode.
> > [13818] Wed May 30 12:29:41 2012 :: New connection from:
> > wuncler.uits.indiana.edu:36921
> > [13818] Wed May 30 12:29:42 2012 :: DN
> > /DC=ch/DC=cern/OU=computers/CN=cmspilotjob/vocms157.cern.ch
> successfully
> > authorized.
> > [13818] Wed May 30 12:29:42 2012 :: User prdcms15 successfully
> authorized.
> > [13818] Wed May 30 12:29:43 2012 :: Failure attempting to transfer
> >
> "/var/cream_sandbox/prdcms/_DC_ch_DC_cern_OU_computers_CN_cmspilotjob_v
> ocms1
> >
> 57_cern_ch_cms_Role_production_Capability_NULL_prdcms15/30/CREAM3088156
> 26/IS
> > B/glidein_startup.sh".
> > [13818] Wed May 30 12:29:43 2012 :: Transfer failure:
> > globus_l_gfs_file_open failed.
> > globus_xio: Unable to open file
> >
> /var/cream_sandbox/prdcms/_DC_ch_DC_cern_OU_computers_CN_cmspilotjob_vo
> cms15
> >
> 7_cern_ch_cms_Role_production_Capability_NULL_prdcms15/30/CREAM30881562
> 6/ISB
> > /glidein_startup.sh
> > globus_xio: System error in open: No such file or directory
> > globus_xio: A system call failed: No such file or directory
> >
> >
> > And indeed when I look there is no
> >
> /var/cream_sandbox/prdcms/_DC_ch_DC_cern_OU_computers_CN_cmspilotjob_vo
> cms15
> > 7_cern_ch_cms_Role_production_Capability_NULL_prdcms15 directory.
> >
> > The filesystem is only 50% full, the permissions on the
> > /var/cream_sandbox/prdcms look right (tomcat:prdcms).
> >
> > This is with glite-ce-cream-1.13.4-1.sl5
> >
> > So does any anyone know what is supposed to be creating the sandbox
> dirs and
> > why might it have stopped for this one DN?
> >
> > Many Thanks,
> > Chris.
> >
>
|