On Fri, 27 May 2005, Ian Fisk wrote:

> We are seeing the the lock corruption problem again.    After stable  
> running for 5 days we have two corrupted Grid-manager-monitor lock  
> files.    The machine swapping does not appear to be the cause of the  
> problem, but it is the result of the problem.

I have a hunch the problem is due to your system:

-----------------------------------------------------------------------------
$ globus-job-run cmslcgce.fnal.gov /bin/pwd
/uscms_data/d1/grid_home/dteam006

$ globus-job-run cmslcgce.fnal.gov /bin/df .
Filesystem           1K-blocks      Used Available Use% Mounted on
131.225.207.94:fs_ibrix02
                     2836649880 991990964 1838253096  36% /uscms_data/d1
-----------------------------------------------------------------------------

Did Lisa not send a message to the roll-out list a few days ago,
about problems with the Ibrix file server?

You may want to mount the thing with the "noac" option.

> We have hundreds of the grid-monitor-processes.    Today the users in  
> question are new people and they both came in through the CERN RB.     
> In some cases the grid-monitor-manager doesn't finish so the job  
> state is not updated and users don't get the results, because they  
> never change state.
> 
> The grid-monitor-manager perl script seems to be set to delete itself  
> after it starts and it's submitted with the job, so I'm not sure how  
> to debug.   The instance of the script recorded in the gass cache  
> appears to be consistent across users.   At the moment we have to  
> watch and respond before the system has so few resources that it has  
> to be rebooted to clear them.
> 
> -Ian
> 
> 
> On May 24, 2005, at 12:13 PM, David Smith wrote:
> 
> > On Tue, 24 May 2005, Ian Fisk wrote:
> >
> >
> >> We could use some guidance on solving this grid-manager-monitor  
> >> problem.
> >>
> >> The lock file for one user is corrupted and we are seeing the grid-
> >> manager-monitor thrashing reported a few days ago.    Somehow the  
> >> problem
> >> seems reproducible for the user, but he is submitting through the  
> >> same RB as
> >> several other people and using the same infrastructure to  
> >> configure the jobs.
> >> He has submitted 200 jobs, but another user without the problem  
> >> has also
> >> submitted more than 200.
> >>
> >> The gateway has all 4GB of real memory used and 2GB of swap used.
> >>
> >> The lock file is corrupted for cms002.    It should be about 16  
> >> bytes long
> >>
> >
> > Hello Ian,
> >
> > I wasn't able to understand exactly what was happening - there was a
> > problem with some internal locks for the cms002 user
> > (~cms002/.lcgjm/*lock) which was making the job managers hang. I  
> > think it
> > is likely this was from earlier problems with the machine running  
> > low on
> > memory. I was able to correct that, and now things appear to running
> > smoothly. However since I wasn't able to understand the corruption  
> > of the
> > lock file nor the cause of the original problem exactly it is quite
> > possible there is a further problem or bug that needs to be found.  
> > I'll
> > try to keep a period check on your CE, and likewise if you notice any
> > problem send another mail, and CC me directly on it.
> >
> > Yours,
> > David
> > --
> > ---------------------------------------------------------------------- 
> > ---
> > David Smith       e-mail: [log in to unmask]        tel: +41 22 76  
> > 74462
> > Address: D. Smith, CERN G06610, Bat 28 R-007, 1211 Geneva 23,  
> > Switzerland
> > ---------------------------------------------------------------------- 
> > ---
> >
>