Hi Antun, hi Maarten,
thanks to both of you.
As you suspected, the jobs hanging in the ice_fl "queue" got matched to
CREAM CEs at CERN. We will do some cleanup following Antun's recipe.
Let's hope that this gets sorted out soon. All WMS service providers
should have suffered from that...
Cheers, Christoph
On Tue, 30 Jun 2009 14:44:43 +0200
Maarten Litmaath <[log in to unmask]> wrote:
> Hallo Christoph,
>
> > we observe the problem that on some of our WMSes the
> >
> > var/glite/ice/ice_fl
> >
> > file keeps growing and reaches the limit at some point. This then makes the WMS unusabel.
> >
> > To my understanding (can be completely wrong) this is only used by CREAM based submission. For that style of jobs we have only a very limited number, basically SAM tests only. So reaching the limits here is quite surprising.
> >
> > Is a limit of 500 (default) not reasonable? Is there a way of cleaning the ice_fl? The number of jobs in there seems not to decrease even with the WMS in draining mode.
> >
> > On other observation, the service glite-wms-ice is not running and fails to start when I try. Could this be the reason?
> >
> > As usual, any hint and/or pointers to further documentation is very welcome.
> >
> > BTW, just as a user I see also other (off-site) WMSes suffering from the same problem.
>
> The trouble appears to be caused by CERN: it publishes its CREAM CEs
> as Production instead of Special. This causes the WMS to match jobs
> to those CEs, but then those jobs cannot be submitted, since ICE is
> still broken. I have opened a top-priority ticket against CERN.
>
> The next WMS version fixes the problem with ICE; it is on the PPS
> and should go to production fairly soon.
>
> Meanwhile just zap /var/glite/ice/ice_fl when it gets full... :-(
|