> from CREAM side, there is the mechanism for automatic job purging.
>
> You can see the following link:
> https://wiki.italiangrid.it/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#3_18_Job_purging
Yes, and this would be marvellous if it actually worked nicely :) If I kill 8000 jobs in the scheduling system the CREAM will be busy state checking and reporting to WMS etc for ages with MySQL load at 100%. I've even seen BNotifier just die under the load. It seemed to me at the time that it tried to report ALL jobs to the WMS (might have been only one WMS had sent them) together so it consumed as much RAM as I gave the node and died. I at some point gave the CREAM VM 22GB of RAM that BNotifier consumed in half a day and still it was stuck at the time.
This time I wanted to prevent such an occurrence by killing all the jobs and cleaning the CREAM database as well to reset everything to 0 and sadly causing the jobs to be "lost". However it seems this time CREAM managed to survive as we did the killings in bulks of 100 jobs with queues in draining so nothing new was coming in at the time.
Still it would be good to know how to clean this stuff also manually because last time I checked the manual Job purge script failed miserably because it couldn't connect to DB through Java.
Mario Kadastik, PhD
Researcher
---
"Physics is like sex, sure it may have practical reasons, but that's not why we do it"
-- Richard P. Feynman
|