Hi Kevin,
Kejun (Kevin) Dong wrote:
> Now I find the weblink
> https://twiki.grid.iu.edu/twiki/bin/view/Troubleshooting/GlobusError3
> and it suggests to "Ask the grid admin to check/restart services using
> large amounts of memory and verify that your account has sufficient
> space to run the jobs you are trying to submit."
> Does anybody know any methods to check which service eats the memory so
> much? Thanks.
>
From my experience, the worst memory-eater on the CE is R-GMA. After
around 100 days of continuous operation there are plenty of java
processes (22M each) plus accompanying mysql instances (11M each).
Restarting the service helps only partially, because some of the
processes survive this step. Besides, the number of java processes
quickly (couple of days) increases to the previous level. These memory
parasites slow down the machine considerably, although I didn't notice
Globus errors no. 3. Nevertheless, after about 100 days, they impair the
speed of pbs in considerable extent, so the job submission may fail
occasionally.
The definitive solution is to restart the CE machine, because then the
situation goes back to "normal" (whatever it is) and you have another
100 days ;-). It doesn't bother me very much, because usually in 100
days there are plenty of other reasons to restart the machine at some
point.
And still the R-GMA is still much more thrifty than for example DPM on
the SE, which produces dpm processes (50M each) much faster than R-GMA
does with it's java progeny.
If you want to look which processes occupy most of the memory, try the
'M' switch in top. Or you can try something like this:
ps auxww | head -n 1; ps auxww | sort -rn +3 | head
Anyways, if you have 8GB of RAM, and it's being eaten so quickly, I
suggest you to look at the users' processes in the first place, not the
system daemons. Try to filter out the processes belonging to root,
edguser, mysql etc. and see what remains. If you don't know how to do
it, you can try
ps -auxww |sort -rn +3 |egrep -v mysql\|root\|edguser\|rgma | less
or something like that
good luck,
Adam
|