Hello,
First apologies for long email and outputs but I guess I'd better be
more precise than not.
I've seen the re-occurrence of a problem I had first seen on a test SL4 CE
a while ago.
The BDII (replacement of globus-mds and not the site BDII) kept dying.
It happened every hour just after the gatekeeper had received a kill
signal and restarted itself.
This problem disappeared after a fresh re-install of the CE. The CE had
been stable for weeks until last Thursday when the output of a job filled
up the partition containing exported pool accounts. As a result, new grid
jobs failed and the problem I had observed with my test CE reappeared.
Stopping the gatekeeper and the marshals, killing all the agents and
cleaning up the pool accounts and tmps directories, and rebooting the CE
did not fix the problem.
As I was upgrading my DPM head node from SL3 to SL4 yesterday, I felt I
should focus on this rather than debugging the CE and I decided to
re-install a fresh CE (update 21), which I can trivially do. After the
installation, I ran into permissions problem causing jobs to fail. I've
described this in the "recent optimizations in lcg-ce" thread.
Job submissions finally worked, but the CE got in a funny state again.
The gatekeeper receives a kill signal every hour:
[root@epgce2 ~]# grep sig /var/log/messages | tail -n 10
May 3 10:05:37 epgce2 GRAM gatekeeper[28152]: Gatekeeper received
signal:15
May 3 10:05:37 epgce2 GRAM gatekeeper[28152]: Gatekeeper shutdown on
signal:15
May 3 11:05:38 epgce2 GRAM gatekeeper[31837]: Gatekeeper received
signal:15
May 3 11:05:38 epgce2 GRAM gatekeeper[31837]: Gatekeeper shutdown on
signal:15
May 3 12:05:38 epgce2 GRAM gatekeeper[6170]: Gatekeeper received
signal:15
May 3 12:05:38 epgce2 GRAM gatekeeper[6170]: Gatekeeper shutdown on
signal:15
May 3 13:00:24 epgce2 GRAM gatekeeper[9963]: Gatekeeper received
signal:15
May 3 13:00:24 epgce2 GRAM gatekeeper[9963]: Gatekeeper shutdown on
signal:15
May 3 13:05:38 epgce2 GRAM gatekeeper[4933]: Gatekeeper received
signal:15
May 3 13:05:38 epgce2 GRAM gatekeeper[4933]: Gatekeeper shutdown on
signal:15
-->
I did a "service globus-gatekeeper restart", so ignore the 13:00:24
entries.
One corresponding message in the gatekeeper log is:
<--
JMA 2008/05/03 13:00:58 GATEKEEPER_JM_ID
2008-05-03.13:00:53.0000007749.0000000000 mapped to prdatl07 (1
04007, 103001)
JMA 2008/05/03 13:00:58 GATEKEEPER_JM_ID
2008-05-03.13:00:53.0000007749.0000000000 has GRAM_SCRIPT_JOB_I
D 1209816058:lcgpbs:internal_266204464:7751.1209816053 manager type lcgpbs
JMA 2008/05/03 13:00:59 GATEKEEPER_JM_ID
2008-05-03.13:00:53.0000007749.0000000000 JM exiting
TIME: Sat May 3 13:05:38 2008
PID: 4933 -- Notice: 5: Gatekeeper received signal:15
Failure: Gatekeeper shutdown on signal:15
TIME: Sat May 3 13:05:38 2008
PID: 4933 -- Failure: Gatekeeper shutdown on signal:15
TIME: Sat May 3 13:05:38 2008
PID: 8289 -- Notice: 6: /opt/globus/sbin/globus-gatekeeper pid=8289
starting at Sat May 3 13:05:38 200
8
TIME: Sat May 3 13:05:38 2008
PID: 8289 -- Notice: 6: GRAM contact:
epgce2.ph.bham.ac.uk:2119:/C=UK/O=eScience/OU=Birmingham/L=Partic
[log in to unmask]
TIME: Sat May 3 13:05:38 2008
PID: 8289 -- Notice: 0: GATEKEEPER_ACCT_FD=6
(/var/log/globus-gatekeeper.log)
TIME: Sat May 3 13:06:00 2008
PID: 8411 -- Notice: 6: Got connection 130.209.239.17 at Sat May 3
13:06:00 2008
TIME: Sat May 3 13:06:00 2008
-->
Shortly after or a the same time, this happens my bdii dies:
[root@epgce2 ~]# service bdii status
lock file exists but PID 7420 died
I've attached traces with times of the gatekeeper and bdii processes in
the files strace-gk and strace-bdii.
The globus processes on the CE are:
<--
[root@epgce2 ~]# ps -ef | grep globus
root 5434 1 0 13:00 ? 00:00:00 globus-gass-cache-marshal:
accepting connections
root 5446 1 0 13:00 ? 00:00:00
globus-job-manager-marshal: accepting connections
root 8321 1 0 13:05 ? 00:00:00
/opt/globus/sbin/globus-gatekeeper -conf
/opt/globus/etc/globus-gatekeeper.conf
root 8342 1 0 13:05 ? 00:00:00
/opt/globus/sbin/globus-gridftp-server -p 2811 -d error,warn,info -l
/var/log/gridftp-session.log -Z /var/log/globus-gridftp.log -s
prdatl07 8618 1 0 13:10 ? 00:00:00 globus-job-manager -conf
/opt/globus/etc/globus-job-manager.conf -type fork -rdn jobmanager-fork
-machine-type unknown -publish-jobs
-->
A new error has made its way in the bdii log after this fresh install. I
think this is irrelevant and just a new oddity as I had not seen it in the
past.
<--
Updating DB on port 2171
Waiting 180 s for query results.
GIP: Can't call method "eof" on an undefined value at
/opt/glite/libexec/glite-info-generic line 16.
Time for searches: 0 s
current port: 44175 - OK
-->
The status of the marshals is misleading has they're actually running
fine as can be seen in the process list above.
[root@epgce2 ~]# service globus-gass-cache-marshal status
globus-gass-cache-marshal dead but pid file exists
[root@epgce2 ~]# service globus-job-manager-marshal status
globus-job-manager-marshal dead but pid file exists
Job submissions works fine otherwise, and I could have a trigger
restarting my bdii, but this is not a clean solution.
Has anyone experienced a similar problem. Most, importantly does anyone
understand why this happen and/or how to recover the gatekeeper without
doing a fresh install and hoping for the best?
Thank you,
Yves
13:02:32 select(5, [4], NULL, [4], {21, 557000}) = 0 (Timeout)
13:02:53 select(5, [4], NULL, [4], {60, 0}) = 0 (Timeout)
13:03:53 select(5, [4], NULL, [4], {60, 0}) = 0 (Timeout)
13:04:53 select(5, [4], NULL, [4], {60, 0}) = ? ERESTARTNOHAND (To be restarted)
13:05:38 --- SIGTERM (Terminated) @ 0 (0) ---
13:05:38 time(NULL) = 1209816338
13:05:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:38 send(5, "<29>May 3 13:05:38 GRAM gatekee"..., 72, MSG_NOSIGNAL) = 72
13:05:38 time(NULL) = 1209816338
13:05:38 write(2, "TIME: Sat May 3 13:05:38 2008\n "..., 86) = 86
13:05:38 close(4) = 0
13:05:38 write(2, "Failure: Gatekeeper shutdown on "..., 42) = 42
13:05:38 time(NULL) = 1209816338
13:05:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:38 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:38 send(5, "<27>May 3 13:05:38 GRAM gatekee"..., 76, MSG_NOSIGNAL) = 76
13:05:38 time(NULL) = 1209816338
13:05:38 write(2, "TIME: Sat May 3 13:05:38 2008\n "..., 87) = 87
13:05:38 write(3, "/opt/globus/lib/opt/globus/lib/e"..., 216) = 216
13:05:38 exit_group(1) = ?
13:03:00 accept(3, {sa_family=AF_INET, sin_port=htons(45838), sin_addr=inet_addr("147.188.46.4")}, [16]) = 4
13:03:29 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:03:29 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:03:29 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:03:29 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:03:29 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:03:29 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 5
13:03:29 ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:03:29 _llseek(5, 0, [0], SEEK_CUR) = 0
13:03:29 fstat64(5, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:03:29 fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
13:03:29 read(5, "2171\n", 4096) = 5
13:03:29 close(5) = 0
13:03:29 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:03:29 getpeername(4, {sa_family=AF_INET, sin_port=htons(45838), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:03:29 getpeername(4, {sa_family=AF_INET, sin_port=htons(45838), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:03:29 time(NULL) = 1209816209
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 time(NULL) = 1209816209
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 write(1, "20080503_130256 Forked process 8"..., 140) = 140
13:03:29 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f3bae8) = 8072
13:03:29 time(NULL) = 1209816209
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 time(NULL) = 1209816209
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 close(4) = 0
13:03:29 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:03:29 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:03:29 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:03:29 accept(3, 0xbfe61ff0, [4096]) = ? ERESTARTSYS (To be restarted)
13:03:29 --- SIGCHLD (Child exited) @ 0 (0) ---
13:03:29 sigreturn() = ? (mask now [])
13:03:29 rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
13:03:29 rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
13:03:29 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 4
13:03:29 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:03:29 _llseek(4, 0, [0], SEEK_CUR) = 0
13:03:29 fstat64(4, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:03:29 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:03:29 read(4, "2171\n", 4096) = 5
13:03:29 close(4) = 0
13:03:29 waitpid(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG) = 8072
13:03:29 time(NULL) = 1209816209
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 time(NULL) = 1209816209
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:03:29 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:03:29 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:03:29 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:03:29 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:03:29 accept(3, {sa_family=AF_INET, sin_port=htons(45907), sin_addr=inet_addr("147.188.46.4")}, [16]) = 4
13:04:03 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:04:03 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:04:03 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:04:03 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:04:03 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:04:03 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 5
13:04:03 ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:04:03 _llseek(5, 0, [0], SEEK_CUR) = 0
13:04:03 fstat64(5, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:04:03 fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
13:04:03 read(5, "2172\n", 4096) = 5
13:04:03 close(5) = 0
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:04:03 getpeername(4, {sa_family=AF_INET, sin_port=htons(45907), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:04:03 getpeername(4, {sa_family=AF_INET, sin_port=htons(45907), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 write(1, "20080503_130329 Forked process 8"..., 194) = 194
13:04:03 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f3bae8) = 8150
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 close(4) = 0
13:04:03 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:04:03 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:04:03 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:04:03 accept(3, 0xbfe61ff0, [4096]) = ? ERESTARTSYS (To be restarted)
13:04:03 --- SIGCHLD (Child exited) @ 0 (0) ---
13:04:03 sigreturn() = ? (mask now [])
13:04:03 rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
13:04:03 rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
13:04:03 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 4
13:04:03 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:04:03 _llseek(4, 0, [0], SEEK_CUR) = 0
13:04:03 fstat64(4, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:04:03 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:04:03 read(4, "2172\n", 4096) = 5
13:04:03 close(4) = 0
13:04:03 waitpid(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG) = 8150
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 time(NULL) = 1209816243
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:03 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:04:03 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:04:03 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:04:03 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:04:03 accept(3, {sa_family=AF_INET, sin_port=htons(45940), sin_addr=inet_addr("147.188.46.4")}, [16]) = 4
13:04:36 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:04:36 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:04:36 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:04:36 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:04:36 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:04:36 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 5
13:04:36 ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:04:36 _llseek(5, 0, [0], SEEK_CUR) = 0
13:04:36 fstat64(5, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:04:36 fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
13:04:36 read(5, "2172\n", 4096) = 5
13:04:36 close(5) = 0
13:04:36 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:04:36 getpeername(4, {sa_family=AF_INET, sin_port=htons(45940), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:04:36 getpeername(4, {sa_family=AF_INET, sin_port=htons(45940), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:04:36 time(NULL) = 1209816276
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 time(NULL) = 1209816276
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 write(1, "20080503_130403 Forked process 8"..., 140) = 140
13:04:36 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f3bae8) = 8163
13:04:36 time(NULL) = 1209816276
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 time(NULL) = 1209816276
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 close(4) = 0
13:04:36 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:04:36 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:04:36 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:04:36 accept(3, 0xbfe61ff0, [4096]) = ? ERESTARTSYS (To be restarted)
13:04:36 --- SIGCHLD (Child exited) @ 0 (0) ---
13:04:36 sigreturn() = ? (mask now [])
13:04:36 rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
13:04:36 rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
13:04:36 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 4
13:04:36 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:04:36 _llseek(4, 0, [0], SEEK_CUR) = 0
13:04:36 fstat64(4, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:04:36 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:04:36 read(4, "2172\n", 4096) = 5
13:04:36 close(4) = 0
13:04:36 waitpid(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG) = 8163
13:04:36 time(NULL) = 1209816276
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 time(NULL) = 1209816276
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:04:36 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:04:36 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:04:36 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:04:36 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:04:36 accept(3, {sa_family=AF_INET, sin_port=htons(46014), sin_addr=inet_addr("147.188.46.4")}, [16]) = 4
13:05:09 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:05:09 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:05:09 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe61de8) = -1 EINVAL (Invalid argument)
13:05:09 _llseek(4, 0, 0xbfe61e20, SEEK_CUR) = -1 ESPIPE (Illegal seek)
13:05:09 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:05:09 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 5
13:05:09 ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:05:09 _llseek(5, 0, [0], SEEK_CUR) = 0
13:05:09 fstat64(5, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:05:09 fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
13:05:09 read(5, "2172\n", 4096) = 5
13:05:09 close(5) = 0
13:05:09 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:05:09 getpeername(4, {sa_family=AF_INET, sin_port=htons(46014), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:05:09 getpeername(4, {sa_family=AF_INET, sin_port=htons(46014), sin_addr=inet_addr("147.188.46.4")}, [16]) = 0
13:05:09 time(NULL) = 1209816309
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 time(NULL) = 1209816309
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 write(1, "20080503_130436 Forked process 8"..., 140) = 140
13:05:09 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f3bae8) = 8222
13:05:09 time(NULL) = 1209816309
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 time(NULL) = 1209816309
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 close(4) = 0
13:05:09 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:05:09 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:05:09 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:05:09 accept(3, 0xbfe61ff0, [4096]) = ? ERESTARTSYS (To be restarted)
13:05:09 --- SIGCHLD (Child exited) @ 0 (0) ---
13:05:09 sigreturn() = ? (mask now [])
13:05:09 rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
13:05:09 rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
13:05:09 open("/opt/bdii/var/bdii-fwd.conf", O_RDONLY|O_LARGEFILE) = 4
13:05:09 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe62c68) = -1 ENOTTY (Inappropriate ioctl for device)
13:05:09 _llseek(4, 0, [0], SEEK_CUR) = 0
13:05:09 fstat64(4, {st_mode=S_IFREG|0664, st_size=5, ...}) = 0
13:05:09 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
13:05:09 read(4, "2172\n", 4096) = 5
13:05:09 close(4) = 0
13:05:09 waitpid(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG) = 8222
13:05:09 time(NULL) = 1209816309
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 time(NULL) = 1209816309
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
13:05:09 waitpid(-1, 0xbfe62ff8, WNOHANG) = -1 ECHILD (No child processes)
13:05:09 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
13:05:09 rt_sigaction(SIGCHLD, {0x5a3757, [], SA_RESTORER, 0xb779c8}, {0x5a3757, [], SA_RESTORER, 0xb779c8}, 8) = 0
13:05:09 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
13:05:09 accept(3,
|
