 |
 |
Hi, My below email did'nt attract any response, so I am giving another try. 1. We have to find replacement hardware (soon will be out of warranty ) for software area server, So what would be minimum or ideal specs for it ? 2. As atlas already use cvmfs, Is that OK to keep ATLAS LOCAL Area on non-raid machine with 4 cores. ? 3. Which file system people used xfs, ext3 or ext4 ? You are more than welcome to reply on last email as well :) Thanks Govind On Fri, Sep 7, 2012 at 10:35 AM, Govind Songara <[log in to unmask]> wrote: > Hi, > > One of our storage server which host software area for other VO's and > atlas > local area having problem for very long time, it happens 3-4 times in a > month average. > > > Sep 7 03:05:01 storage001 kernel: INFO: task nfsd:5696 blocked for more > than 120 seconds. > Sep 7 03:05:01 storage001 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Sep 7 03:05:01 storage001 kernel: Call Trace: > Sep 7 03:05:01 storage001 kernel: [<ffffffff8006468c>] > __down_read+0x7a/0x92 > Sep 7 03:05:01 storage001 kernel: [<ffffffff886dbcc4>] > :xfs:xfs_iget+0x142/0x17a > Sep 7 03:05:01 storage001 kernel: [<ffffffff886fa220>] > :xfs:xfs_fs_get_dentry+0x3e/0xae > > The load on server increased dramatically up 256 then it crashed nfsd and > need power cycle to reboot. > > > We have applied various tuning but none of them help. > =================== > # to avoid high load > echo "64" > /sys/block/sdb/queue/max_sectors_kb > blockdev --setra 16384 /dev/sdb1 > > # RAID consumes all memory when high load > echo 524288 > /proc/sys/vm/min_free_kbytes > > /sbin/blockdev --setra 16384 /dev/sda > /sbin/blockdev --setra 16384 /dev/sdb > > > echo 3 > /proc/sys/vm/dirty_background_ratio > echo 10 > /proc/sys/vm/dirty_ratio > echo 20 > /proc/sys/vm/overcommit_ratio > sync > echo 0 > /proc/sys/vm/zone_reclaim_mode > echo 524288 > /proc/sys/vm/min_free_kbytes > echo 1546176 2061568 3092552 > /proc/sys/net/ipv4/tcp_mem > echo 4096 10000000 16777216 > /proc/sys/net/ipv4/tcp_wmem > echo 4096 10000000 16777216 > /proc/sys/net/ipv4/tcp_rmem > echo 16777216 > /proc/sys/net/core/wmem_max > echo 16777216 > /proc/sys/net/core/rmem_max > echo 10000000 > /proc/sys/net/core/wmem_default > echo 10000000 > /proc/sys/net/core/rmem_default > =================== > > > Worker nodes mount software area with following options, if there anything > wrong ?? > storage001.cm.cluster:/raid1/exp_software /vosw nfs > rsize=32768,wsize=32768,hard,intr,async 0 0 > > > We are thinking to move atlas local area (which is just few MB, only setup > scripts) to some other server. > > Does anyone keeping it on non-raid server? > We use nfs3, does nfs4 make any difference on it or SL6 would be better > for > it.? > Any other suggestions? > > Thanks > Govind >