Hi,

We are in the works with an important HW upgrade to our site
(RO-09-UTCN), and we have to decide for a storage architecture.

We have 64 WN nodes, which have at least two Gbit interfaces(4 of them
have 4 interfaces) bonded together, and a single 146 GB SAS drive each.

We also have two other machines with 6 drives each, identical to those
on the WNs. We will be configuring them in RAID1+0 mode, and there we'll
be running Xen or OpenVZ(TBD) VMs for all the servers. These machines
will be mirrored using DRBD and will be using Linux-HA to maximize
availability.

The SE node will run virtualized along with the others and will have
access to a dedicated iSCSI volume. 

We can have at most 2 bonded Gbit interfaces for the internal network
connection, which will be shared by all the VMs. Would this be enough
for the SE and CE storage performance?

The problem we are facing now is about choosing the storage layout we
are to use for the shared home directory. How much space should we
allocate for this volume? Also, what is the usage pattern we should
expect from such a site? I'm asking because our old site is quite
small(~10 nodes) and pretty much unused so our statistics may not be
relevant since the usage will most likely increase. 

Regarding the filesystem, we should choose among NFS, Lustre, glusterfs
and GFS2. What any of these do you suggest for our site?
Would it worth the effort of having full redundancy for the /home
storage, or only regular backups?

Here are some ideas we had:

1) mirrored(or not?) NFS server (I incline for this so far)
- Would it be enough for our number of nodes, even if we will use
bonding? In case not, see the other alternatives.
- Would it be ok to be installed on the same hardware with a WN node
designated for SAM jobs only?

2) sacrifice 4 WNs (those with 4 interfaces) for a small Lustre cluster
Is it ok to run WN along with Lustre on these nodes?
Configurations:
	- 1 MDS and 3 OSS - no redundancy
	- 2 MDS and 2 OSS - mirrored, in case we need redundancy - we would
loose an interface out of 4 (it needs to dedicated to DRBD)
drawbacks:
	-we need to have Lustre packages installed and the Lustre kernel, which
doesn't work from the CE if we use OpenVZ
	-I have no knowledge of a Yum repo for Lustre, and Sun's download sites
really suck. Updating Lustre will be a small nightmare in case of future
kernel vulnerabilities.

3) what about using glusterfs instead of Lustre? I favor it a bit over
Lustre because it's a fully userspace solution, no kernel package is
needed and we can use it in OpenVZ VMs.

4) use all the WNs as Lustre OSS nodes and have a virtualized MDS node
running in Xen.
	- really a nightmare in case of kernel issues that need updates.

5) GFS2 cluster on the 4 more gifted WNs, with data shared using NFS to
all the others so that the NFS load would be evenly-distributed among
them (~15 clients/storage node).

What option do you think it would be more suited for my site, or say if
you have any other suggestions that I may be missing here..

Best regards,
Cristi

-- 
Cristi Măgherușan, alumnus System/Network Engineer
Technical University of Cluj-Napoca, Romania
http://cc.utcluj.ro  +40264 401247