Hi,
I installed the LCG-1 software on our testbed here at the FZK
and I wanted to write down my experiences with it. Maybe my
(incomplete) notes help some other sites, but at least I hope
somebody can answer the questions that came to my mind during
the installation. I apologize for this long message, I hope
somebody has the time to read through until the end as there
are a number of questions. There are some more questions
throughtout the text.
Let me first introduce you to the setup. It consists of one CE,
one SE, one UI, and 8 WNs (so far I only installed 2 of the WNs
as I'm going to use the other ones for further installation
testing). An important point is that the CE, SE, and UI machines
have two network cards -- one connected to the Intra/Internet and
one connected to an internal network. All WNs are only connected
to this internal network using private IP addresses. Our setup
is therefore somewhat different from most people's (I believe)
and a number of the installation problems were due to this setup.
Let me also say that most problems were due to me not being an
LCFG expert. Although I worked with LCFGng for about ten days
right before starting to install LCG1, parts of it are still
a mystery to me.
Well, I started out with a working LCFGng server, but I wanted
to have a fresh copy of all the installation RPMs, so I tried
using updaterep. After waiting a long time and not really getting
far, I decided to use another method downloading only the required
packages with updateRPMCache.pl. This method is not really
convenient as it downloads files from only one RPM list at a
time, but at least it seemed much faster to me. Therefore the
first question: is there an easier method that fills my RPM
repository but avoids downloading GBs of unnecessary files?
(Side note: some files were not downloaded correctly and the
hd files could not be generated. This gave me some trouble later.)
Fixing the given configuration files was not much of a headache,
as most stuff was already correctly filled in (at least that's
what I thought at that time -- more on this later). So I started
by installing the UI as this seemed to be the one with the
easiest setup.
First thing I had to do was switch from grub to lilo as our
machines don't seem to like grub very much. (I also modified
the lilo lcfg object as I didn't get a symlink to vmlinuz
automatically.) After a few tries I had the system running,
but the second ethernet interface was not there. Adding a few
lines to the configuration fixed this problem, but a bigger one
was lying ahead: routing was not correctly set up. The file
/etc/sysconfig/static-routes is needed for this but I couldn't
find a way to have it created automatically. I therefore wrote
my own LCFG object to handle this. Is there another, already
existing way to do this?
This fixed the networking problem, so I could start installing
CE and SE. Installation worked more or less okay, but I got
a bit confused when shortly after starting the installation,
the machines didn't do anything (at least anything observable)
for about five minutes. I thought they crashed and rebooted
them a couple of times before waiting long enough. This didn't
happen with the UI or the WNs. Did anybody else encounter this
behavior? What's going on during that time?
And now the configuration problems started. I had to figure
out what should be mounted where. I have doubts about the
preconfigured mounting scheme. The CE mounts /flatfiles/SE00
from the SE and the SE mounts the gridmapdir from the CE.
I don't see how this could work without manual intervention
as there is (at the maximum) only a very short time period
where a machine exported its filesystems but did not mount
external ones yet. And these time periods would have to match
at the CE and the SE... I therefore decided to have the SE
export all needed directories and have the CE (and WNs) mount
them. Works fine. I also mounted our local home directories
on the UI for a couple of local accounts. I also had to change
the timezone (in my case to Europe/Berlin). The preconfigured
value CEST didn't link to anything, so the machines booted up
using UTC.
Now I started installing the WNs. I had to prepare a special
bootdisk as the EDG one didn't boot properly on the WN that
has an additional graphics card (because the internal one is
broken). Again, this seems to be a problem with grub. I used
to like grub, but now I'm not so convinced anymore...
Starting the installation was kind of a hassle, as the WNs
are on a different subnet than the LCFG server. I therefore
couldn't use the DHCP server running on the LCFG box and had
to reconfigure a different DHCP server. I also had to change
the whole network configuration for the WNs as they have
different gateways, hostnames etc. Other than that, the
isntallation went fine.
A problem was name resolving. As all WNs are on a private
network, they do not have a DNS entry. Name resolving relies on
a hosts file distributed via NIS. So I also had to configure
NIS on all the machines. This works okay, but I couldn't persuade
LCFG to automatically start ypbind at boot time. I added the line
REPLACE(chkconfig.services,network,network ypbind) to my
configuration, but to no avail. Does anybody know how to do
this? (Side note: I tried to use numerical IP addresses in the
beginning. Some LCFG objects do not like this at all and fail.
You have to use names and they need to be resolvable!)
Another thing I was stuck for some time was that changes in
the configuration were not automatically enacted by the
LCFG objects. New profiles were fetched, but I had to call
changed objects by hand. The reason for this was that some
script (I think /usr/bin/om) does some kind of double reverse
lookup of the hostname and this only works if the domain is
properly set. And it wasn't in my case. Adding the FQDN to
/etc/hosts solved this, i.e. changing
10.20.1.101 c20-001-101 c20-001-101
to
10.20.1.101 c20-001-101.fzk.de c20-001-101
I must have made a mistake in the configuration somewhere as
it worked fine on the CE and SE...
Having configured the WNs I tried to submit jobs. Submitting
PBS jobs directly worked well, and so did globus-job-run on
the CE using fork. Submitting to a PBS queue failed with the
error "Host key verification failed." Surprisingly, going to
lunch, it suddenly worked! I found out today that this must
have been due to SSH keys being moved around by a script
(edg-pbs-knownhosts) that is invoked every couple of hours by
cron. The 13:03 invocation fixed all my problems ;-)! Is this
script supposed to be called during installation? Do I have
to invoke it by hand after configuring everything? Could this
please be written down somewhere?
Well, since then, the testbed seems to work properly...
I do have some more questions, though:
* Is it possible to add/remove RPMs to/from the list of RPMs to
install without touching the RPM lists in lcg1/rpmlist (like
I can change the LCFG configuration without changing files in
lcg1/source)?
* In the same vein, lcg1/source/redhat73-cfg.h contains the
definition of RPMDIR right at the end. Could this be replaced by
#ifndef RPMDIR
#define RPMDIR /export/local/linux/7.3/RPMS
#endif
so that I can change the directory in my site-cfg.h?
* Is mounting the gridmapdir directory on the WNs necessary?
* The LCFG object nfsmount does not automatically enact changes.
Adding the line "nfsmount.ng_reconfig restart" had the desired
effect. Is this a good idea? Why is it not done anyway?
* lcg1/rpmlist/rh73-release-rpm.h contains rpm-4.0.2-7x.edg as
well as rpm-4.0.4-7x.18. Unfortunately, the older version gets
installed. What's the reason for this? Can I safely replace it
with the 4.0.4 version? I need it to get ucd-snmp running...
Sorry for the long message. I hope I helped somebody with
my explanations.
Thanks for reading,
Peer
--
Peer Hasselmeyer [log in to unmask]
Forschungszentrum Karlsruhe Tel.: +49-7247-828601
Institute for Scientific Computing / Inst. f. Wissenschaftliches Rechnen
Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany
|
