Hi Peer.

Peer Hasselmeyer wrote:
>
> Hi,
>
> I installed the LCG-1 software on our testbed here at the FZK
> and I wanted to write down my experiences with it. Maybe my
> (incomplete) notes help some other sites, but at least I hope
> somebody can answer the questions that came to my mind during
> the installation. I apologize for this long message, I hope
> somebody has the time to read through until the end as there
> are a number of questions. There are some more questions
> throughtout the text.

I'll try answering a few of your questions...

> Let me first introduce you to the setup. It consists of one CE,
> one SE, one UI, and 8 WNs (so far I only installed 2 of the WNs
> as I'm going to use the other ones for further installation
> testing). An important point is that the CE, SE, and UI machines
> have two network cards -- one connected to the Intra/Internet and
> one connected to an internal network. All WNs are only connected
> to this internal network using private IP addresses. Our setup
> is therefore somewhat different from most people's (I believe)
> and a number of the installation problems were due to this setup.

You are right: we never tried this configuration before. I have a few
nodes with double network interface, but only used one of the two.

> Let me also say that most problems were due to me not being an
> LCFG expert. Although I worked with LCFGng for about ten days
> right before starting to install LCG1, parts of it are still
> a mystery to me.

I have been working with it for 1.5 years and quite a few parts are
still a mistery to me :-) or also :-(

> Well, I started out with a working LCFGng server, but I wanted
> to have a fresh copy of all the installation RPMs, so I tried
> using updaterep. After waiting a long time and not really getting
> far, I decided to use another method downloading only the required
> packages with updateRPMCache.pl. This method is not really
> convenient as it downloads files from only one RPM list at a
> time, but at least it seemed much faster to me. Therefore the
> first question: is there an easier method that fills my RPM
> repository but avoids downloading GBs of unnecessary files?
> (Side note: some files were not downloaded correctly and the
> hd files could not be generated. This gave me some trouble later.)

This remark is correct: updaterep wants to do a full mirror copy of the
rpm repository in Lyon (7-8 GB now, I think) and takes a long time. The
good part is that, after the first mirror, you only get updates and this
is usually very fast.

UpdateRPMCache.pl is more clever (cleverer?) as it only downloads the
rpms which are needed for a specific rpm list and does not care about
the infinte number of obsolete versions of the same rpm.

We should think of better ways of creating the first repository...

> Fixing the given configuration files was not much of a headache,
> as most stuff was already correctly filled in (at least that's
> what I thought at that time -- more on this later). So I started
> by installing the UI as this seemed to be the one with the
> easiest setup.
>
> First thing I had to do was switch from grub to lilo as our
> machines don't seem to like grub very much. (I also modified
> the lilo lcfg object as I didn't get a symlink to vmlinuz
> automatically.) After a few tries I had the system running,
> but the second ethernet interface was not there. Adding a few
> lines to the configuration fixed this problem, but a bigger one
> was lying ahead: routing was not correctly set up. The file
> /etc/sysconfig/static-routes is needed for this but I couldn't
> find a way to have it created automatically. I therefore wrote
> my own LCFG object to handle this. Is there another, already
> existing  way to do this?

Not that I know of. The network object is only able to define a default
gateway but I think you have a more complex routing schema.

Can you please upload to CVS the configuration files you are using and
send us the object you wrote? It should be part of the general
distribution.

> This fixed the networking problem, so I could start installing
> CE and SE. Installation worked more or less okay, but I got
> a bit confused when shortly after starting the installation,
> the machines didn't do anything (at least anything observable)
> for about five minutes. I thought they crashed and rebooted
> them a couple of times before waiting long enough. This didn't
> happen with the UI or the WNs. Did anybody else encounter this
> behavior? What's going on during that time?

I have not observed this behaviour before. Does this happen during the
inital installation or after the first reboot? In anny case, I will keep
an eye open when we will install our nodes.

> And now the configuration problems started. I had to figure
> out what should be mounted where. I have doubts about the
> preconfigured mounting scheme. The CE mounts /flatfiles/SE00
> from the SE and the SE mounts the gridmapdir from the CE.
> I don't see how this could work without manual intervention
> as there is (at the maximum) only a very short time period
> where a machine exported its filesystems but did not mount
> external ones yet. And these time periods would have to match
> at the CE and the SE... I therefore decided to have the SE
> export all needed directories and have the CE (and WNs) mount
> them. Works fine. I also mounted our local home directories
> on the UI for a couple of local accounts. I also had to change
> the timezone (in my case to Europe/Berlin). The preconfigured
> value CEST didn't link to anything, so the machines booted up
> using UTC.

This NFS cross-mounting configuration is a kind of relic from times now
past: I think we should change it to something like the configuration
you are using. At CERN we mount everything from a disk server.

About the time zone, we always used Europe/Paris (which was included in
the example site-cfg.h file). It is indeed strange that CEST is not
correctly handled, though. I should note this in the installation
instructions.

> Now I started installing the WNs. I had to prepare a special
> bootdisk as the EDG one didn't boot properly on the WN that
> has an additional graphics card (because the internal one is
> broken). Again, this seems to be a problem with grub. I used
> to like grub, but now I'm not so convinced anymore...
>
> Starting the installation was kind of a hassle, as the WNs
> are on a different subnet than the LCFG server. I therefore
> couldn't use the DHCP server running on the LCFG box and had
> to reconfigure a different DHCP server. I also had to change
> the whole network configuration for the WNs as they have
> different gateways, hostnames etc. Other than that, the
> isntallation went fine.
>
> A problem was name resolving. As all WNs are on a private
> network, they do not have a DNS entry. Name resolving relies on
> a hosts file distributed via NIS. So I also had to configure
> NIS on all the machines. This works okay, but I couldn't persuade
> LCFG to automatically start ypbind at boot time. I added the line
> REPLACE(chkconfig.services,network,network ypbind) to my
> configuration, but to no avail. Does anybody know how to do
> this? (Side note: I tried to use numerical IP addresses in the
> beginning. Some LCFG objects do not like this at all and fail.
> You have to use names and they need to be resolvable!)

I had the same problem with an old layout where we used NIS for user
accounts: the ypbind startup was done by hand (well, we had a script for
this). The chkconfig can be used but the syntax should probably be

EXTRA(chkconfig.services) ypbind

but you should check if this is really all you need to start the NIS
client. I attached to this e-mail a (somewhat old) guide to NIS
configuration I wrote for the old EDG testbed.

> Another thing I was stuck for some time was that changes in
> the configuration were not automatically enacted by the
> LCFG objects. New profiles were fetched, but I had to call
> changed objects by hand. The reason for this was that some
> script (I think /usr/bin/om) does some kind of double reverse
> lookup of the hostname and this only works if the domain is
> properly set. And it wasn't in my case. Adding the FQDN to
> /etc/hosts solved this, i.e. changing
> 10.20.1.101   c20-001-101 c20-001-101
> to
> 10.20.1.101   c20-001-101.fzk.de c20-001-101
> I must have made a mistake in the configuration somewhere as
> it worked fine on the CE and SE...

I just checked: on our nodes /etc/hosts contains the full host name.

> Having configured the WNs I tried to submit jobs. Submitting
> PBS jobs directly worked well, and so did globus-job-run on
> the CE using fork. Submitting to a PBS queue failed with the
> error "Host key verification failed." Surprisingly, going to
> lunch, it suddenly worked! I found out today that this must
> have been due to SSH keys being moved around by a script
> (edg-pbs-knownhosts) that is invoked every couple of hours by
> cron. The 13:03 invocation fixed all my problems ;-)! Is this
> script supposed to be called during installation? Do I have
> to invoke it by hand after configuring everything? Could this
> please be written down somewhere?

Looking at the code, I saw that the pbskownhosts object only creates the
edg-pbs-knownhosts configuration file but does not run the script teh
first time. This means that yes, you either wait for cron to run the
script or you call it by hand. This is another issue we should consider.

>
> Well, since then, the testbed seems to work properly...
>
> I do have some more questions, though:
>
> * Is it possible to add/remove RPMs to/from the list of RPMs to
>    install without touching the RPM lists in lcg1/rpmlist (like
>    I can change the LCFG configuration without changing files in
>    lcg1/source)?

You may create a wrapper around the main rpm list file, called, e.g.,
CE-local-rpm, which goes like:

#include CE-rpm
<extra rpm to install>
+<rpm I want to install with a version different from the default one>
-<rpm I do not want to install>

and then adding to the node configuration file a line like

+updaterpms.rpmcfg               CE-local-rpm

which will replace the default rpm list, CE-rpm, with your new wrapper
file. You may look at the updaterpms man page to see all the details on
how to specify an rpm list.

> * In the same vein, lcg1/source/redhat73-cfg.h contains the
>    definition of RPMDIR right at the end. Could this be replaced by
> #ifndef RPMDIR
> #define RPMDIR                   /export/local/linux/7.3/RPMS
> #endif
>    so that I can change the directory in my site-cfg.h?

This is a good idea: site-cfg.h is definitely the place you want to set
this symbol. Will be done for the next release.

> * Is mounting the gridmapdir directory on the WNs necessary?

WNs need the /etc/grid-security/certificates directory in order to be
able to verify the identity of the servers they contact. As having a
big, possibly huge, number of WNs all around teh world trying and
contacting the CA certs distribution sites at the same time to get the
latest greatest CRL does not seem a good idea ;-), we chose to have a
single shared directory and let only one node doing the job for
everybody.

> * The LCFG object nfsmount does not automatically enact changes.
>    Adding the line "nfsmount.ng_reconfig restart" had the desired
>    effect. Is this a good idea? Why is it not done anyway?

I think I remember discussing this issue long time ago and the outcome
was that triggering an automatic update can cause big trouble to the
users of the node if the changes include the umount-ing a file system
while it is being used. The author then decided to have it run only
either at boot or by somebody who knows what's going on the node. I am
open to discussion on this, though...

> * lcg1/rpmlist/rh73-release-rpm.h contains rpm-4.0.2-7x.edg as
>    well as rpm-4.0.4-7x.18. Unfortunately, the older version gets
>    installed. What's the reason for this? Can I safely replace it
>    with the 4.0.4 version? I need it to get ucd-snmp running...

This is a long standing issue: the way I understand this, LCFG needs
some feature from the old rpm version which was removed with the new.
EDG then took action and created the EDG-specific version of the rpm.
Other people reading this list may have a more detailed view on the
subject.

> Sorry for the long message. I hope I helped somebody with
> my explanations.

I think that your message is exactly the kind of feedback needed to get
LCG into a production state. It would be very useful for everybody if
you could send all the final configuration files and the exact chanes
you had to apply to the default files to get your configuration up and
running.

> Thanks for reading,

Thank you

             Emanuele

> Peer
> --
> Peer Hasselmeyer                             [log in to unmask]
> Forschungszentrum Karlsruhe                        Tel.: +49-7247-828601
> Institute for Scientific Computing / Inst. f. Wissenschaftliches Rechnen
> Hermann-von-Helmholtz-Platz 1,  76344 Eggenstein-Leopoldshafen,  Germany

--
/------------------- Emanuele Leonardi -------------------\
| eMail: [log in to unmask] - Tel.: +41-22-7674066 |
|  IT division - Bat.31 2-012 - CERN - CH-1211 Geneva 23  |
\---------------------------------------------------------/


LCFG is not able to fully handle the NIS client activation. This poses a
problem when installing an new node which wants to use it. At CERN we use NIS
as the basis for access to service nodes and for UI account definition.

This is a partly manual recipe to setup and activate NIS on a geenric EDG node
handled with LCFG..

1) Install the node as usual. In the node's LCFG configuration file add the
partial NIS configuration from the nisclient-pro-cfg.h in the lxshare0371
source directory.

@------------------------------
[root@lxshare0371 source]# cat nisclient-pro-cfg.h
+auth.nsswitch ignore
+update.ypserver              edg-nfs00.cern.ch
EXTRA(boot.services)          nsswitch
EXTRA(boot.run)               nsswitch
EXTRA(profile.components)     nsswitch
+nsswitch.mods_passwd   compat
+nsswitch.mods_shadow   files nis
+nsswitch.mods_group    files nis
+nsswitch.mods_hosts    files dns [NOTFOUND=return]
+nsswitch.mods_netgroup files nis
@------------------------------

2) If the node was installed from scratch, then the nsswitch has already been
executed. If you added the nisclient section to an already installed node, then
you have to run the nsswitch object by hand:

> /etc/obj/nsswitch start

3) Setup the remaining part of the NIS client with the following operations:

- set the domain name with

  > domainname edg-tb

- make this setting persistent by editing the /etc/sysconfig/network file
  and adding the line "NISDOMAIN=edg-tb"
- add the line "+::::::" at the end of the /etc/passwd file
- add the line "+:::" at the end of the /etc/group file
- start the ypbind daemon with

  > /etc/rc.d/init.d/ypbind start

  Note: the daemon might report a successful connection to lxshare072d: this
  is the physical name of edg-nfs00.
- make the daemon startup persistent with

  > /sbin/chkconfig ypbind on

4) All directories exported by edg-nfs00 can be mounted on a node by adding the
nfsmount-cfg.h script in the node LCFG configuration file.

Please note: while a UI node mounts the user home directories from the UIhome
area on edg-nfs00, all service nodes either have a local /home area (RB,MDS)
or share a common home area from CEhome on edg-nfs00. This means that on
service nodes users will not have a perosnal home directory: this is consistent
with the plan of only allowing them to run sudo commands there. In special
cases, users may want to have a personal /home directory on service nodes. In
this case the node manager can create it locally or create it in CEhome
(/shift/lxshare072d/data02/site_pro-1.2/CEhome on edg-nfs00), depending on the
node.