Hi,
You don't need pnfs installed on the pool node. There should only be one pnfs running in your entire dcache instance.
What you need to do is nfs mount the pnfs service from your admin node on the pool node.
On the admin node, go to /pnfs/fs/admin/etc/exports and copy the file 127.0.0.1 to a file that has the ip address of your pool node as the name, then cd into the trusted directory and do the same.
Then on the pool node add something like
gw04.hep.ph.ic.ac.uk:/fs /pnfs/fs nfs hard,intr,rw,noac,auto 0 0
to your /etc/fstab, create the /pnfs/fs directory, mount it and create the symlink
Then, to run a gridftp door on your pool node:
Set the /opt/d-cache/etc/door_config file on the pool node to
ADMIN_NODE gw04.hep.ph.ic.ac.uk
door active
--------------------
GSIDCAP no
GRIDFTP yes
SRM no
and then run /opt/d-cache/install/install_doors.sh
and then start dcache-opt.
dCache uses the name of things to work out which one to use. so if you have two gridftp doors named GFTP then the one that started last gets used. What the install_doors script does is change the name to something more unique (GFTP-<hostname -s> basically) so you don't get name collisions. You can change the names yourself in the /opt/d-cache/config/*.batch files
Derek
-----Original Message-----
From: GRIDPP2: Deployment and support of SRM and local storage management [mailto:[log in to unmask]]On Behalf Of Mona Aggarwal
Sent: 17 May 2005 14:16
To: [log in to unmask]
Subject: dCache- clients transfer files
Hi all,
At IC we have the following dCache nodes:
admin_node --- gw04.hep.ph.ic.ac.uk
pool_node --- gw03.hep.ph.ic.ac.uk
We want to configure the pool node such that clients transfer
files directly to/from the pool node, and not via the admin node.
We installed pnfs and dcache-opt rpms on the pool node
and started pnfs and dcache-opt (including gsiftp) services.
> /opt/pnfs.3.1.10/pnfs/install/pnfs-install.sh
NB. There was no pnfs configuration (this seems to be part of the
dcache-core package, which of course we did not want to install on the
pool node!) so:
>/sbin/service pnfs start
>pnfs unrecognised service
so we used
> /opt/pnfs.3.1.10/pnfs/bin/pnfs start
then we manually created the
/pnfs/fs/usr/data/dteam
directory and set the ownership and permissions and
> ln -s /pnfs/fs/usr /pnfs/hep.ph.ic.ac.uk
We also started dcahe-opt service on the pool node
>[root@gw03 root]# /opt/d-cache/bin/dcache-opt start
>Starting dcache services: Starting gridftpdoorDomain 6 5 4 3 2 1 0 Done (pid=30168)
>Starting gsidcapdoorDomain 6 5 4 3 2 1 0 Done (pid=30250)
>Starting srmDomain 6 5 4 3 2 1 0 Done (pid=30333)
Then we conducted tests with srmcp and following are our observations.
1. If pnfs service is not running on the pool_node then it is not
selected by the dCache as a TURL. Admin_node is selected
as a TURL by default and srmcp is succesfull.
2. If pnfs service is running on the pool_node then it fails, if the TURL is selected through it.
TURL=gsiftp://gw03.hep.ph.ic.ac.uk:2811///dteam/testfile1721
// Error message
-------------------------------------------------------------------------------------------------
[root@gw04 root]# /opt/d-cache/srm/bin/srmcp -debug=true -x509_user_proxy=./mtmproxy file:////root/testfile srm://gw04.hep.ph.ic.ac.uk:8443/pnfs/hep.ph.ic.ac.uk
/data/dteam/testfile1721
SRM Configuration:
debug=true
gsissl=true
help=false
pushmode=false
userproxy=true
buffer_size=2048
tcp_buffer_size=0
config_file=/root/.srmconfig/config.xml
glue_mapfile=/opt/d-cache/srm/conf/SRMServerV1.map
webservice_path=srm/managerv1.wsdl
webservice_protocol=https
gsiftpclinet=globus-url-copy
protocols_list=http,gsiftp
save_config_file=null
srmcphome=/opt/d-cache/srm
urlcopy=/opt/d-cache/srm/bin/url-copy.sh
x509_user_cert=/root/k5-ca-proxy.pem
x509_user_key=/root/k5-ca-proxy.pem
x509_user_proxy=./mtmproxy
x509_user_trusted_certificates=/root/.globus/certificates
retry_num=20
retry_timeout=10000
wsdl_url=null
use_urlcopy_script=false
connect_to_wsdl=false
from[0]=file:////root/testfile
to=srm://gw04.hep.ph.ic.ac.uk:8443/pnfs/hep.ph.ic.ac.uk/data/dteam/testfile1721
Tue May 17 13:49:10 BST 2005: starting SRMPutClient
Tue May 17 13:49:10 BST 2005: SRMClient(https,srm/managerv1.wsdl,true)
Tue May 17 13:49:10 BST 2005: connecting to server
Tue May 17 13:49:10 BST 2005: connected to server, obtaining proxy
SRMClientV1 : connecting to srm at httpg://gw04.hep.ph.ic.ac.uk:8443/srm/managerv1
Tue May 17 13:49:13 BST 2005: got proxy of type class org.dcache.srm.client.SRMClientV1
SRMClientV1 : put, sources[0]="/root/testfile"
SRMClientV1 : put, dests[0]="srm://gw04.hep.ph.ic.ac.uk:8443/pnfs/hep.ph.ic.ac.uk/data/dteam/testfile1721"
SRMClientV1 : put, protocols[0]="http"
SRMClientV1 : put, protocols[1]="dcap"
SRMClientV1 : put, protocols[2]="gsiftp"
SRMClientV1 : put, contacting service httpg://gw04.hep.ph.ic.ac.uk:8443/srm/managerv1
doneAddingJobs is false
copy_jobs is empty
Tue May 17 13:49:17 BST 2005: srm returned requestId = -2147483536
Tue May 17 13:49:17 BST 2005: sleeping 1 seconds ...
Tue May 17 13:49:19 BST 2005: FileRequestStatus with SURL=srm://gw04.hep.ph.ic.ac.uk:8443/pnfs/hep.ph.ic.ac.uk/data/dteam/testfile1721 is Ready
Tue May 17 13:49:19 BST 2005: received TURL=gsiftp://gw03.hep.ph.ic.ac.uk:2811///dteam/testfile1721
doneAddingJobs is false
copy_jobs is not empty
copying CopyJob, source = file:////root/testfile destination = gsiftp://gw03.hep.ph.ic.ac.uk:2811///dteam/testfile1721
GridftpClient: connecting to gw03.hep.ph.ic.ac.uk on port 2811
GridftpClient: gridFTPClient tcp buffer size is set to 1048576
GridftpClient: gridFTPWrite started, source file is java.io.RandomAccessFile@140984b destination path is //dteam/testfile1721
GridftpClient: parallelism: 10
GridftpClient: adler 32 for file java.io.RandomAccessFile@140984b is 82af088a
GridftpClient: waiting for completion of transfer
GridftpClient: gridFtpWrite: starting the transfer in emode to //dteam/testfile1721
GridftpClient: DiskDataSink.close() called
-----------------------------------------------------------------------------------
The client then hangs for a large time before failing with Java errors
saying no such file or directory for /pnfs/hep.ph.ic.ac.uk/data/dteam
(which is why we manually created it on gw03).
//java error
-----------------------------------------------------------------------------------
GridftpClient: DiskDataSink.close() called
org.globus.ftp.exception.ServerException: Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 553 /dteam/testfile1721: No such file or directory. : /dteam]. Nested exception is org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message: Unexpected reply: 553 /dteam/testfile1721: No such file or directory. : /dteam
at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:167)
GridftpClient: transfer exception
org.globus.ftp.exception.ServerException: Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 553 /dteam/testfile1721: No such file or directory. : /dteam]. Nested exception is org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message: Unexpected reply: 553 /dteam/testfile1721: No such file or directory. : /dteam
at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:167)
GridftpClient: closing client : org.dcache.srm.util.GridftpClient$FnalGridFTPClient@1a1399
GridftpClient: closed client
copy failed with the error
org.globus.ftp.exception.ServerException: Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 553 /dteam/testfile1721: No such file or directory. : /dteam]. Nested exception is org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message: Unexpected reply: 553 /dteam/testfile1721: No such file or directory. : /dteam
at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:167)
try again
sleeping for 30000 before retrying
copying CopyJob, source = file:////root/testfile destination = gsiftp://gw03.hep.ph.ic.ac.uk:2811///dteam/testfile1721
GridftpClient: connecting to gw03.hep.ph.ic.ac.uk on port 2811
GridftpClient: gridFTPClient tcp buffer size is set to 1048576
GridftpClient: gridFTPWrite started, source file is java.io.RandomAccessFile@18e8541 destination path is //dteam/testfile1721
GridftpClient: parallelism: 10
GridftpClient: adler 32 for file java.io.RandomAccessFile@18e8541 is 82af088a
GridftpClient: waiting for completion of transfer
GridftpClient: gridFtpWrite: starting the transfer in emode to //dteam/testfile1721
GridftpClient: DiskDataSink.close() called
-----------------------------------------------------------------------------------
We're not completely clear when/how gw03 selected over gw04 for the
TURL: if pnfs is not running on one, the other is always selected, but
if pnfs is running on both the choice seems almost to depend on which
service was last restarted!
Any suggestions?
Thanks,
Mona & Owen
===========================================================
Imperial College London
High Energy Physics Department
The Blackett Laboratory
Prince Consort Road, London, SW7 2BW
