Hi Jeremy,
Some comments:
(1) FTS3 originally supported MyProxy but this was removed when it was found that no one needed this functionality. Maybe as an alternative using MyProxy and a cron which runs fts-delegation-init may work? (I've never tried this).
(2) There is no limit on the number of files in a single transfer job (at least at the level Lydia is talking about). The limit she's seeing results from a single statement being sent to the database whose size exceeds max_allowed_packet. This can be increased of course, but first it would be good to know how many files Lydia would like to have in a single job, as this will determine how big max_allowed_packet will need to be.
Note that having millions of files (even hundreds of thousands of files) in a single job is generally not what people do and is new territory.
(3) Information about transfers is only kept in the main database tables for 7 days then moved to backup tables. If this wasn't done FTS would grind to a halt very quickly. fts-transfer-status has a "-a" option for querying the archive.
Regards,
Andrew.
________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Jeremy Coles [[log in to unmask]]
Sent: Monday, July 27, 2015 11:49 PM
To: [log in to unmask]
Subject: Fwd: questions and issues raised on transfering data to RAL
Dear All,
I think this email may be of wider interest and will come up at our ops meeting when we discuss DIRAC, so I forward it to this list.
Jeremy
Begin forwarded message:
Date: 23 July 2015 17:46:55 BST
Reply-To: Lydia Heck <[log in to unmask]<mailto:[log in to unmask]>>
From: Lydia Heck <[log in to unmask]<mailto:[log in to unmask]>>
Subject: questions and issues raised on transfering data to RAL
To: <[log in to unmask]<mailto:[log in to unmask]>>
Hi Brian,
as promised here are my questions/requests/suggestions:
(1) I can create voms proxies that have very long lifetimes, however the
server system to which I transfer the data will only authorize for 24 hours from time of submission.
As I am transfering large amounts of files and data, this 24 hour window is
very crippling as I cannot say in advance how long individual jobs will take.
Jobs that have been submitted with a valid proxy will fail if they have not completed within the 24 hour window of the proxy under which they have been submitted.
(2) I have experimented and found that per `job' I can only submit <=2048 <4096 files. I have experimented and I do know that 4096 files is too many. Then I settled for 2048; so I am not sure if 2048 is the maximum.
This is very limiting, as I have more than 88M files to transfer, which will mean ~43,000 submissions. Coupled with the proxy limitations, the job will become arduous indeed. Could that limit be raised?
(3) When trying to follow the progress of jobs, I got some very useful links
from you and Jens, however I would not remember the strings of the web addresses, if I had not copied them into some documentation.
I would like to have an easily remembered web address that would, with some clicks, be the portal to all the other useful information about the jobs and files that have been transmitted. Would that be possible?
Currently I have
https://lcgcadm04.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/vo.dirac.ac.uk/DiRAC/tape/durham.ac.uk/
which is most helpful and possibly as simple as it can get.
I have now sussed (I think) how to use the
https://lcgfts3.gridpp.rl.ac.uk:8449/fts3/ftsmon/#/
site and I have managed to identify easier what has failed.
So maybe this will become more routine as I go along and maybe I have the "easily" remembered webpages already.
However that page only allows me to check the jobs for 7 days. Is there any page
which can give a more complete history or a custom time window?
Are there any other pages that I could/should use ?
Best wishes,
Lydia
|