 |
 |
-------- Forwarded Message -------- Subject: Data transfer technologies Date: Sat, 20 Jun 2015 15:46:06 +0100 From: David Colling <[log in to unmask]> Reply-To: National E-Infrastructure Project Directors Group <[log in to unmask]> To: [log in to unmask] Dear PDG members, I am not sure if this is the best list to which to send this but it was the best that I can think of at at the moment. If anybody has suggestion for a better list then please suggest. In conversations with members of this group it has become clear that we are in increasing need of a data transfer system capable of handling large amounts of data in a flexible and easy to use manner. The HEP community already has a tool to do this called FTS but it is not a simple standalone tool and generally assumes that quite a lot of the software that our community uses is installed. So I asked Sam Skipsey to look into what we should suggest that other communities with who GridPP works should use. Sam reported back to our fortnightly GridPP technical meeting yesterday. He has only had a chance to do fairly superficial work and he is going to look more deeply but I still think that he report was interesting. His presentation can be found at: https://indico.cern.ch/event/403303/ David F. joined our meeting yesterday as well so if I misrepresent anything he can correct me. The bottom line would appear to be that there are two possibilities for the sort of functionality that I think people are looking for. These are Globus Connect or WebFTS. Both work on the same underlying technology and offer a very similar service however both have pros and cons. Sam has a direct comparison on slide 16 but for those too busy to read Sam's talk I will give a very brief summary below. The Globus service is pretty good and has much (if not quite all) the functionality that people are looking for. It supports a range of third party authentication methods as well as the default X.509 certificates (and even when using certificates it can do so in a hiden way). The downside is that while some of the basic functionality is free Globus have this to make money and to get the full range of functionality it costs $90K/year/grouping. WebFTS is still only a beta release and is currently much less polished than Globus. It's range of supported third authentication is more limited and is not as smooth to set up (yet). However it is open source and is free and will remain so. I am almost ambivalent as to what we choose as HEP will still use FTS for the transfers of large amounts of data (not using any web interface) and from Sam's review both of the leading options seem to be pretty good if at different stages of development. My thoughts were that if the UK communities are going to use Globus en masse then we should probably try to get a UK wide price from Globus rather than each community paying these few $K for the specific range of services and data volume that they want to move. If the UK decides that the free WebFTS is the way forward then I think that it would probably be a good thing to pay somebody to work with the developers (for a year or two) to make sure that we get the functionality that we want in there reasonably soon and that nothing that matters to us is overlooked. I said that I was almost ambivalent and the reason that I am only almost is because I think that if we worked with the WebFTS people we would have a far greater possibility of getting support in there for things like JISC Assent and safe share than we would with Globus. However that is just my feeling and I have no real evidence to back this up. Anyway, I thought it worth disseminating this information as it is a topic that we have often discussed. BTW ... the next GridPP Technical meeting in just under a fortnight will focus on traceability in clouds and anybody interested is welcome to come along. Best, david