-------- Forwarded Message --------
Subject: Data transfer technologies
Date: Sat, 20 Jun 2015 15:46:06 +0100
From: David Colling <[log in to unmask]>
Reply-To: National E-Infrastructure Project Directors Group
<[log in to unmask]>
To: [log in to unmask]

Dear PDG members,

I am not sure if this is the best list to which to send this  but it was
the best that I can think of at at the moment. If anybody has suggestion
for a better list then please suggest.

In conversations with members of this group it has become clear that we
are in increasing need of a data transfer system capable of handling
large amounts of data in a flexible and easy to use manner. The HEP
community already has a tool to do this called FTS but it is not a
simple standalone tool and generally assumes that  quite a lot of the
software that our community uses is installed. So I asked Sam Skipsey
to look into what we should suggest that other communities with who
GridPP works should use. Sam reported back to our fortnightly GridPP
technical meeting yesterday. He has only had a chance to do fairly
superficial work and he is going to look more deeply but I still think
that he report was interesting. His presentation can be found at:
https://indico.cern.ch/event/403303/

David F. joined our meeting yesterday as well so if I misrepresent
anything he can correct me.

The bottom line would appear to be that there are two possibilities for
the sort of functionality that I think people are looking for. These are
Globus Connect or WebFTS. Both work on the same underlying technology
and offer a very similar service however both have pros and cons. Sam
has a direct comparison on slide 16 but for those too busy to read Sam's
talk I will give a very brief summary below.


The Globus service is pretty good and has much (if not quite all) the
functionality that people are looking for. It supports a range of third
party authentication methods as well as the default X.509 certificates
(and even when using certificates it can do so in a hiden way). The
downside is that while some of the basic functionality is free Globus
have this to make money and to get the full range of functionality it
costs $90K/year/grouping.

WebFTS is still only a beta release and is currently much less polished
than Globus. It's range of supported third authentication is more
limited and is not as smooth to set up (yet).  However it is open source
and is free  and will remain so.

I am almost ambivalent as to what we choose as HEP will still use FTS
for the transfers of large amounts of data (not using any web interface)
and from Sam's review both of the leading options seem to be pretty good
if at different stages of development. My thoughts were that if the UK
communities are going to use Globus en masse then we should probably try
to get a UK wide price from Globus rather than each community paying
these few $K for the specific range of services and data volume that
they want to move.
If the UK decides that the free WebFTS is the way forward then I think
that it would probably be a good thing to pay somebody to work with the
developers (for a year or two)  to make sure that we get the
functionality that we want in there reasonably soon and that nothing
that matters to us is overlooked.

I said that I was almost ambivalent and the reason that I am only almost
is because I think that if we worked with the WebFTS people we would
have a far greater possibility of getting support in there for things
like JISC Assent and safe share than we would with Globus. However that
is just my feeling and I have no real evidence to back this up.

Anyway, I thought it worth disseminating this information as it is a
topic that we have often discussed.

BTW ... the next GridPP Technical meeting in just under a fortnight will
focus on traceability in clouds and anybody interested is welcome to
come along.

Best,
david