Hi,

So, it's worth noting that every single one of the technologies that
Matthew mentions by name (except iRODS) is a UDP-based "network
saturating" transfer tool, with exactly the same aim (to remove the
perceived deficiencies of TCP as a high-bandwidth data transfer
protocol for high-latency-high-bandwidth or nonzero packetloss
systems).

Some of them also come with a support ecosystem which sits reliable
transfer / syncing etc interfaces on top of them, but the real selling
point for each of them is that they do "really fast transfers by not
being TCP".

As far as I have ever been able to tell, the commercial operators have
very slick products, but also charge enough that they don't want their
prices on the public web. Tsunami (and the other open source products)
do about 90% as well, but are much less "slick", as with many open
source technologies.
(In any case, I didn't consider them because I have no evidence that
any of the communities that GridPP directly talks to are using them,
probably because of the cost of buy-in.)

If we're interested in post-TCP technologies, though...
While GridFTP is a tool designed to "do fast transfers *despite* TCP",
mostly by optimising TCP stream characteristics and allowing parallel
transfer streams, there was actually work done to allow it to use
non-TCP mechanisms for transfer (Barney Garrett did a bit of this work
at Edinburgh in about 2006, I recall, and the instructions for
enabling UDT in GridFTP are actually in the Globus documentation
nowadays - http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/admin/#gridftp-config-udt
. A study, for those who are interested, from 2009, is here
http://www.mcs.anl.gov/~kettimut/publications/PFLDNeT09.pdf ). I don't
believe that WLCG has ever required or supported GridFTP-over-UDT, but
it doesn't seem that FTS itself would notice the difference (as it
just orchestrates the transfers, and is somewhat agnostic as to how
they get done). Similarly, the documentation on Globus Connect does
not mention if the GridFTP implementation used will configure to use
UDT automatically, but I can't see that Globus Connect would mind
which protocol the transfer happened over.

I think there are several different things being conflated into one
topic here - my presentation was intended to cover the higher-level
"orchestration" tools, but the forwarded reply mixes this up with
post-TCP data transfer protocols, and the choice of actual transfer
tool (GridFTP v etc) . (Even in WLCG, we have at least 3 different
transfer applications, and it isn't clear that GridFTP is necessarily
the best of them; but it is the most guaranteed to exist at the
moment.) There's also the vexed socio-political question of the policy
concerning closed v open tools and standards, which has also not been
addressed here.

Sam

On 22 June 2015 at 09:12, David Colling <[log in to unmask]> wrote:
> -------- Forwarded Message --------
> Subject: Re: Data transfer technologies
> Date: Mon, 22 Jun 2015 07:33:36 +0000
> From: Matthew Dovey <[log in to unmask]>
> Reply-To: National E-Infrastructure Project Directors Group
> <[log in to unmask]>
> To: [log in to unmask]
>
>> My thoughts were that if the UK communities are going to use
>> Globus en masse then we should probably try to get a UK wide price from Globus
>> rather than each community paying these few $K for the specific range of
>> services and data volume that they want to move.
>
> In principle, this is something that Jisc may be able to help with -
> depending on cost and demand.
>
> However, one thing I am curious about is the lack of consideration of
> some of the commercial offerings in this space (e.g. FASP, FileCatalyst,
> etc.) as well as other open-source ones (Tsunami, I note the PDG data
> infrastructure report mentions iRods, etc.), or some of the data slicing
> topologies.
>
> Of course some of these saturate the bandwidth so may need to be used in
> conjunction with Janet developments to offer virtual dedicated paths
> across the backbone.
>
> Any pointers to work looking at how these address the various
> requirements of different communities would be welcome.
>
> Matthew
>
> Matthew Dovey
> Principal Consultant (Research e-Infrastructures) and Customer Advocate
> (Research), Jisc
> Chair, EGI Council and Executive Board
>
> T: +44 (0)203 006 6016
> Skype: mdovey
> Twitter: @mdovey
>
> Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG
>
> Jisc is a registered charity (number 1149740) and a company limited by
> guarantee which is registered in England under Company No. 5747339, VAT
> No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower
> Hill, Bristol, BS2 0JA. T 0203 697 5800.
>
> Jisc Services Limited is a wholly owned Jisc subsidiary and a company
> limited by guarantee which is registered in England under company number
> 2881024, VAT number GB 197 0632 86. The registered office is: One Castle
> Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.