We (ATLAS) have used s3: at the Openstack instance at Datacentred.
This was using credentials stored in the panda server and configured
as a native Rucio endpoint. It was a quick test but there are many
details to sort through of which people on this list are aware of. RAL
is blazing a trail here but I hope T2-types keep an open mind!

Cheers,
Peter

On 19 January 2016 at 00:07, Christopher J. Walker
<[log in to unmask]> wrote:
> Alastair,
>
>          Whilst I'm no longer with GridPP, would an S3 interface to storage
> make Tier-1 storage more applicable to wider communities. Notably
> bioinformatics - possibly in relation to the EU Tier-0.
>
> Then a technical question.
>
>
> AIUI StoRM, and DPM (and almost certainly dCache) can do http/https with
> X509 certificate authentication -  how's that different from S3?
>
> Wikipedia claims Openstack swift provides S3 compatible storage, and
> http://docs.openstack.org/developer/swift/overview_auth.html looks
> interesting - particularly when you know that Rackspace are paying for
> someone at CERN to do federated authentication for openstack.
>
>
>  Chris
>
>
> On 18/01/16 14:51, Jensen, Jens (STFC,RAL,SC) wrote:
>>
>> Hi Alastair,
>>
>> Thanks for the description. We actually discussed the topic two weeks
>> ago (nearly) in the storage meeting - see item 3 of
>>
>>          http://storage.esc.rl.ac.uk/weekly/20160106-minutes.txt
>>
>> but I am afraid there wasn't much interest among the T2s for either CEPH
>> or for S3. So I don't think there is a good case for using GridPP T2
>> funding for the work... perhaps the climate modellers would be more
>> interested?
>>
>> Cheers
>> --jens
>>
>> On 18/01/2016 10:56, Alastair Dewhurst wrote:
>>>
>>> Hi Jens, all
>>>
>>> Recently I have been trying to push the testing of S3 endpoints within
>>> the WLCG framework.
>>> - Brian, Andrew L and myself have been looking at FTS transfers to and
>>> from S3 endpoints.
>>> - I have been speaking to the ATLAS DDM team about what bits are still
>>> missing for them to an S3 only endpoint as a space token. (I have spoken to
>>> the other WLCG VOs.  ALICE aren’t interested in S3 currently.  CMS and LHCb
>>> are interested but aren’t currently committing much/any effort towards it).
>>> - I have spoken to Oliver Keeble (http task force) and Alejandro (FTS
>>> lead developer).
>>>
>>> While I don’t feel I have a complete picture yet, it is becoming obvious
>>> that the largest current problem being encountered with S3 is the
>>> authentication (username / password) and how to get this to work within the
>>> WLCG framework while keeping things secure!  New(er) features such as
>>> pre-signed urls and authentication tokens look like good methods but it all
>>> needs development work.   The current thinking to have a central service
>>> which stores the credentials securely and then hands out limited access as
>>> required (either via pre-signed urls or authentication tokens).  Slightly
>>> worryingly it seems that the same thing is being developed by different
>>> people!
>>>
>>> For those of you not aware, STFC has a graduate scheme, where recent
>>> graduates are given ~6 month projects for a couple of years before being
>>> placed somewhere permanent.  I thought it would be good to apply for one of
>>> these who could then be put to work on this.  This would allow us to
>>> influence the direction of the development work and hopefully deliver
>>> something beneficial for the new storage at RAL, other sites that provide an
>>> S3 endpoint and any VOs GridPP supports who choose to use S3.
>>>
>>> Now the reason for this email is simple - money.  I asked Andrew Sansum
>>> if there was any way this project could be funded.  Initially he couldn’t
>>> think of any, but then he realised that there is a slight underspend on the
>>> GridPP storage budget, which if combined with a slight underspend elsewhere
>>> can be squeezed into enough money for a Graduate project.  Obviously Jens is
>>> in charge of storage budget so I need his approval, but Andrew Sansum also
>>> said that I need to get at least some support from the GridPP storage
>>> community in general as he doesn’t want lots of objections at the PMB.
>>>
>>> Unfortunately this is rather last minute and I need to submit the request
>>> for a project by the 20th January.  Once submitted it would be possible to
>>> withdraw the project request up until the 26th January.  I am happy to
>>> attend the storage meeting this Wednesday morning if you think this would be
>>> something worth discussing.
>>>
>>> Thanks
>>>
>>> Alastair
>>>
>>>
>>>
>>> Project proposal is below:
>>> Title: Developing authentication software for S3 and Swift
>>>
>>> Description:
>>> The Scientific Computing Department’s new Ceph based object store (Echo)
>>> provides S3 and Swift APIs for users.  These APIs do not support Certificate
>>> based authentication commonly used by the scientific community on the Grid.
>>> S3 and Swift authentication is by username/password and if these are stored
>>> insecurely or shared by many users within a VO, pose an unacceptable
>>> security risk to the site.  S3 and Swift provide features such as pre-signed
>>> urls and authentication tokens which grant limited access to the storage and
>>> would greatly reduce the security risk.  This project aims to develop
>>> software tools that larger VOs could integrate into their existing data
>>> management systems to allow transfers without the proliferation of access
>>> credentials.  For smaller VOs or new H2020 projects the aim would be to have
>>> a central service that securely stores the access credentials and generates
>>> pre-signed urls or authentication tokens when presented with a valid grid
>>> certificate (or other tru
>
> sted authentication method).
>>>
>>>
>>>
>>>
>>>
>