Hi Ewan et al,

You said ...

> What we need is for the PMB to listen to the 
> site admins and feed these facts upwards. 
> 
> Given the near universal and quite vehement condemnation from 
> the site admins and various coordinators I'm rather surprised 
> that that isn't what's happening already.
> 

I can assure you that everyone is fully aware of the UK sys admin views.
These views are not just being ignored!

As has already been explained (by Jeremy, Stephen, John and others) ... 

There are serious concerns about multi-user pilot jobs running without
the identity switching. All user jobs run under the id of the pilot
framework itself meaning that every user owns the framework, thereby
being able to do whatever they want and write whatever they want into
the logs. This does not meet the requirements for traceability. 

Blocking a whole VO (when it is large and important, e.g. ATLAS) in
general just does not work. And even if we do block it, if we don't know
which user was the problem one, we can't fix the problem and hence we
can never re-enable the VO.

So... the only real question is to support pilot jobs or not? Three of
the four LHC VOs say they *have* to have them.

We have been working (and will continue to work) hard on many other
controls for pilot jobs (an agreed policy, restricting submission of
pilot jobs to a few trusted people who have signed the policy, code
review of glexec and reviews of the pilot frameworks), i.e. all we can
to mitigate the risks of running glexec in setuid mode.

We discussed this issue at length at last week's Joint Security Policy
Group meeting (chaired by me). Here are the draft notes from that
discussion (not yet approved as official). I should add that we were not
just discussing the LHC needs but pilot jobs in general for all VOs in
EGEE, OSG etc.

We now need to see what the response is from other (non-UK) WLCG sites.
If the WLCG MB does impose the pilot/switch-id, I will be happy to
participate in any local negotiation between GridPP staff and your local
University security officers, IT directors, cluster managers etc.

So ... please don't think we are not listening. I would be much happier
if the need for pilot jobs would just go away, but that seems rather
unlikely.

Dave K

------------------------------------------------

JSPG meeting
29/30 Oct 2007
CERN
Draft version 2 - 6 Nov 07

Summary of discussion on Pilot Jobs and related issues
------------------------------------------------------
(these are D Kelsey's personal notes - not yet approved by JSPG)

We discussed the current draft policy (v0.3) on "Multi User Pilot Jobs"
See https://edms.cern.ch/document/855383/1 

Discussion focussed on whether we should require Sites to run the
Grid-provided utility (today glexec) in the identity switching mode or
whether Sites should have the right to choose not to switch identities. 

The view of the participants in the meeting room at CERN was that there
are significant security risks in not switching identity (namely the
users' workload runs under the same identity as the pilot job framework
resulting in the ability of users to take control of the framework and
to interfere with the audit logs) and that we should indeed therefore
require identity switching. It was appreciated that this is contentious
and may not be acceptable to some EGEE sites and/or the EGEE management,
but was put forward as a proposal for discussion.

The view of the OSG participants, including Ruth Pordes, who were
participating remotely by phone was that they had not been able to
consult their Sites or discuss this suggestion internally and so they
could not agree to the proposal to require identity switching at this
time. They have concerns as to what extent they can impose such policies
on their Sites. 

JSPG concluded that this is clearly a very contentious issue and that we
need to discuss with the Sites and the Grid managements to see if
consensus can be reached as to whether identity switching is required or
not.

JSPG also discussed the current stated requirement that pilot jobs must
clean up all local data files between different user jobs within one
pilot job. This had been a strong requirement coming out of discussion
at the August GDB meeting, but Oxana noted that this may conflict with
experiment requirements. JSPG agreed that there are security concerns,
e.g. with one user job leaving infected files which the next job may
pick up. Again we concluded that further consultation is required.

We did not produce a new version of the policy document as this seemed
pointless until these issues are resolved. 

On the second day, JSPG revisited the topic to decide the best way
forward. 

Ian N expressed the view that we should not need a separate policy for
pilot jobs as all of the issues should be covered in other more general
documents. Dave K reminded JSPG that we started trying to write a
general policy document on "Operation of Grid Services", but decided
this would take too long, so we decided to prepare the more specific
policy on pilot jobs, as this is needed rather urgently. 

Bob C presented a new draft policy he had prepared to address not only
pilot jobs but also the issues related to Grid portals, noting that many
of the policy issues are the same. We came to no definite conclusion on
whether to take this further at this stage, but given the urgency for a
policy on pilot jobs (for WLCG) it is probably best to finalise this
first. This also allows for the fact that EGEE has a working group on
portals looking into the various issues, which will not conclude its
work before Spring 2008. The pilot jobs policy may well then be replaced
later by something more general. 

We then decided to concentrate on the requirements for traceability and
logging. These are general requirements which apply not only to
multi-user pilot jobs, but also to all other forms of job submission
including, for example, Grid portals. We wish to get agreement on these
general principles which can then be applied to the consideration of any
particular service, such as pilot jobs.

JSPG produced some draft words for the new "Policy on Traceability and
Logging" (which we also worked on during the first day of the meeting).
This will replace the old policy on "Audit Requirements". The words are
not yet final and still need more work.

The main points are: 

Risk management is crucial for Grid operations. 

When security incidents happen it must be possible to identify the cause
so that it can be contained while keeping services operational. It must
also be possible to take action to prevent the incident happening again.


The response to an incident needs to be commensurate with the scale of
the problem. Banning a whole large VO, rather than just one user, is in
most cases impossible as this affects too many users. 

Understanding and fixing the cause of an incident is essential before
re-enabling access. If a VO has been blocked in its entirety, unless
there is a way of understanding what happened, it will be impossible to
fix and hence impossible to re-enable access. 

The minimum level of traceability for Grid usage is to be able to
identify the source of all actions (executables, file transfers, pilot
jobs, portal jobs, etc) that are part of an incident and the individual
who initiated them. In addition, sufficiently fine-grained controls,
such as blocking the originating user and monitoring to detect abnormal
behaviour, are necessary for keeping services operational.

There are trade-offs between knowledge of individual user identity and
controlling the code which can be executed. Anonymous user access may
well be possible if there is no possibility for the user to submit
or modify executable code, i.e. the user just provides input parameters
and/or data to control a job. 

------------------------------------------------
Dr David Kelsey
Particle Physics Department
Rutherford Appleton Laboratory
Chilton, DIDCOT, OX11 0QX, UK

e-mail: [log in to unmask]
Tel: [+44](0)1235 445746 (direct)
Fax: [+44](0)1235 446733
------------------------------------------------
>