Crispin
Many thanks for your comments on the draft InteropAbility specification.
This is exactly the type of critique that we need, in order to test the
draft and address practical usability issues. I should perhaps stress
here, on behalf of the project team, that the specification is a draft,
as a first step to further work on the road (perhaps) to a usable
standard. We expect a considerable amount of development work in the
future.
The final report has been available from 5 August via both the wiki
(http://www.alanpaull.co.uk/InteropAbility/InteropAbility_Project_Report
_Final_1.2.pdf) and the JISC website. The report may help to address
some of points you have raised and should at least inform you a bit more
about how the draft specification was developed.
See my inline comments on your comments. I'm sure others will also wish
to respond.
Best wishes.
Alan Paull
[log in to unmask]
From: Discussion List for Skill and Competence Recording
[mailto:[log in to unmask]] On Behalf Of Crispin Weston
Sent: 24 August 2011 07:10
To: [log in to unmask]
Subject: Re: InteropAbility Project reports to JISC
Alan,
Here are my comments on the InteropAbility outputs, as requested.
1. I think the comprehensibility of the specification would benefit
from:
- examples, showing how it is envisaged that the specification might be
used;
- more reader-friendly diagrams of the data types;
- a rationale, showing how this compared to approaches taken by others
to competencies, such as RCD and the current work by ISO, SIFA and IEEE
LTSC;
- a definition of key terms, which are notoriously slippery in this area
(competence, competency, proficiency, ability etc.).
[AP: Some of these points are covered in the Final Report. I hope that
colleagues in the InteropAbility project will be able to add material as
appropriate.]
2. To my mind, the fundamental characteristic of an effective competency
definition is its "referencability". The current spec seems to be rather
vague in its use of identifiers and references. First, I think you
muddle the two - I would use the term <identifier> for the thing being
identified and <reference> for a pointer to an external thing which has
an <identifier>. Second, there is no clarity as to what your <reference>
contains. Following Dublin Core, you recommend a URI (we should surely
be in mandatory territory here) and then you say that you recommend that
this should point at some human readable text, which immediately makes
the spec impenetrable to machines trying to manipulate competency
frameworks. It is machine readability not human, which is fundamental.
Without a secure means of referencing competency definitions, those
definitions strike me as having little practical use.
[AP: Regarding your first point, it's possible that you may have
misunderstood the draft specification. We use <identifier> to identify
things, and we use <abilityRef> in <relationship> elements to refer to
other abilities. Where an ability is included as part of another
ability, rather than referenced, we include the whole ability element.
Regarding URIs, I have presumed (possibly wrongly) that cool URIs would
be used, so that servers would provide human readable stuff to humans
and appropriate machine readable content to machines. I should have made
this explicit. We considered at this stage of development that provision
of human readable material was essential, because part of what we're
trying to achieve is exposure for human-controlled import/export. I
agree that machine readability is fundamental in the longer term, but we
are at the start of a longer process, I believe.]
3. I agree with your assertion that "The InteropAbility view is that the
structure of an ability is an integral part of its definition". However,
I am not sure about your follow-through on this point. Questions about
structural integrity intersect with questions of ownership because the
owner of a definition must be able to control not only the atomic
definition but also those relationships which are "structural". Your own
requirement 7 says "Permit the owner of the data to control its use",
and you question this with the comment "I'm not certain this is correct.
Sounds more like a security or implementation issue than one for the
specification". I agree with you to the extent that it is not necessary
(or desirable) for the owner to control the *use* of a definition, but
it *is* necessary for the owner to control the *integrity* of the
definition. And we have established above, that the structure of
sub-abilities is an aspect of that integrity.
[AP: I agree.]
I don't like SKOS/Zthes-type relationships, firstly because they lack
control over this issue of ownership. Let's say that EdExcel defines
ability X with sub-ability Y and we have already agreed that Y is "an
integral part of the definition" of X. What is to stop Jo Bloggs coming
along and defining ability Z which also defines itself as a sub-ability
of X? If you look at the graph from Jo Bloggs' perspective (or from the
perspective of a system that has imported both definitions) then Jo
Bloggs has *changed* EdExcel's definition of ability X - and that is
surely wrong and something that EdExcel is quite unhappy about.
I would represent these definitional, structural relationships in the
form of nested XML elements, incapable of ambivalent definition,
securely published on the owner's website and beyond the reach of any
third-party to change and where they can be globally referenced. Other
people can then come along and claim equivalences and create derivative
definitions by linked data relationships, without undermining the
integrity of the original structures.
[AP: SKOS / Zthes-type relationships have the advantage of being there
in existence and used. We state our own ambiguity about this approach in
the report para 117. You raise an important issue of data integrity and
control versus re-use. We've only started to scratch the surface of this
problem, as we recognise in the section of the report on Version Control
(para 115). There's nothing fundamentally wrong with Jo Bloggs changing
Edexcel's definition, as long as there is a recognition that this type
of re-use requires version control and re-publishing with appropriate
acknowledgements. We've also pointed out that some form of 'reciprocal
approval' for references would be useful to provide authority to
relationships (para 114). Our current state of play is to make something
useful within the context of partner systems, in order to take a step
forward. We recognise that this single, relatively small project leaves
many problems unresolved. Part of the role of the project was to raise
issues and, in some cases, start to address them.]
4. I also think that your vocabulary of relationships muddles two types
of relationship: organisational/structural (e.g.
isSubAbility/hasSubAbility) and functional (isUnitOf/hasUnit,
leadsTo/leadsFrom etc.). This begs two questions: (a) "what does being a
sub-ability *mean*?" If it means that it "is a unit of", then why have
two terms which mean the same thing?. (b) what does the relationship "is
a unit of" mean in respect of a competency definition - I suspect the
answer is "not a lot". It seems to me that you have cut-and-pasted a
vocabulary that is intended for use with learning activities, not
competency definitions.
[AP: I agree that all the relationship vocabulary items need to be
defined - this has not yet been done. isSubAbility/hasSubAbility are
structural - this ability has subAbilities. isUnitOf/hasUnit are to do
with unit-based entities, such as qualification or course structures, so
'unit' is a formal learning or assessment structure. In fact this isn't
a cut-and-pasted vocabulary, as there don't seem to be many
relationship-type vocabularies for this domain in the UK - it's more of
a 'suck-it-and-see' vocabulary that we would envisage will require more
definitive work than we had time for in the project.]
5. A second reason (in addition to point 3) why I do not like the
SKOS/Zthes-type relationships is that they create polyhierarchical
graphs. I would say that the meaning of the sub-ability relationship is
that the sub-ability represents some sort of component of the parent
ability, so that a claim to possess more or less proficiency in the
sub-ability will automatically affect the claim of proficiency in the
parental ability. Exactly *how* the sub-ability affects the proficiency
claimed against the parent will depend on some kind of roll-up rule,
which does not feature in your spec. You start to get some nasty
consequences if you find that a particular component of the carburettor
is also a component of the fuel injection system.
With polyhierarchical graphs, a single sub-ability may affect multiple
parents. To take a Medbiquitous example, you might say that nurses and
doctors both require communication skills. If you then decide that
doctors need some sort of extra communication skills, not currently
represented by your "communication skills" competency, you add this to
the requirements for "communications skills" and then find you have,
unwittingly, changed the requirements for nurses as well as doctors. I
think you will find that the scope for unintended consequences in
polyhierarchical graphs is severe, the readability of the graph is
opaque, and maintainability is poor. For that reason, I think that it is
a much better bet to represent competency frameworks as
mono-hierarchical trees, with mapping relationships between them. So,
you might have something like this (I express it in a kind of XML pseudo
code):
<ability name="medical_communications_skills"/>
<ability name="doctoring">
<subAbility name="doctoring_general_communications_skills"
maps_to="medical_communications_skills"/>
<subAbility name="doctoring_specialist_communications_skills/>
</ability>
<ability name="nursing">
<subAbility name="nursing_general_communications_skills"
maps_to="medical_communications_skills"/>
<subAbility name="nursing_specialist_communications_skills/>
</ability>
The difference between our two structures is that in yours, "doctoring"
and "nursing" share *by definition* the same type of communications
skills, whereas in mine, doctoring and nursing have by definition
different communication skills which, at the moment, might happen to be
defined as the same or as similar. The mapping between the two could be
optimised by some sort of weighting or other qualifier, with the option
that future versions of "doctoring_general_communications_skills" and
"nursing_general_communications_skills" can develop along separate
paths. This draws a distinction between two completely different kinds
of relationships: structural relationships which are true by definition;
and mapping relationships which are contingent and subject to change.
[AP: I'm not sure that I follow your argument here - it's probably my
fault though. The draft specification does support the kind of
relationships you've got in this example. There's no necessity in
InteropAbility to structure things in the wrong way as you seem to be
suggesting. Why would InteropAbility per se force you to have a common
undifferentiated set of communication skills for doctoring and nursing?
And why would it not permit post-hoc differentiation? It seems to me
that this is more a question of version control and responsible
authorship and authority.]
6. The issue above is related to versioning, which I think is critical
and needs a more thorough treatment than you have given it. If they are
to be useful to anyone, claims to proficiency and competence must be
verifiable. If I claim to be a competent pilot, but crash the plane
every time I try and land, it can be said that my claim is false (or at
least, increasingly unlikely to be true as I continue to crash). If the
process used to validate the original claim is shown regularly to
produce false claims, then the process will be regarded at least to be
unreliable, and it is likely to be found, when attempts are made to
improve the process, that the competency framework itself also needs to
be improved. In a situation where the reliability of competency data can
be checked against real outcomes, competency frameworks are likely to be
undergoing constant optimisation. You then face questions like "if Dr
Jones claims to be proficient in
"doctoring_general_communications_skills" version 1, on what basis can
he claim to be proficient in "doctoring_general_communications_skills"
version 2?
[AP: I agree that versioning needs to be addressed in more detail, as we
say in the report. We have a starting position only.]
7. I don't really understand why you need separate <ability> and
<interopAbility> elements. This seems to me to militate against what
should be infinitely nestable definition objects. If an interopAbility
element needs to group together numerous ability elements (again, you do
not explain what this grouping signifies), then why do individual
<ability> elements not need to do the same? Incidentally, while
"interopAbility" may be a catchy name for the project, I personally find
it a bit kooky as the name of a element, which should strike a neutral,
clear, descriptive, technical tone. It is easily misread as
"interoperability" and suggests that other things (including plain
vanilla <ability> elements) do not need to be interoperable. Perhaps
this needs to be some kind of <document> or <instance> wrapper, with
e.g. publication information, rather than what appears to be a sub-class
of <ability>.
[AP: The purpose of the InteropAbility element is to provide the top
element of a collection. This enables a system to identify a starting
point for traversing the framework, which would be particularly
important if a set of abilities was dispersed over the internet. We did
consider the efficacy of a simple 'wrapper' element, but this would be
distinctively different from our basic <ability> building block. The
<InteropAbility> is intended to be an ability element that is
hierarchically top of a framework. I make no particular comment about
the naming.]
8. You state in your email that you have "established that our approach
works well for a wide range of institutional and industry partners in a
wide range of occupational contexts". But I can see no evidence from the
wiki that this spec has been piloted at all, let alone by "a wide range
of institutional and industry partners". If I am correct in this, then
you have not established that your approach works for anyone, and some
of the rhetoric used in your covering email is seriously misleading. You
are now asking for comments at the same time as you say that you have
already submitted your final report to JISC but before it has been
published on the wiki, and are about to be presented at a range of
conferences and exhibitions as a new specification. This suggests to me
a rather haphazard process.
[AP: Your assumptions are not correct. We were conscious that some time
had elapsed between the start of the project and the likely publication
of the final report and felt that there was sufficient material
available on the wiki for stakeholders, such as yourself, to start to
review our approach - as indeed you have, for which I am very grateful.
Ideally we would have published the final report earlier, and preferably
in draft, so that it could be reviewed by stakeholders prior to
submission. The project time scales did not permit this unfortunately.
Again, project resources did not permit extension of the wiki to cover
all our work, particularly as we wanted to have something substantive to
present. The final report was made available on 5 August, which was 3
days after the email to the group, in which I said: "Project outputs can
be found at: http://www.interopAbility.org/, where the final approved
project report will be located shortly; it will also be available from
the JISC website." I thought it was clear from the wiki that the project
consisted of 5 leading practitioner organisations (I do not include
myself in this), and that these organisations had worked with the draft
specification as part of the project. I apologise if this was not clear
enough in the published material. It's clear in the report, which is now
available from the wiki and from the JISC website. There are also
resources and examples available from the InteropAbility Resources page
- http://www.interopability.org/wiki/InteropAbility_resources - that I
hope will help to convince you that our process has not been haphazard.
Please bear in mind that the major output of the project is a starting
point, not an end point. The major purpose of this work is to stimulate
discussion. I would suggest that this aspect has been to some extent
successful.]
9. On a relatively trivial point, I have never liked the langstring,
which is an inefficient way of handling multi-lingual requirements. Much
better, surely, would be something along the lines of the following,
which would offer greater transparency and better control over
multiplicity:
<documentation language="en_UK">
<title>xxx</title>
<shortTitle>xxx</shortTitle>
<description>xxx</description>
<leadIn>xxx</leadIn>
...
</documentation>
[AP: Greater minds than mine have commented on this.]
10. Lumping together several of these comments into a single comment on
general approach, there seems to be a particular concern within the
project plan to achieve compatibility with a small selection of
standards used by the HE community (XCRI, Zthes, Dublin Core). Not only
does this approach lead to a lack of analysis of the core requirements
for competency definitions (see comments 4 and 8 on cut-and-paste and
the lack of piloting); but also is part of a general picture in which
different communities of practice (SIFA, IMS, ISO, SCORM, ISCS ISB)
increasingly burrow into their own isolated bunkers, creating
dependences between "families" of their own favourite standards, but
virtually no interrelationship between the separate families. This then
leads to aggressive manipulation of organisations like CEN and ISO, as
different communities of practice try and win one over their rivals by
getting the supposed (but actually meaningless) authority of formal
accreditation. The opposite approach needs to be taken: a strong
proposal for competency definitions should be decoupled from other
standards and vocabularies and based on a really strong analysis of the
core requirement, with solid evidence that it works in practice.
[AP: The project was a JISC-funded one that was to build upon
interoperability work that has gone before. Our work specifically
included relationships between competence frameworks and qualifications
and learning opportunities, which are important for creating joined-up
systems that include data of different types, not just competences. This
includes, for example, extensive work in the East Midlands linking job
profiles, careers, courses, qualifications, training,
information/advice/guidance, labour market information, and
competencies. For these kinds of networks we need standards, usually
accredited standards, so that vendors, education and training
organisations, local, regional, national and international organisations
and other stakeholders can have the confidence to spend significant
resources on services and products that will join this stuff together.
As an incidental aside, I wouldn't say that Zthes and Dublin Core are
necessarily creatures of the HE community. These are established and
widely used standards across many industries.
You've presumed, wrongly as it happens, that the project was required to
analyse core requirements for competency definitions. In fact, the
project was not required to address these core definitions, but rather
to concentrate on interoperability and relationships between
competencies and entities that use them. Across our 5 partner
organisations, we had a very wide range of views on what 'competence'
and 'competence framework' might mean, as well as different operational
approaches to the content of a competence object. As there was a
significant danger that we would get seriously bogged down in analysis
of definitions of content, rather than practical interoperability
issues, all partners agreed very early in the project to take a very
broad view of the entity that we decided to call 'ability'. The report
describes our process at some length.]
Achieving a common set of competency definitions is an extremely
important objective, but one that needs to be established across the
board and needs extensive piloting and buy-in from a wide variety of
stakeholders. If this specification is to go forwards usefully, then I
think you must be clear that it is still a proposal which is at a very
early stage of development and that the next step is piloting and
iterative development of the spec, not implementation in a final sense
or formal standardisation.
[AP: Your first sentence is an important one, though conclusions upon it
may not be easily, or possibly even usefully, reached. It seems to me
similar to attempting to reach a generally acceptable definition of
'course'. Probably a futile exercise, because the term 'course' is used
successfully in many different contexts with different meanings and
different content. What's important is that we can link like things
together and re-use stuff where it's useful to do so.
I hope that I've made it clear (and it is in our report) that the
project team would agree completely with your final sentence. In no
sense do we believe that the draft specification is anything other than
an initial proposal that must be tested, developed iteratively and
improved. If my communications have given any other impression, then I
apologise whole-heartedly. Alan Paull]
Crispin.
From: Discussion List for Skill and Competence Recording
[mailto:[log in to unmask]] On Behalf Of Alan Paull
Sent: 02 August 2011 09:30
To: [log in to unmask]
Subject: InteropAbility Project reports to JISC
InteropAbility Project reports to JISC
The InteropAbility Project has completed its work on a draft
Interoperable Competences Framework Specification. A final report has
been submitted to JISC. Project outputs can be found at:
http://www.interopAbility.org/ <http://www.interopAbility.org/> , where
the final approved project report will be located shortly; it will also
be available from the JISC website.
Invitation to comment
You are cordially invited to review the work of the project and to offer
a critique of the approach and the detail. Please post your comments on
this mailbase, send them to Alan Paull ([log in to unmask]
<mailto:[log in to unmask]> ), or engage directly in discussions with
one of the InterCom Consortium partners (TAG Developments Ltd,
MyKnowledgeMap Ltd, Newcastle University, The University of Nottingham,
Pebble Learning Ltd and APS Ltd).
Having established that our approach works well for a wide range of
institutional and industry partners in a wide range of occupational
contexts, we believe that the moment has come to engage in discussions
with others involved in similar work. We would welcome your views on our
work so far.
Further dissemination
Partners in the InteropAbility Project will be introducing the draft
specification at a range of exhibitions and conferences attended by
interested parties. Please take the opportunity to find out more at any
of the events.
The first such event was EPIC '11 in early July. We will post here the
events where partners plan to introduce the new InteropAbility
Specification.
Best wishes.
Alan Paull
--
Alan Paull
Project Manager for the InteropAbility Project
[log in to unmask] <mailto:[log in to unmask]>
Notes:
InterOpAbility: What is it?
The InteropAbility Project has produced a draft Interoperable
Competences Framework Specification for representing competences and
competence frameworks across a range of vocational and accreditational
contexts.
Who produced it?
The work was undertaken by a consortium of organisations that have an
interest in this space: TAG Developments Ltd, MyKnowledgeMap Ltd,
Newcastle University, The University of Nottingham, Pebble Learning Ltd
and APS Ltd.
