Crispin

Many thanks for your comments on the draft InteropAbility specification. This is exactly the type of critique that we need, in order to test the draft and address practical usability issues. I should perhaps stress here, on behalf of the project team, that the specification is a draft, as a first step to further work on the road (perhaps) to a usable standard. We expect a considerable amount of development work in the future.

The final report has been available from 5 August via both the wiki (http://www.alanpaull.co.uk/InteropAbility/InteropAbility_Project_Report_Final_1.2.pdf) and the JISC website. The report may help to address some of points you have raised and should at least inform you a bit more about how the draft specification was developed.

See my inline comments on your comments. I’m sure others will also wish to respond.

Best wishes.

Alan Paull

[log in to unmask]

Alan,

Here are my comments on the InteropAbility outputs, as requested.

1. I think the comprehensibility of the specification would benefit from:

- examples, showing how it is envisaged that the specification might be used;

- more reader-friendly diagrams of the data types;

- a rationale, showing how this compared to approaches taken by others to competencies, such as RCD and the current work by ISO, SIFA and IEEE LTSC;

- a definition of key terms, which are notoriously slippery in this area (competence, competency, proficiency, ability etc.).

[AP: Some of these points are covered in the Final Report. I hope that colleagues in the InteropAbility project will be able to add material as appropriate.]

2. To my mind, the fundamental characteristic of an effective competency definition is its “referencability”. The current spec seems to be rather vague in its use of identifiers and references. First, I think you muddle the two – I would use the term <identifier> for the thing being identified and <reference> for a pointer to an external thing which has an <identifier>. Second, there is no clarity as to what your <reference> contains. Following Dublin Core, you recommend a URI (we should surely be in mandatory territory here) and then you say that you recommend that this should point at some human readable text, which immediately makes the spec impenetrable to machines trying to manipulate competency frameworks. It is machine readability not human, which is fundamental. Without a secure means of referencing competency definitions, those definitions strike me as having little practical use.

[AP: Regarding your first point, it’s possible that you may have misunderstood the draft specification. We use <identifier> to identify things, and we use <abilityRef> in <relationship> elements to refer to other abilities. Where an ability is included as part of another ability, rather than referenced, we include the whole ability element. Regarding URIs, I have presumed (possibly wrongly) that cool URIs would be used, so that servers would provide human readable stuff to humans and appropriate machine readable content to machines. I should have made this explicit. We considered at this stage of development that provision of human readable material was essential, because part of what we’re trying to achieve is exposure for human-controlled import/export. I agree that machine readability is fundamental in the longer term, but we are at the start of a longer process, I believe.]

3. I agree with your assertion that “The InteropAbility view is that the structure of an ability is an integral part of its definition”. However, I am not sure about your follow-through on this point. Questions about structural integrity intersect with questions of ownership because the owner of a definition must be able to control not only the atomic definition but also those relationships which are “structural”. Your own requirement 7 says “Permit the owner of the data to control its use”, and you question this with the comment “I'm not certain this is correct. Sounds more like a security or implementation issue than one for the specification”. I agree with you to the extent that it is not necessary (or desirable) for the owner to control the *use* of a definition, but it *is* necessary for the owner to control the *integrity* of the definition. And we have established above, that the structure of sub-abilities is an aspect of that integrity.

[AP: I agree.]

I don’t like SKOS/Zthes-type relationships, firstly because they lack control over this issue of ownership. Let’s say that EdExcel defines ability X with sub-ability Y and we have already agreed that Y is “an integral part of the definition” of X. What is to stop Jo Bloggs coming along and defining ability Z which also defines itself as a sub-ability of X? If you look at the graph from Jo Bloggs’ perspective (or from the perspective of a system that has imported both definitions) then Jo Bloggs has *changed* EdExcel’s definition of ability X – and that is surely wrong and something that EdExcel is quite unhappy about.

I would represent these definitional, structural relationships in the form of nested XML elements, incapable of ambivalent definition, securely published on the owner’s website and beyond the reach of any third-party to change and where they can be globally referenced. Other people can then come along and claim equivalences and create derivative definitions by linked data relationships, without undermining the integrity of the original structures.

[AP: SKOS / Zthes-type relationships have the advantage of being there in existence and used. We state our own ambiguity about this approach in the report para 117. You raise an important issue of data integrity and control versus re-use. We’ve only started to scratch the surface of this problem, as we recognise in the section of the report on Version Control (para 115). There’s nothing fundamentally wrong with Jo Bloggs changing Edexcel’s definition, as long as there is a recognition that this type of re-use requires version control and re-publishing with appropriate acknowledgements. We’ve also pointed out that some form of ‘reciprocal approval’ for references would be useful to provide authority to relationships (para 114). Our current state of play is to make something useful within the context of partner systems, in order to take a step forward. We recognise that this single, relatively small project leaves many problems unresolved. Part of the role of the project was to raise issues and, in some cases, start to address them.]

4. I also think that your vocabulary of relationships muddles two types of relationship: organisational/structural (e.g. isSubAbility/hasSubAbility) and functional (isUnitOf/hasUnit, leadsTo/leadsFrom etc.). This begs two questions: (a) “what does being a sub-ability *mean*?” If it means that it “is a unit of”, then why have two terms which mean the same thing?. (b) what does the relationship “is a unit of” mean in respect of a competency definition – I suspect the answer is “not a lot”. It seems to me that you have cut-and-pasted a vocabulary that is intended for use with learning activities, not competency definitions.

[AP: I agree that all the relationship vocabulary items need to be defined – this has not yet been done. isSubAbility/hasSubAbility are structural – this ability has subAbilities. isUnitOf/hasUnit are to do with unit-based entities, such as qualification or course structures, so ‘unit’ is a formal learning or assessment structure. In fact this isn’t a cut-and-pasted vocabulary, as there don’t seem to be many relationship-type vocabularies for this domain in the UK – it’s more of a ‘suck-it-and-see’ vocabulary that we would envisage will require more definitive work than we had time for in the project.]

5. A second reason (in addition to point 3) why I do not like the SKOS/Zthes-type relationships is that they create polyhierarchical graphs. I would say that the meaning of the sub-ability relationship is that the sub-ability represents some sort of component of the parent ability, so that a claim to possess more or less proficiency in the sub-ability will automatically affect the claim of proficiency in the parental ability. Exactly *how* the sub-ability affects the proficiency claimed against the parent will depend on some kind of roll-up rule, which does not feature in your spec. You start to get some nasty consequences if you find that a particular component of the carburettor is also a component of the fuel injection system.

With polyhierarchical graphs, a single sub-ability may affect multiple parents. To take a Medbiquitous example, you might say that nurses and doctors both require communication skills. If you then decide that doctors need some sort of extra communication skills, not currently represented by your “communication skills” competency, you add this to the requirements for “communications skills” and then find you have, unwittingly, changed the requirements for nurses as well as doctors. I think you will find that the scope for unintended consequences in polyhierarchical graphs is severe, the readability of the graph is opaque, and maintainability is poor. For that reason, I think that it is a much better bet to represent competency frameworks as mono-hierarchical trees, with mapping relationships between them. So, you might have something like this (I express it in a kind of XML pseudo code):

<ability name=”medical_communications_skills”/>

<ability name=”doctoring”>

          <subAbility name=”doctoring_general_communications_skills” maps_to=”medical_communications_skills”/>

          <subAbility name=”doctoring_specialist_communications_skills/>

</ability>

<ability name=”nursing”>

          <subAbility name=”nursing_general_communications_skills” maps_to=”medical_communications_skills”/>

          <subAbility name=”nursing_specialist_communications_skills/>

</ability>

The difference between our two structures is that in yours, “doctoring” and “nursing” share *by definition* the same type of communications skills, whereas in mine, doctoring and nursing have by definition different communication skills which, at the moment, might happen to be defined as the same or as similar. The mapping between the two could be optimised by some sort of weighting or other qualifier, with the option that future versions of “doctoring_general_communications_skills” and “nursing_general_communications_skills” can develop along separate paths. This draws a distinction between two completely different kinds of relationships: structural relationships which are true by definition; and mapping relationships which are contingent and subject to change.

[AP: I’m not sure that I follow your argument here – it’s probably my fault though. The draft specification does support the kind of relationships you’ve got in this example. There’s no necessity in InteropAbility to structure things in the wrong way as you seem to be suggesting. Why would InteropAbility per se force you to have a common undifferentiated set of communication skills for doctoring and nursing? And why would it not permit post-hoc differentiation? It seems to me that this is more a question of version control and responsible authorship and authority.]

6. The issue above is related to versioning, which I think is critical and needs a more thorough treatment than you have given it. If they are to be useful to anyone, claims to proficiency and competence must be verifiable. If I claim to be a competent pilot, but crash the plane every time I try and land, it can be said that my claim is false (or at least, increasingly unlikely to be true as I continue to crash). If the process used to validate the original claim is shown regularly to produce false claims, then the process will be regarded at least to be unreliable, and it is likely to be found, when attempts are made to improve the process, that the competency framework itself also needs to be improved. In a situation where the reliability of competency data can be checked against real outcomes, competency frameworks are likely to be undergoing constant optimisation. You then face questions like “if Dr Jones claims to be proficient in “doctoring_general_communications_skills” version 1, on what basis can he claim to be proficient in “doctoring_general_communications_skills” version 2?

[AP: I agree that versioning needs to be addressed in more detail, as we say in the report. We have a starting position only.]

7. I don’t really understand why you need separate <ability> and <interopAbility> elements. This seems to me to militate against what should be infinitely nestable definition objects. If an interopAbility element needs to group together numerous ability elements (again, you do not explain what this grouping signifies), then why do individual <ability> elements not need to do the same? Incidentally, while “interopAbility” may be a catchy name for the project, I personally find it a bit kooky as the name of a element, which should strike a neutral, clear, descriptive, technical tone. It is easily misread as “interoperability” and suggests that other things (including plain vanilla <ability> elements) do not need to be interoperable. Perhaps this needs to be some kind of <document> or <instance> wrapper, with e.g. publication information, rather than what appears to be a sub-class of <ability>.

[AP: The purpose of the InteropAbility element is to provide the top element of a collection. This enables a system to identify a starting point for traversing the framework, which would be particularly important if a set of abilities was dispersed over the internet. We did consider the efficacy of a simple ‘wrapper’ element, but this would be distinctively different from our basic <ability> building block. The <InteropAbility> is intended to be an ability element that is hierarchically top of a framework. I make no particular comment about the naming.]

8. You state in your email that you have “established that our approach works well for a wide range of institutional and industry partners in a wide range of occupational contexts”. But I can see no evidence from the wiki that this spec has been piloted at all, let alone by “a wide range of institutional and industry partners”. If I am correct in this, then you have not established that your approach works for anyone, and some of the rhetoric used in your covering email is seriously misleading. You are now asking for comments at the same time as you say that you have already submitted your final report to JISC but before it has been published on the wiki, and are about to be presented at a range of conferences and exhibitions as a new specification. This suggests to me a rather haphazard process.

[AP: Your assumptions are not correct. We were conscious that some time had elapsed between the start of the project and the likely publication of the final report and felt that there was sufficient material available on the wiki for stakeholders, such as yourself, to start to review our approach – as indeed you have, for which I am very grateful. Ideally we would have published the final report earlier, and preferably in draft, so that it could be reviewed by stakeholders prior to submission. The project time scales did not permit this unfortunately. Again, project resources did not permit extension of the wiki to cover all our work, particularly as we wanted to have something substantive to present. The final report was made available on 5 August, which was 3 days after the email to the group, in which I said: “Project outputs can be found at: http://www.interopAbility.org/, where the final approved project report will be located shortly; it will also be available from the JISC website.” I thought it was clear from the wiki that the project consisted of 5 leading practitioner organisations (I do not include myself in this), and that these organisations had worked with the draft specification as part of the project. I apologise if this was not clear enough in the published material. It’s clear in the report, which is now available from the wiki and from the JISC website. There are also resources and examples available from the InteropAbility Resources page - http://www.interopability.org/wiki/InteropAbility_resources - that I hope will help to convince you that our process has not been haphazard. Please bear in mind that the major output of the project is a starting point, not an end point. The major purpose of this work is to stimulate discussion. I would suggest that this aspect has been to some extent successful.]

9. On a relatively trivial point, I have never liked the langstring, which is an inefficient way of handling multi-lingual requirements. Much better, surely, would be something along the lines of the following, which would offer greater transparency and better control over multiplicity:

<documentation language=”en_UK”>

          <title>xxx</title>

          <shortTitle>xxx</shortTitle>

          <description>xxx</description>

          <leadIn>xxx</leadIn>

          ...

</documentation>

[AP: Greater minds than mine have commented on this.]

10. Lumping together several of these comments into a single comment on general approach, there seems to be a particular concern within the project plan to achieve compatibility with a small selection of standards used by the HE community (XCRI, Zthes, Dublin Core). Not only does this approach lead to a lack of analysis of the core requirements for competency definitions (see comments 4 and 8 on cut-and-paste and the lack of piloting); but also is part of a general picture in which different communities of practice (SIFA, IMS, ISO, SCORM, ISCS ISB) increasingly burrow into their own isolated bunkers, creating dependences between “families” of their own favourite standards, but virtually no interrelationship between the separate families. This then leads to aggressive manipulation of organisations like CEN and ISO, as different communities of practice try and win one over their rivals by getting the supposed (but actually meaningless) authority of formal accreditation. The opposite approach needs to be taken: a strong proposal for competency definitions should be decoupled from other standards and vocabularies and based on a really strong analysis of the core requirement, with solid evidence that it works in practice.

[AP: The project was a JISC-funded one that was to build upon interoperability work that has gone before. Our work specifically included relationships between competence frameworks and qualifications and learning opportunities, which are important for creating joined-up systems that include data of different types, not just competences. This includes, for example, extensive work in the East Midlands linking job profiles, careers, courses, qualifications, training, information/advice/guidance, labour market information, and competencies. For these kinds of networks we need standards, usually accredited standards, so that vendors, education and training organisations, local, regional, national and international organisations and other stakeholders can have the confidence to spend significant resources on services and products that will join this stuff together.

As an incidental aside, I wouldn’t say that Zthes and Dublin Core are necessarily creatures of the HE community. These are established and widely used standards across many industries.

You’ve presumed, wrongly as it happens, that the project was required to analyse core requirements for competency definitions. In fact, the project was not required to address these core definitions, but rather to concentrate on interoperability and relationships between competencies and entities that use them. Across our 5 partner organisations, we had a very wide range of views on what ‘competence’ and ‘competence framework’ might mean, as well as different operational approaches to the content of a competence object. As there was a significant danger that we would get seriously bogged down in analysis of definitions of content, rather than practical interoperability issues, all partners agreed very early in the project to take a very broad view of the entity that we decided to call ‘ability’. The report describes our process at some length.]

Achieving a common set of competency definitions is an extremely important objective, but one that needs to be established across the board and needs extensive piloting and buy-in from a wide variety of stakeholders. If this specification is to go forwards usefully, then I think you must be clear that it is still a proposal which is at a very early stage of development and that the next step is piloting and iterative development of the spec, not implementation in a final sense or formal standardisation.

[AP: Your first sentence is an important one, though conclusions upon it may not be easily, or possibly even usefully, reached. It seems to me similar to attempting to reach a generally acceptable definition of ‘course’. Probably a futile exercise, because the term ‘course’ is used successfully in many different contexts with different meanings and different content. What’s important is that we can link like things together and re-use stuff where it’s useful to do so.

I hope that I’ve made it clear (and it is in our report) that the project team would agree completely with your final sentence. In no sense do we believe that the draft specification is anything other than an initial proposal that must be tested, developed iteratively and improved. If my communications have given any other impression, then I apologise whole-heartedly.  Alan Paull]

Crispin.