Print

Print


Hi Chris
    As you know  I recently discussed the question of 'measures' in a post
entitled "How Do We Measure the Effectiveness of Institutional
Repositories?" - see
http://ukwebfocus.wordpress.com/2011/02/24/how-do-we-measure-the-effectivene
ss-of-institutional-repositories/
   I first suggested some key purposes of an IR: "(1) maximising access to
research publications and (2) ensuring long-term preservation of research
publications" and then looked at metrics which may provide either a direct
or a proxy indication of the success of achieving these goals.
   I agree with the comment you made that there is a need for "detailed
individual data, showing referrals etc, rather than just totals".
   No0te that  a comment from Steve Hitchcock also indicated the need for a
better understanding of the stats provided by the IR Stats module.
    Comments on these approaches are welcomed.

Brian


-----Original Message-----
From: Chris Rusbridge [mailto:[log in to unmask]] 
Sent: 07 March 2011 17:08
Subject: Re: What makes a good repository?

Andy, you may be the only one so far to directly address "measures". I think
I might agree with making the distinction between the general public and the
academic community, except it's very blurred... I'm not confident which side
I sit in! And I definitely agree with the idea of the "proportion of the
total scholarly output within the target community of depositors", other
than the inconvenient fact that the divisor is unknown (which I guess is why
we mostly opt for uninteresting totals rather than the interesting unknowns,
or estimates).

Your point about (c1, deposit by the academics themselves) being a good
proxy for usefulness may be a good one. Apart from another inconvenient fact
that in most cases it's very low (estimates of proportions of self-deposited
stuff without mandates being in the low single digit percentages of total
output).

I definitely wasn't aiming to conclude that most repositories are not very
useful (as I think that most are, or will be). (Biased research? Moi?)

But I AM keen to see more suggestions for measure of usefulness.

Sorry for the blitz of replies, I'm just back from a pleasant sunny walk
along the Stratford Canal (yes, I'm on THAT side of the academic/general
public divide!).

--
Chris Rusbridge
Mobile: +44 791 7423828
Email: [log in to unmask]




On 7 Mar 2011, at 15:15, Andy Powell wrote:

> Citation strikes me as primarily being an indicator of reading (i.e. use)
rather than re-use? (OK, you/I need to define 're-use' here! :-) ).
> 
> Under d I think you need to separate out usefulness to the academic
community (i.e. to other researchers?) from usefulness to the general public
(which surely is of secondary importance?).
> 
> I'm probably stating the obvious here... but counts of 'deposits',
'downloads' and 'citations' (roughly your c1, c2, and c3) are indicative of
usefulness to depositors, readers and other researchers (roughly your d1, d2
and d3(ish - see above). In other words, c is the measure for d ??
> 
> I think that c1 can only be measured as a proportion of the total
scholarly output within the target community of depositors (i.e. an absolute
number of deposits isn't all that interesting). It would be nice to measure
c2 in a similar way (downloads from repository as a proportion of total
downloads including paid-for accesses or somesuch) but I can't really see
how that might be done. As you indicate, c3 is problematic to measure given
current citation practice.
> 
> I think one could make the argument that c1 (as undertaken by researchers
themselves) is a good single metric for overall usefulness (on the basis
that no-one would willingly take time to deposit something in a repository
unless they were confident of it leading to increased downloads and
citation). Mandates completely skew that unfortunately... and, in any case,
it is/was presumably hard to determine real researcher-initiated deposits
from those down by intermediaries (which, again, skews the picture)?
> 
> Sorry... just thinking out loud. No real help at all!
> 
> Andy
> 
> --
> Andy Powell
> Research Programme Director
> Eduserv
> t: 01225 474319
> m: 07989 476710
> twitter: @andypowe11
> blog: efoundations.typepad.com
> 
> www.eduserv.org.uk 
> 
> 
> -----Original Message-----
> From: Repositories discussion list
[mailto:[log in to unmask]] On Behalf Of Chris Rusbridge
> Sent: 07 March 2011 11:08
> To: [log in to unmask]
> Subject: What makes a good repository?
> 
> I'm interested in the question "what makes a good repository?". Or
perhaps, given a particular repository, how could we assess whether it is
doing its job well? Or, well enough... to be sustainable?
> 
> I've been given various answers starting from
> 
> a) the repository meets its (defined) goals.
> 
> OK, sounds reasonable, but the goals were probably defined in the past,
perhaps even before the repository existed. That was then; this is a
different world. How about...
> 
> b) the repository meets real needs.
> 
> Yes, I like that. But what are those real needs? I can think of two groups
that sound similar but are subtly different...
> 
> c) the repository is (well) used
> c1) by depositors
> c2) by readers
> c3) by re-users.
> 
> (There are probably more important subtypes of users.) This is the set we
often measure: c1 by total deposited items or by rates of deposit, c2 by
accesses and downloads. We less often measure c3, but citations and in-links
could be reasonable proxies. Both are slightly muddy as many repositories
contain substitutes for the version of record, and good practice is to cite
the latter (but perhaps more often link to the substitute). But how about...
> 
> d) the repository is useful
> d1) to depositors
> d2) to its owner
> d3) to the public in general
> 
> (Again this might not be the right set of subtypes.) The first of these,
d1 is not the same as c1; repositories might be used without being useful to
depositors. This might be because of mandates, perhaps, or by being "used"
by librarians acting for the depositors without much motivation by the
depositors. Much better where the repository is useful to the depositor.
This (I think) is what the various "Negative Click Repository" posts were
about (see posts in
http://digitalcuration.blogspot.com/search/label/Negative%20click), and I
think it's part of the thrust of Steve Hitchcock's DepositMO project
(http://blogs.ecs.soton.ac.uk/depositmo/).
> 
> Sustainability is in part about continuing to convince decision makers to
keep paying the costs, so being demonstrably useful to the owner (d2) seems
pretty important.
> 
> The last subtype (d3) I've made as general as possible, believing that
there is a real public-spirit, philanthropic nature to most institutions
that run repositories, as well as a belief that good deeds can come back to
reward us (casting our bread upon the waters?).
> 
> I'm interested in any comments on these ideas, and particularly interested
in any suggestions for measures of the (d) group. Does this make sense?
> 
> --
> Chris Rusbridge
> Mobile: +44 791 7423828
> Email: [log in to unmask]