This is a timely discussion as we're currently looking at how we might
measure the success of the student portal we're about to start developing,
with the intention of collecting comparison data before we start the
project to compare with post-launch results. We're looking at quality and
satisfaction measures rather than value for money so it would be really
useful to consider how to measure VFM. (Any pointers to resources, however
basic, would be really helpful here.)
We've come up with these so far and would be interested to find out what
others are doing:
Gather stats on number of users/user adoption rates, number of repeat
users and frequency of use (Analytics)
Check user satisfaction in terms of functionality, ease of use, quality of
content, 'findability', 'productivity' (user testing/feedback forms and
questionnaires)
Develop reputation/student experience measures, eg measurements around the
IT facilities which students are faced with. Identify how it might change
the student.s relationship with the University as a corporate entity, as
well as with their department/college. (Ask in focus groups and on student
questionnaire.)
Monitor how many requests for change there are (assume the more requests,
the more engaged people are with the system)
Evaluate how well the software/hardware works technically (how reliable it
is).
Compare with other channels in terms of the numbers of students who use
this one rather than others. .Would you mind if we took it away..
Monitor NSS rating on student satisfaction with IT facilities and
particularly student comments where it is possible to access them.
One of the issues is that doing this work can be time-consuming and itself
can cost money, certainly for the user testing. Plus it's not currently
clear to me how compelling the 'softer' qualitative measures are in terms
of their persuasiveness with senior management. We're concerned about
reputational issues due to feedback from our IT NSS scores and other
sources, but how could we pin down an increase in satisfaction with IT
services (for example) to the development of a student portal?
I know this is a bit of a diversion from issues of 'why should HE continue
to spend money on a web team' but we need to prove that the projects we
work on are successful and valuable, so this kind of work matters too.
Kriss
-------------------------------------------------------------------------
Web Coordinator HB30 Tel: (01904) 434682 Fax: 434685
University of York, UK 9-5.15, Mon-Fri http://www.york.ac.uk/weboffice/
Visit the Communications web pages at www.york.ac.uk/communications
and www.york.ac.uk/news-and-events
http://www.york.ac.uk/docs/disclaimer/email.htm
On Wed, 17 Nov 2010, Christopher Saward wrote:
> I'm bemused by some of the rather defensive responses to the Telegraph's
> report. Whatever may be said on the methodology or quality of the report,
> institutional websites developed with public money are surely a legitimate
> subject for journalistic investigation.
>
> Asking about the spend on the Telegraph's own website isn't really relevant;
> it's not developed using public money. They are not accountable in the same
> way and must answer to shareholders rather than taxpayers.
>
> Similarly BBC online isn't a helpful comparison: the nature and sheer scale
> of their operation put it in a totally different field from our institutional
> sites.
>
> I think one should be honest and accept that sometimes there may be a case to
> answer here. A public body spending a five- or six-figure sum on a web
> design ought to be able to justify it as value for money.
>
> A tendency of some management or marketing types is to become seduced by the
> superficial (animated pages, sexy graphics, etc), in the quest for a 'Wow!'
> factor. No doubt we've all come across this. It's an easy way to spend
> money, but does it meet the real needs of people using the online resource ?
>
> The most seductive design in the world is no use if the underlying content
> isn't also of high quality. If some or all of the content is poorly written,
> out of date or hard to navigate, or the information people need simply isn't
> there, the look won't rescue it. I believe that's partly what the
> Telegraph's article is trying to get at.
>
> Best wishes - Christopher
>
> From Brian Kelly on 16/11/2010 15:41:
>> Conrad Quilty-Harper, author of the Daily Telegraph article has
>> commented on Twitter:
>>
>> Article by @briankelly criticises my university website spending
>> investigation http://bit.ly/d4tAVh bit busy now but i'll write a
>> response
>> http://twitter.com/#!/Coneee/status/4556170362621952
>>
>> He has a blog at:
>> http://blogs.telegraph.co.uk/news/author/conradquiltyharper/
>>
>> and is a Data Mapping Reporter for Telegraph.co.uk | Visiting lecturer
>> in online journalism @cityjournalism
>>
>> He has been responding to comments on his Twitter account:
>> http://twitter.com/#!/Coneee
>>
>> Note response tyo:
>> 'Universities spend "millions" on websites': http://is.gd/hcsx2 Dear
>> Daily Telegraph, that's how much ANY corporate website costs. Even
>> yours.
>> is
>> right, but: a) everyone pays for university websites (unlike
>> Tgraph.co.uk) b) there's little correlation between quality& cost
>>
>> SO here's an opportunity to respond to the journalist.
>>
>> Brian
>>
>>
>>
>>
>>
>> Quoting Brian Kelly<[log in to unmask]>:
>>
>>> Thanks Ranjit (and all)
>>>
>>> A very timely article as demonstrating the value of institutional Web
>>> sites was the discussed at Strathclyde University on Friday. Ranjit
>>> and I were in agreement that we need to be able to respond in financial
>>> terms and we need to be gathering evidence which can be equated to
>>> financial value.
>>>
>>> I've written a quick post on "University Web Sites Cost Money!"
>>> http://ukwebfocus.wordpress.com/2010/11/16/university-web-sites-cost-money/
>>>
>>> Note I suggest looking at the JISC Value For Money reports for 2006 and
>>> 2009 for ways of estimated ROI of networked services.
>>>
>>> We also need to fight collectively, comrades.
>>>
>>> Have to dash
>>>
>>> Brian
>>>
>>> Quoting Ranjit Sidhu<[log in to unmask]>:
>>>
>>>> Hi All,
>>>>
>>>> And thanks for bringing this to my attention.
>>>>
>>>> Well, it has started-we knew this was coming-talked about this at
>>>> the IWMW10 and again on Friday at the Scottish Web folk meeting.The
>>>> similarity to the government web cull should not be missed-again
>>>> the stats are out of context:
>>>>
>>>> Just to distribute my advice that has been asked by a couple of
>>>> people of what specifically they can do:
>>>>
>>>> To take this seriously and communicate (ignoring is a dangerous
>>>> strategy) that, as Claire mentioned, this survey is very limited
>>>> and does not take into account the multifaceted role of the
>>>> University Website, in particular:
>>>>
>>>> Marketing:
>>>> The university website is an important revenue generator for
>>>> universities. From our work with institutions I would make sure you
>>>> emphasise the role the website has in bringing in international
>>>> student revenue ( for examples see my slides at the IWMW10 ) as
>>>> even simple estimations will blow these figures out of the water
>>>> (revenue will be in the £millions) Also, the online use for
>>>> research into PG courses (again big bucks).
>>>>
>>>> Savings
>>>> Do quick estimations of the savings from online prospectus
>>>> downloads and application forms etc... this will go close to
>>>> covering these cost of redesigns (we would be looking at £100,000s
>>>> over a year). Advantage of this is that they are very clear cut
>>>> figures.
>>>>
>>>> Dissemination of information of current students and staff (cost of
>>>> distribution)
>>>> A look at the number of your current students and staff accessing
>>>> the site will give a clear indication of the amount of information
>>>> disseminated to different user groups. Instead of doing the cost
>>>> per student as the article I would look at the cost per visit to
>>>> the website of current student ie the cost of distributing
>>>> information to current students- this will be far lower, give
>>>> better context and will be lower than any offline costs.
>>>>
>>>> My advice is fight fire with fire: Revenue generation, savings and
>>>> cost of distribution is a good start to putting context to these
>>>> misleading figures.
>>>>
>>>> Sector Response
>>>>
>>>> Would be indeed wise (Brian ?), let me know if there is anything I
>>>> can do to help (but obviously this is dependent on the individual
>>>> organisation we work withs approval)
>>>>
>>>>
>>>> Hope that helps I am working on a distributable presentation of
>>>> Friday's talk of University Evidence for Their Value (strangely
>>>> very relevant now!)
>>>>
>>>> Sid
>>>>
>>>>
>>>> Ranjit Sidhu
>>>> t: @rssidhu
>>>> e: [log in to unmask]
>>>> t: 07540094760
>>>> http://www.sidspace.info
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 16 Nov 2010, at 10:20, Adrian Tribe wrote:
>>>>
>>>>> Amusing/interesting survey at http://is.gd/hcsx2.
>>>>>
>>>>> Questionable methodology methinks.
>>>>>
>>>>> Any other views?
>>>>>
>>>>> Best wishes,
>>>>> Adrian
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> ------------------------
>>> Brian Kelly
>>> UKOLN, University of Bath, BATH, BA2 7AY
>>> Blog: http://ukwebfocus.wordpress.com/
>>
>>
>>
>> ------------------------
>> Brian Kelly
>> UKOLN, University of Bath, BATH, BA2 7AY
>> Blog: http://ukwebfocus.wordpress.com/
>
>
|