"Is the problem here really the kind of nightmarish outsourced, less than useful IT arrangements too many LAs have got themselves embroiled in, arrangement which are simply incapable of properly supporting users…?"Undoubtedly. I have been caught between the "you can't have that commercial software because there's an ongoing subscription cost/the initial outlay is too expensive" rock and the "you can't have that free open source software because there's no support for it" hard place. And our outsourced IT have still not restored our server (9 working days and counting folks...) - doesn't say much for our data security does it?RachelRachel Grahame
Archaeologist (HER)Tees Archaeology, Sir William Gray House, Clarence Road, Hartlepool, TS24 8BT
Email [log in to unmask] | Tel 01429 523457 |
www.teesarchaeology.com*Please note I normally work Monday-Thursday*
From: Issues related to Historic Environment Records [mailto:[log in to unmask]] On Behalf Of Paul Cripps
Sent: 20 November 2014 15:47
To: [log in to unmask]
Subject: Re: [HERFORUM] E Conference Day 3 - What criteria are essential or desirable?
Some comments on issues arising. The world of data is changing and this is a good opportunity to build robust foundations for heritage data management.
It sounds like whilst NHRE may be retired, at least in its current form, and HER records becoming the definitive record, there will still be a need for EH to record and maintain some information. So there will need to be two sets of records of some kind, but with a unified index…? So are we actually talking about a process whereby the NHRE is stripped back to only the added value data EH need to record, records not in HERs are created (in that tiny minority of cases) and a more technological solution is implemented to allow HERs and NHRE to publish as eg Linked Data and via web services so as to share their bits of records to form a full virtual record? This way, the full records from each end (EH and HERs) can form part of records at the other end (and fed across to planning systems, LA websites, EH overviews, etc as needed) with each party maintaining their own bit.
For example, there would be no need for NHRE to incorporate spatial information, monument types, dating, etc; the HER would be definitive. Equally, any content created and maintained by EH relating to eg monitoring done by EH could be propagated across to HERs. Crucially there would be a unified index (the HER), which (imo) is the key to this whole proposal and the most vital and significant change, we just seem to be getting a bit bogged down in what to do with the attributes of sites/monuments, the detailed content of the records, beyond the unique identifier. Why not instead embrace the difference…?
In other words, an alternative approach such as this could:
a) Provide the benefits of integration and a single, unified index whilst allowing and appreciating the need for different data to be recorded by different users, crucially against a unified index of sites maintained at the HER end.
b) Streamline the integration process as it would become largely a ‘this is the same as that’ process with minimal record creation in HERs for missing sites. No need to try to assimilate all the contents of each record at once. Plus option to say (where discrepancies/differences are present) which eg set of spatial information, dating, classifications, etc should be taken forward and which deprecated. With options for manual intervention/editing of course. And flagging of record status en route. Such a process could be more successfully automated than full on reconciliation of two sets of records into one.
c) Provide cached offline use for all users (the web is just a transfer mechanism for linking systems, like how the Heritage Gateway uses the web; no-one is suggesting web based applications as far as I can see)
d) Allow both parties (EH and HERs) to have complete and dynamically updated records either by a push or a periodic pull mechanism.
e) Align neatly with other similar scenarios such as how to include PAS data, heritage (and other) designations data, etc as well as access to resources we rarely discuss eg museum collections – external resources could all be managed and maintained and shared using this approach and technological framework. So tackles broader issues of disparate data too; win win!
f) Not affect local management of data. This approach would involve tiers of applications and services on top of existing data/application layers to push data out and pull it in. And be platform agnostic so could be made to work against HBSMR+Mapinfo/QGIS/ArcGIS or Arches or a homebrew system or whatever system EH decide they wish to use. There just need to be tools to publish and cache/consume, many components already exist and could be tailored for deployment in the myriad LA environments.
So there’s a potential way of reaching the goals of this project whilst not getting bogged down in manual reconciliation of all content of all records for the whole country (with all the time and expense that would involve). Instead, use some of that resource to target some development work to build an infrastructure ready for today and tomorrow. And sidestepping some of the technical spanners in the works that have been noted. On which note, some more general points on matter arising:
Duplication of data.
Yes, there are sometimes good reasons for data to be duplicated, duplication is not necessarily bad per se, as Crispin and Martin point out. Creating security copies is obviously important and could be achieved by periodic output to eg ADS. by Where duplication is problematic is when it exists for no good reason and requires massive amounts of effort, repeatedly, to resolve. The proposal behind this e-conference has the potential to remove one such unnecessary duplication which drains our limited historic environment resources on so many levels.
Web. Online/Offline. Access.
Using the web for data sharing is not the same as ‘putting data on a website’. Adding a hyperlink to an external website is not the same as using a desktop application to automatically pull full records from an external source into the records within an HER system, creating a local complete cache of externally maintained data. Linked Data systems often cache records to improve performance, provide offline access, archival/security copies (this is all good and part of the paradigm; this is good, useful duplication). Web services and Linked Data approaches can be used to share data between desktop systems, websites, web based applications and feed existing corporate systems (eg LA websites, planning systems, etc). Professional access to data provided by such means can be managed, charged for, etc; increased use of web services and/or Linked Data to improve shared access could provide consistent, systematic delivery mechanisms between user groups (EH, HERs, planners, end users) and free up HER staff time. Commercial access to good quality data feeds could be seen as enabling HEROs to focus more on heritage advice rather than cutting HER staff out of the loop just because they are not running queries and emailing data: HER staff could and should of course continue to provide advice in addition to data, indeed they would have more time to do so. Such self service models could build on the Heritage Gateway infrastructure eg to feed desktop GIS to undertake DBAs; but that is most definitely not the same as saying use the existing Heritage Gateway website for professional use.
Data structures. Editing. Roles/Responsibilities.
Publishing data from existing systems as Linked Data and/or web services would not (necessarily) replace desktop systems. But it would facilitate interoperability between them. There would be no need for EH staff to edit HER records, as someone suggested; EH could maintain their own EH specific records linked to and incorporating HER records with all data shared between all parties so everyone has the full records locally and properly maintained. Or vice versa for HEROs using EH statutory records (eg to record additional information about LBs). There would be no need to modify core HER data structures; additional functionality to consume externally maintained resources could be added on; redundant bits of systems could be retired (eg NHRE spatial components) and it would be made clear who is responsible for the maintenance of data. All this could provide a single coherent (virtual) ‘record’ which actually comprises bits from multiple internal/external records. A big strength of Linked Data approaches is that groups can manage the data they have responsibility for, publish it (automated dump to a local/remote repository/warehouse) and links can be used to consume related resources (in local systems, cached as needed, etc) so they can be incorporated, augmented, etc.
Dependence on LA infrastructure
This is an issue but not insurmountable. Changes in government procurement policies are heading towards standards based rather than platform based procurement. Which will open up the marketplace. For the limited use to which HERs put GIS, Open Source applications are ideal. There are setup costs, support costs, training costs with any solution, just without the ongoing licensing costs. Some of the licensing costs can be used to eg pay for on tap developers. This seems to work nicely in Wales where they use an Open Source and single developer model. Also, HBSMR supports an Open Source GIS option (QGIS) so that would surely suffice in terms of support and demonstrates Open Source is perfectly useable? And there are many options for paid support packages from numerous providers not to mention free training. Appreciated, graphical interfaces may not be as polished as those from commercial packages straight out of the box, but the money saved on licenses more than pays to have your own bespoke set of customisations. Is the problem here really the kind of nightmarish outsourced, less than useful IT arrangements too many LAs have got themselves embroiled in, arrangement which are simply incapable of properly supporting users…?
Distributed vs unified
Either approach could be used to achieve goals here. HBSMR already comes as a cloud hosted solution so the concept of shared access to a remote (unified) system is demonstrable. Maintaining a unified system could bring benefits (eg lower total cost of ownership) whilst a distributed system with aggregation could provide more flexibility for LAs (HEROs would get to choose their own systems and have greater control over how they are set up, which has pros and cons). And the idea that a unified approach would mean HERs maintaining part of a national system is just a matter of emphasis. It could equally be said they are maintaining a local HER, just through a shared, cloud based system. Key point is these days, it matter little where data and applications reside rather how they work and support users; some of my data is in Europe, some of my apps are in California, it makes no odds to me as I use them here at home.
Ownership/Control
The notion that there needs to be ‘gatekeepers’ of data is rapidly becoming outdated. It cuts across government Open Data initiatives and whilst HER advice (when given and/or available) is often useful (and appreciated), the gatekeeper role often comprises just running a query and emailing data. The HERO is not adding much value to the process, in many cases because they do not have enough time to do everything that is asked of them. Opening up resources could free up staff time to allow HEROs to really contribute using their invaluable local knowledge and expertise rather than spending time doing mechanistic tasks. As has been demonstrated through numerous examples, community engagement from fellow professionals and the public can be very rewarding for eg improving quality of records.
Paul J Cripps BA MSc
Archaeogeomancy
Skype: pauljcripps
www.linkedin.com/company/archaeogeomancy
www.facebook.com/Archaeogeomancy
plus.google.com/+ArchaeogeomancyNet
From: Issues related to Historic Environment Records [mailto:[log in to unmask]] On Behalf Of Louisa Matthews
Sent: 19 November 2014 12:13
To: [log in to unmask]
Subject: Re: E Conference Day 3 - What criteria are essential or desirable?
Hi,
I’d like to come back to a point made/hinted at by Jude Plouviez yesterday, Sharon Clough this morning, and Principle 6 of the HIAS. “…data or knowledge should not be at risk of loss, fragmentation, inundation (in data), or system obsolescence” I see this as essential. In order to achieve this, Historic England (my preferred acronym is HEn so as not to confuse with Historic Environment!) must have access to an interrogatable national dataset for the purposes of plan and decision making (designation decisions and planning casework being 2 things that immediately spring to mind). I assume (correct me if I’m wrong) that until now EH have used AMIE for this purpose, since nobody from Designations has ever asked me about the number of horse engines or sheep washes we have on the HER with a view to determining the significance of something proposed for designation (although they do consult us at a later stage in the process), and neither have I ever been asked for data to support planning recommendations or SMC decisions.
The Heritage Gateway is not designed for this sort of professional access (nor should it be) and I cannot see how with dwindling capacity HERs could run potentially innumerable complex queries and create outputs to support not just our own DM people, but HEn’s as well.
My worry is that without a central resource for HEn in place, built as part of this project (or running in tandem with it), we would have subsets of HER/AMIE data being kept, and worse still, edited, by individual case officers or teams because we just can’t get back to them quickly enough. In other words, we would have a situation akin to that which can occur in local authorities where case work staff (e.g. Conservation Officers) are separated from, and unable to use, the HER – they keep their own non-MIDASHeritage compliant data set, which nobody understands and isn’t fed back to the HER. Data supply and interoperability with planning systems was a key concept for HER 21, and equally, interoperability/delivery to HEn should be essential to this project.
Furthermore, it also needs to be established what will happen to future/on-going projects at HEn which would previously have been recorded in AMIE – for example the National Mapping Programme (or is this included under the ‘National Datasets’ in principle 2)? This is still on-going and it is my understanding that monument data is entered into AMIE before being sent to us. It seems to me that there would have to be a practical alternative to this – in theory you could have projects inputting straight into HERS but I can see a number of practical problems (projects that run across several counties running different systems, no HER in place etc.) that would need to be addressed by this project before it could go forward.
In conclusion, I don’t think that we can adhere to Principle 6 (or indeed Principle 1) without having a clear idea of whether there will be some form of NRHE for casework and data entry purposes, or whether it is proposed that HERs become this tool. If the former, then to my mind, the development of any new system must be occur in tandem with this project. If the latter, then we need to build in access and interoperability before we start filling HERs with NRHE data.
Louisa
~~~~~~~~~~~~~
Ms Louisa Matthews BSc(hons) MSc
Historic Environment Record Officer
Waste and Countryside Services
Business and Environmental Services
North Yorkshire County Council
County Hall
Northallerton DL7 8AH
Direct Dial: 01609 532331
Request a search via our website: www.northyorks.gov.uk/her
or browse our records via the Heritage Gateway
From: Issues related to Historic Environment Records [mailto:[log in to unmask]] On Behalf Of Graham Tait
Sent: 19 November 2014 08:42
To: [log in to unmask]
Subject: E Conference Day 3 - What criteria are essential or desirable?
Hi all,
Thinking further about this project, are there some things that need to happen if this takes place?
For example, I would suggest that if this happens English Heritage should stop adding or amending terrestrial NRHE records (otherwise there’d still be duplication of effort!).
Are there any other criteria that need to be considered??
As a starting point, here’s the list of ‘principles’ from the Heritage Information Access Strategy. Which of these are essential criteria? Which are desirable criteria? Which are undesirable or not important criteria? What else needs to happen if this proposal goes ahead?
Principle 1: Local Authority HERs should be the first point of call for and primary trusted source of investigative research data and knowledge
Principle 2: English Heritage should be the first point of call for and primary trusted source of national datasets such as the National Maritime Record
Principle 3: English Heritage should continue to champion the development, maintenance and implementation of standards for the creation, management and storage of digital historic environment data
Principle 4: Investigative research data or knowledge should be readily uploaded, validated and accessed online
Principle 5: The national overview should continue to be delivered online through the Heritage Gateway
Principle 6: Such data or knowledge should not be at risk of loss, fragmentation, inundation (in data), or system obsolescence
Principle 7: English Heritage should, on behalf of the nation, ensure that a security copy of all such data exists in accordance with Principles 3 and 6
Principle 8: Digital data should be supported by material archives in safe repositories accessible to the public.
Best regards,
Graham
--
Graham Tait
Archaeologist
Historic Environment Team
Devon County Council
Lucombe House
County Hall
Exeter
EX2 4QD
Telephone: 01392 382214
Email: [log in to unmask]
Web: http://www.devon.gov.uk/historicenvironmentTwitter: @DevonHistEnv
Disclaimer: http://www.devon.gov.uk/email
Access your county council services online 24 hours a day, 7 days a week at www.northyorks.gov.uk.WARNING
Any opinions or statements expressed in this e-mail are those of the individual and not necessarily those of North Yorkshire County Council.
This e-mail and any files transmitted with it are confidential and solely for the use of the intended recipient. If you receive this in error, please do not disclose any information to anyone, notify the sender at the above address and then destroy all copies.
North Yorkshire County Council's computer systems and communications may be monitored to ensure effective operation of the system and for other lawful purposes. All GCSX traffic may be subject to recording and/or monitoring in accordance with relevant legislation.
Although we have endeavoured to ensure that this e-mail and any attachments are free from any virus we would advise you to take any necessary steps to ensure that they are actually virus free.
If you receive an automatic response stating that the recipient is away from the office and you wish to request information under either the Freedom of Information Act, the Data Protection Act or the Environmental Information Regulations please forward your request by e-mail to the Data Management Team ([log in to unmask]) who will process your request.
North Yorkshire County Council.
Please consider the environment before printing this e-mail
Hartlepool Council
On the Web www.hartlepool.gov.uk
On Facebook www.facebook.com/hartlepoolcouncil
On Twitter www.twitter.com/HpoolCouncil
**********************************************************************************************
This document is strictly confidential and is intended only for the use by the addressee.
If you are not the intended recipent, any disclosure, copying, distribution or other action taken in reliance of the information contained in this email is strictly prohibited.
Any view expressed by the sender of this message are not necessarily those of Hartlepool Borough Council.
If you have received this transmission in error, please use the reply function to tell us and then permanently delete what you have received.
Please note: Incoming and outgoing e-mail messages are routinely monitored for compliance with our policy on the use of electronic communications.
**********************************************************************************************