Thanks everyone for the answers and additional thoughts. What a can of
worms!
Joe - looks interesting. Have you come across date_parse (see
http://stackoverflow.com/questions/15350309/heuristic-fuzzy-date-extraction-from-the-string<http://stackoverflow.com/questions/15350309/heuristic-fuzzy-date-extraction-from-the-string?answertab=votes#tab-top>)
which is supposedly very good at short pieces of text - for example if you
have a date field, but the values are inconsistently formatted. I was
trying it though on longer text input such as "The Prime Minister, Mr
Winston Churchill, lights a cigar as he and General Montgomery set out in a
jeep to go inland. Photograph taken during the Prime Minister's short visit
to the Normandy coast, 12 June 1944." and was surprised how it frequently
failed to spot anything even as obvious. And also it always tries to return
a precise date, which kind of defeats the object of what we're talking
about here.
Ken - didn't realise you were on here :) Decimal places is something I had
thought of, and I wonder if there are any other examples in mainstream use?
I'm guessing you could do something roughly similar using trailing zeros
on Unix timestamps, but I can certainly see a few problems, for example if
something were located precisely at e.g. 51.000000000, 1.00000000 or
happened at exactly 09 / 09 / 01 @ 1:46:40am UTC (I'll let you look that
one up!). I personally prefer the approach of Flickr and Google which both
use a precise measurement but store a measure of accuracy/granularity/zoom.
Sarah - that sounds like the most pragmatic solution I've heard, though I
can think of scenarios where you'd probably have to be over cautious and
thereby lose some of the relative precision other solutions might offer
(e.g. 'post-War' - 1945-19..?)
Florian - really interesting examples, thanks. Lots of food for thought!
Richard - I've done a few experimental things in OpenLayers using aggregate
pins, custom markers to depict accurate vs vague etc. The other possible
approach is to show and hide pins with differing levels of accuracy
at different zoom levels, but I've yet to find anything really elegant. And
your last point I think is proven by this discussion!
Food for thought.
James
On Mon, Nov 4, 2013 at 5:29 PM, Richard Light <[log in to unmask]>wrote:
> It's an interesting reflection on the quality of machine-processible data
> that we are generating as a community, if it's necessary to use natural
> language processing or regular expressions to extract usable date ranges
> from CMS [1] data. The advice to record earliest and latest date in a
> consistent processible format has been around since the 1970s, and remains
> in SPECTRUM to this day.
>
> Of course, you still have to distinguish between duration of an event and
> uncertainty over when exactly it happened. The CIDOC CRM SIG have provided
> guidance on date recording [2] which aims to address this issue as "ongoing
> throughout" vs. "at some time within". When looking into a possible design
> pattern for dates in a Linked Data context [3], I found that the W3C date
> types allow you to express less precise dates, e.g. "year" [4]. This might
> be useful.
>
> For places, there is in principle the option of using a bounding box or
> boundary-line to indicate the size/scope of the place (which isn't quite
> the same thing as imprecision, of course). Geonames has the former (e.g.
> [5]); Ordnance Survey the latter [6], at least for administrative units in
> the area it covers. Of course, having this information, and getting to use
> it in an interactive context, are two different challenges. Has anyone
> implemented boundary box or boundary-line-based "pins", e.g. in OpenLayers?
>
> One problem I am finding with geolocation is that when you convert
> descriptive place name data to coordinates, every object from the "same"
> place ends up with exactly the same coordinates. So to add to the spurious
> precision of the place itself, you have a spurious collocation of objects.
> I am trying to address this issue by generating a single "pin" with a
> group description of the objects it contains, unless there is only one.
> [7] (Apart from other considerations, popular web mapping applications
> will often only display one pin for a single coordinate, so one is rather
> forced into this approach.)
>
> The requirement to display collections data on maps and in timelines is
> possibly a useful wake-up call as regards the usefulness of our data.
>
> Richard
>
> [1] i.e. Collections Management System, not the other one
> [2] http://www.cidoc-crm.org/docs/How_to%20implement%20CRM_Time_
> in%20RDF.pdf
> [3] http://light.demon.co.uk/wordpress/?p=600 - all comments welcome, BTW
> [4] http://www.w3.org/TR/xmlschema-2/#isoformats
> [5] http://ws.geonames.org/get?geonameId=2644667
> [6] http://data.ordnancesurvey.co.uk/datasets/os-linked-data/about
> [7] http://light.demon.co.uk/wordpress/?page_id=446&modes_
> query={Word%20search}=*{oakham}&page_id=537<http://light.demon.co.uk/wordpress/?page_id=446&modes_query=%7BWord%20search%7D=*%7Boakham%7D&page_id=537>
>
>
> On 04/11/2013 13:16, James Morley wrote:
>
>> Hi David
>>
>> I've been looking at it from that angle too, thought not quite in as much
>> detail as it sounds like you have! And I've also been thinking about the
>> 'middle' bit that you allude to - once you've deciphered them, how you
>> might store these things in a unified way.
>>
>> I'm working on a prototype crowdsourcing toolkit for cultural collections
>> and will be presenting the concept in a quick open-mic session at next
>> week's UKMW13. The aim is not just to give users tools, but also tap into
>> natural language processing (e.g. OpenCalais, Alchemy, Zemanta) to suggest
>> tags, locations, dates, people, events etc that users can then verify (or
>> otherwise!).
>>
>> Give me a shout if you'd like to have a look at this in a day or two, when
>> I should have something ready for testing.
>>
>> Cheers, James
>>
>>
>> ---
>> James Morley
>> www.jamesmorley.net / @jamesinealing
>> www.whatsthatpicture.com / @PhotosOfThePast
>> www.apennypermile.com / @APennyPerMile
>> <http://www.apennypermile.com>
>>
>>
>>
>> On Mon, Nov 4, 2013 at 11:54 AM, David Croft <[log in to unmask]>
>> wrote:
>>
>> I've been working on this problem on an off for a while now, but from the
>>> other side as it were. Trying to extract the dates that the record author
>>> meant from what they actually wrote.
>>> There are a LOT of different date formats out there and I've yet to see a
>>> really good solution.
>>> I'm coming at this problem from a software angle, trying to decode dates
>>> automatically, so my desires for date formats may be different to yours.
>>> But I really, really, really wish that date information was stated
>>> explicitly and consistently.
>>>
>>> Plenty of collections use modifiers like 'circa', 'early' or 'first
>>> half',
>>> but then don't use these consistently.
>>> In one record 'late 20th century' means 1950 to 2000, in another place it
>>> will mean 1975 to 2000.
>>> These sort of date modifiers never seem to get explicitly defined for the
>>> collection which means that what one collection means by 'circa' is
>>> different to what another collection means.
>>> The modifiers also mean different things to different dates, 'circa 1950'
>>> may mean 1945 to 1955 but is `circa 1950s' 1950 to 1959 or 1945 to 1965?
>>> There are lots of records with dates like '80s' where you just have to
>>> assume the century information or '1940-50s' where you assume it means
>>> 1940
>>> to 1959.
>>>
>>> So for me, the best way is just to provide the upper and lower bounds for
>>> date period in full, i.e. not `circa 1955' but instead `1950/1/1 to
>>> 1959/12/31`.
>>> Or if that's not possible, define exactly what you mean by 'circa',
>>> 'late', 'early' etc and make that information available where anyone
>>> looking at your records can see it.
>>> For example, are you going to use the word 'circa'? or just put a 'c' on
>>> the front of the date i.e. 'c1950'?
>>> If there are two dates in a field does the circa apply to just the first
>>> one or both? i.e. is 'circa 1950 to 1960' the same as 'circa 1950 to
>>> circa
>>> 1960'?
>>> If you are saying 'circa 19th century' do you mean up to 25 years either
>>> side? 50 years? 75?
>>> Software can decode any format you use as long as we know what the rules
>>> are.
>>>
>>> P.S
>>> There are some truly interesting date fields out there and I've been
>>> keeping a list as part of my really tricky testing data.
>>> Some of my favourites are '25 feb ?', 'circa pre world war two', 'early
>>> or
>>> late 19th or 20 century' and 'c18-1 to c--01?'
>>>
>>> David
>>>
>>> ****************************************************************
>>> website: http://museumscomputergroup.org.uk/
>>> Twitter: http://www.twitter.com/ukmcg
>>> Facebook: http://www.facebook.com/museumscomputergroup
>>> [un]subscribe: http://museumscomputergroup.org.uk/email-list/
>>> ****************************************************************
>>>
>>> ****************************************************************
>> website: http://museumscomputergroup.org.uk/
>> Twitter: http://www.twitter.com/ukmcg
>> Facebook: http://www.facebook.com/museumscomputergroup
>> [un]subscribe: http://museumscomputergroup.org.uk/email-list/
>> ****************************************************************
>> .
>>
>>
> --
> *Richard Light*
>
>
> ****************************************************************
> website: http://museumscomputergroup.org.uk/
> Twitter: http://www.twitter.com/ukmcg
> Facebook: http://www.facebook.com/museumscomputergroup
> [un]subscribe: http://museumscomputergroup.org.uk/email-list/
> ****************************************************************
>
****************************************************************
website: http://museumscomputergroup.org.uk/
Twitter: http://www.twitter.com/ukmcg
Facebook: http://www.facebook.com/museumscomputergroup
[un]subscribe: http://museumscomputergroup.org.uk/email-list/
****************************************************************
|