JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for DC-RDA Archives


DC-RDA Archives

DC-RDA Archives


DC-RDA@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

DC-RDA Home

DC-RDA Home

DC-RDA  February 2009

DC-RDA February 2009

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: datasets for testing rda at scale

From:

Rob Styles <[log in to unmask]>

Reply-To:

List for discussion on Resource Description and Access (RDA)

Date:

Fri, 13 Feb 2009 09:05:17 +0000

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (138 lines)

Hey Alistair, this is great work and very interesting. I'm keen to see  
where the analysis and mapping to RDA goes.

What was the rationale for converting to MODS first rather than  
mapping straight from the MARC to RDA?

rob


On 12 Feb 2009, at 10:30, Alistair Miles wrote:

> Hi all,
>
> This is just an update to say that I've converted the LOC/scriblio
> data to marc xml and from there to mods xml. My next step is to do
> some analysis of the loc data in mods xml to get an overview of the
> elements used, then to try to design at least a partial mapping from
> mods xml to RDF using the RDA and FRBR schemas.
>
> FYI the marc xml and mods xml versions of the LOC/scriblio data can be
> downloaded from the links below...
>
> http://dcmi-rda.s3.amazonaws.com/locdata/part01-marcxml.tar.gz
> http://dcmi-rda.s3.amazonaws.com/locdata/part01-modsxml.tar.gz
> http://dcmi-rda.s3.amazonaws.com/locdata/part02-marcxml.tar.gz
> http://dcmi-rda.s3.amazonaws.com/locdata/part02-modsxml.tar.gz
> [...]
> http://dcmi-rda.s3.amazonaws.com/locdata/part29-marcxml.tar.gz
> http://dcmi-rda.s3.amazonaws.com/locdata/part29-modsxml.tar.gz
>
> Each download is a gzipped tar containing a *set* of up to 25 xml
> files. Each of these files is a 10,000 record split of the data in the
> corresponding part. I broke each part into 10,000 record splits so I
> could process the transformations more easily.
>
> N.B. there is a bug in part 13 split 25, for some reason the marc xml
> output was incomplete so up to 10,000 records could be missing.
>
> FWIW I initially tried the conversions without splitting each
> part. I.e. I converted each original marc file into a single marc xml
> file, then tried to transform that to a mods xml file via
> xsltproc. However I found you need more than 7GB ram to do the marcxml
> to modsxml transform on a whole part (I tried it on a large ec2
> instance), so that's when I decided to split each part into smaller
> chunks, which I figured would be faster to process and more amenable
> to parallel processing (transforming all the splits from marcxml to
> modsxml took a couple of hours on a c1.xlarge ec2 instance, running up
> to 10 transformations in parallel; it can also be done on a laptop,
> but takes ~10 times longer).
>
> Btw if anyone else has experience of the marcxml->modsxml transform on
> a file of similar size do let me know, I don't do a lot of xslt-ing so
> may be missing some tricks for making it work on smaller computers.
>
> Cheers,
>
> Alistair
>
>
> On Mon, Dec 22, 2008 at 03:31:50PM -0500, Ed Summers wrote:
>> Hey Alistair:
>>
>> On Mon, Dec 22, 2008 at 1:16 PM, Alistair Miles
>> <[log in to unmask]> wrote:
>>> Any tips for how I could turn these data into RDF?
>>
>> If you want to work specifically with that dataset you could download
>> the different parts Karen pointed you to, and convert to MARCXML  
>> using
>> an efficient tool like yaz-marcdump [2]. yaz-marcdump is nice it will
>> convert from MARC-8 to UTF-8.
>>
>> Once you've got it in MARCXML you could then use a stylesheet like
>> LC's [2] to convert to DublinCore flavored RDF. This might be kinda
>> lossy for your RDA work though, so you might want MARCXML->MODS [3],
>> and then use the MODS->RDF conversion that the Simile folks created
>> (which Karen also pointed you to) [4].
>>
>> In fact Simile used that stylesheet on their own MIT Library Catalog
>> MARC data (Barton) and still seem to have the result online [5]. So
>> perhaps just using the Barton data is the quickest way to begin
>> playing with what once was MARC data as RDF? To my knowledge Stefano
>> Mazzocchi simply created an RDF vocabulary that mirrors the  MODS XML
>> Schema, but I haven't looked at it in a while.
>>
>> Another thing worth checking out might be Rob Styles work [6] with
>> other people at Talis at converting MARC with full fidelity to RDF.
>> Perhaps he has some tools (or data) at his disposal? Rob you are on
>> here right?
>>
>> I'd be willing to lend a hand with some of this if necessary, so just
>> let me know if you think I can help.
>>
>> //Ed
>>
>> [1] http://www.indexdata.com/yaz/doc/yaz-marcdump.tkl
>> [2] http://www.loc.gov/standards/marcxml/xslt/MARC21slim2RDFDC.xsl
>> [3] http://www.loc.gov/standards/mods/v3/MARC21slim2MODS3.xsl
>> [4] http://simile.mit.edu/wiki/MARC/MODS_RDFizer
>> [5] http://simile.mit.edu/wiki/Dataset:_Barton
>> [6] http://events.linkeddata.org/ldow2008/papers/02-styles-ayers-semantic-marc.pdf
>
> -- 
> Alistair Miles
> Senior Computing Officer
> Image Bioinformatics Research Group
> Department of Zoology
> The Tinbergen Building
> University of Oxford
> South Parks Road
> Oxford
> OX1 3PS
> United Kingdom
> Web: http://purl.org/net/aliman
> Email: [log in to unmask]
> Tel: +44 (0)1865 281993

Rob Styles
tel: +44 (0)870 400 5000
fax: +44 (0)870 400 5001
mobile: +44 (0)7971 475 257
msn: [log in to unmask]
irc: irc.freenode.net/mmmmmrob,isnick
web: http://www.talis.com/
blog: http://www.dynamicorange.com/blog/
blog: http://blogs.talis.com/panlibus/
blog: http://blogs.talis.com/nodalities/
blog: http://blogs.talis.com/n2/
 
Please consider the environment before printing this email.

Find out more about Talis at www.talis.com 

shared innovationTM

Any views or personal opinions expressed within this email may not be those of Talis Information Ltd or its employees. The content of this email message and any files that may be attached are confidential, and for the usage of the intended recipient only. If you are not the intended recipient, then please return this message to the sender and delete it. Any use of this e-mail by an unauthorised recipient is prohibited.

Talis Information Ltd is a member of the Talis Group of companies and is registered in England No 3638278 with its registered office at Knights Court, Solihull Parkway, Birmingham Business Park, B37 7YB.

Top of Message | Previous Page | Permalink

JISCMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
August 2012
July 2012
May 2012
April 2012
March 2012
February 2012
January 2012
October 2011
September 2011
August 2011
June 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
June 2010
February 2010
January 2010
December 2009
November 2009
October 2009
June 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
August 2007
July 2007
June 2007
May 2007
April 2006
February 2006
January 2006
December 2005


WWW.JISCMAIL.AC.UK

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager