JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCP4BB Archives


CCP4BB Archives

CCP4BB Archives


CCP4BB@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCP4BB Home

CCP4BB Home

CCP4BB  August 2007

CCP4BB August 2007

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Depositing Raw Data

From:

"Winter, G (Graeme)" <[log in to unmask]>

Reply-To:

Winter, G (Graeme)

Date:

Fri, 17 Aug 2007 15:50:04 +0100

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (204 lines)

Hi,

On the question of a "uniform format" for this data, I believe that
imgCIF has been working towards this end for a number of years. As a
very vocal supporter of this I would like to say that this is an ideal
archival format for the following reasons:

 - the terms are clearly defined (or are currently in the process of
such)
 - the images are compressed, typically by a factor of 2-3
 - some data reduction packages (Mosflm, XDS) can read them in this
compressed form 

Now, I would be telling lies if I said that this was all finished but I
think it is fair to say that this is already a long way down the path.
As soon as I am convinced that you can go losslessly to and from imgCIF,
and the data reduction programs will give precisely the same results, I
will convert thus freeing up 5 firewire disks.

For more information on this take a look at medsbio.org.

Cheers,

Graeme

-----Original Message-----
From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf Of
Mischa Machius
Sent: 17 August 2007 15:07
To: [log in to unmask]
Subject: [ccp4bb] Depositing Raw Data

Since there are several sub-plots in that mammoth thread, I thought
branching out would be a good idea.

I think working out the technicalities of how to publicly archive raw
data is fairly simple compared to the bigger picture.

1. Indeed, all the required meta-data will need to be captured just like
for refined coordinates. This will be an additional burden for the
depositor, but it's clearly necessary, and I do consider it trivial.
Trivial, as in the sense of "straightforward", i.e., there is no
fundamental problem blocking progress. As mentioned, current data
processing software captures most of the pertinent information already,
although that could be improved. I am sure that the beamlines,
diffraction-system manufacturers and authors of data- processing
software can be convinced to cooperate appropriately, if the community
needs these features.

2. More tricky is the issue of a unified format for the images, which
would be very helpful. There have been attempts at creating unified
image formats, but - to my knowledge - they haven't gotten anywhere.  
However, I am also convinced that such formats can be designed, and that
detector manufacturers will have no problems implementing them,
considering that their detectors may not be purchased if they don't
comply with requirements defined by the community.

3. The hardware required to store all those data, even in a highly
redundant way, is clearly trivial.

4. The biggest problem I can see in the short run is the burden on the
databank when thousands of investigators start transferring gigabytes of
images, all at the same time.

5. I think the NSA might go bonkers over that traffic, although it
certainly has enough storage space. Imagine, they let their decoders go
wild on all those images. They might actually find interesting things in
them...

So, what's the hold-up?

Best - MM



On Aug 17, 2007, at 3:23 AM, Winter, G (Graeme) wrote:

> Storing all the images *is* expensive but it can be done - the JCSG do

> this and make available a good chunk of their raw diffraction data.
> The
> cost is, however, in preparing this to make the data useful for the 
> person who downloads it.
>
> If we are going to store and publish the raw experimental measurements

> (e.g. the images) which I think would be spectacular, we will also 
> need to define a minimum amount of metadata which should be supplied 
> with this to allow a reasonable chance of reproduction of the results.

> This is clearly not trivial, but there is probably enough information 
> in the harvest and log files from e.g. CCP4, HKL2000, Phenix to allow 
> this.
>
> The real problem will be in getting people to dig out that tape / dvd 
> with the images on, prepare the required metadata and "deposit" this 
> information somewhere. Actually storing it is a smaller challenge, 
> though this is a long way from being trivial.
>
> On an aside - firewire disks are indeed a very cheap way of storing 
> the data. There is a good reason why they are much cheaper than the 
> equivalent RAID array. They fail. Ever lost 500GB of data in one go?
> Ouch. ;o)
>
> Just MHO.
>
> Cheers,
>
> Graeme
>
> -----Original Message-----
> From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf Of 
> Phil Evans
> Sent: 16 August 2007 15:13
> To: [log in to unmask]
> Subject: Re: [ccp4bb] The importance of USING our validation tools
>
> What do you count as raw data? Rawest are the images - everything 
> beyond that is modellling - but archiving images is _expensive_!
> Unmerged intensities are probably more manageable
>
> Phil
>
>
> On  16 Aug 2007, at 15:05, Ashley Buckle wrote:
>
>> Dear Randy
>>
>> These are very valid points, and I'm so glad you've taken the 
>> important step of initiating this. For now I'd like to respond to one

>> of them, as it concerns something I and colleagues in Australia are
>> doing:
>>>
>>> The more information that is available, the easier it will be to 
>>> detect fabrication (because it is harder to make up more information

>>> convincingly). For instance, if the diffraction data are deposited, 
>>> we can check for consistency with the known properties of real 
>>> macromolecular crystals, e.g. that they contain disordered solvent 
>>> and not vacuum. As Tassos Perrakis has discovered, there are 
>>> characteristic ways in which the standard deviations depend on the 
>>> intensities and the resolution. If unmerged data are deposited, 
>>> there
>
>>> will probably be evidence of radiation damage, weak effects from 
>>> intrinsic anomalous scatterers, etc. Raw images are probably even 
>>> harder to simulate convincingly.
>>
>> After the recent Science retractions we realised that its about time 
>> raw data was made available. So, we have set about creating the 
>> necessary IT and software to do this for our diffraction data, and 
>> are
>
>> encouraging Australian colleagues to do the same. We are about a week

>> away from launching a web-accessible repository for our recently 
>> published (eg deposited in PDB) data, and this should coincide with 
>> an
>
>> upcoming publication describing a new structure from our labs. The 
>> aim
>
>> is that publication occurs simultaneously with release in PDB as well

>> as raw diffraction data on our website.
>> We hope to house as much of our data as possible, as well as data 
>> from
>
>> other Australian labs, but obviously the potential dataset will be 
>> huge, so we are trying to develop, and make available freely to the 
>> community, software tools that allow others to easily setup their own

>> repositories.  After brief discussion with PDB the plan is that PDB 
>> include links from coordinates/SF's to the raw data using a simple 
>> handle that can be incorporated into a URL.  We would hope that we 
>> can
>
>> convince the journals that raw data must be made available at the 
>> time
>
>> of publication, in the same way as coordinates and structure factors.
>
>> Of course, we realise that there will be many hurdles along the way 
>> but we are convinced that simply making the raw data available ASAP 
>> is
>
>> a 'good thing'.
>>
>> We are happy to share more details of our IT plans with the CCP4BB, 
>> such that they can be improved, and look forward to hearing feedback
>>
>> cheers


------------------------------------------------------------------------
--------
Mischa Machius, PhD
Associate Professor
UT Southwestern Medical Center at Dallas
5323 Harry Hines Blvd.; ND10.214A
Dallas, TX 75390-8816; U.S.A.
Tel: +1 214 645 6381
Fax: +1 214 645 6353

Top of Message | Previous Page | Permalink

JISCMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007


WWW.JISCMAIL.AC.UK

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager