Possibly deviating from the OT, but I have a related question:
Why, exactly, is it that storing 20 GB of image data in 3600 separate
files is so much slower than concatenating them all into one big file?
I'm honestly not sure where the bottleneck is. Internally, HDF5 is
organized very much like a disk-based file system. Sort of like the
ISO9660 file system you create as a file on disk before burning it to
CD, or Blu-Ray or what have you. You can also format a regular file as
any other file system you wish, such as XFS, EXT4, JFS, BTRFS, GPFS,
NTFS, HFS, FAT32, etc. Then you can mount it with the "-o loop" option,
copy your image files to it, and then you have everything in one big
file without sacrificing the advantages of individual file access. The
only difference seems to be that for HDF5 the file system access is
implemented separately in each user-space program, but for a real file
system it is implemented in the kernel. Over the years lots of
functionality has "started" in userspace and then ends up in the kernel
after about 1-2 years. These days, you can even "mount" a tar archive
and access it like a file system. So how is HDF5 any different from,
say, HFS? Is it because permissions and security checks must be done
for every file access? Perhaps after communicating with a remote LDAP
server? If that is the case, it seems to me this is a failing of the
implementation of file systems, and not an advantage of HDF5.
Or am I missing something?
-James Holton
MAD Scientist
On 3/12/2016 8:17 AM, Graeme Winter wrote:
> Dear All
>
> Thank you to everyone who sent data in response to this initial request. We are working through this now & hope to ensure these are all supported shortly.
>
> A particular thanks to those who sent both the cbf images and the hdf5 as that is superb for ensuring the hdf5 handling is correct.
>
> An observation on these hdf5 files. If you rename them after collection they stop working... The links don't work any more...
>
> Thanks & best wishes Graeme
>
>> On 8 Mar 2016, at 19:57, <[log in to unmask]> <[log in to unmask]> wrote:
>>
>> Hello World,
>>
>> A few times over the last week we at the xia2 helpdesk have been asked about support for Eiger detectors, and particularly local dialects of CBF file which are created on Eiger powered beamlines. Often we are able to get a single image which is helpful but from there it is hard to prove everything is working right.
>>
>> Please could beamline scientists who have an Eiger and use local software to make this into miniCBF files get in touch, ideally with an example data set, so we can test the support in xia2 and DIALS and make corrections where necessary.
>>
>> Thanks & best wishes Graeme
>> --
>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
|