I've been playing around with some large HDS files and one thing I'm
noticing is that there are always temp files, tNNNN.sdf, being created
and they are very very large (presumably copies of the input data files).
I'm getting errors such as from stats or ndftrace:
!! Requested data extends beyond the end of the record; record length is
! 947912704 bytes (possible corrupt HDS container file
! /home/timj/scicom/acsis/specwrite/tc99.sdf).
! DAT_COPY: Error copying an HDS object to a new structure component.
! ARY_COPY: Error making a copy of an array.
! NDF_COPY: Error copying an NDF to a new location.
! STATS: Error computing simple statistics for an NDF's pixels.
The above error is caused because the input file is 64bit and the temp
file is not. Now, that's not really the point of this email (but it is a
worry and indicates that the only sane way to switch to 64bit is to
default HDS_64BIT to 1 inside HDS, once Brian has fixed the datCopy bug),
but it does indicate a performance problem since
1. stats and ndftrace are opening the file readonly.
2. the temporary file is enormous and simply copying it takes a long
time.
The latter point is especially annoying when using ndftrace (is it mapping
the data array for dertermining whether bad pixels are present?). HDSTRACE
lists the file really quickly.
Why is ndftrace creating these temporary files when an NDF is opened for
read? Who is doing the call to NDF_COPY (it's not stats or ndftrace)?
AIF_GETVM uses DAT_TEMP not NDF.
Clearly, if we weren't making a copy of the 6 GB file then things would be
a lot faster. I'm just trying to get a handle on these issues since they
do have a performance impact.
Of course, if I'm going to start handling 8GB data arrays (2 giga pixels
of _REAL should still be usable without overflowing a fortran INTEGER)
then these things are really going to hurt...
--
Tim Jenness
JAC software
http://www.jach.hawaii.edu/~timj
