 |
 |
On Mon, 6 Mar 2006, Tim Jenness wrote: > I've been playing around with some large HDS files and one thing I'm > noticing is that there are always temp files, tNNNN.sdf, being created > and they are very very large (presumably copies of the input data files). > > I'm getting errors such as from stats or ndftrace: > > !! Requested data extends beyond the end of the record; record length is > ! 947912704 bytes (possible corrupt HDS container file > ! /home/timj/scicom/acsis/specwrite/tc99.sdf). > ! DAT_COPY: Error copying an HDS object to a new structure component. > ! ARY_COPY: Error making a copy of an array. > ! NDF_COPY: Error copying an NDF to a new location. > ! STATS: Error computing simple statistics for an NDF's pixels. > > The above error is caused because the input file is 64bit and the temp > file is not. Now, that's not really the point of this email (but it is a > worry and indicates that the only sane way to switch to 64bit is to > default HDS_64BIT to 1 inside HDS, once Brian has fixed the datCopy bug), > but it does indicate a performance problem since > > 1. stats and ndftrace are opening the file readonly. > 2. the temporary file is enormous and simply copying it takes a long > time. > > The latter point is especially annoying when using ndftrace (is it mapping > the data array for dertermining whether bad pixels are present?). HDSTRACE > lists the file really quickly. > > Why is ndftrace creating these temporary files when an NDF is opened for > read? Who is doing the call to NDF_COPY (it's not stats or ndftrace)? > AIF_GETVM uses DAT_TEMP not NDF. > > Clearly, if we weren't making a copy of the 6 GB file then things would be > a lot faster. I'm just trying to get a handle on these issues since they > do have a performance impact. > > Of course, if I'm going to start handling 8GB data arrays (2 giga pixels > of _REAL should still be usable without overflowing a fortran INTEGER) > then these things are really going to hurt... Hi Tim, HDS will make temporary copies of mapped data when it's not in the right format, not just the right type. So in this case are we talking about a bigendian PPC mapping data created on a littlendian PC? If that's the case KAPPA:NATIVE should make the problem go away (or an NDFCOPY)./ Cheers, Peter.